evidence & practice / pilot study

SHARING EXPERIENCES

Why undertake a pilot in a qualitative PhD

study? Lessons learned to promote success
Wray J, Archibong U, Walton S (2017) Why undertake a pilot in a qualitative PhD study? Lessons learned to promote success.
Nurse Researcher. 24, 3, 31-35. Date of submission: 14 July 2015; date of acceptance: 8 April 2016. doi: 10.7748/nr.2017.e1416

Jane Wray Abstract

PhD student, Faculty of Health Background Pilot studies can play an important role in qualitative studies. Methodological and
Studies, University of Bradford, practical issues can be shaped and refined by undertaking pilots. Personal development and
UK researchers’ competence are enhanced and lessons learned can inform the development and
quality of the main study. However, pilot studies are rarely published, despite their potential to
Uduak Archibong improve knowledge and understanding of the research.
Professor of diversity and
director of the Centre for Aim To present the main lessons learned from undertaking a pilot in a qualitative PhD study.
Inclusion and Diversity, Discussion This paper draws together lessons learned when undertaking a pilot as part of a
University of Bradford, UK qualitative research project. Important methodological and practical issues identified during
the pilot study are discussed including access, recruitment, data collection and the personal
Sean Walton development of the researcher. The resulting changes to the final study are also highlighted.
Academic developer, Quality
Conclusion Sharing experiences of and lessons learned in a pilot study enhances personal
and Enhancement Office, development, improves researchers’ confidence and competence, and contributes to the
University of Salford, UK understanding of research.

Correspondence Implications for practice Pilots can be used effectively in qualitative studies to refine the final design,
jwray@[Link] and provide the researcher with practical experience to enhance confidence and competence.

Peer review Keywords

This article has been subject nursing research, novice researcher, pilot study, qualitative methods, reflexivity
to external double-blind
peer review and checked
for plagiarism using
automated software Introduction The terms ‘pilot’ and ‘feasibility’ have been
Pilot or feasibility studies serve an important used interchangeably (Van Teijlingen and
Conflict of interest function before a main study (Halberg 2008). Hundley 2001) and sometimes erroneously
None declared Research methods and protocols can be tested, (Thabane et al 2010) to varying definitions
pre-empting future difficulties and enabling (Arain et al 2010). According to the NIHR
adjustments (Kim 2011). However, pilot Evaluation, Trials and Studies Coordinating
studies are not always published (Arain et al Centre (NETSCC), a pilot study is a smaller
2010), despite their potential to contribute to version of the main study used to test
the understanding of research (Secomb and whether the mechanisms of the main study
Smith 2011). References to pilot studies are would work as planned (NETSCC 2013). By
often limited to cursory, one-line mentions in contrast, feasibility studies focus on study
published papers, and the lessons learned and parameters, such as clinicians’ willingness to
experiences gained remain unavailable to the recruit, the time required to collect data and
wider research community. Publishing pilot analyse them, the design of outcome measures,
studies can provide important information compliance, and adherence (NETSCC 2013).
to other researchers (Van Teijlingen and Arnold et al (2009) preferred not to use the
Hundley 2001) and prevent further resources term ‘feasibility’, differentiating between ‘pilot
being spent on studies that are impracticable work’ (background work that informs future
(Thabane et al 2010). Publishing information research); a ‘pilot study’ (which has specified
about pilot studies is essential for shared objectives and methodology) and a ‘pilot trial’
learning and might not necessarily relate to the (a stand-alone study including randomisation).
findings, as the results ‘may not be meaningful With this variability in the usage of terms,
and have not been reported, (but) the outcomes publishing details about what work was
and experiences are’ (Secomb and Smith 2011). actually undertaken becomes important, to

[Link] volume 24 number 3 / January 2017 / 31

evidence & practice / pilot study

inform others in the research community – requires a substantial amount of time and
‘Every attempt should be made to publish’ effort for little return (Kaba and Beran 2014);
(Thabane et al 2010). failure to recruit also has implications for study
Pilot or feasibility studies are common in timelines (McCance and Mcilfatrick 2008),
quantitative research (Arnold et al 2009), reliability and validity (Jessiman 2013).
and are increasing being reported in studies Before beginning recruitment, it was
using qualitative approaches (Sampson necessary to secure ethical approval. It is
2004, Kim 2011, Secomb and Smith 2011). essential to seek approval specifically for a
They can provide a clearer understanding pilot, as its purpose, risks and benefits to
of the topic being investigated and explore participants are different to those of the main
procedural elements of a study (Jessiman study (Thabane et al 2010). The implications
2013). For novice researchers, they can provide of taking part must be explained in the pilot
engagement in the practicalities of research, as study’s information sheet and consent forms
a way to develop understanding and experience so that potential participants can make an
(Kezar 2000). Thabane et al (2010) argued informed decision as to whether to participate.
that there is an ethical as well as scientific Approval for the pilot was secured to
obligation to publish pilot work; although they recruit final year, pre-registration nursing
were referring to phase III trials, this obligation students from a higher education institution
should also apply to qualitative studies. not involved in the main study. Although
This paper is concerned with reporting approval had been obtained and contact was
the benefits for qualitative researchers of made with the relevant programme lead by the
undertaking a pilot study, detailing the researcher, communication was also required
experiences gained, the lessons learned and with a number of other faculty staff to ensure
resulting changes to the main study as the information was sent to eligible participants.
result of a pilot study undertaken as part Identifying the ‘right’ person to achieve this
of a PhD that sought to explore newly proved to be the most important lesson learned
qualified nurses’ perceptions of culturally at this stage, as this person might not be the
competent practice. programme or department lead. The researcher
There are numerous potential reasons had to negotiate with potential gatekeepers
for undertaking a pilot (van Teijlingen and and navigate bureaucratic systems (Kaba and
Hundley 2001, Arain et al 2010), and this Beran 2014), to create an open and ongoing
study had four stated objectives (Thabane et al communication chain involving all the relevant
2010): to determine whether the planned people. As a result, additional time was built
approach to recruitment would generate into the recruitment timeline of the main study
volunteers (Secomb and Smith 2011); to to allow for this process.
test the tools for collecting data to ensure Information was sent to potential
they elicited the type and range of responses participants by email and the university’s
required (van Teijlingen and Hundley 2001); virtual learning environment (VLE). This and a
to explore procedural elements, primarily reminder email sent two weeks later generated
to see whether email and telephone were only three expressions of interest. Generating
more effective than face-to-face contact for enthusiasm and interest in a study is necessary
communication and collecting data (Jessiman to engage potential participants (Kaba and
2013); and to reflect on personal skills Beran 2014), and so with the agreement
and abilities as a researcher, and explore of the programme director, the researcher
self in a study that used an interpretative gave a brief presentation to the cohort. This
phenomenological methodology (Kim 2011). introduced the main study, and outlined the
For each objective, decisions were made about purpose and implications of participation in
to whether to proceed as planned, modify the pilot study. Personalising the information
or change the approach in the final study in this way provided an additional impetus to
(Arain et al 2010, Thabane et al 2010). Data recruitment, securing five more expressions of
were collected between May and August 2014. interest. Timing may also have been a pertinent
factor, as the information was initially
Main lessons learned circulated just before the cohort’s submission
Sample, recruitment and access date for an assignment.
The pilot study sought to determine whether To clarify any issues, all those who had
the planned approach to recruitment would expressed interest were contacted by email
result in volunteers, as recruitment can be and/or SMS message before they completed
challenging, especially with a volunteer sample their consent forms. Five people participated in
(Berry and Bass 2012). Recruitment often the pilot; one did not complete the interview,

32 / January 2017 / volume 24 number 3 [Link]

although it is not known why – SMS and pre-registration nurse education to capture Online archive
For related information,
email reminders were sent to the participant reflections on practice (Bulman et al 2012). visit nurseresearcher.
providing additional opportunities on Descriptions of practice and nurses’ thoughts com and search using
alternative times and dates, but as no response and reflections on that practice were generated the keywords
was received, further contact was deemed however, the amount of information provided
potentially intrusive and the participant was varied considerably. Some sections were
considered to have withdrawn. detailed while others contained only two or
The key lesson learned about recruitment three words.
was that a more personalised and Only minor amendments to the directed
comprehensive strategy was needed (Secomb reflection were considered necessary for the
and Smith 2011). The main study’s recruitment main study and additional information was
strategy was modified to include a formal added in the introduction to guide participants.
presentation to be delivered at a time sensitive The directed reflection would not be used
to participants’ demands (Harris et al 2008). in isolation (Jacelon and Imperio 2005) and
Successful recruitment in a pilot study does not the subsequent interview would provide an
guarantee success in the main study but testing opportunity to address information deficits and
an approach reduces the likelihood of time clarify any ambiguities.
and resources being invested in unsuccessful After they had completed the directed
methods (Kaba and Beran 2014). reflection, participants were contacted to set
up a suitable time and date for the interview.
Data collection tools Telephone interviews were used, as they make
A directed reflection and the topic guide for it easier to access busy professionals and
semi-structured interviews were also tested geographically diverse populations (Harris et al
in the pilot to establish whether they were 2008, Mealer and Jones 2014). They are a
user-friendly and if rephrasing or additional versatile, resource-efficient approach (Novick
questions or prompts were needed. Participants 2008) and can produce data that are
were asked to complete their reflections in comparable to interviews conducted face-to-
the form of a structured diary sheet and were face (Sturges and Hanrahan 2004). It had been
then asked specific questions regarding its estimated that the telephone interviews would
completion during the subsequent interview. take between 30 and 40 minutes, and the pilot
In directed or solicited reflection, confirmed this.
participants record their actions, thoughts By undertaking the interviews, it was
and/or feelings, at the request of a researcher possible to reflect on the topic guide. Although
(Clayton and Thorne 2000). Such reflections the main areas did not require amendment,
are used in conjunction with interviews, as changes to the order and additional
they provide researchers with initial data on prompts were added. The topic guide had
a topic or issue that can be explored though initially begun with general questions about
further discussion (Jackson et al 2008, Smith participants’ experience to ease them into
2008). They can also provide participants with the interview, with discussion of the directed
greater control over how their experiences are reflection in the latter part of the interview.
represented (Woll 2013). This was changed, as participants started
In this pilot, the directed reflection was talking about the directed reflection at the
developed from existing literature and with start of the interview, which proved to be a
the support of a patient and carer reference simple and natural interview opener (Smith
group. Participants were asked to describe a 2008), leading easily into further questions
recent interaction with a patient from a diverse to encourage clarification and elaboration.
background and detail their thoughts, feelings, A possible limitation to this was that in the
actions and behaviour. The reflection was subsequent discussion of cultural competence,
divided into sections, each section starting with the participants appeared to understand and
a question to direct or prompt the participant interpret further questions within the same
– for example, ‘How would you describe your broad theme of diversity set by the reflection.
feelings during this event?’ It is important that participants’ feel
During the interviews, participants comfortable during an interview and
were asked about the readability and their conversation and dialogue are encouraged
comprehension of the directed reflection. rather than imposed (Arthur et al 2014).
All responded positively, confirming Therefore, additional prompts were added
that completion of it was not especially to the topic guide so that if a participant
problematic. Participants were familiar with remained focused on one diversity
the approach as it is commonly used in characteristic in the main study, they could

[Link] volume 24 number 3 / January 2017 / 33

evidence & practice / pilot study

be encouraged to give examples from other proved invaluable in ensuring that interviews
diversity groups. went ahead as scheduled.
Undertaking the interviews also provided As a result of these experiences, some minor
important experiential learning about a changes were made to the main study. These
research method, as well as an opportunity included using email primarily for sending
to consider the benefits and challenges of the and receiving information and documents,
approach. Scrutiny of the transcripts and and using text messages mainly to engage and
re-listening to recordings of the interviews retain participants, since although recruitment
confirmed that participants’ pauses and silences and data collection were the same for all
had not always been responded to successfully. participants, the time lapse between expression
One of the main lessons learned was how to of interest, consent, completion of reflection
manage silences when conducting a telephone and interview did vary.
interview, when non-verbal encouragement Another benefit of experiential learning was
to elaborate is not available (Trier-Bieniek the ability to understand and acknowledge that
2012). The researchers’ ability to communicate participants had other, more pressing demands
rapport with the participant can be limited by on their time and that engagement would vary
the lack of face-to-face contact (Novick 2008). and depend on participants’ circumstances.
This is a potential problem with telephone
interviews, particularly for researchers with Self
little experience in this approach (Mealer and A journal and field notes – written and audio
Jones 2014). Rapport can be improved when – were used for personal reflection throughout
using telephone interviewing by ensuring that the pilot to explore personal assumptions and
verbal contact has already been made before the influence of previous experiences (Rapport
the interview (Carr and Worth 2001, Sturges 2004, Hill 2006). Recording reflections
and Hanrahan 2004, Harris et al 2008), and so provided a valuable source of data (Dickson-
this was added to the main study’s protocol. Swift et al 2007), as well as a useful reference
point when re-examining the interviews.
Procedural issues Reflexivity is an important tool in qualitative
The benefits and challenges of using email and/ approaches but requires practice to develop
or text communication were also considered effectively (Jootun et al 2009) – the pilot
in the pilot study. Email was the preferred was indispensable in this development. To
approach of the university and the usual aid reflection, the researcher added questions
means of communicating with the target group and prompts to the journal to structure and
(Berry and Bass 2012, Kaba and Beran 2014, guide reflections after interviews. Examples
Mason and Ide 2014). However, text messages include: ‘How well did I listen to what was
appeared to be participants’ preferred contact said?’; ‘Was I able to establish a good level
method, as they responded more quickly to of trust and rapport?’; ‘Did the interview
them, and other research had shown them flow or was it stilted?’; ‘Did I agree with
to be effective (Berry and Bass 2012, Mason them too readily or prompt too quickly
and Ide 2014). Emails and texts were used curtailing elaboration?’; and ‘Did I clarify
throughout the pilot to encourage continued any ambiguity?’ Considering these questions
engagement in the study, although email was encouraged personal consideration of how
more effective when supplemented by a text. well each interview was conducted and
Email proved useful in collecting data, as aided reflection on how the interviews might
all participants had access to a computer and be improved in the main study (Dickson-
email. The directed reflection was emailed Swift et al 2007).
to them and they completed it electronically. Listening to the recordings of the two
Four were then returned within seven days, interviews and examining the notes indicated
one within 14 days following a text-message that there were some challenges to address
prompt. Only one of these was handwritten in subsequent interviews. It appeared that
– it was written up by the researcher and the attention was divided between the interviewee,
original scanned and kept. The electronic the interview topic guide and the recording
responses were provided in a format that did equipment. The recording devices had caused
not require transcription and had no additional considerable anxiety and two devices were
resource implications for participants used in case one failed.
(Novick 2008). Participants were texted before the interview
In the latter two interviews, text-message started to ensure that they were prepared. On
reminders were sent one day before and 10 reflection, this notification was also important
minutes before the interview started, which for the researcher as it acted as a sign to be

34 / January 2017 / volume 24 number 3 [Link]

Write for us mindful of ‘the space and place’ (Gagnon et al Undertaking a pilot study as part of a
[Link]/r/ 2015). The interview schedule and directed qualitative PhD enhances the understanding
nr‑author‑guidelines
reflection were re-read, equipment tested, and of important research processes, including
the physical area of the researcher prepared to access, recruitment and data collection.
minimise potential distractions and intensify Personal development is enhanced, and
focus. researchers’ confidence and competence
improves. Establishing the main objectives
Conclusion of a pilot study then enables the researcher
Undertaking a pilot study to experience to decide whether these objectives were met
research and develop personal skills and and refine and re-shape the main study as
abilities can make a significant contribution to a consequence.
the main study (Sampson 2004). In the pilot Pilot studies remain poorly described in
study described in this paper, the researchers’ the literature despite the potential benefit
competence and confidence – particularly in of sharing insights into methodological and
relation to telephone interviewing – improved practical issues in qualitative research. By
with each subsequent interview. In addition, reporting on these insights and experiences,
the breadth and scope of personal reflection this paper adds to the small but growing
contained in the audio and written journal body of work being shared about the value
notes improved exponentially. of pilot studies.

References

Arain M, Campbell MJ, Cooper CL et al (2010) What Gagnon M, Jacob JD, McCabe J (2015) Locating the Kaba A, Beran T (2014) Twelve tips to guide effective Qualitative Methodologies in Health and Social Care
is a pilot or feasibility study? A review of current qualitative interview; reflecting on space and place participant recruitment for interprofessional Research. Routledge, New York NY, 99-122.
practice and editorial policy. BMC Medical Research in nursing research. Journal of Research in Nursing. education research. Medical Teacher. 36, 7, 578-584.
Methodology. 10, 67. 20, 3, 203-215. Sampson H (2004) Navigating the waves: the
Kezar A (2000) The importance of pilot studies: usefulness of a pilot in qualitative research.
Arnold DM, Burns KE, Adhikari NK et al (2009) The Hallberg IR (2008) Surveys. In Watson R, McKenna H, beginning the hermeneutic circle. Research In Higher Qualitative Research. 4, 3, 383-402.
design and interpretation of pilot trials in clinical Cowman S et al (Eds) Nursing Research: Designs and Education. 41, 3, 385-400.
research in critical care. Critical Care Medicine. 37, 1 Methods. Churchill Livingstone Elsevier, Edinburgh Secomb JM, Smith C (2011) A mixed method pilot
Suppl, 569-574. 179-188. Kim Y (2011) The pilot study in qualitative inquiry: study: the researchers’ experience. Contemporary
identifying issues and learning lessons for culturally Nurse. 39, 1, 31-35.
Arthur S, Mitchell M, Lewis J et al (2014) Designing Harris R, Kelly D, Hunt JA et al (2008) Accessing elite competent research. Qualitative Social Work. 10, 2,
fieldwork. In Ritchie J, Lewis J, McNaughton Nicholls C nurses for research: reflections on the theoretical and 190-206. Smith JA (2008) Qualitative Psychology: A Practical
et al (Eds) Qualitative Research Practice: A Guide For practical issues of telephone interviewing. Journal of Guide to Research Methods. Second edition. Sage
Social Science Students and Researchers. Second Research in Nursing. 13, 3, 236-248. Mason DM, Ide B (2013) Adapting qualitative research Publications, London.
edition. Sage Publications, London, 147-176. to technology savvy adolescents. Nurse Researcher.
Hill ML (2006) Representin(g): negotiating multiple 21, 5, 40-45. Sturges JE, Hanrahan KJ (2004) Comparing telephone
Berry DM, Bass CP (2012) Successfully recruiting, roles and identities in the field and behind the desk. and face-to-face qualitative interviewing: a research
surveying and retaining college students: a Qualitative Inquiry. 12, 5, 926-949. McCance T, Mcilfatrick S (2008) Phenomenology. note. Qualitative Research. 4, 1, 107-118.
description of methods for the Risk, Religiosity, and In Watson R, McKenna H, Cowman S et al (Eds)
Jacelon CS, Imperio K (2005) Participant diaries Nursing Research: Designs and Methods. Churchill Thabane L, Ma J, Chu R et al (2010) A tutorial on
Emerging Adulthood Study. Research in Nursing and pilot studies: the what, why and how. BMC Medical
Health. 35, 6, 659-670. as a source of data in research with older adults. Livingstone Elsevier, Edinburgh, 231-241.
Qualitative Health Research. 15, 7, 991-997. Research Methodology. 10, 1.
Bulman C, Lathlean J, Gobbi M (2012) The concept of Mealer M, Jones J (2014) Methodological and ethical
Jackson D, Daly J, Davidson P (2008) Interviews. issues related to qualitative telephone interviews on Trier-Bieniek A (2012) Framing the telephone
reflection in nursing: qualitative findings on student interview as a participant-centred tool for qualitative
and teacher perspectives. Nurse Education Today. In Watson R, McKenna H, Cowman S et al (Eds) sensitive topics. Nurse Researcher. 21, 4, 32-37.
Nursing Research: Designs and Methods. Churchill research: a methodological discussion. Qualitative
32, 5, 8-13. NIHR Evaluation, Trials and Studies Coordinating Research. 12, 6, 630-644.
Livingstone Elsevier, Edinburgh, 281-288.
Carr ECJ, Worth A (2001) The use of telephone Centre (NETSCC) (2013) Glossary. [Link].
Jessiman WC (2013) ‘To be honest, I haven’t even [Link]/glossary (Last accessed: 5 July 2016.) van Teijlingen ER, Hundley V (2001) The importance of
interview for research. NT Research. 6, 1, 511-25. pilot studies. Social Research Update. 35.
thought about it’ – recruitment in small-scale,
Clayton AM, Thorne T (2000) Diary data enhancing qualitative research in primary care. Nurse Novick G (2008) Is there a bias against telephone
interviews in qualitative research? Research in Woll H (2013) Process diary as methodological
rigour: analysis framework and verification tool. Researcher. 21, 2, 18-23. approach in longitudinal phenomenological research.
Journal of Advanced Nursing. 32, 6, 1514-1521. Nursing and Health. 31, 4, 391-398.
Jootun D, McGhee G, Marland GR (2009) Reflexivity: The Indo-Pacific Journal of Phenomenology. 13, 2, 1-11.
Dickson-Swift V, James EL, Kippen S et al (2007) Doing promoting rigour in qualitative research. Nursing Rapport N (2004) From the porter’s point of
sensitive research: what challenges do qualitative Standard. 23, 23, 42-46. view: participant observation by the interpretive
researchers face? Qualitative Research. 7, 3, 327-353. anthropologist in the hospital. In Rapport F (Ed) New

[Link] volume 24 number 3 / January 2017 / 35