Pilot study

Aim

To explore certain aspects, as feasibility, before undertaking a large-scale study.

 

Requirements

  • Undertake a pilot study when needed, especially when data are collected in the study.

 

Documentation

Note important decisions that have been made while undertaking the pilot study, note the results of the pilot study and describe the possible adaptations that have been made in the main study in your electronic logbook.

 

Responsibilities

Executing researcher:
  • Make sure each member of the project group considers the need of a pilot study;
  • Clearly document decisions while undertaking the pilot study;
  • Undertake a pilot study as you are performing the larger scale study.
Project leaders:
  • Consider the need of a pilot study;
  • Support the executing researcher in undertaking the pilot study.
Research assistant: N.a.

 

How To

The aim of pilot studies is to explore certain issues before undertaking a large-scale study [1]. Pilot studies have other objectives than a large-scale study; they are rarely just a small version of a larger study.

 

Aims of a pilot study
Certain aspects of the large-scale study can be tested out in a pilot study. This may involve testing feasibility in practice or improving the methodological quality of parts of the study. Examples will be provided for both of these:

 

Examples of questions about feasibility
  • What is the anticipated number of patients/participants that can be included? Patient inclusion usually appears to be much lower than the total number of eligible patients, and also much lower than the numbers doctors (and other recruiters) promise to deliver.
  • How many of the potential patients satisfy the inclusion and exclusion criteria?
  • Will the eligible participants agree to be randomised for comparison of the interventions?
  • Why do some participants not want to take part or what are the reasons for drop out? You might be able to interview these people to understand the reasons for non-participation.
  • Are all medical files or information from other data sources retrievable?
  • Have all the necessary details been registered in statuses (doctors are sometimes overly positive about this issue), and is the quality of the data adequate?
  • Can all measurements be easily implemented and are they not too bothersome for the research participants?
  • What is the best recruitment method?
  • Is the intervention appropriate for the intended target group (e.g. language use, type of intervention)?
  • Is it possible to carry out the intervention in the target group with respect to acceptability and compliance?
  • Is it possible to the implement the intervention and/or study in the organisation?
  • How much time does the measurement procedure take, and how many times can this be undertaken per day?
  • What is the best way to approach the target group?

 

Examples of questions about methodological quality
  • Is it possible to blind the patients for the type of intervention?
  • Have the measurement instruments good content validity: i.e. are all questions relevant, are no questions missing and do the respondent understand all questions properly?
  • Are the measurements reproducible: Are your research assistants performing the measurements in exactly the same way?
  • Are the questionnaires being completed properly? Is the wording of the questions clear and comprehensible?
  • What is the most efficient measurement method for reproducible measurements (see for instance Observations: Who, when and how often?)
  • Is exposure (for instance in a cohort study) being properly measured?
Pilot studies are less suitable for estimates of effect size. The reason for this is that pilot studies are too small to achieve a reliable estimate of the effect. It is not justified to calculate the number of required participants in a large trial out of the effect size determined in a pilot study. Beurskens et al. [1] has presented an illustrative example of how misleading a pilot study can be. The correct interpretation is that all values in the confidence interval could represent the actual effect, meaning that when the confidence interval includes the neutral value (0 for differences and 1 for ratios), the effect could actually be both positive and negative. However, pilot studies can be used to provide an indication of the variance in measurement, that is, the standard deviation which can be used for the power calculation for the main study.

 

Organisation: Internal or external pilot studies
External pilot:
Pilot studies may precede larger studies and may include a clear go/no go decision. If the research aim/question of the pilot study is formulated in concrete, measurable terms, then a clear decision can be taken on the basis of the results. This decision may be for instance, that clear modifications in study design are required or that the larger study is not feasible.
Internal pilot:
If the pilot study exactly replicates the large study, e.g. a trial, then it may turn out that nothing needs to be altered based on the pilot study Therefore, why shouldn’t patients in the pilot study be included as the first participants in the larger study? There is nothing to stop you from doing this. However, you do need to make sure that the decision not to change the procedures is made, because one wants to add the patients of the pilot study to the larger study. It helps if clear agreements have been made in advance about this.

 

Appendices/references/links

  • Beurskens AJHM, de Vet HCW, Kant IJ. Dwalingen in de methodologie (Methodological erros). VIII. Pilot onderzoeken: zin en onzin (Pilot studies: Sense and nonsense). Ned T Geneesk., 1998; 142:2142-2145.

 

Audit questions

  1. Are data being collected in this study?
    1. If yes, was a pilot study conducted?
      1. If no, why not?

 

V3.0: 10 Aug 2016: Revision guideline
V2.0: 12 May 2015: Revision format
V1.1:  1 Jan 2010: Translated into English
V1.0: 14 Feb 2007