Aim

- Observing and solving problems during data collection on time.
- Monitoring the quality of the data collected during the data collection process.

Description

It is important during the data collection phase that the data continue to be collected accurately and in a standardised fashion. Data collectors therefore need to be intensively supported and monitored. The data collected also need to be monitored. This support and monitoring may lead to changes in the protocol, supplementary training or revision training.

Methods for quality monitoring during the data collection/fieldwork will be discussed for each measurement instrument. These agreements are confirmed in writing. Examples of quality monitoring include multiple testing (test-retest by the same and/or different data collectors), calibration, monitoring completed questionnaires and participating in intra-and interlaboratory calibration or other laboratory quality programmes. Interviews/tests can be randomly checked by the researcher or another data collector. Audio or video recordings can also be made of the tests/interviews.

Quality of the actual data collection is monitored during the data collection phase, to allow for any potential adjustments to be made. The results of, for instance, multiple measurements or calibration are immediately verified. The quality of an interview will be discussed with the interviewer, and potentially with other data collectors and researchers. Questionnaires will be assessed for completeness. The data collector will attempt to retrieve any omitted data as quickly as possible (e.g. by telephone).

Create a list in advance for questionnaires, distinguishing between core variables for which missing data will always need to be retrieved immediately (e.g. primary outcome measure), and variables for which missing data are less important (e.g. 1 item from a multi-item scale in a questionnaire). The data collectors should be provided with clear instructions about retrieving missing data for the various types of variables.

Clear (written) instructions should be given to data collectors about what they need to do if measurements have not succeeded or data are missing for other reasons.

Data collectors should maintain a logbook and record what the most significant problems were during the data collection process. Agreements should be made in advance about maintaining the logbook. Problems will be discussed regularly with the researchers. It is important to determine whether problems are incidental or structural.

The logbook needs to be handed over to the data entry clerks along with the data.

Analysis of the initial data at variable level can provide an insight into the quality of the data, the data collector(s) and the measurement instruments. Review, for instance, the distribution of the variables and look out for outliers, impossible values and missing values. This analysis may lead to modifications to the test protocol. This analysis may also lead to modifications in the number of participants to be included or tested on the basis of a new power calculation (see guideline 1.1-04). The analysis of the initial data should definitely not be used to answer the research question.

Recently collected datasets can also repeatedly be inspected at variable level (“continuous monitoring”) during the data collection process.

Modifications to the agreed test protocol should only be made after consultation with all researchers concerned. Modifications should be included immediately in the test protocol.

V1.1:  1 Jan 2010: English translation.
V1.0:  8 Jul 2008.

1)         Has the quality monitoring of the data collection been confirmed in writing?
2)         Which forms of quality monitoring have been carried out?
3)         Has the quality of the data collection during the data collection phase been monitored?
4)        Have problems during the data collection been recorded in a logbook, and how have these been solved?
5)         Have agreements been made about retrieving missing data?
6)         Have data been monitored at variable level during the data collection process?
7)        Have test protocols been changed during the data collection process? If so, on what basis and has this been confirmed in writing?