Aim

Ensuring reliable measurements by:

• determining the measurement properties of the instrument;
• monitoring whether the instrument satisfies accuracy requirements;
• checking if the measurement protocol fulfils the quality assurance requirements.

Description

Often in epidemiological research various physical characteristics have to be determined for study participants, such as: Height, weight, blood pressure, exercise capacity, lung volume, etc. Physical measuring instruments are used to determine these variables. The measures described in this guideline are needed to exclude the possibility of wrong outcomes by: An incorrectly working instrument, incorrect measurement procedure, or the observed effect being too small to reliably measure with the instruments. The guideline, 1.2-05 Data provided bij Third Party, should be consulted for measurements not undertaken by yourself. For instance, those carried out by a laboratory.
In brief, the following steps should be taken:

1)        Determine the following for each physical instrument:

• What is the desired measurement accuracy (precision)?
• What is the desired range of measurement?
• What is an acceptable level of measurement error for the instrument?

2)        Choose a measurement instruments fulfilling the predefined requirements.

3)        Calibrate the instrument, i.e. demonstrate that the equipment satisfies the requirements stipulated in point 1.

4)        Determine the measurement error and, if required, the intra- and inter-observer reliability. Validate your measurement procedure.

5)        Check whether the equipment is still working properly for each measurement.

6)        Write down your requirements, instrument, type of instrument, calibrations, checks, maintenance and deviations in a logbook.

7)        Describe your measurement procedure in a measurement protocol. 

See the details section for further information.

Precision
Precision is related to the desired unit in which you want to measure. If you want to measure weight, you need to decide whether to measure in grams or kilograms. Also for laboratory work you need to think about the minimal precision necessary for your measurements (how many decimal places needed) Read also 1.2-04: Third party data.

Range of measurement
You also need to decide the desired measurement range. For instance, when weighing adults, the scales need to produce an accurate measurement between 30 to 150 kg.

• Measurement error
• It is very important to think about what level of measurement error is acceptable for your instrument. For example: If you want to measure accurately by the kilogram, then a deviation (measurement error) in the scales of plus or minus half a kilogram is just about acceptable. If you need to accurately know within 100 grams, then this level of deviation is not acceptable. The maximum acceptable measurement error depends to a great extent on the intended effects you want to measure (e.g. differences between people, or differences in people over time). Decide what the minimal (clinically) relevant difference is you wish to measure with the instrument. The measurement error always needs to be smaller than this minimal (clinically) relevant difference. In many cases you will have to determine the measurement error of the instrument yourself (refer below for calibration and guideline 1.1B-08c Evaluation of measurement properties ).

For some measurement instruments you will have to explore a number of issues in trials and properly describe them fully. Example a movement tracker. You will firstly need to determine the kind of movements you want to record (biking, walking, running, etc), whether the instrument can distinguish between the various types of movement, how reliably the movements can be measured (refer also to guideline 1.1B-08c Evaluation of measurement properties), and/or on which position on the body this can be measured. These trials can only be undertaken if you have the measurement instrument available. If you are unable to borrow to instrument, you will have to ask whether suppliers will provide instruments on trial. Approach suppliers and manufacturers with these types of questions.

2. Choice of instrument
Once you have determined the desired level of precision, measurement range and measurement error, you will need to find an instrument fulfilling these criteria. For example: If you want to accurately weigh adults by kilogram within a range of + 0.5 kg, then you will need a set of scales able to provide this. For instance, a set of scales which the manufacturer states has an accuracy of + 0.3 for a measurement range of 20 - 150 kg. You should also consider the following points when choosing an instrument:

• How much does the instrument cost and what are the costs associated with use and maintenance?
• Is the equipment easy to use?
• Reliability of the instrument;
• Could you borrow it from another project?
• Do others have (good) experience(s) using this instrument?

3. Calibration
You will also need to calibrate an instrument to ensure the instrument is doing what it should be doing. For a set of scales this could be done by taking a number of measurements with standard weights. A series could include: 10, 20 and 30 kg. Choose measurement points at the bottom, middle and top of the measurement range. Measure the series at least 3 times. If this is working properly, then the values measured should fall within the range you have set. After this you should use a graph to plot the values measured against the measurement points (0, 10, 20, 30). If this has worked, then you should see a large correlation in the shape of a straight line and the instrument has been calibrated. If you want to be really thorough, you could also use standards certified by an official certification agency. In some cases there will be no accepted (gold) standards available to use in comparing the measurement instrument. In this case it is advisable to determine the construct validity of the instrument. For instance, the instrument could be compared with other instruments believed to correlate either strongly or weakly with the instrument. Prior hypotheses regarding the anticipated association are tested in this process. Refer to guideline 1.1B-08c Evaluation of measurement properties for more information.

Some cases will require more than a simple calibration procedure such as this. This applies, for instance, to situations where a complex and expensive instrument is being replaced by a simpler, cheaper method (e.g. for application in large studies). The question is then whether this simple method is able to measure as well as the more expensive and complex instruments. In cases such as this a separate validation study will need to be set up in which both tests are presented to a group of patients to demonstrate that the new instrument is just as valid, accurate and reliable as the old instrument. The Bland & Altman method can be used for this. Refer to guideline 1.1B-08c Evaluation of measurement properties for more information.

4. Determining measurement error and intra- and inter-observer reliability
In many cases you will need to determine the measurement error of an instrument yourself (comparable to a test-retest for questionnaires). Repeated measures from the same individuals will be needed for this. The measurement error can then be determined using the Bland & Altman (limits of agreement), or expressed using the Standard Error of Measurement (SEM). When measurements are dependent on, or could be influenced by those using the measurement instruments you need to determine the intra-observer reliability. To what extent the same results are obtainable from repeat measurements. Example, a manual blood pressure monitor, where the measurement is dependent on what someone hears.
When different people use the measurement equipment e.g. when different research assistants measure patients throughout the study.  The level of agreement between research assistants needs to be determined (inter-observer reliability). Repeated measures from the same individuals are needed for this. The intra- or inter-observer reliability can also be determined using the Bland & Altman method or with the SEM (absolute measurement error). Reliability is often also expressed as Cohen's Kappa or an intra-class correlation coefficient (relative measurement error). Refer to guideline 1.1B-08c Evaluation of measurement properties for more information. The 1.1C-05 Recruiting and training data collectors guideline can also be consulted.

5. Monitoring
Before measurements or a series of measurements, it is advisable to check whether the instrument is still working properly. For a set of scales: Choose a standard (an object with a known, set weight) and check whether the scales reflect the right weight. A tape measure, skin fold callipers, or other measures can be regularly checked by measuring a fixed object in the same way. Record the date, instrument and measurement value in the logbook.

6. Annotations
The calibration mode and method should be noted in the logbook. This also applies to the maintenance. It is advisable to fully calibrate the equipment after each maintenance session and before using it again. Determining the measurement error and/or intra- or inter-observer reliability of a new or existing measurement instrument is often worth the effort. You can publish it as a separate article (provided the research group is large enough). Consult someone from the clinimetrics group (Riekie de Vet or Caroline Terwee) or a statistician (preferably before the measurements start!).

7. Measurement
Use a measurement protocol to determine how exactly the measurements are performed and under which circumstances (measurement method). Ensure that everybody does this in the same way. Refer also to guidelines 1.1C-05: Recruiting and training data collectors, and 1.2-01: Mentoring and monitoring data collectors.

The following should also be considered:

• On which part(s) of the body are the measurements going to be carried out, how are this/these position(s) determined?
• Describe the standard measurement process;
• How frequently are measurements repeated?
• How can you read the instrument most accurately?  
• Are there environmental factors that may influence the measurement? If this is the case, then standardise the conditions as much as possible and record the conditions present during the measurement. E.g. very high moisture and room temperature, air moisture during exercise measurement, time of day;
• If you are expecting very small differences, you could also explore how environmental factors could influence your measurements and correct for these as much as possible;
• Write down instructions for the participants;
• Can the participant have a practice session first or not?
• How should I check the instruments functioning?
• What is the appropriate calibration method;
• Record things such as who is taking the measurement, which instrument is being used.

Test the protocol properly and review each of the above points. The measurement method will affect the measurement error. Therefore, standardise measurements as much as possible and determine the measurement error under the same conditions as, later on, in the real study.

Finally
A specific check, calibration method and validation of the measurement protocol will be required for each type of instrument. Use your common sense and ask colleagues or the supplier how these issues should be approached.

1.1B-08a, Selecting instruments;
1.1B-08b, Developing instruments;
1.1B-08c Evaluation of measurement properties
1.1C-05, Recruiting and training data collectors
1.2-01, Supporting and monitoring data collectors.
1.2-04, Data Provided by Third parties
Bland & Altman article

Calibration:      Act of comparing an instrument's measuring accuracy to a known, traceable, standard.

V 1.2: 30 Jan 2011. References to guideline 1.1B-08 replaced by the updated guidelines
V 1.1:  1 Jan 2010: Translation into English.

1. Have the desired accuracy, measurement range and error been determined?
2. Has the right instrument been used?
3. Does the instrument satisfy the criteria applied?
4. Has the instrument been calibrated and has its functioning been regularly checked?
5. Has this been clearly recorded?
6. Is there a clear measurement protocol?
7. Are there any logbooks?
8. Has the inter- or intra-observer reliability been determined?