Post-hoc and sensitivity analyses

Aim

To specify and correctly implement post-hoc and sensitivity analyses.

 

Requirements

  • Post-hoc and sensitivity analyses are performed when needed.

 

Documentation

  • Data analysis plan: Describe which post-hoc tests and sensitivity analyses will be performed.
  • Syntax: Document the post-hoc tests and sensitivity analyses.
  • Logbook: Document the decisions you made based on the results of post-hoc tests and sensitivity analyses.

 

Responsibilities

Executing researcher:
  • To make sure you do post-hoc tests and sensitivity analyses when needed;
  • To discuss the results with your supervisors;
  • To document the points mentioned under 3 (documentation).
Project leaders:
  • To inspect the data analysis plan on post-hoc tests and sensitivity analyses when needed;
  • To discuss the results of sensitivity analyses with the executing researcher
Research assistant: N.a.

 

How To

Post-hoc analyses
Post-hoc analyses are required when a significant relationship has been found between the dependent variable and a categorical, independent variable with more than two categories.  An example of this is variance analysis, in which so-called post-hoc tests can be used (examples include Tukey, Duncan, Scheffé of Bonferroni tests). The tests differ in the way they correct for multiple testing.

 

Sensitivity analyses
There is always more than one way to carry out an analysis. In order to be more certain about the results it is advisable to redo the analyses in a slightly different way, often by changing one or more (external) parameters. There are a number of cases where a sensitivity analyses is almost always desirable. These will be discussed here.
Firstly, when a cut-off has been selected for the dependent or independent variable for which there is, as yet, no consensus. Even if there is a consensus, there is the question of whether this cut-off is applicable to the study population. It is advisable to repeat the analyses with different cut-off values.
Secondly, there may be variables with “unused” categories. This may be either missing values or variables that are composed of data from various sources where there may occasionally be conflicts between both sources. An example of the latter is a disease diagnosis based on data provided by the general practitioner and the respondent. Missing values can be substituted, meaning the respondent can be retained for the analysis. Advanced statistical imputation methods can be used for this. Substitution can also be based on the basis of a “best guess”. It is good practice to carry out the analyses both with and without respondents with missing values, and to compare the results.
An example of this is an uncertain diagnosis where all uncertain cases are set at “no disease” in one analysis, and as “diseased” in another. All uncertain cases can be omitted in a third analysis.
Finally, sensitivity analyses are a standard component of economic evaluations. The opportunities for multivariable analysis in economic evaluations are very limited, owing to the fact that the distribution of cost data is skewed. Sensitivity analyses are used to study the effect of, for instance, the value of cost points on outcomes. Often subgroup analyses and analyses with imputed missing values are carried out as sensitivity analyses (see Drummond et al., 1997).

 

Appendices/references/links

Drummond MF, O’Brien BJ, Soddart, GL and Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. Oxford New York Toronto: Oxford University Press, Second Edition, 1997.

 

Audit questions

  1. Have post-hoc tests been carried out following the “omnibus” tests?
    1. If not, why not?
  2. Have sensitivity analyses been carried out? Would it still be useful to do this for some of the variables?
  3. Are cost variables being used? Are sensitivity analyses needed for this?

 

V3.1: 23 Jan 2017: Minor revision
V3.0: 13 Oct 2016: Minor revision
V2.0: 12 May 2015: Revision format
V1.1: 1 Jan 2010: English translation
V1.0: 23 Apr 2007