Aim

Specifying and correctly implementing post-hoc and sensitivity analyses.

Description

Post-hoc analyses

Post-hoc analyses are required when a significant relationship has been found between the dependent variable and a categorical, independent variable with more than two categories. This allows researchers to ascertain to which categories the significance can be ascribed. For logistic or Cox regressions the output provides both the “overall” significance for categorical variables as well as the significance of the OR or RR of the separate categories in respect of the reference category. The latter are, in fact, post-hoc analyses (albeit not corrected for repeated testing). However, there are also analysis methods in which the output does not automatically provide this specification. It is tempting to decide which categories differ significantly by “eyeballing” the results. However, additional analyses need to be undertaken in order for this to be determined. An example of this is variance analysis, in which so-called post-hoc tests can be used (examples include Tukey, Duncan, Scheffé of Bonferroni tests).

Sensitivity analyses

There is always more than one way to carry out an analysis. In order to be more certain about the results it is advisable to redo the analyses in a slightly different way, often by changing one or more (external) parameters. There are a number of cases where a sensitivity analyses is almost always desirable. These will be discussed here.
Firstly, when a cut-off has been selected for the dependent or independent variable for which there is, as yet, no consensus. Even if there is a consensus, there is the question of whether this cut-off is applicable to the study population. It is advisable to repeat the analyses with different cut-off values.
Secondly, there may be variables with “unused” categories. This may be either missing values or variables that are composed of data from various sources where there may occasionally be conflicts between both sources. An example of the latter is a disease diagnosis based on data provided by the general practitioner and the respondent. Missing values can be substituted, meaning the respondent can be retained for the analysis. Advanced statistical imputation methods can be used for this. Substitution can also be based on the basis of a “best guess”. It is good practice to carry out the analyses both with and without respondents with missing values, and to compare the results.
An example of this is an uncertain diagnosis where all uncertain cases are set at "no disease" in one analysis, and as "diseased" in another. All uncertain cases can be omitted in a third analysis.
A third situation when sensitivity analysis is desirable is  with longitudinal data. For instance, there may be data at two time points and the analysis concerns the definition of “change” in the dependent variable. There is an ongoing lively discussion around this topic in the literature: Whether the choice should be for a difference score or for one or another definition of “relevant” change, it is advisable to carry out the analyses using different definitions. A similar strategy is recommended in all cases in which there is uncertainty regarding the best choice of statistical measure or procedure.
Finally, sensitivity analyses are a standard component of economic evaluations. The opportunities for multivariate analysis in economic evaluations are very limited, owing to the fact that the distribution of cost data is skewed. Sensitivity analyses are used to study the effect of, for instance, the value of cost points on outcomes. Often subgroup analyses and analyses with imputed missing values are carried out as sensitivity analyses (see Drummond et al., 1997).  

• Drummond MF, O'Brien BJ, Soddart, GL and Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. Oxford New York Toronto: Oxford University Press, Second Edition, 1997.

V1.1: 1 Jan 2010: English translation.
V1.0: 23 Apr 2007.

• Have post-hoc tests been carried out following the “omnibus” tests? If not, why not?
• Have sensitivity analyses been carried out? Would it still be useful to do this for some of the variables?
• Are cost variables being used? Are sensitivity analyses needed for this?