Multiple comparisons problem

The alpha error accumulation, often called α error inflation referred to in the statistics, the global increase in the alpha error probability ( type 1 error ) by multiple testing in the same sample.

Graphically formulated, the more hypotheses testing for a record, the higher the probability that one of them ( in error ) is assumed to be applicable.

Multiple testing

Often in a study not only a null hypothesis is defined, but one wants to answer several questions using the data obtained. This can be further null hypotheses, but also confidence intervals or estimates.

In the case of multiple null hypotheses then one speaks of a multiple testing problem.

In such cases, one encounters the following two problems:

1) inconsistencies (eg )

Suppose someone wants to compare the expected values ​​. Pairwise test all null hypotheses are not rejected, only the hypothesis is rejected.

2) inflation of the α - error

For multiple testing issues a local (only the relevant single hypothesis ) distinguished α - α level, and the global level ( for the entire hypothesis family). If the tests are independent, may be zero for each hypothesis, the local α is adjusted based on the global level using the following formula: where k = number of hypotheses.

Adjustment of the global α - level

But how can you counteract or correct it this α error inflation?

Bonferroni correction

The Bonferroni correction is the simplest and most conservative form, adapt the multiple α - level. Here, the global α - level is distributed equally among the individual tests:

It follows by means of the Bonferroni inequality that each individual test below the level (and not α ) will be used: For valid

The very conservative approach to the Bonferroni correction has the disadvantage that the result must have a very low α - value to qualify as statistically significant can. This attempt advancements such as avoid the Bonferroni - Holm procedure.

Bonferroni - Holm procedure

An extension of the Bonferroni correction, the Bonferroni - Holm procedure dar. Here, the following algorithm comes into play:

Step 1:

Determination of the global α - level

Step 2:

Execution of all individual tests and determination of p- values

Step 3:

Sort the p- values ​​from smallest to largest

Step 4:

Calculating the local α levels as the ratio α of a global level for the number of tests - i, where:

Step 5

Compare the p- values ​​with the calculated α - stocked local levels (starting with ) and repeat this step until the p-value is greater than the corresponding value.

Step 6

All null hypotheses whose p were smaller than the local α value will be rejected. With the null hypothesis, the p was greater than the local α - level, all of the following null hypotheses are accepted, under the global α - level.

The Bonferroni - Holm procedure is less conservative than the Bonferroni correction. Only the first test must be statistically significant at the level required in the Bonferroni correction, then the necessary level drops steadily. However, this procedure has also, as well as the Bonferroni correction has the disadvantage that any logical and stochastic dependencies between the test statistics are not used.

Other methods

In addition to the adjustments described, there are further opportunities to adapt to a global α - level. These include for example:

• Tukey T method
• Dunnett's procedure

Documents

• Test theory
• Fault Management