Multiple Comparisons
Alex Bäcker, May 12, 1998
Clarified slightly June 2000
Let us assume we are making an experiment in which we are trying to decide if any of a set of measurements under experimental condition A is different from the corresponding set of measurements made under the negative control condition B. For example, we might be trying to tell whether a spike train in response to a stimulus is different from spike trains under a control condition where there is no stimulus; one measurement might be the # of spikes within a time window, the set might be given by a series of successive windows. The null hypothesis is that both sets are indistinguishable: that the response to A is no different from that to B.
If making many measurements and reporting any deviation from the value expected given the null hypothesis, the probability of finding a value equal or greater than X given the null hypothesis is not given by the probability of finding that value if one were performing a single measurement --even if one uses the probability for the measurement that actually gave the deviation.
P = p(any of N measurements >= X) = 1 - p(all N measurements < X)
If all N measurements are independent, we can write
P = 1 - p(M1<X).p(M2<X)…(p(MN>=X)
For example, if we are ma
Furthermore, if all N measurements are drawn from the same distribution and thus have the same p-values
P = 1 - [1-p(one measurement >= X)]N
Thus if we are making two independent measurements, and we wish to be as strict as if we were doing a single measurement and using a p-value of 0.05, we must make P above equal to 0.05:
But what if we do not know if the measurements are independent, or if we suspect they are not?
There are at least two possible empirical solutions:
This page has been visited times since August 26th, 1999.