[1] 4.372651
Part 2: Multiple Comparison Procedures
We know, of course, that the overall error rate when we make multiple tests is larger than \(\alpha\) (and possibly much larger). To compensate for this, several different multiple comparison procedures have been proposed to control various error rates related to the overall error rate.
There are several of these; we will consider the following:
Fisher’s LSD Procedure
Tukey’s W Procedure
Scheffe’s Procedure
These are different procedures that each controls a different kind of error rate and each is more or less conservative than others. Each has its set of fans among researchers.
Each procedure is constructed to control a certain kind of error rate and it is important for a user to be aware of what error rate is controlled by a procedure before using it. We will try to state how conservative each one is as we discuss it.
The procedure is used for making all possible comparisons between pairs of means \(H_0: \mu_i - \mu_j = 0\) vs. \(H_a: \mu_i - \mu_j\neq 0\). It presumes we rejected \(H_0: \mu_1 = \mu_2 = \cdots =\mu_t\).
For equal sample sizes \(n_1 = n_2 = \cdots = n_t = n\), consider the \(t\)-test of the hypothesis above. \[t = \frac{(\bar{y}_{i.} - \bar{y}_{j.})}{ (s_W \sqrt{2/n} \;)}\]
We reject \(H_0\) when \(|t| \geq t_{\alpha/2}\), This is equivalent to rejecting \(H_0\), for a pair of (\(i,j\)) whenever \[|\bar{y}_{i.} - \bar{y}_{j.}| \geq t_{\alpha/2} \, s_W \, \sqrt{2/n } \ .\]
The right hand member of this inequality is not a function of \(i\) or \(j\). It is constant for a specified \(\alpha\) and \(n\), and is called the Least Significant Difference or LSD.
Once the LSD is calculated, doing the tests for the pairs of differences of the form \(H_0: \mu_i - \mu_j = 0\) is simple: Form all possible absolute differences \(|\bar{y}_{i.} - \bar{y}_{j.}|\) and reject the corresponding \(H_0\) if this difference exceeds or equals the LSD.
Testing the hypotheses is thus easy, but reporting the results of all those tests can be messy. For \(t\) means, there are \(t(t-1)/2\) differences to test.
To minimize the number of comparisons we need to make, first arrange the \(\bar{y}_{i.}\)’s ordered smallest to largest in value. If we use the notation \(\bar{y}_{(i)}\) for the \(i\)-th smallest \(\bar{y}\), the ranked means may be represented as \(\bar{y}_{(1)} \leq \bar{y}_{(2)} \leq \bar{y}_{(3)} \leq \cdots \leq \bar{y}_{(t)}\)
For e.g., \(\bar{y}_{(1)}\) might be \(\bar{y}_7\) if \(\bar{y}_{7}\) is the smallest; Now note that if the difference \(\bar{y}_{(t)}- \bar{y}_{(1)}\), for example, does not exceed the LSD, then all the differences \(\bar{y}_{(t)}- \bar{y}_{(2)}\), \(\bar{y}_{(t)}- \bar{y}_{(3)}\ldots\), \(\bar{y}_{(t)}- \bar{y}_{(t-1)}\) will not exceed the LSD.
It follows that in this case we are spared from computing all the above differences and comparing them to the LSD. The following procedure is based on this idea:
First write the ordered means on a line identified by their corresponding treatment names above them.
For Example – we might have
| trt5 | trt3 | trt1 | trt4 | trt2 |
| 9.5 | 10.5 | 11.6 | 12.2 | 13.5 |
Take each column in turn, and on a separate line below the list, starting from column 1 connect the means by underlining those pairs of means that are not significantly different from the mean in the current column, in the following way.
Start the comparison of the mean \(\bar{y}_{(1)}\) with the mean on the last column \(\bar{y}_{(t)}\). We know that if this pair is less than the LSD value, then none of the differences \(|\bar{y}_{(1)}-\bar{y}_{(t-1)}|\) will exceed the LSD value. If so, underline the means connecting \(\bar{y}_{(1)}\) with \(\bar{y}_{(t)}\)
Otherwise, move left to the next largest mean \(\bar{y}_{(t-1)}\) and compare \(\bar{y}_{(1)}\) with \(\bar{y}_{(t-1)}\), and so on.
Begin underlining at that column the the difference is found to be less than the LSD value and extend all the way to the left to column 1 (or the column where you started)
This line implies that those means that are connected with this line are not significantly different from the mean in column 1 and all means between.
Now restart at column 2 (i.e., \(\bar{y}_{(2)}\) and repeat the procedure the same way as above. The new set of underlines will be displayed in a separate line. For Example – we might have
| trt5 9.5 | trt3 10.5 | trt1 11.6 | trt4 12.2 | trt2 13.5 |
Supposed that the computed sample means of six treatments with equal sample size 5 (i.e. \(n=5\)) are:
\[\bar{y}_{1.} = 505, \ \bar{y}_{2.} = 528, \ \bar{y}_{3.} = 564, \
\bar{y}_{4.} = 498, \ \bar{y}_{5.} = 600, \ \bar{y}_{6.} = 470\]
Since MSE \(=s^2_W=2,451\) with 24 d.f. Thus the LSD is:
LSD \(=2.064\sqrt{2(2451)/5}=64.63\).
Ordered smallest to largest, the means are:
\(\bar{y}_{6.},\,\bar{y}_{4.},\,\bar{y}_{1.},\,\bar{y}_{2.},\,\bar{y}_{3.},\,\bar{y}_{5.},\equiv 470,\, 498,\, 505,\, 528,\, 564,\, 600\)
Prepare table to be used in the underlining procedure:
Using LSD \(= 64.63\), underlining procedure is done as follows:
| trt6 470 | trt4 498 | trt1 505 | trt2 528 | trt3 564 | trt5 600 |
These may lead to one or more of the following conclusions:
\(\mu_6, \mu_4, \mu_1, \mu_2\) are significantly different from \(\mu_5\).
\(\mu_6, \mu_4\) are significantly different from \(\mu_3\).
None of \(\mu_6, \mu_4, \mu_1\) is significantly different from \(\mu_2\).
None of \(\mu_6, \mu_4\) is significantly different from \(\mu_1\).
\(\mu_6\) is not significantly different from \(\mu_4\).
When sample sizes are not equal the above procedure is not feasible. In this case, we may construct confidence intervals for all pairs of differences \(\mu_i - \mu_j\) using \[\bar{y}_{i.} - \bar{y}_{j.} \pm t_{\alpha/2} \, s_W \, \sqrt{\frac{1}{n_i} + \frac{1}{n_j}}\] where \(t_{\alpha/2}\) is again the percentile from the t-table with degrees of freedom same as that of the within mean square \(s_W^2\).
The protected LSD has a per-comparison error rate of \(\alpha\), i.e., the probability of a Type I error is \(\alpha\) for any single comparison (or test). However, as we already discussed, the overall error rate when multiple tests are made can be much larger than \(\alpha\), i.e., the probability of making one or more Type I errors exceeds \(\alpha\).
The protected part involves making sure that \(H_0: \mu_1 = \mu_2 = \cdots = \mu_t\) is tested using the analysis of variance F-test prior to using the multiple comparison procedure.
The LSD analysis is carried out only when \(H_0\) is rejected. There is some evidence, based on simulation studies that the experimentwise error rate for protected LSD may be near \(\alpha\).
The experiment-wise error rate is the probability of observing an experiment with one or more pairwise comparisons falsely declared significant.
Protected LSD is not a very conservative method. We would not be surprised to see it falsely declare several pairwise comparisons significant in an experiment involving several treatments when all possible differences are tested.
This procedure should not be used to make tests suggested after the experiment has been conducted and the sample means computed. At the planning stage of an experiment, the experimenter must state all questions that needs to be answered in terms of possible comparisons. These comparisons are called pre-planned or apriori comparisons.
Instead of pre-planned comparisons, a part of the plan for the experiment may require testing all differences or only some of them. The intent of LSD is to not to perform all paiwise comparisons routinely.
In any case, it is not recommended that any kind of comparisons be devised after first looking at the \(\bar{y}\)’s. The problem with testing based on comparisons suggested by looking at the data is that it changes the \(\alpha\) level of the test i.e, Type I error rate is not controlled at the specified \(\alpha\) level anymore.
For example — an extreme case — say you look at the sample means and see that the largest is much greater than the smallest, so you decide to test their difference for significance. On average — across experiments — you will seldom fail to reject \(H_0\) when you do this, so the Type I error rate is probably not \(\alpha\).
This method for comparing all possible pairs is more conservative than LSD (i.e., it tends to be more resistant to falsely declaring significance.)
The method is based on comparisons of \(|\bar{y}_{i.} - \bar{y}_{j.}|\) to the value \[W = q_\alpha(t,\nu) \sqrt{\frac{s_W^2}{n}}\] where \(s_W^2\) is the mean square within samples all of size \(n\), \(\nu\) is the degrees of freedom for \(s_W^2\), \(t\) is number of population means, \(\mu_i\) compared, and \(\alpha\) is the chosen significance level.
Then if \[|\bar{y}_{i.} - \bar{y}_{j.}| \geq W\] we declare that the mean pair \(\mu_i\) and \(\mu_j\) are significantly different.
The value of \(q_\alpha(t,\nu)\) is determined by the Studentized range distribution (R: xtukey() functions).
The the sample means are ordered smallest to largest as before. Then make all possible pairwise comparisons using the value of \(W\) and underlining method may then be used to display results.
The ANOVA table resulting from an experiment involving 6 treatments and \(n=5\) per treatment is:
| Source of Variation | DF | SS | MS | F |
|---|---|---|---|---|
| Between Treatments | 5 | 847.05 | 169.41 | 14.37 |
| Within Treatments | 24 | 282.93 | 11.79 | |
| Total | 29 | 1129.98 |
The sample treatment means are: \(\bar{y}_{1.}= 28.8, \bar{y}_{2.} = 24.0, \bar{y}_{3.} = 14.6, \bar{y}_{4.} = 19.9, \bar{y}_{5.} = 13.3, \bar{y}_{6.} =18.7\)
The ordered means table:
| trt5 | trt3 | trt6 | trt4 | trt2 | trt1 |
| 13.3 | 14.6 | 18.7 | 19.9 | 24.0 | 28.8 |
[1] 4.372651
The underlining procedure gives:
| trt5 13.3 | trt3 14.6 | trt6 18.7 | trt4 19.9 | trt2 24.0 | trt1 28.8 |
Means that have an underline in common are declared not significantly different from each other.
Thus we find that \(\mu_5,\, \mu_3,\, \mu_6,\) and \(\mu_4\) are not different from \(\mu_1\), and \(\mu_5\) and \(\mu_3\) are not different from \(\mu_2\).
Just as with LSD, Tukey’s method can be used when sample sizes are not all the same, but the above procedure is not feasible. In this case we may construct confidence intervals for all pairs of comparisons \(\mu_i - \mu_j\). Its form is \[\bar{y}_{i.} - \bar{y}_{j.} \pm q_\alpha (t,v) \sqrt{\frac{s_W^2}{2}
\left(\frac{1}{n_i} + \frac{1}{n_j} \right) }\] The value of \(q_\alpha (t,\nu)\) from Table 10 is obviously the same for all comparisons as well as \(s_W^2\).
This procedure is ultra conservative. It controls experiment-wise error rate. The probability of observing an experiment with one or more contrasts (from the set of all possible contrasts) falsely declared significant is the selected \(\alpha\).
Scheffe’s method can be used to test all possible differences of means (recall that simple differences are contrasts). However it is usually used where contrasts that are not all simple differences are to be tested together with any pairwise differences.
To test \(H_0: \ell = \sum_i a_i\mu_i = 0 \ \rm{vs.}~ H_a: \ell \neq 0\)
we base the test statistic on the estimate \(\hat{\ell} = \sum a_i \bar{y}_{i.}\)
Compute the quantity \(S\) based on a \(F\)-distribution as \[S = \sqrt{\hat{V}(\hat{\ell})} \ \sqrt{(t-1)F_{\alpha,df_1,df_2}}\]
Here \(df_1 = t-1\), and \(df_2 = \nu\)
The variance estimate of \(\hat{\ell}\) is \(\hat{V}(\hat{\ell}) = s_W^2 \sum_i \frac{a_i^2}{n_i}\) where \(s_W^2\) has \(\nu\) degrees of freedom.
We reject \(H_0\) when \(|\hat{\ell}| > S\).
Now the underlining procedure is applied in the same way as described for the LSD or Tukey procedures.
Using the emmeans package, we can easily access this functionality. For all pairwise comparisons, we can use the pairs() function on the object produced by emmeans().
STAT 801A