FROM MANOVA TO GLM: MORE BASICS OF PARAMETERIZATION

David P. Nichols

From SPSS Keywords, Number 65, 1997

In the last issue, I began discussing the differences between the ways in which the MANOVA and GLM procedures parameterize linear models with categorical or factor variables. I noted that GLM always uses one particular kind of parameterization, referred to as an overparameterized indicator design matrix, while MANOVA produces a design matrix based specifically on what kind of contrasts the user specifies, or else on defaults. The parameterization used in GLM was referred to as canonical or basic because it represents the model in a somewhat more fundamental way than do the MANOVA parameterizations.

An important feature of such a model illustrated by the GLM design matrix is the fact that there is no unique solution to the set of equations defined by this matrix. Another way to say this is that the model is _overparameterized_. As noted last time, GLM deals with this by using a generalized inverse produced via a symmetric sweep operator applied to the X'X matrix. The solution produced for a particular problem by MANOVA will depend on the contrast specifications. Figure 1 presents the design matrix used by GLM and the default design matrix used by MANOVA for the case of a single three level factor A.

Figure 1: Design Matrices for GLM and MANOVA defaults
------------------------------------------------------------------------------- 
                    GLM Design Matrix                 MANOVA Design Matrix
Level of A      C       A1      A2      A3             C       A1      A2
    1           1       1       0       0              1       1       0 
    2           1       0       1       0              1       0       1
    3           1       0       0       1              1      -1      -1
------------------------------------------------------------------------------- 
 

Perhaps the most obvious difference between the two matrices is the number of columns in each. The GLM matrix reflects the overparameterized nature of the model by identifying four parameters. The MANOVA matrix has been reduced to the number of nonredundant parameters available for estimation (three: one for each group mean), via what is known as _reparameterization_. That is, the MANOVA design matrix can be constructed from the more general GLM design matrix by factoring the GLM matrix into the product of two matrices: the MANOVA design or basis matrix, and a contrast matrix whose rows show the interpretations of the parameter estimates produced by use of the MANOVA design matrix.

Another notable difference is the fact that the GLM design matrix contains only ones and zeros, so that when computing the predicted value for a given case via the linear model equation

^    ^
Y = XB

you would need only to identify which parameter estimates are used (have a 1 for that row), and sum these. The design matrices from MANOVA will have numbers other than 0 and 1, including negative numbers, and sometimes very complicated decimal values. This makes reproducing predictions much easier in GLM than in MANOVA.

As noted above, with three groups, there are three means to estimate, and the basic model, represented by the GLM design matrix, is overparameterized. GLM handles such situations by aliasing redundant or linearly dependent parameters to 0. In this example, the parameter A3 is fixed at 0, since the A3 column is equal to C-A1-A2. This method is sometimes referred to as using set to 0 restrictions. MANOVA handles this by reparameterizing the model, using a method that is sometimes referred to as using sum to 0 restrictions (note that each A column in the design matrix sums to 0). MANOVA offers a variety of ways to define the design or basis matrix, each one producing a different set of parameter estimates and contrast coefficients. All of these methods produce K-1 contrasts for a K level factor, and all imply design matrix columns that sum to 0 for each of these K-1 columns.

Since the A3 parameter in GLM is fixed to 0, and we know that the predicted value for a case in group 3 is that group's mean for a one factor model, we know that the C parameter in GLM must be estimating the mean of the third group. Since the predicted values for cases in the other groups are given by C+A1 and C+A2, A1 and A2 must be estimating the differences between the other two group means and the final one. As noted last issue, these are the same interpretations resulting from a model using just two dummy variables representing each of the first two groups.

With MANOVA's sum to 0 approach, the estimate for the constant column always produces the unweighted average of the cell means in a single factor design. The particular contrast coding used here (the default), produces estimates for A1 and A2 that are the deviations of each of the first two groups from this unweighted grand mean. The deviation of the final group from the mean is the negative of the sum of the other deviations. While all of the contrast options in MANOVA produce the same type of constant or intercept term, the estimates for the A terms will differ depending on what type of contrasts are specified.

The important point to note here is that while the parameters estimated by the two procedures (or the different choices in MANOVA) are different, the underlying model, the predicted values for a given case, and the summary measures of prediction error (such as the sum of squared residuals) are always the same. With regard to parameters, while there is no unique value resulting for the constant or intercept, nor for any of the A terms, any contrast (linear combination whose coefficients sum to 0) among the A terms will have an unique value, regardless of how the model is parameterized. The invariance of contrasts among the A parameters is of such importance that it occupies a central place in theoretical treatments of statistical estimation. That central concept will be the focus of the next installment.