Download Applications of the IML Procedure for Multiple Response Permutation Tests

APPLICATIONS OF THE IML PROCEDURE FOR MULTIPLE
RESPONSE PERMUTATION TESTS
William D. Johnson and Donald E. Mercante
William D. Johnson, Lonisiana State University Medical Center
New Orleans, LA 70112-1393
Keywords :PROC IML, Multivariate Analysis,
Multiresponse Permutation Procedures, Clinical
Trials
adjusted p-values may be better than their unadjusted
counterparts, they are not suitable alternatives to a
p-value based on an appropriate global test.
Abstract
The usual parametric statistics require two related
sets of assumptions regarding the data. First, the data
for each subject must be measured on a common
scale at least On the ordinal level. Second, the data
must satisfy the distributional assumptions Of the
statistical technique. Typically, the data must be
normally distributed or amenable to a transformation
which will make the data normally distributed.
This article discusses statistical methods for the
analysis of multivariate data in experiments with a
relatively small number of subjects. Methods based
on the assumption of multivariate normality have
power that decreases rapidly as the number of
variates increases for a fixed sample size. In some
investigations, the number of variates exceeds the
sample size creating an awkward dilemma in
classical multivariate analysis. Permutation tests
have many useful applications especiaJly where the
assumptions such as normal distributions and
hornescedasticity are in question. Additionally, the
number of variates relative to the sample size is not
an issue with these tests ..
Permutation tests based' on randomization theory
provide an alternative class of tests that are often
overlooked in multivariate analysis. Although
permutation tests are well known and frequently used
in the analysis of univariate data, they are used
infrequently in multivariate analysis presumably
either because the procedures are not well known or
computer programs are not readily available for the
computations. Mielke and his colleagues described
a multiresponse permutation procedure and demonstrated its use in a variety of applications especially
in meteorological investigations [1-7]. The test is
exact but may require an inordinate amount of
computing time for a large number. of permutations.
Mielke [1] developed an accurate approximation to
the exact permutation test based on the beta
distribution that is appropriate for large sample sizes.
Our experience has been that the approximation
yields p-values very close to those of the exact test
even for modest sample sizes and the computer time
required is negligible.
We discuss a multivariate permutation test and
present a readily implemented IML procedure for
constructing these tests. Software for the multiple
response permutation tests was available previously
in the SAS® Supplemental Library as PROC MRPP
but has been discontinued in SAS Version 6. Our
IML program in, SAS Version 6 fills this void as
illustrated in the three examples presented.
Introduction
Clinical trials and other experiments often involve a
small number of subjects and a relatively large
number of observations on each subject; In some
instances, the multivariate data structure involves a
single variate observed repeatedly either under
different experimental conditions and/or longitudinally over time. The analysis is often approached
through a repeated measures or split-plot analysis of
variance in these situations. When a number of
different types of variates must be dealt with in the
analysis and the sample size is small, many analysts
focus on one variate at a time and perform separate
univariate analyses. Some analysts attempt to adjust
the resulting p-values for multiple testing, but while
The method is based on the concept of multidimensional distances between subjects and utilizes
permutation tests to make group comparisons. The
response variates must be measured on at least an
ordinal scale. For a given set ofdata, the analyst can
choose a metric from a family of distance functions.
In most applications, either Euclidean or squared
Euclidean distance is an appropriate choice. To
carry out tests of hypotheses, a permutation
procedure can be used to determine the exact p-value
or it can be estimated by the approximate procedure.
1027
Alternatively, one could use the pennutation
procedure to estimate the p-value by evaluating a
random sample of all possiblepennlltations.
An exact probability requires the complete
enumeration ·of all pennutations of the n subjects
considered nJ, n2, ... , ng at a time. There are M =
n!(nl!,n2!, ... ,ng!)-1 such pennutations. The test
statistic d is calculated for each of the M
pennutations. Under the null hypothesis, each of the
M pennutations has probability ~ of occurrence, the
exact p-value is detennined as the proportion of ~
values less than or equal to the realized (observed) d
value. In practice, the number of pennutations can
quickly become fonnidable even for moderate
sample sizes. If the maximum number of pennutations to be perfonned is large, we switch to random
sampling of all possible pennutations to detenniDe
the significance of the observed data or implement
an approximate test based on the beta distributioil.
For brevity we do not discuss the approximate test
here but rather refer the reader to Mielke [I].
The purpose of this article is to describe pennutation
tests for the analysis of small sample multivariate
data sets; The next section gives a mathematical
description of the multivariate pennuta\ion test. The
subsequent section provides a brief description of the
statements required to execute the program. Its use
is further illustrated with examples in later sections.
Multivariate Permutation Test
The test criterion is based on a weighted group
average ofa mean between subject distance function
for all subjects within a group. The distance function
may be based on the raw data or linear functions of
the raw data. The linear functions are used to
generate data structures for testing certain hypotheses of interest such as parallelism for
longitudinal data. In this context, the hypothesis to
be tested is
H. : Switching subjects among groups should have
no effect on the averag~ within and among
group distances,
and the alternative is
HA : Switching subjects among groups increases
the average within group distances and
decreases the average among group distance.
Let g denote the number of groups to be compared in
a completely randomized design with nJ,n,. ... ,ng
subjects in each group where n,,, 2; and n = ~n;.
Let y~ = {YI]'Y21, ••• ,YpI} denote p commensurate
observations on subject 1. Let each· pair of pdimensional response vectors be summarized by a
symmetric distance function such as
dIJ=
d; = ( n;
2
)
-I
.
To illustrate the use of the program consider the data
in Table I. These data represellt two samples
(freannents 1 and 2) of bivariate data (y, and Y J
with sample size n, "" 4 and n, = 3, respectively.
Table 1. Sample data to illustrate program.
Treatment 1
Y,
!dIJ
f wid"~
=2 for squared
be the average distance
I<J
cards;
4 I
34
where Wi is an arbitrary weighting
!if > 0 with i=1f W; = I
Y,
Y,
6
3
3
4
7
4
2
7
6
3
2
3
Data MRPP;
Input Y, Y,;
i=1
constant chosen to be w, =
Treatment 2
A SAS data set was created in the usual way and
used as input into the IML program. Alternatively,
the data can be input directly into the Y matrix in
IML. The following SAS statements provide an
example of how to use the program. Expressions in
bold face may need to be modified for the particular
application at hand.
between subjects for all subjects within the ;"' group,
(; = 1,2, ... , g). Then the t~ statistic of interest is
the weighted average ofthe d; values and is given by
d=
Y,
4
[~I (YkI-Ykl)2Y
where v = I for Euclidean distance, v
Euclidean distance, etc.
Let
Program Execution
in
this paper.
1028
Example: Application of an IN VIVO Receptor
Binding Assay
6 3
PROCIML;
Start;
UseMRPP;
Read ALL into Y ;
j* Begin User-Specified Parameter Settings *j
N_Vec = { 4, 3 } ; j* Sample Size Vector *j
M = I( NCOL(Y) ); j* Transfonnation Matrix *j
j* End User-Specified Parameter Settings *j
Mercante and Johnson [8) presented data to illustrate
use of the multivariate permutation test. Data on
thirteen mice were collected to assess the acute
effects of lorazepam and alprazolam on the. binding
of 3!lC i [3H]RoI5-1788 in selected brain
regions. The mice were randomly assigned to 3.2
mglkg lorazepam, 1.8 mglkg alprazolam ot saline.
AU mice underwent chronic administration of the
treatment and received 3 !lCi [3 H]R. 15 - 1788
approximately 24 hours following the chronic phase
of treatment. After twenty minutes the mice were
decapitated and their brain tissue analyzed. Binding
in cpmlmg was determined in six sections of .the
brain: Y I = cerebellum, Y, = brain stem,Y, = cortex,
Y. = hypothalamus, Y, = striatum and Y. = hippocampus.
(remainder ofIML program).
.The permutation test statistics for the 35 permutations comprise the complete null distribution
Permutation number one,
shown in Table 2.
corresponding to the original data (ie, no permutation
of the data) was found to be the most extreme over
the pennutation distribution (p-value = 1/35). We
conclude that there is a statistically significant
difference in the response distribution for the two
treatments. The responses Y I tended to be higher for
Treatment 2 but the converse trend was observed for
response Y,.
For the observed data the MRPP test statistic was
-
d = 1l.3. From a total of 90,090 permutations,
3,585 yielded more extreme test statistic values
giving a p-value of 0.0398. For purposes of
comparison, the approximate test resulted in a
A traditional multivariate
p-value of 0.0354.
analysis of variance yielded the following:
Test
Table 2. All possible test statistics for the multiple
response permutation test performed on
the data in Table I.
Perm
Perm
#
Test
Stat
Perm
#
Test
Stat
I
2.79
13
4.06
25
2
4.3
2.98
14
4.07
26
4.32
3
3.29
15
4.09
27
4.33
4
3.59
16
4.15
28
4.33
5
3.6
17
4.15
29
4.34
#
Test
Stat
6
3.65
18
4.1 7
30
4.34
l
,l
7
3.72
19
4.18
31
4.4
i
8
3.81
20
4.19
32
4.42
9
3.84
21
4.2
33
4.43
10
3.85
22
4.22
34
4.46
II
3.94
23
4.24
35
4.52
12
4.06
24
4.28
i
,~
P-Value
Wilk'sLR
0.094
Pillai's Trace
0.089
Hotelling-Lawley Trace
0.119
Roy's Greatest Root
0.025
Thus, the MRPP detected a statistically significant
difference among the groups and in this example
appears to be more sensitive to group differences
compared to the traditional multivariate parametric
tests except for Roy's Greatest Root.
Example:
Application in Generalized Anxiety
Disorder
A double-blind, placebo-controlled randomized
clinical trial was conducted to investigate the
possible axiolytic properties of a psychotropic
compound. Nine subjects were randomly allocated
to the placebo group and fifteen to the active
treatment group.
Five response variates were
observed: anxiety as determined using the total
score from the 21-item Hamilton Anxiety Scale
1029
Severity of illness and depression as measured on the
SCL-90 instrument were not statistically significant.
The M matrix used to generate a transformation of
the data suitable for testing no treatment by time
interaction (or, equivalently, parallel treatment
profiles over time) is given by M = M'®Is where
(Ham A), depression as determined using the total
score from the Hamilton Depression Scale (Ham D),
severity of illness (SJ), depression as determined
using the total score from the depression cluster of
the SCL-90 questionnaire, and anxiety as determined
using the total score from the anxiety cluster of the
SCL-90 questionnaire.
The patients were required to take the medication
daily throughout the 10-week study. They were also
required to return to the clinic biweekly for
evaluation resulting in response values measured on
each of 6 visits. Each response vector can thus be
viewed as multivariate longitudinal data and are
described in further detail in Mercante and Johnson
[9]. There were a total of 5 x 6 = 30 possible
responses for each patient. A few patients dropped
out of the study and so data were missing on their
later visits. We used the last observation carried
forward method to substitute data for the missing
values.
-I 0 0 0 0
o -I 0 0 0
o 0 -I 0 0
o 0 0 -I 0
o 0 0 0 -I
Because of the small sample sizes, a Hotelling T 2
test for treatment by time interaction on all 5 types,of
response simultaneously, was not feasible. However,
the permutation test was statistically significant
(p=0.023). We can average the response profiles
over time by iaking M = 16®Is where 16 is a 6xl
vector of ones. We should be cautious in' our
interpretation of the second test if the corresponding
test for interaction is significant. The results are
presented here merely to illustrate the test
constructions.
The total number of subjects in the study was 24 with
30 response variables and therefore an overall
multivariate analysis of variance using traditional
parametric methods was not possible. The permutation tests were carried out on the transformed data
obtained by multiplying the 2x30 data matrix by the
appropriate M matrix (chosen to generate functions
of the daia relevant to the analysis). The test of the
global hypothesis of no between treatment group
location shift in the Ix30 response vectors was
obtained by iaking the transformation matrix M to be
the 30x30 identity matrix I,.. The p-values were
0.037 and 0.035 for the exact and approximate tests,
respectively. Sub-hypotheses pertaining to the five
scale types were also tested by iaking M to select the
response at each of the six time points for each
variate in turn. For example, for the sub-hypothesis
pertaining to Ham A, we take
Hotelling's T2 tests for treatment effect on averages
over time were not significant. Conversely, the
permutation test was significant for all variate types
except SCL-90 depression. Here we see an entirely
different interpretation of these data because the
power of the parametric multivariate test is so low in
this case. Even when response is averaged over time
or linear effects over time are considered, the
conclusions that can be made are in conflict with
those resulting from the permutation tests.
Concluding Remarks
The fact that Hotelling's T2 and other traditional
multivariate tests lose power as the number of
response variables increases relative to the sample
size is well known. Multivariate permutation tests
offer a method of analysis that is not only
theoretically sound and requires minimal assumptions for validity but is also valid when the number
of response variates exceeds the total sample size.
Software for the multiple response permutation tests
was available previously in the SAS Supplemental
Library as PROC MRPP but has been discontinued
in SAS Version 6. We have developed the IML
program in SAS Version 6 to fill this void. We
M-[ 0::.6 ]
where 0""" is a 24x6 matrix of zeros. The tests
corresponding to the five sub-hypotheses yielded
statistically sigDificant tests for Hamilton anxiety
and depression. The tests for severity of illness and
SCL-90 depression were borderline significant while
that for SCL-90 anxiety was not statistically
significant.
From these results, we conclude that the active
treatment was effective in decreasing both anxiety
and depression as measured on the Hamilton scales.
1030
illustrated its use in three specific examples.
However, it can be used in a wide range of problems
in data analysis. The method is appropriate when the
sample size is small relative to the number of
response variates and is especially useful for
perfonning global testing when the number of
response variates exceeds the sample size. The
response vectors must be measured on at least the
ordinal scale, otherwise, many of the same strategies
employed in parametric data analysis can be used.
This may require linear transformations on the raw
data as in the third example.
3.
Mielke PW, KJ Berry and GW Brier,
Applications of multi-response permutation
procedures for examining seasonal changes in
monthly mean sea-level pressure patterns,
Monthly Weather Review 109 (1981) 120126.
4.
Mielke PW, KJ Berry, PJ Brockwell, and JS
Williams, A class of nonparametric tests based
on multi-response permutation procedures,
Biometrika 68 (1981) 720-724.
5.
Berry KJ and PW Mielke, Computation of
finite population parameters and approximate
probability
values
for
multi-response
permutation procedures (MRPP), Communications in Statistics: Simulation and Computations B-12 (1983) 83-107.
6.
Berry KJ and PW Mielke, Computation of
exact probability values for multi-response
permutation procedures (MRPP), Communications in Statistics: Simulation and Computations 13-3 (1984) 417-432.
7.
Zimmerman GM, H Goetz and PW Mielke,
Use of an improved statistical method for
group comparisons to study the effects of
prairie fire, Ecology 66-2 (1985) 606-611.
8.
Mercante DE and WD Johnson, The analysis
of small sample multivariate data with·
applications in cliuical trials, Journal of Biopbarmaceutical Statistics, in press.
9.
Mercante DE and WD Johnson, The analysis
of small sample multivariate data in clinical
trials, Proceedings of the Biopharmaceutical
Subsection, Joint Meetings of the American
Statistical Association, Biometric Society and
IMS, Boston, 1992.
SAS is a registered trademark of SAS Institute, Inc.
In the USA and other countries. ® indicates USA
registration.
Autbo..s· CUJ"rent Address:
Department of Biometry and Genetics
Louisiana State University Medical Center
190 I Perdido Street
New Orleans, LA 70112-1393
(504) 568-6150
BTINET: PATHDEM@NNOMED
R.eferences
I.
Mielke PW, On asymptotic non-normality of
null distributions of MRPP statistics,
Communications in Statistics: Theory and
Methods A8-15 (1979) 1541-1550.
2.
Mielke PW, KJ Berry and ES Johnson,
Multi-response permutation procedures for a
priori classifications, Communications in
Statistics: Theory and Methods AS-14 (1976)
1409-1424.
r
Ig
I
t,
I
l
1031