Chapter 2 The Experimental Design

Now that we have introduced the basics of what constitutes a power analysis we can move onto how to use Superpower as tool for power analyses. The first step is to define and specify the details of the experimental design. Currently, Superpower only provides power analysis capabilities for ANOVAs (and MANOVA for repeated measures) so all designs are currently defined by the ANOVA_design function.

2.1 ANOVA_design function

Currently the ANOVA_design function can create designs with up to three factors, for both within, between, and mixed designs. It requires the following input: design, n, mu, sd, r, and optionally allows you to set labelnames.

  1. design: string that specifies the design (see below).
  2. n: the sample size for each between subject condition. Can be specified as a single value (when all conditions have the same sample size), or as a vector of values specifying the sample size for each condition.
  3. mu: a vector with the means for each condition.
  4. sd: the population standard deviation. Assumes homogeneity of variances (only one standard deviation can be provided). Can be specified as a single value (when all conditions have the same standard deviation), or as a vector of values specifying the standard deviation for each condition.
  5. r: the correlation(s) for within designs (or 0 for between designs). Can be specified as a single value (when all pairs of variables have the same correlation), or as a correlation matrix specifying the correlation between each pair.
  6. labelnames: This is an optional vector of words that indicates factor names and level names (see below).
  7. A final optional setting is to specify if you want to automatically print a plot or not (plot = TRUE or FALSE)

2.1.1 Specifying the design using design

The design option is used to specify the design. Every factor is specified with a number, indicating the number of levels of the factor, and a letter, b or w, to indicate whether the factor is manipulated between or within participants. For example, a 2b design has two between-participant groups. A 12w design has one factor with 12 levels, all manipulated within-participants. A 2b*3w is a design with two factors (a 2b factor and a 3w factor), the first of which has 2 between participant levels (2b), and the second of which has 3 within participants levels (3w). If there are multiple factors (the functions take up to three different factors) separate factors with a * (asterisk). An example of a 2b*3w design is a group of people in one condition who get a drug, and a group of people in another condition who get a placebo (hence 2b), and we measure their health before they take the pill, one day after they take the pill, and a week after they take the pill (hence the 3w).

2.1.2 Specifying the means using mu

Note that for each cell in the design, a mean must be provided. Thus, for a 2b*3w design, 6 (i.e., 2*3=6) means need to be entered.

Means need to be entered in the correct order. ANOVA_design outputs a plot so you can check if you entered all means as you intended. Always carefully check if the plot that is generated matches your expectations.

The general principle is that the code generates factors, indicated by the factor names you entered in the labelnames variable, (i.e., condition and time). Levels are indicated by factor names and levels (e.g., control_time1, control_time2, control_time3, etc).

If your design has just one factor, just enter the means in the same order as the labelnames (see below). For more factors, note the general pattern in the example below. Means are entered in the following order for a 3 factors design (each with 2 levels):

  1. a1 b1 c1
  2. a1 b1 c2
  3. a1 b2 c1
  4. a1 b2 c2
  5. a2 b1 c1
  6. a2 b1 c2
  7. a2 b2 c1
  8. a2 b2 c2

So if you enter the means 1, 2, 3, 4, 5, 6, 7, 8 the first 4 means correspond to level 1 of factor 1, the second 4 means correspond to level 2 of factor 1. Within the first 4 means, the first 2 correspond to level 1 of factor 2, and within those 2 means, the first corresponds to level 1 of factor 3.

The plot below visualizes means from 1 to 8 being entered in a vector: mu = c(1, 2, 3, 4, 5, 6, 7, 8) so you can see how the basic ordering works.

2.1.3 Specifying label names

To make sure the plots and tables with simulation results are easy to interpret, it really helps to name all factors and levels. You can enter the labels in the ‘labelnames’ variable. You can also choose not to specify names. Then all factors are indicated by letters (a, b, c) and all levels by numbers (a1, a2, a3).

For the 2x3 design we have been using as an example, where there are 2 factors (condition and time of measurement), the first with 2 levels (placebo vs. medicine) and the second with three levels (time1, time2, and time3) we would enter the labels as follows:

c("condition", "placebo", "medicine", "time", "time1", "time2", "time3")

As you can see, you follow the order of the design (2b*3w), and first write the FACTOR label (condition) followed by the levels of that factor (placebo and medicine). Then you write the second factor name (time) followed by the three labels for each LEVEL (time1, time2, time3). Do not use spaces or special characters in the names (so not “time 1” or “time_1” but “time1”).

Some examples:

  1. One within factor (time with 2 levels), 2w: c("time", "morning", "evening")
  2. Two between factors (time and group, each with 2 levels), 2b*2b: c("time", "morning", "evening", "group", "control", "experimental")
  3. Two between factors (time and group, first with 4 levels, second with 2 levels), 4b*2b: c("time", "morning", "afternoon" ,"evening", "night", "group", "control", "experimental")

2.1.4 Specifying the correlation

Depending on whether factors are manipulated within or between, variables are correlated, or not. You can set the correlation for within-participant factors. You can either assume all factors have the same correlation (e.g., r = 0.7), or enter the correlations for each pair of observations separately by specifying a correlation matrix.

In a 2x2 design, with factors A and B, each with 2 levels, there are 6 possible comparisons that can be made.

  1. A1 vs A2
  2. A1 vs B1
  3. A1 vs B2
  4. A2 vs B1
  5. A2 vs B2
  6. B1 vs B2

The number of possible comparisons is the product of the levels of all factors squared minus the product of all factors, divided by two. For a 2x2 design where each factor has two levels, this is:

(((2 * 2) ^ 2) - (2 * 2))/2
## [1] 6

The number of possible comparisons increases rapidly when adding factors and levels for each factor. For example, for a 2x2x4 design it is:

(((2 * 2 * 4) ^ 2) - (2 * 2 * 4))/2
## [1] 120

Each of these comparisons can have their own correlation if the factor is manipulated within subjects (if the factor is manipulated between subjects the correlation is 0). These correlations determine the covariance matrix. Potvin and Schutz (2000) surveyed statistical tools for power analysis and conclude that most software packages are limited to one factor repeated measure designs and do not provide power calculations for within designs with multiple factor (which is still true for software such as G*Power). Furthermore, software solutions which were available at the time (DATASIM by Bradley, Russel, & Reeve, 1996) required researchers to assume correlations were of the same magnitude for all within factors, which is not always realistic. If you do not want to assume equal correlations for all paired comparisons, you can specify the correlation for each possible comparison.

The order in which the correlations are entered in the vector should match the covariance matrix. The order for a 2x2 design is given in the 6 item list above. The general pattern is that the matrix is filled from top to bottom, and left to right, illustrated by the increasing correlations in the table below. The diagonal is generated dynamically (based on the standard deviation).

Table 2.1: Correlation Matrix
a1_b1 a1_b2 a2_b1 a2_b2
a1_b1 1.00 0.91 0.92 0.93
a1_b2 0.91 1.00 0.94 0.95
a2_b1 0.92 0.94 1.00 0.96
a2_b2 0.93 0.95 0.96 1.00

We would enter this correlation matrix in the ANOVA_design function as:

design_result <- ANOVA_design(design = "2w*2w",
                              n = 80,
                              mu = c(1.1, 1.2,
                                     1.3, 1.4),
                              sd = 1,
                              r <- c(0.91, 0.92,
                                     0.93, 0.94,
                                     0.95, 0.96),
                              labelnames = c("COLOR",
                                             "red", "blue",
                                             "SPEED",
                                             "fast", "slow"),
                              plot = TRUE)

We can check the correlation matrix by printing it from the design_result object to check if it was entered the way we wanted:

design_result$cor_mat
Table 2.2: Correlation Matrix
red_fast red_slow blue_fast blue_slow
red_fast 1.00 0.91 0.92 0.93
red_slow 0.91 1.00 0.94 0.95
blue_fast 0.92 0.94 1.00 0.96
blue_slow 0.93 0.95 0.96 1.00

We should also check the covariance-variance matrix to ensure the ANOVA_design function is working properly. The variance should be the diagonal element while the off-diagonal elements should be equal to covariance = correlation*variance or \(cov_{x,y}=\frac{\sum_{i=1}^{N}(x_{i}-\bar{x})(y_{i}-\bar{y})}{N-1}\). In this case, it is identical to the correlation matrix because the variance is equal to 1.

design_result$sigmatrix
Table 2.3: Covariance-Variance Matrix
red_fast red_slow blue_fast blue_slow
red_fast 1.00 0.91 0.92 0.93
red_slow 0.91 1.00 0.94 0.95
blue_fast 0.92 0.94 1.00 0.96
blue_slow 0.93 0.95 0.96 1.00

2.1.5 Specifying the sample size

You can set the sample size per condition by setting a value for n. It is beneficial to have a fully balanced design (the same sample size in each condition) in case of any violations of assumptions (such as homogeneity of variances), but this might not always be possible in real life, and it can be more efficient to collect a larger sample size in some conditions (e.g., if the population from which other participants are drawn is more difficult to collect). Superpower allows you to enter a single value (the same sample size in each condition) or a vector of values (different sample sizes in each condition).

This means for the 2w*2w design above there is a total of 80 participants/subjects with a total of 320 observations. Instead of 80 participants in each condition, for a 2b*2b between subjects design, we could also specify the sample size as n = c(60, 80, 80, 60) if we plan to collect 60 participants in 2 conditions and 80 participants in 2 other conditions.

2.1.6 Specifying the standard deviation

You can set the standard deviation(s) by setting a value of sd. You can enter either a single standard deviation for all conditions, or you can violate the assumption of homogeneity of variance by entering a vector with different standard deviations for each conditions (e.g., sd = c(0.8, 1.2, 0.92, 1.12) for a 2b*2b design. Violation the homogeneity of variances assumption will affect the type I error rate if the differences in sd between conditions are extreme. Note that there is always some uncertainty in which values you can expect in the study you are planning. It is therefore useful to perform sensitivity analyses (e.g., running the simulation with the expected standard deviation, but also with more conservative or even worst-case-scenario values).