Power and Sample Size for an Intraclass Correlation Test

Computes the power of the F test of an intraclass correlation coefficient (ICC) used by icc_test(), or solves for the number of subjects, the number of measurements per subject, the ICC under the alternative, or the significance level that gives a target power. Exactly one of n, k, rho1, power and sig.level must be NULL; that one is computed from the others, as in stats::power.t.test().

Usage

power_icc(
  n = NULL,
  k = 2,
  rho0 = 0,
  rho1 = NULL,
  sig.level = 0.05,
  power = NULL,
  model = c("oneway", "twoway"),
  unit = c("single", "average"),
  missing = 0,
  alternative = c("two.sided", "greater", "less"),
  method = c("exact", "approximate")
)

Arguments

n: Number of subjects.
k: Number of measurements (items: trials, raters or occasions) per subject. When solved for, the smallest whole number of at least 2 that reaches the target is returned.
rho0: The ICC under the null hypothesis (default 0).
rho1: The ICC under the alternative hypothesis.
sig.level: Significance level (Type I error rate; default 0.05).
power: Power of the test (1 minus the Type II error rate).
model: "oneway" (default) or "twoway". "twoway" means the consistency ICC, ICC(C,1) of icc_test(); the absolute agreement ICC is not supported (see Details).
unit: "single" (default) for the reliability of a single measurement, or "average" for the reliability of the mean of the k measurements (ICC(k) or ICC(C,k)), as in icc_test(). With "average", the ICC values are on the average-measure scale and k cannot be solved for. See Details.
missing: The expected proportion of measurements (subject-by-item cells) that will be missing, analysed with icc_test(method = "reml"). The default, 0, gives the exact complete-data calculation; values above 0 give an approximation (see Details). For subjects who will be lost entirely, reduce n instead.
alternative: "two.sided" (default), "greater", or "less", as in icc_test(). A design planned with the defaults matches the analysis run with the defaults.
method: "exact" (default) computes power from the exact distribution of the F statistic. "approximate" uses the closed form of Walter, Eliasziw and Donner (1998), available only for model = "oneway".

Value

An object of class "power.htest", printed by stats::print.power.htest(), with elements n, k, rho0, rho1, sig.level, power, alternative, method and note.

Details

Exact power. For the one-way and the two-way consistency ICC with complete data, the F ratio of the subject mean square to the error mean square is a scaled central F variable whatever the true ICC $\rho$: $$F \sim \frac{1 + (k-1)\rho}{1 - \rho} F_{n-1, \nu_2},$$ with $\nu_2 = n(k-1)$ (one-way) or $(n-1)(k-1)$ (two-way). The test of $H_0: \rho = \rho_0$ (McGraw & Wong, 1996) divides the F ratio by the same factor at $\rho_0$, so its power is a single F probability (Donner & Eliasziw, 1987). Non-integer n is allowed, as in stats::power.t.test(); round up to the next whole number of subjects.

Approximate power uses the closed form of Walter, Eliasziw and Donner (1998), $$n = 1 + \frac{2k (z_{1-\alpha} + z_{1-\beta})^2}{(k-1)(\log C_0)^2}, \quad C_0 = \frac{1 + k\theta_0}{1 + k\theta_1}, \quad \theta = \frac{\rho}{1 - \rho},$$ with $\alpha/2$ in place of $\alpha$ for a two-sided test. Walter et al. (and the PASS software) write n for the number of measurements per subject and k for the number of subjects; here n is always the number of subjects and k the number of measurements per subject. The exact method is preferred; the approximation is provided to reproduce published tables.

Two-sided tests. When solving for rho1 with alternative = "two.sided", the solution above rho0 is returned; use alternative = "less" (with the significance level halved) for the one below.

Average-measure ICCs. The average-measure ICC is the Spearman-Brown projection of the single-measure ICC to k measurements, $k\rho / (1 + (k-1)\rho)$, so a test about ICC(k) or ICC(C,k) is the same F test as the test about the corresponding single-measure ICC, and its interval is the projection of the single-measure interval. With unit = "average" the calculations are therefore exact, not approximations; k must be given because it defines the quantity being planned for.

Missing measurements (approximate). With missing = p > 0, the calculation uses the observed-data quantities that icc_test(method = "reml") uses for incomplete data: $N = nk(1 - p)$ observations, $\bar{k} = N/n$ measurements per subject in the scale factor of the F ratio and in the interval, and error degrees of freedom $N - n - k + 1$ (two-way) or $N - n$ (one-way). This is an approximation: the F distribution is exact only for complete data. In simulations of the REML analysis (one-way and two-way ICCs, 15 or 40 subjects, 3 or 5 items, 15% or 30% of measurements missing completely at random or 30% at random depending on the first measurement; 1000 replicates per setting), the approximate power was within 0.035 of the simulated power (Monte Carlo SE 0.016) and the approximate median interval width within 2.2% of the simulated median. The REML test itself was slightly liberal with 30% missing (type I error up to 0.069 missing completely at random and 0.073 at random, for a nominal 0.05), which the simulated power includes. A warning is given above 30% missing. The calculation assumes each subject keeps at least one measurement.

Planning other designs. These functions are exact only for the one-way and two-way consistency ICCs with complete, normally distributed data. They can still guide the planning of other ICC studies, as long as the direction of the error is kept in mind:

Absolute agreement ICC (ICC(A,1), and the CCC for two methods). Not supported exactly: its test has no exact F distribution, and because its denominator degrees of freedom depend on the sample, its power "cannot be determined exactly" (McGraw & Wong, 1996, footnote 3). When the items (trials, raters, methods) differ little systematically, ICC(A,1) is close to ICC(C,1) and the consistency calculation is a good guide. When they differ, the agreement test and interval also depend on the item mean square, which has only k - 1 degrees of freedom: power is lower and intervals wider than the consistency calculation says, most of all with 2 or 3 items. Treat the consistency result as a lower bound on the number of subjects; simulation would be needed for a precise answer.
Missing measurements. Use missing for measurements missing here and there (see below). For subjects who will be lost entirely, plan for the subjects who complete the study and recruit $n / (1 - d)$ for an expected dropout d.
Non-normal data. The exact calculations assume normality. With skewed or heavy-tailed data the F-based intervals are wider and cover less than planned (see icc_test() and reli_err_test() for simulations), so allow a margin or plan for the bootstrap with a larger sample.

References

Donner, A., & Eliasziw, M. (1987). Sample size requirements for reliability studies. Statistics in Medicine, 6(4), 441-448. doi:10.1002/sim.4780060404

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30

Walter, S. D., Eliasziw, M., & Donner, A. (1998). Sample size and optimal designs for reliability studies. Statistics in Medicine, 17(1), 101-110. doi:10.1002/(SICI)1097-0258(19980115)17:1<101::AID-SIM727>3.0.CO;2-E

Examples

# Subjects needed to show an ICC above 0.6 when it is 0.8, with 2 trials
power_icc(rho0 = 0.6, rho1 = 0.8, k = 2, power = 0.8,
          alternative = "greater")
#> 
#>      Power calculation for the one-way ICC, ICC(1) (exact F distribution) 
#> 
#>               n = 38.90078
#>               k = 2
#>            rho0 = 0.6
#>            rho1 = 0.8
#>       sig.level = 0.05
#>           power = 0.8
#>     alternative = greater
#> 
#> NOTE: n is the number of subjects; round up to a whole number.
#>       n = number of subjects, k = measurements per subject
#> 

# Power with 30 subjects and 3 trials
power_icc(n = 30, k = 3, rho0 = 0.5, rho1 = 0.75, alternative = "greater",
          model = "twoway")
#> 
#>      Power calculation for the two-way consistency ICC, ICC(C,1) (exact F distribution) 
#> 
#>               n = 30
#>               k = 3
#>            rho0 = 0.5
#>            rho1 = 0.75
#>       sig.level = 0.05
#>           power = 0.8835267
#>     alternative = greater
#> 
#> NOTE: n = number of subjects, k = measurements per subject
#> 

# Trials needed with 20 subjects
power_icc(n = 20, k = NULL, rho0 = 0.5, rho1 = 0.8, power = 0.9,
          alternative = "greater")
#> 
#>      Power calculation for the one-way ICC, ICC(1) (exact F distribution) 
#> 
#>               n = 20
#>               k = 3
#>            rho0 = 0.5
#>            rho1 = 0.8
#>       sig.level = 0.05
#>           power = 0.9
#>     alternative = greater
#> 
#> NOTE: n = number of subjects, k = measurements per subject
#>