Skip to contents

Equivalence Test for ANOVA Results

equ_anova.Rd

[Stable]

Performs equivalence or minimal effect testing on the partial eta-squared (pes) value from ANOVA results to determine if effects are practically equivalent to zero or meaningfully different from zero.

Usage

equ_anova(object, eqbound, MET = FALSE, alpha = 0.05)

Arguments

object

An object returned by either Anova, aov, or afex_aov.

eqbound

Equivalence bound for the partial eta-squared. This value represents the smallest effect size considered meaningful or practically significant.

MET

Logical indicator to perform a minimal effect test rather than equivalence test (default is FALSE). When TRUE, the alternative hypothesis becomes that the effect is larger than the equivalence bound.

alpha

Alpha level used for the test (default = 0.05). Note that this argument is currently accepted but not used: the returned p.equ should be compared against the desired alpha level by the user.

Value

Returns a data frame containing the ANOVA results with equivalence tests added. The following columns are included in the table:

  • effect: Name of the effect.

  • df1: Degrees of Freedom in the numerator (i.e., DF effect).

  • df2: Degrees of Freedom in the denominator (i.e., DF error).

  • F.value: F-value.

  • p.null: p-value for the traditional null hypothesis test (probability of the data given the null hypothesis).

  • pes: Partial eta-squared measure of effect size.

  • eqbound: Equivalence bound used for testing.

  • p.equ: p-value for the equivalence or minimal effect test.

Details

This function tests whether ANOVA effects are practically equivalent to zero (when MET = FALSE) or meaningfully different from zero (when MET = TRUE) using the approach described by Campbell & Lakens (2021).

The function works by:

  1. Extracting ANOVA results from the input object

  2. Converting the equivalence bound for partial eta-squared to a non-centrality parameter

  3. Performing an equivalence test or minimal effect test for each effect in the ANOVA

For equivalence tests (MET = FALSE), a significant result (p < alpha) indicates that the effect is statistically equivalent to zero (smaller than the equivalence bound).

For minimal effect tests (MET = TRUE), a significant result (p < alpha) indicates that the effect is meaningfully different from zero (larger than the equivalence bound).

For details on the calculations in this function see vignette("the_ftestTOSTER").

Multi-factor and within-subjects designs

Campbell & Lakens (2021) derived the test for one-way ANOVA and multivariable regression. This function generalizes it by computing the non-centrality parameter as \(\lambda = \frac{\Delta}{1 - \Delta} (df_1 + df_2 + 1)\), which reduces exactly to the published expression whenever \(df_1 + df_2 + 1 = N\). The quantity \(df_1 + df_2 + 1\) is the effective sample size of the error stratum in which an effect is tested, which is not in general the total number of observations; in a mixed design, for example, it recovers the number of subjects for a between-subjects effect. Simulation work covering one-way within-subjects, factorial between-subjects, and mixed designs found the test maintained its nominal Type I error rate at the boundary of the null hypothesis in each case.

Interpreting bounds in within-subjects designs

eqbound is a bound on partial eta-squared, which excludes subject variance from its denominator. In repeated-measures designs it is therefore not a bound on the share of total variance in the data, and the gap between the two widens as the number of levels of the within-subjects factor falls. This is a property of partial eta-squared rather than of the equivalence test, but a bound chosen as "a negligible share of the variance I observe" will not be the bound the test applies.

Limitation: sphericity corrections are not applied

For within-subjects factors with more than two levels, this function uses the uncorrected univariate degrees of freedom and does not apply Greenhouse-Geisser or Huynh-Feldt corrections, even when the object was fit with a correction requested (e.g. afex::aov_car(..., anova_table = list(correction = "GG"))). As a result both p.null and p.equ may differ from the corrected values reported by the fitting function. When sphericity is badly violated the equivalence test can become substantially anti-conservative. Assess sphericity on the fitted model and treat results with caution when it is in doubt.

References

Campbell, H., & Lakens, D. (2021). Can we disregard the whole model? Omnibus non‐inferiority testing for R2 in multi‐variable linear regression and in ANOVA. British Journal of Mathematical and Statistical Psychology, 74(1), 64-89. doi: 10.1111/bmsp.12201

See also

Other f-test: equ_ftest()

Examples

# One-way ANOVA
data(iris)
anova_result <- aov(Sepal.Length ~ Species, data = iris)

# Equivalence test with bound of 0.1
equ_anova(anova_result, eqbound = 0.1)
#>        effect df1 df2  F.value       p.null       pes eqbound p.equ
#> 1 Species       2 147 119.2645 1.669669e-31 0.6187057     0.1     1

# Minimal effect test with bound of 0.1
equ_anova(anova_result, eqbound = 0.1, MET = TRUE)
#>        effect df1 df2  F.value       p.null       pes eqbound        p.equ
#> 1 Species       2 147 119.2645 1.669669e-31 0.6187057     0.1 3.563826e-10

# Two-way ANOVA with lower equivalence bound
anova_result2 <- aov(Sepal.Length ~ Species * Petal.Width, data = iris)
equ_anova(anova_result2, eqbound = 0.05)
#>                effect df1 df2    F.value       p.null        pes eqbound
#> 1 Species               2 144 137.828968 3.577896e-34 0.65686339    0.05
#> 2 Petal.Width           1 144  22.571323 4.848845e-06 0.13550545    0.05
#> 3 Species:Petal.Width   2 144   1.655197 1.946683e-01 0.02247224    0.05
#>       p.equ
#> 1 1.0000000
#> 2 0.9711723
#> 3 0.1175566