Permutation t-test

Performs t-tests with permutation-based p-values and confidence intervals. This function supports standard hypothesis testing alternatives as well as equivalence and minimal effect testing, with optional trimmed means (Yuen's approach) for robust inference. It can handle one-sample, two-sample (independent), and paired designs.

Usage

perm_t_test(x, ...)

# Default S3 method
perm_t_test(
  x,
  y = NULL,
  alternative = c("two.sided", "less", "greater", "equivalence", "minimal.effect"),
  mu = 0,
  paired = FALSE,
  var.equal = FALSE,
  alpha = 0.05,
  tr = 0,
  R = NULL,
  symmetric = TRUE,
  perm_se = TRUE,
  p_method = NULL,
  keep_perm = TRUE,
  ...
)

# S3 method for class 'formula'
perm_t_test(formula, data, subset, na.action, ...)

Arguments

x: a (non-empty) numeric vector of data values.
...: further arguments (currently ignored).
y: an optional (non-empty) numeric vector of data values.
alternative: the alternative hypothesis: - "two.sided": different from mu (default) - "less": less than mu - "greater": greater than mu - "equivalence": between specified bounds - "minimal.effect": outside specified bounds
mu: a number or vector specifying the null hypothesis value(s): * For standard alternatives: for standard alternatives (two.sided, less, greater): a single value (default: 0 * For equivalence/minimal.effect: either a single value (symmetric bounds will be created) or a vector of two values representing the lower and upper bounds
paired: a logical indicating whether you want a paired t-test.
var.equal: a logical variable indicating whether to treat the two variances as being equal. If TRUE then the pooled variance is used to estimate the variance otherwise the Welch (or Satterthwaite) approximation to the degrees of freedom is used. Default is FALSE.
alpha: significance level (default = 0.05).
tr: the fraction (0 to 0.5) of observations to be trimmed from each end before computing the mean and winsorized variance. Default is 0 (no trimming). When tr > 0, the function performs Yuen's trimmed t-test.
R: the number of permutations. Default is NULL, which computes all exact permutations. If R is specified and is less than the maximum number of possible permutations, randomization (permutation with replacement) is used instead.
symmetric: a logical variable indicating whether to assume symmetry in the two-sided test. If TRUE (default) then the symmetric permutation p-value is computed, otherwise the equal-tail permutation p-value is computed. Only relevant for alternative = "two.sided".
perm_se: a logical variable indicating whether to recompute the standard error for each permutation (studentized permutation test). If TRUE (default), the standard error is recomputed for each permuted sample, following the studentized permutation approach of Janssen (1997) and Chung & Romano (2013). This test is exact under the sharp null hypothesis and asymptotically valid for the weak null hypothesis of equal means, even under heteroscedasticity (see "Sharp and Weak Null Hypotheses" in Details). If FALSE, the standard error from the original sample is used for all permutations, which is computationally faster and still exact under the sharp null, but is not valid for the weak null when variances and group sizes differ.
p_method: the method for computing permutation p-values. Options are: * NULL (default): Automatically selects "exact" for exact permutation tests (when all permutations are enumerated) and "plusone" for randomization tests (when permutations are sampled with replacement). * "exact": Uses b/R where b is the number of permutation statistics at least as extreme as observed. Appropriate when all permutations are enumerated. * "plusone": Uses (b+1)/(R+1), which guarantees p > 0 and provides exact Type I error control for randomization tests where permutations are sampled with replacement (Phipson & Smyth, 2010).
keep_perm: logical. If TRUE (default), the permutation distribution of the test statistic and effect sizes are stored in the output. Set to FALSE for large datasets to save memory.
formula: a formula of the form lhs ~ rhs where lhs is a numeric variable giving the data values and rhs either 1 for a one-sample test or a factor with two levels giving the corresponding groups. For paired tests, use the default method with x and y vectors instead of the formula method.
data: an optional matrix or data frame (or similar: see model.frame) containing the variables in the formula formula. By default the variables are taken from environment(formula).
subset: an optional vector specifying a subset of observations to be used.
na.action: a function which indicates what should happen when the data contain NAs. Defaults to getOption("na.action").

Value

A list with class "htest" containing the following components:

"statistic": the value of the t-statistic.
"parameter": the degrees of freedom for the t-statistic.
"p.value": the permutation p-value for the test.
"stderr": the standard error of the mean (difference).
"conf.int": a permutation confidence interval, obtained by inverting the permutation test, appropriate to the specified alternative hypothesis.
"estimate": the estimated mean or difference in means (or trimmed means if tr > 0).
"null.value": the specified hypothesized value(s) of the mean or mean difference.
"alternative": a character string describing the alternative hypothesis.
"method": a character string indicating what type of permutation t-test was performed.
"data.name": a character string giving the name(s) of the data.
"call": the matched call.
"R": the requested number of permutations.
"R.used": the actual number of permutations used (may differ if exact permutations computed).
"perm.stat": (if keep_perm = TRUE) the permutation distribution of the test statistic.
"perm.eff": (if keep_perm = TRUE) the raw permutation distribution of the effect (permuted mean differences shifted by the estimate). This is provided for inspection only and is not used to construct the confidence interval.

Details

This function performs permutation-based t-tests, providing more robust inference than standard parametric t-tests. It supports one-sample, two-sample (independent), and paired designs, as well as five different alternative hypotheses.

The permutation procedure follows these steps:

Calculate the test statistic from the original data
Generate R permutation samples by randomly reassigning observations to groups
Calculate the test statistic for each permutation sample
Compute the p-value by comparing the original test statistic to the permutation distribution
Calculate confidence intervals by inverting the permutation test

Confidence Intervals

Confidence intervals are obtained by inverting the same (studentized) permutation test that produces the p-value. Because the permutation distribution \(T^*\) is generated from the unshifted data, it does not depend on the null value, and the interval has the closed form \(\hat\theta - SE \cdot q\), where \(\hat\theta\) is the estimate, \(SE\) is the standard error from the original sample, and \(q\) is the order statistic of \(T^*\) at which the permutation p-value (using the same p_method counting rule and, for two-sided tests, the same symmetric setting) crosses alpha. As a result, the confidence interval and p-value always agree: a null value lies inside the interval if and only if its p-value exceeds alpha. When perm_se = TRUE this yields a studentized interval that remains valid under unequal variances. When perm_se = FALSE it reduces to a (reflected) percentile interval of the permuted differences.

If the number of permutations is too small for the test to ever reject at the requested alpha, the corresponding confidence limit is infinite.

For one-sample and paired tests, permutation is performed by randomly flipping the signs of the (centered) observations or difference scores.

For two-sample tests, permutation is performed by randomly reassigning observations to the two groups. When perm_se = TRUE (default), the studentized permutation approach of Janssen (1997) and Chung & Romano (2013) is used, which recomputes the standard error for each permutation and remains asymptotically valid under heteroscedasticity. When perm_se = FALSE, the standard error from the original sample is used for all permutations, which is faster but assumes equal variances (see "Sharp and Weak Null Hypotheses" below).

When tr > 0, the function uses Yuen's trimmed t-test approach:

Trimmed means are computed by removing the fraction tr of observations from each tail
Winsorized variances are used in place of standard variances
This provides robustness against outliers and heavy-tailed distributions

For different alternatives, the p-values are calculated as follows:

"two.sided": Either symmetric (proportion of |T\*| >= |T|) or equal-tail (2 * min of one-sided p-values) depending on the symmetric argument
"less": Proportion of T\* <= T
"greater": Proportion of T\* >= T
"equivalence": Maximum of two one-sided p-values (for lower and upper bounds)
"minimal.effect": Minimum of two one-sided p-values (for lower and upper bounds)

For two-sample tests, the test is of \(\bar x - \bar y\) (mean of x minus mean of y). For paired samples, the test is of the difference scores (z), wherein \(z = x - y\), and the test is of \(\bar z\) (mean of the difference scores). For one-sample tests, the test is of \(\bar x\) (mean of x).

If the number of possible permutations is less than R, all exact permutations are computed and a message is printed to the console.

Permutation Approach to Equivalence and Minimal Effect Testing

When alternative = "equivalence" or alternative = "minimal.effect", the function performs two one-sided permutation tests and combines them following the nonparametric combination (NPC) framework of Arboretti, Pesarin, and Salmaso (2021).

The "equivalence" alternative implements the intersection-union (IU) approach: the null hypothesis is non-equivalence and the alternative is equivalence. Two one-sided p-values are computed and the global p-value is their maximum. The "minimal.effect" alternative implements the union-intersection (UI) approach: the null hypothesis is equivalence and the alternative is non-equivalence. The global p-value is the minimum of the two one-sided p-values. Both partial tests are evaluated against the same permutation distribution (i.e., the same set of permuted samples), which preserves the negative dependence between the two test statistics as required by the NPC theory.

The permutation distribution is constructed under the exchangeability null (i.e., by permuting the unshifted data), while the observed test statistics are centered at the equivalence bounds. This differs from the shifted-data algorithm described in Arboretti et al. (2021), which permutes margin-shifted pooled samples. Under the default studentized permutation approach (perm_se = TRUE), the two methods are asymptotically equivalent (Janssen, 1997; Chung & Romano, 2013). The exchangeability-based approach avoids the computational cost of constructing and permuting two separate shifted datasets.

Note that these are uncalibrated (naive) procedures in the terminology of Arboretti et al. (2021). For the IU direction ("equivalence"), the naive approach can be conservative when sample sizes or equivalence margins are small relative to the variability in the data. For the UI direction ("minimal.effect"), the impact of not calibrating is smaller because the calibrated critical value lies in the narrower interval \([\alpha/2, \alpha]\) rather than \([\alpha, (1+\alpha)/2)\).

Choosing a Method

A set of simulations covering two-sample and paired designs (small to moderate samples, normal, skewed, heavy-tailed, and discrete data) indicated the following. These are rules of thumb, not guarantees.

Independent groups: this function most consistently held its nominal Type I error rate, including with unequal variances and heavy-tailed data.
Paired data: the test flips the signs of the differences, so it relies on the differences being symmetric. With strongly skewed differences it was too liberal, and boot_t_test() with the studentized interval (boot_ci = "stud") was the better choice. With symmetric (including heavy-tailed) differences, this function performed well.
Skewed paired data with outliers: use trimmed means (e.g., tr = 0.2); the two functions then performed similarly.
Groups that differ in shape or spread (e.g., one skewed group): no method was uniformly reliable. All of them, including Welch's t-test, could be modestly liberal in one tail with small samples, particularly when the more variable or skewed group was the smaller one. Trimming reduced the problem. Treat borderline results in this setting with caution.

Sharp and Weak Null Hypotheses

Two different null hypotheses are relevant when interpreting a permutation test:

The sharp null (Fisher) states that there is no effect at all: the two groups have identical distributions (for one-sample and paired tests, the observations or difference scores are symmetric about zero). Under the sharp null the group labels (or signs) are exchangeable, so the permutation test is exact at any sample size. This holds for any test statistic, including the studentized (Welch) statistic used by default, and for both p_method options.
The weak null (Neyman) states only that the means (or trimmed means) are equal, while the distributions may otherwise differ, for example in variance or shape. Under the weak null the observations are not exchangeable and no permutation test is exact. The studentized test (perm_se = TRUE) is asymptotically valid (Janssen, 1997; Chung & Romano, 2013), so it controls the Type I error rate for the weak null in large samples while remaining exact for the sharp null (Wu & Ding, 2020). The non-studentized test (perm_se = FALSE) does not have this property and can have incorrect Type I error rates for the weak null even in large samples when group sizes and variances differ.

Because validity for the weak null is only asymptotic, the studentized test can be anti-conservative in very small samples with unequal variances, particularly when the smaller group has the larger variance, and it offers no protection against the small-sample problems of comparing means of differently skewed distributions. In such cases the Welch t-test or boot_t_test() may perform comparably or better.

Exactness under the sharp null applies to the test of no effect (mu = 0). For tests against a non-zero mu, for the equivalence and minimal effect tests, and for the confidence interval, the permutation distribution of the unshifted data is used as the reference distribution (see above), so these are approximate rather than exact, even under a constant-shift model.

Choosing between permutation and bootstrap methods. In a randomized experiment, the permutation distribution corresponds to the actual randomization distribution, so the test is justified by the design itself and is exact under the sharp null. This makes perm_t_test() a natural choice for randomized experiments, especially small ones. When data are instead random samples from two populations (e.g., observational comparisons), the permutation test remains exact for the null of identical population distributions and asymptotically valid for the weak null, but this justification rests on an exchangeability assumption rather than the design; in that setting the bootstrap (boot_t_test()), which targets the weak null directly, is an equally natural choice. Neither approach supports population inference from non-random (e.g., convenience) samples.

Purpose

Use this function when:

You need more robust inference than provided by standard t-tests
You want to perform equivalence or minimal effect testing with permutation methods
Sample sizes are very small where standard or bootstrap approaches may be less reliable
You prefer the standard htest output format for compatibility with other R functions
You want an alternative that tests means, mean difference, or difference in means that can also utilize trimming

Comparison with Other Packages

Results from perm_t_test may differ slightly from other permutation test implementations such as MKinfer::perm.t.test or coin::oneway_test. These differences arise from methodological choices:

P-value formula: This function uses the (b+1)/(R+1) formula by default (where b is the count of permutation statistics at least as extreme as observed), which provides exact p-values that are never zero and guarantees correct Type I error control (Phipson & Smyth, 2010). Some packages use b/R with strict inequality (>), which excludes tied values and can produce p-values of exactly zero.
Studentized permutation: By default (perm_se = TRUE), this function recalculates the standard error for each permutation, following the studentized approach of Janssen (1997). This provides valid inference even under heteroscedasticity but may produce different results than non-studentized implementations.
Exact permutation detection: This function automatically detects when exact enumeration is feasible and computes all possible permutations, while some packages may default to Randomization sampling regardless of sample size.

All approaches produce valid permutation tests; the differences reflect trade-offs between conservatism, exactness, and robustness to assumption violations.

References

Arboretti, R., Pesarin, F. & Salmaso, L. (2021). A unified approach to permutation testing for equivalence. Stat Methods Appl 30, 1033-1052. doi: 10.1007/s10260-020-00548-0

Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman and Hall/CRC.

Janssen, A. (1997). Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens-Fisher problem. Statistics & Probability Letters, 36, 9-21.

Chung, E., & Romano, J. P. (2013). Exact and asymptotically robust permutation tests. The Annals of Statistics, 41(2), 484-507.

Yuen, K. K. (1974). The two-sample trimmed t for unequal population variances. Biometrika, 61(1), 165-170.

Phipson, B., & Smyth, G. K. (2010). Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology, 9(1), Article 39.

Wu, J., & Ding, P. (2020). Randomization tests for weak null hypotheses in randomized experiments. Journal of the American Statistical Association, 116(536), 1898-1913. doi: 10.1080/01621459.2020.1750415

Examples


# Example 1: Basic two-sample test with formula notation
data(sleep)
result <- perm_t_test(extra ~ group, data = sleep)
#> Computing all 184756 exact permutations.
result
#> 
#> 	Exact Permutation Welch Two Sample t-test
#> 
#> data:  extra by group
#> t-observed = -1.8608, df = 17.776, p-value = 0.08145
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#>  -3.3822417  0.2222417
#> sample estimates:
#>           mean of group '1'           mean of group '2' 
#>                        0.75                        2.33 
#> mean difference ('1' - '2') 
#>                       -1.58 
#> 

# Example 2: One-sample permutation t-test
set.seed(123)
x <- rnorm(20, mean = 0.5, sd = 1)
perm_t_test(x, mu = 0, R = 199)
#> Note: Number of permutations (R = 199) is less than 1000. Consider increasing R for more stable p-value estimates.
#> 
#> 	Randomization Permutation One Sample t-test
#> 
#> data:  x
#> t-observed = 2.9501, df = 19, p-value = 0.015
#> alternative hypothesis: true mean is not equal to 0
#> 95 percent confidence interval:
#>  0.1560004 1.1272472
#> sample estimates:
#> mean of x 
#> 0.6416238 
#> 

# Example 3: Paired samples test
before <- c(5.1, 4.8, 6.2, 5.7, 6.0, 5.5, 4.9, 5.8)
after <- c(5.6, 5.2, 6.7, 6.1, 6.5, 5.8, 5.3, 6.2)
perm_t_test(x = before, y = after,
            paired = TRUE,
            alternative = "less",
            R = 199)
#> 
#> 	Randomization Permutation Paired t-test
#> 
#> data:  before and after
#> t-observed = -17, df = 7, p-value = 0.005
#> alternative hypothesis: true difference in means is less than 0
#> 95 percent confidence interval:
#>        -Inf -0.3794767
#> sample estimates:
#> mean of the differences (z = x - y) 
#>                              -0.425 
#>