Permutation t-test

Performs t-tests with permutation-based p-values and confidence intervals. This function supports standard hypothesis testing alternatives as well as equivalence and minimal effect testing, with optional trimmed means (Yuen's approach) for robust inference. It can handle one-sample, two-sample (independent), and paired designs.

Usage

perm_t_test(x, ...)

# Default S3 method
perm_t_test(
  x,
  y = NULL,
  alternative = c("two.sided", "less", "greater", "equivalence", "minimal.effect"),
  mu = 0,
  paired = FALSE,
  var.equal = FALSE,
  alpha = 0.05,
  tr = 0,
  R = NULL,
  symmetric = TRUE,
  perm_se = TRUE,
  p_method = c("plusone", "original"),
  keep_perm = TRUE,
  ...
)

# S3 method for class 'formula'
perm_t_test(formula, data, subset, na.action, ...)

Arguments

x: a (non-empty) numeric vector of data values.
...: further arguments (currently ignored).
y: an optional (non-empty) numeric vector of data values.
alternative: the alternative hypothesis: - "two.sided": different from mu (default) - "less": less than mu - "greater": greater than mu - "equivalence": between specified bounds - "minimal.effect": outside specified bounds
mu: a number or vector specifying the null hypothesis value(s): * For standard alternatives: for standard alternatives (two.sided, less, greater): a single value (default: 0 * For equivalence/minimal.effect: either a single value (symmetric bounds will be created) or a vector of two values representing the lower and upper bounds
paired: a logical indicating whether you want a paired t-test.
var.equal: a logical variable indicating whether to treat the two variances as being equal. If TRUE then the pooled variance is used to estimate the variance otherwise the Welch (or Satterthwaite) approximation to the degrees of freedom is used. Default is FALSE.
alpha: significance level (default = 0.05).
tr: the fraction (0 to 0.5) of observations to be trimmed from each end before computing the mean and winsorized variance. Default is 0 (no trimming). When tr > 0, the function performs Yuen's trimmed t-test.
R: the number of permutations. Default is NULL, which computes all permutations. If R is specified and is less than the maximum number of possible permutations, Monte Carlo sampling is used instead.
symmetric: a logical variable indicating whether to assume symmetry in the two-sided test. If TRUE (default) then the symmetric permutation p-value is computed, otherwise the equal-tail permutation p-value is computed. Only relevant for alternative = "two.sided".
perm_se: a logical variable indicating whether to recompute the standard error for each permutation (studentized permutation test). If TRUE (default), the standard error is recomputed for each permuted sample, following the studentized permutation approach of Janssen (1997) and Chung & Romano (2013), which is valid even under heteroscedasticity. If FALSE, the standard error from the original sample is used for all permutations, which is computationally faster but assumes homoscedasticity.
p_method: the method for computing permutation p-values. Options are: * "plusone" (default): Uses the formula (b+1)/(R+1) where b is the number of permutation statistics at least as extreme as the observed. Because permutations are sampled without replacement in this implementation, this formula provides exact p-values that are never zero and guarantee correct Type I error control (Phipson & Smyth, 2010). * "original": Uses b/R with a minimum of 1/R. This can produce p-values of exactly 1/R when no permutation statistic exceeds the observed, which may be problematic for multiple testing corrections and does not guarantee exact Type I error control.
keep_perm: logical. If TRUE (default), the permutation distribution of the test statistic and effect sizes are stored in the output. Set to FALSE for large datasets to save memory.
formula: a formula of the form lhs ~ rhs where lhs is a numeric variable giving the data values and rhs either 1 for a one-sample test or a factor with two levels giving the corresponding groups. For paired tests, use the default method with x and y vectors instead of the formula method.
data: an optional matrix or data frame (or similar: see model.frame) containing the variables in the formula formula. By default the variables are taken from environment(formula).
subset: an optional vector specifying a subset of observations to be used.
na.action: a function which indicates what should happen when the data contain NAs. Defaults to getOption("na.action").

Value

A list with class "htest" containing the following components:

"statistic": the value of the t-statistic.
"parameter": the degrees of freedom for the t-statistic.
"p.value": the permutation p-value for the test.
"stderr": the standard error of the mean (difference).
"conf.int": a permutation percentile confidence interval appropriate to the specified alternative hypothesis.
"estimate": the estimated mean or difference in means (or trimmed means if tr > 0).
"null.value": the specified hypothesized value(s) of the mean or mean difference.
"alternative": a character string describing the alternative hypothesis.
"method": a character string indicating what type of permutation t-test was performed.
"data.name": a character string giving the name(s) of the data.
"call": the matched call.
"R": the requested number of permutations.
"R.used": the actual number of permutations used (may differ if exact permutations computed).
"perm.stat": (if keep_perm = TRUE) the permutation distribution of the test statistic.
"perm.eff": (if keep_perm = TRUE) the permutation distribution of the effect (mean differences).

Details

This function performs permutation-based t-tests, providing more robust inference than standard parametric t-tests. It supports one-sample, two-sample (independent), and paired designs, as well as five different alternative hypotheses.

The permutation procedure follows these steps:

Calculate the test statistic from the original data
Generate R permutation samples by randomly reassigning observations to groups
Calculate the test statistic for each permutation sample
Compute the p-value by comparing the original test statistic to the permutation distribution
Calculate confidence intervals using the percentile method

For one-sample and paired tests, permutation is performed by randomly flipping the signs of the (centered) observations or difference scores.

For two-sample tests, permutation is performed by randomly reassigning observations to the two groups. When perm_se = TRUE (default), the studentized permutation approach of Janssen (1997) and Chung & Romano (2013) is used, which recomputes the standard error for each permutation and is valid even under heteroscedasticity. When perm_se = FALSE, the standard error from the original sample is used for all permutations, which is faster but assumes equal variances.

When tr > 0, the function uses Yuen's trimmed t-test approach:

Trimmed means are computed by removing the fraction tr of observations from each tail
Winsorized variances are used in place of standard variances
This provides robustness against outliers and heavy-tailed distributions

For different alternatives, the p-values are calculated as follows:

"two.sided": Either symmetric (proportion of |T\*| >= |T|) or equal-tail (2 * min of one-sided p-values) depending on the symmetric argument
"less": Proportion of T\* <= T
"greater": Proportion of T\* >= T
"equivalence": Maximum of two one-sided p-values (for lower and upper bounds)
"minimal.effect": Minimum of two one-sided p-values (for lower and upper bounds)

For two-sample tests, the test is of \(\bar x - \bar y\) (mean of x minus mean of y). For paired samples, the test is of the difference scores (z), wherein \(z = x - y\), and the test is of \(\bar z\) (mean of the difference scores). For one-sample tests, the test is of \(\bar x\) (mean of x).

If the number of possible permutations is less than R, all exact permutations are computed and a message is printed to the console.

Purpose

Use this function when:

You need more robust inference than provided by standard t-tests
You want to perform equivalence or minimal effect testing with permutation methods
Sample sizes are very small where standard or bootstrap approaches may be less reliable
You prefer the standard htest output format for compatibility with other R functions
You want an alternative that tests means, mean difference, or difference in means that can also utilize trimming

Comparison with Other Packages

Results from perm_t_test may differ slightly from other permutation test implementations such as MKinfer::perm.t.test or coin::oneway_test. These differences arise from methodological choices:

P-value formula: This function uses the (b+1)/(R+1) formula by default (where b is the count of permutation statistics at least as extreme as observed), which provides exact p-values that are never zero and guarantees correct Type I error control (Phipson & Smyth, 2010). Some packages use b/R with strict inequality (>), which excludes tied values and can produce p-values of exactly zero.
Studentized permutation: By default (perm_se = TRUE), this function recalculates the standard error for each permutation, following the studentized approach of Janssen (1997). This provides valid inference even under heteroscedasticity but may produce different results than non-studentized implementations.
Exact permutation detection: This function automatically detects when exact enumeration is feasible and computes all possible permutations, while some packages may default to Monte Carlo sampling regardless of sample size.

All approaches produce valid permutation tests; the differences reflect trade-offs between conservatism, exactness, and robustness to assumption violations.

References

Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman and Hall/CRC.

Janssen, A. (1997). Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens-Fisher problem. Statistics & Probability Letters, 36, 9-21.

Chung, E., & Romano, J. P. (2013). Exact and asymptotically robust permutation tests. The Annals of Statistics, 41(2), 484-507.

Yuen, K. K. (1974). The two-sample trimmed t for unequal population variances. Biometrika, 61(1), 165-170.

Phipson, B., & Smyth, G. K. (2010). Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology, 9(1), Article 39.

Examples


# Example 1: Basic two-sample test with formula notation
data(sleep)
result <- perm_t_test(extra ~ group, data = sleep)
#> Computing all 184756 exact permutations.
result
#> 
#> 	Exact Permutation Welch Two Sample t-test
#> 
#> data:  extra by group
#> t-observed = -1.8608, df = 17.776, p-value = 0.08142
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#>  -3.34  0.18
#> sample estimates:
#> mean of x mean of y 
#>      0.75      2.33 
#> 

# Example 2: One-sample permutation t-test
set.seed(123)
x <- rnorm(20, mean = 0.5, sd = 1)
perm_t_test(x, mu = 0, R = 199)
#> Note: Number of permutations (R = 199) is less than 1000. Consider increasing R for more stable p-value estimates.
#> 
#> 	Monte Carlo Permutation One Sample t-test
#> 
#> data:  x
#> t-observed = 2.9501, df = 19, p-value = 0.015
#> alternative hypothesis: true mean is not equal to 0
#> 95 percent confidence interval:
#>  0.1769754 0.9878550
#> sample estimates:
#> mean of x 
#> 0.6416238 
#> 

# Example 3: Paired samples test
before <- c(5.1, 4.8, 6.2, 5.7, 6.0, 5.5, 4.9, 5.8)
after <- c(5.6, 5.2, 6.7, 6.1, 6.5, 5.8, 5.3, 6.2)
perm_t_test(x = before, y = after,
            paired = TRUE,
            alternative = "less",
            R = 199)
#> 
#> 	Monte Carlo Permutation Paired t-test
#> 
#> data:  before and after
#> t-observed = -17, df = 7, p-value = 0.005
#> alternative hypothesis: true difference in means is less than 0
#> 95 percent confidence interval:
#>     -Inf -0.3875
#> sample estimates:
#> mean of the differences 
#>                  -0.425 
#>