Permutation t-test

Performs t-tests with permutation-based p-values and confidence intervals. This function supports standard hypothesis testing alternatives as well as equivalence and minimal effect testing, with optional trimmed means (Yuen's approach) for robust inference. It can handle one-sample, two-sample (independent), and paired designs.

Usage

perm_t_test(x, ...)

# Default S3 method
perm_t_test(
  x,
  y = NULL,
  alternative = c("two.sided", "less", "greater", "equivalence", "minimal.effect"),
  mu = 0,
  paired = FALSE,
  var.equal = FALSE,
  alpha = 0.05,
  tr = 0,
  R = NULL,
  symmetric = TRUE,
  perm_se = TRUE,
  p_method = NULL,
  keep_perm = TRUE,
  ...
)

# S3 method for class 'formula'
perm_t_test(formula, data, subset, na.action, ...)

Arguments

x: a (non-empty) numeric vector of data values.
...: further arguments (currently ignored).
y: an optional (non-empty) numeric vector of data values.
alternative: the alternative hypothesis: - "two.sided": different from mu (default) - "less": less than mu - "greater": greater than mu - "equivalence": between specified bounds - "minimal.effect": outside specified bounds
mu: a number or vector specifying the null hypothesis value(s): * For standard alternatives: for standard alternatives (two.sided, less, greater): a single value (default: 0 * For equivalence/minimal.effect: either a single value (symmetric bounds will be created) or a vector of two values representing the lower and upper bounds
paired: a logical indicating whether you want a paired t-test.
var.equal: a logical variable indicating whether to treat the two variances as being equal. If TRUE then the pooled variance is used to estimate the variance otherwise the Welch (or Satterthwaite) approximation to the degrees of freedom is used. Default is FALSE.
alpha: significance level (default = 0.05).
tr: the fraction (0 to 0.5) of observations to be trimmed from each end before computing the mean and winsorized variance. Default is 0 (no trimming). When tr > 0, the function performs Yuen's trimmed t-test.
R: the number of permutations. Default is NULL, which computes all exact permutations. If R is specified and is less than the maximum number of possible permutations, randomization (permutation with replacement) is used instead.
symmetric: a logical variable indicating whether to assume symmetry in the two-sided test. If TRUE (default) then the symmetric permutation p-value is computed, otherwise the equal-tail permutation p-value is computed. Only relevant for alternative = "two.sided".
perm_se: a logical variable indicating whether to recompute the standard error for each permutation (studentized permutation test). If TRUE (default), the standard error is recomputed for each permuted sample, following the studentized permutation approach of Janssen (1997) and Chung & Romano (2013), which is valid even under heteroscedasticity. If FALSE, the standard error from the original sample is used for all permutations, which is computationally faster but assumes homoscedasticity.
p_method: the method for computing permutation p-values. Options are: * NULL (default): Automatically selects "exact" for exact permutation tests (when all permutations are enumerated) and "plusone" for randomization tests (when permutations are sampled with replacement). * "exact": Uses b/R where b is the number of permutation statistics at least as extreme as observed. Appropriate when all permutations are enumerated. * "plusone": Uses (b+1)/(R+1), which guarantees p > 0 and provides exact Type I error control for randomization tests where permutations are sampled with replacement (Phipson & Smyth, 2010).
keep_perm: logical. If TRUE (default), the permutation distribution of the test statistic and effect sizes are stored in the output. Set to FALSE for large datasets to save memory.
formula: a formula of the form lhs ~ rhs where lhs is a numeric variable giving the data values and rhs either 1 for a one-sample test or a factor with two levels giving the corresponding groups. For paired tests, use the default method with x and y vectors instead of the formula method.
data: an optional matrix or data frame (or similar: see model.frame) containing the variables in the formula formula. By default the variables are taken from environment(formula).
subset: an optional vector specifying a subset of observations to be used.
na.action: a function which indicates what should happen when the data contain NAs. Defaults to getOption("na.action").

Value

A list with class "htest" containing the following components:

"statistic": the value of the t-statistic.
"parameter": the degrees of freedom for the t-statistic.
"p.value": the permutation p-value for the test.
"stderr": the standard error of the mean (difference).
"conf.int": a permutation percentile confidence interval appropriate to the specified alternative hypothesis.
"estimate": the estimated mean or difference in means (or trimmed means if tr > 0).
"null.value": the specified hypothesized value(s) of the mean or mean difference.
"alternative": a character string describing the alternative hypothesis.
"method": a character string indicating what type of permutation t-test was performed.
"data.name": a character string giving the name(s) of the data.
"call": the matched call.
"R": the requested number of permutations.
"R.used": the actual number of permutations used (may differ if exact permutations computed).
"perm.stat": (if keep_perm = TRUE) the permutation distribution of the test statistic.
"perm.eff": (if keep_perm = TRUE) the permutation distribution of the effect (mean differences).

Details

This function performs permutation-based t-tests, providing more robust inference than standard parametric t-tests. It supports one-sample, two-sample (independent), and paired designs, as well as five different alternative hypotheses.

The permutation procedure follows these steps:

Calculate the test statistic from the original data
Generate R permutation samples by randomly reassigning observations to groups
Calculate the test statistic for each permutation sample
Compute the p-value by comparing the original test statistic to the permutation distribution
Calculate confidence intervals using the percentile method

For one-sample and paired tests, permutation is performed by randomly flipping the signs of the (centered) observations or difference scores.

For two-sample tests, permutation is performed by randomly reassigning observations to the two groups. When perm_se = TRUE (default), the studentized permutation approach of Janssen (1997) and Chung & Romano (2013) is used, which recomputes the standard error for each permutation and is valid even under heteroscedasticity. When perm_se = FALSE, the standard error from the original sample is used for all permutations, which is faster but assumes equal variances.

When tr > 0, the function uses Yuen's trimmed t-test approach:

Trimmed means are computed by removing the fraction tr of observations from each tail
Winsorized variances are used in place of standard variances
This provides robustness against outliers and heavy-tailed distributions

For different alternatives, the p-values are calculated as follows:

"two.sided": Either symmetric (proportion of |T\*| >= |T|) or equal-tail (2 * min of one-sided p-values) depending on the symmetric argument
"less": Proportion of T\* <= T
"greater": Proportion of T\* >= T
"equivalence": Maximum of two one-sided p-values (for lower and upper bounds)
"minimal.effect": Minimum of two one-sided p-values (for lower and upper bounds)

For two-sample tests, the test is of \(\bar x - \bar y\) (mean of x minus mean of y). For paired samples, the test is of the difference scores (z), wherein \(z = x - y\), and the test is of \(\bar z\) (mean of the difference scores). For one-sample tests, the test is of \(\bar x\) (mean of x).

If the number of possible permutations is less than R, all exact permutations are computed and a message is printed to the console.

Permutation Approach to Equivalence and Minimal Effect Testing

When alternative = "equivalence" or alternative = "minimal.effect", the function performs two one-sided permutation tests and combines them following the nonparametric combination (NPC) framework of Arboretti, Pesarin, and Salmaso (2021).

The "equivalence" alternative implements the intersection-union (IU) approach: the null hypothesis is non-equivalence and the alternative is equivalence. Two one-sided p-values are computed and the global p-value is their maximum. The "minimal.effect" alternative implements the union-intersection (UI) approach: the null hypothesis is equivalence and the alternative is non-equivalence. The global p-value is the minimum of the two one-sided p-values. Both partial tests are evaluated against the same permutation distribution (i.e., the same set of permuted samples), which preserves the negative dependence between the two test statistics as required by the NPC theory.

The permutation distribution is constructed under the exchangeability null (i.e., by permuting the unshifted data), while the observed test statistics are centered at the equivalence bounds. This differs from the shifted-data algorithm described in Arboretti et al. (2021), which permutes margin-shifted pooled samples. Under the default studentized permutation approach (perm_se = TRUE), the two methods are asymptotically equivalent (Janssen, 1997; Chung & Romano, 2013). The exchangeability-based approach avoids the computational cost of constructing and permuting two separate shifted datasets.

Note that these are uncalibrated (naive) procedures in the terminology of Arboretti et al. (2021). For the IU direction ("equivalence"), the naive approach can be conservative when sample sizes or equivalence margins are small relative to the variability in the data. For the UI direction ("minimal.effect"), the impact of not calibrating is smaller because the calibrated critical value lies in the narrower interval \([\alpha/2, \alpha]\) rather than \([\alpha, (1+\alpha)/2)\).

Purpose

Use this function when:

You need more robust inference than provided by standard t-tests
You want to perform equivalence or minimal effect testing with permutation methods
Sample sizes are very small where standard or bootstrap approaches may be less reliable
You prefer the standard htest output format for compatibility with other R functions
You want an alternative that tests means, mean difference, or difference in means that can also utilize trimming

Comparison with Other Packages

Results from perm_t_test may differ slightly from other permutation test implementations such as MKinfer::perm.t.test or coin::oneway_test. These differences arise from methodological choices:

P-value formula: This function uses the (b+1)/(R+1) formula by default (where b is the count of permutation statistics at least as extreme as observed), which provides exact p-values that are never zero and guarantees correct Type I error control (Phipson & Smyth, 2010). Some packages use b/R with strict inequality (>), which excludes tied values and can produce p-values of exactly zero.
Studentized permutation: By default (perm_se = TRUE), this function recalculates the standard error for each permutation, following the studentized approach of Janssen (1997). This provides valid inference even under heteroscedasticity but may produce different results than non-studentized implementations.
Exact permutation detection: This function automatically detects when exact enumeration is feasible and computes all possible permutations, while some packages may default to Randomization sampling regardless of sample size.

All approaches produce valid permutation tests; the differences reflect trade-offs between conservatism, exactness, and robustness to assumption violations.

References

Arboretti, R., Pesarin, F. & Salmaso, L. (2021). A unified approach to permutation testing for equivalence. Stat Methods Appl 30, 1033-1052. doi: 10.1007/s10260-020-00548-0

Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman and Hall/CRC.

Janssen, A. (1997). Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens-Fisher problem. Statistics & Probability Letters, 36, 9-21.

Chung, E., & Romano, J. P. (2013). Exact and asymptotically robust permutation tests. The Annals of Statistics, 41(2), 484-507.

Yuen, K. K. (1974). The two-sample trimmed t for unequal population variances. Biometrika, 61(1), 165-170.

Phipson, B., & Smyth, G. K. (2010). Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology, 9(1), Article 39.

Examples


# Example 1: Basic two-sample test with formula notation
data(sleep)
result <- perm_t_test(extra ~ group, data = sleep)
#> Computing all 184756 exact permutations.
result
#> 
#> 	Exact Permutation Welch Two Sample t-test
#> 
#> data:  extra by group
#> t-observed = -1.8608, df = 17.776, p-value = 0.08142
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#>  -3.34  0.18
#> sample estimates:
#>           mean of group '1'           mean of group '2' 
#>                        0.75                        2.33 
#> mean difference ('1' - '2') 
#>                       -1.58 
#> 

# Example 2: One-sample permutation t-test
set.seed(123)
x <- rnorm(20, mean = 0.5, sd = 1)
perm_t_test(x, mu = 0, R = 199)
#> Note: Number of permutations (R = 199) is less than 1000. Consider increasing R for more stable p-value estimates.
#> 
#> 	Randomization Permutation One Sample t-test
#> 
#> data:  x
#> t-observed = 2.9501, df = 19, p-value = 0.015
#> alternative hypothesis: true mean is not equal to 0
#> 95 percent confidence interval:
#>  0.1769754 0.9878550
#> sample estimates:
#> mean of x 
#> 0.6416238 
#> 

# Example 3: Paired samples test
before <- c(5.1, 4.8, 6.2, 5.7, 6.0, 5.5, 4.9, 5.8)
after <- c(5.6, 5.2, 6.7, 6.1, 6.5, 5.8, 5.3, 6.2)
perm_t_test(x = before, y = after,
            paired = TRUE,
            alternative = "less",
            R = 199)
#> 
#> 	Randomization Permutation Paired t-test
#> 
#> data:  before and after
#> t-observed = -17, df = 7, p-value = 0.005
#> alternative hypothesis: true difference in means is less than 0
#> 95 percent confidence interval:
#>     -Inf -0.3875
#> sample estimates:
#> mean of the differences (z = x - y) 
#>                              -0.425 
#>