Reliability Statistics — reli_stats • SimplyAgree

The reli_stats and reli_aov functions produce reliability statistics described by Weir (2005). This includes intraclass correlation coefficients, the coefficient of variation, and the standard MSE of measurement.

reli_stats(
  measure,
  item,
  id,
  data,
  wide = FALSE,
  col.names = NULL,
  se_type = c("MSE", "ICC1", "ICC2", "ICC3", "ICC1k", "ICC2k", "ICC3k"),
  cv_calc = c("MSE", "residuals", "SEM"),
  conf.level = 0.95,
  other_ci = FALSE,
  type = c("chisq", "perc", "norm", "basic"),
  replicates = 1999
)

reli_aov(
  measure,
  item,
  id,
  data,
  wide = FALSE,
  col.names = NULL,
  se_type = c("MSE", "ICC1", "ICC2", "ICC3", "ICC1k", "ICC2k", "ICC3k"),
  cv_calc = c("MSE", "residuals", "SEM"),
  conf.level = 0.95,
  other_ci = FALSE,
  type = c("chisq", "perc", "norm", "basic"),
  replicates = 1999
)

Arguments

measure: Name of column containing the measurement of interest.
item: Name of column containing the items. If this is a test-retest reliability study then this would indicate the time point (e.g., time1,time2, time3, etc.).
id: Column with subject identifier.
data: Data frame with all data.
wide: Logical value (TRUE or FALSE) indicating if data is in a "wide" format. Default is TRUE.
col.names: If wide is equal to TRUE then col.names is a list of the column names containing the measurements for reliability analysis.
se_type: Type of standard error calculation. The default is to use the mean square error (MSE). Otherwise, the total sums of squares and the ICC are utilized to estimate the SEM, SEE, and SEP.
cv_calc: Coefficient of variation (CV) calculation. This function allows for 3 versions of the CV. "MSE" is the default.
conf.level: the confidence level required. Default is 95%.
other_ci: Logical value (TRUE or FALSE) indicating whether to calculate confidence intervals for the CV, SEM, SEP, and SEE. Note: this will dramatically increase the computation time.
type: A character string representing the type of confidence intervals for the CV, SEM, SEP, and SEE. Only "norm", "basic", and "perc" currently supported for parametric bootstrap CI. An approximate method, chisq, is the default. See ?boot::boot.ci for more details on bootstrap methods.
replicates: The number of bootstrap replicates. Passed on to the boot function. Default is 1999.

Value

Returns single list with the results of the agreement analysis.

icc: Table of ICC results
lmer: Linear mixed model from lme4
anova: Analysis of Variance table
var_comp: Table of Variance Components
n.id: Number of subjects/participants
n.items: Number of items/time points
cv: Coefficient of Variation
SEM: List with Standard MSE of Measurement estimate (est)
SEE: List with Standard MSE of the Estimate estimate (est)
SEP: List with Standard MSE of Predictions (est)
call: the matched call

Details

These functions return intraclass correlation coefficients and other measures of reliability (CV, SEM, SEE, and SEP). The estimates of variances for any of the measures are derived from linear mixed models. When other_ci is set to TRUE, then a parametric bootstrap approach to calculating confidence intervals is used for the CV, SEM, SEE, and SEP.

reli_stats uses a linear mixed model to estimate variance components. In some cases there are convergence issues. When this occurs it is prudent to use reli_aov which instead utilizes sums of squares approach. The results may differ slightly between the functions. If reli_aov is used then rows with missing observations (e.g., if a participant has a missing observation) will be dropped.

The CV calculation has 3 versions. The "MSE" uses the "mean squared error" from the linear mixed model used to calculate the ICCs. The "SEM" option instead uses the SEM calculation and expresses CV as a ratio of the SEM to the overall mean. The "residuals" option u uses the model residuals to calculate the root mean square error which is then divided by the grand mean.

The CV, SEM, SEE, and SEP values can have confidence intervals produced if the other_ci argument is set to TRUE. For the CV, the default method (type = "chisq") is Vangal's modification of the McKay approximation. For the other measures, a simple chi-squared approximation is utilized (Hann & Meeker, 1991). All other methods are bootstrapping based methods (see ?boot::boot). The reli_stats functions utilizes a parametric bootstrap while the reli_aov function utilizes an ordinary (non-parametric) bootstrap method.

References

Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. The Journal of Strength & Conditioning Research, 19(1), 231-240.

Shrout, P.E. and Fleiss, J.L. (1976). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86, 420-3428.

McGraw, K. O. and Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46. See errata on page 390 of same volume.

Hahn, G. J., & Meeker, W. Q. (2011). Statistical intervals: a guide for practitioners (Vol. 92). John Wiley & Sons. pp. 55-56.

Vangel, M. G. (1996). Confidence intervals for a normal coefficient of variation. The American Statistician, 50(1), 21-26.

Examples

data('reps')
reli_stats(data = reps, wide = TRUE, col.names = c("x","y"))
#> Only 2 items in data. It is recommended to use reli_aov instead of reli_stats.
#> 
#> Coefficient of Variation (%):  15.8
#> Standard Error of Measurement (SEM):  0.863
#> Standard Error of the Estimate (SEE):  0.655
#> Standard Error of Prediction (SEP):  1.11
#> 
#> Intraclass Correlation Coefficients with  95 % C.I.
#>            Model         Measures  Type    ICC Lower CI Upper CI
#> 1 one-way random        Agreement  ICC1 0.4963   0.1632   0.7299
#> 2 two-way random        Agreement  ICC2 0.5097   0.1911   0.7355
#> 3  two-way fixed      Consistency  ICC3 0.5384   0.2117   0.7569
#> 4 one-way random   Avg. Agreement ICC1k 0.6634   0.2806   0.8438
#> 5 two-way random   Avg. Agreement ICC2k 0.6753   0.3209   0.8476
#> 6  two-way fixed Avg. Consistency ICC3k 0.7000   0.3495   0.8616
#>