The rater agreement and effect sizes
toolbox.
Measure rater agreement, test paired yes-or-no results and find a number needed to treat.
Keep this toolbox for offline use
10 useful places to start.
Statistical calculators, not medical advice and not a substitute for a statistician when decisions depend on the answer. Interpretation bands such as “fair” or “good” are rules of thumb, not tests.
Cohen’s kappa calculator
Measure agreement between two raters beyond chance, with optional weights for ordered categories.
Fleiss’ kappa calculator (agreement among three or more raters)
Measure agreement beyond chance among several raters who each sort the same subjects into categories: Fleiss’ kappa, per-category kappa, a z test and Landis and Koch labels.
Intraclass correlation (ICC) calculator, all six Shrout and Fleiss forms
Work out the intraclass correlation from a subjects by raters table: all six Shrout and Fleiss forms with ANOVA mean squares, F tests, confidence intervals and a plot.
Cronbach’s alpha (scale reliability)
Measure the internal consistency of questionnaire items with Cronbach’s alpha, alpha if an item is deleted, item-total correlations and a Feldt interval.
McNemar test calculator for paired proportions
Test whether a paired yes/no result changed from a 2×2 table of matched outcomes: exact, mid-p and chi-square McNemar p-values, the discordant odds ratio and an interval for the difference.
Number needed to treat (NNT) calculator with confidence interval
Work out the number needed to treat from event counts or rates: absolute and relative risk reduction, odds ratio and an NNT interval that stays correct when it includes no effect.
2×2 table: risk ratio, odds ratio and Fisher exact test
Analyse a 2×2 table of exposure and outcome counts with risk ratio, odds ratio and exact p-values.
Diagnostic test calculator (sensitivity, specificity, predictive values)
Work out sensitivity, specificity, predictive values, likelihood ratios and more from a 2×2 table, each with a confidence interval.
Wilson proportion interval
Calculate a bounded confidence interval for a count out of a sample.
Multiple comparisons p-value correction
Adjust several p-values with Bonferroni, Holm or Benjamini–Hochberg corrections.
A few things you
can tick off.
- Measure agreement between three or more raters
- Choose the right intraclass correlation for a design
- Test a before-and-after change in yes-or-no results
- Find the number needed to treat with a confidence interval