Intraclass correlation (ICC) calculator, all six Shrout and Fleiss forms
Work out the intraclass correlation from a subjects by raters table: all six Shrout and Fleiss forms with ANOVA mean squares, F tests, confidence intervals and a plot.
A clear answer. Here’s how.
The table has n subjects (rows) rated by k raters or on k occasions (columns). A two-way ANOVA without replication gives the mean squares BMS between subjects (n − 1 degrees of freedom), JMS between raters (k − 1) and EMS for the residual ((n − 1)(k − 1)); a one-way ANOVA that ignores the raters gives WMS within subjects (n(k − 1)). The six forms of Shrout and Fleiss (1979) are ICC(1,1) = (BMS − WMS) ÷ (BMS + (k − 1) WMS), ICC(2,1) = (BMS − EMS) ÷ (BMS + (k − 1) EMS + k (JMS − EMS) ÷ n), ICC(3,1) = (BMS − EMS) ÷ (BMS + (k − 1) EMS), and for the mean of k ratings ICC(1,k) = (BMS − WMS) ÷ BMS, ICC(2,k) = (BMS − EMS) ÷ (BMS + (JMS − EMS) ÷ n) and ICC(3,k) = (BMS − EMS) ÷ BMS. In McGraw and Wong’s (1996) names these are the one-way random effects model, the two-way random effects model (absolute agreement) and the two-way mixed effects model (consistency), each for a single rater or the mean of k raters. F = BMS ÷ WMS on n − 1 and n(k − 1) degrees of freedom tests the one-way forms against zero, and F = BMS ÷ EMS on n − 1 and (n − 1)(k − 1) tests the two-way forms. The limits for the one-way and the fixed-rater forms are (FL − 1) ÷ (FL + k − 1) and (FU − 1) ÷ (FU + k − 1) with FL = F ÷ F(1 − α/2; df1, df2) and FU = F × F(1 − α/2; df2, df1), and 1 − 1/FL and 1 − 1/FU for the mean forms. The limits for ICC(2,1) use the approximate degrees of freedom ν of Shrout and Fleiss’ Case 2, evaluated at the estimate, and those for ICC(2,k) follow by the Spearman–Brown step k L ÷ (1 + (k − 1) L); both are exactly as R’s psych::ICC computes them and differ from some versions of SPSS. Which form to report is a design choice you must make (each is described in the note under the results); the tool shows all six and makes no choice for you. Limits can lie below 0 or above 1 and are rough for a few subjects, an ICC can be negative, and the bands of Koo and Li (2016) are a convention rather than a test. Only complete rows are used: a subject with a missing rating is left out and listed. It assumes ratings on an interval scale with roughly normal, equal-variance errors; it does not handle ordinal or binary ratings (use kappa), several ratings per subject and rater, or unbalanced designs. It is a calculator, not a substitute for a statistician when decisions depend on the answer.
How this is checked
5 reference checks passed These 5 cases were run when this page was built, on 20 October 2026, and all reproduce within the stated tolerance. The same cases are part of the automated tests that run on every change to Toolmoz. Engine version 1.
See the checks and where each expected answer comes from
Case 1
- Inputs
- The tool’s default inputs
- Expected, and what Toolmoz gave
- Subjects used (n) expected 6, Toolmoz 6Raters (k) expected 4, Toolmoz 4ICC(1,1) expected 0.166, Toolmoz 0.16574176840547544ICC(2,1) expected 0.29, Toolmoz 0.28976377952755905ICC(3,1) expected 0.715, Toolmoz 0.7148407148407149ICC(1,k) expected 0.443, Toolmoz 0.44279713367926865ICC(2,k) expected 0.62, Toolmoz 0.6200505475989891ICC(3,k) expected 0.909, Toolmoz 0.9093155423770695P-value, one-way F expected 0.164769, Toolmoz 0.16476880834463897P-value, two-way F expected 0.000135, Toolmoz 0.00013456651648433712
- Where the expected answer comes from
- Shrout and Fleiss (1979) Table 2 data (6 targets, 4 judges): R’s DescTools::ICC prints 0.166, 0.290, 0.715, 0.443, 0.620 and 0.909 with p = 0.164769 and 0.000135; the values are rounded to 3 decimals, hence the tolerance
- Allowed difference
- ±0.0005
Case 2
- Inputs
- The tool’s default inputs
- Expected, and what Toolmoz gave
- ICC(1,1): 95% CI lower limit expected -0.133, Toolmoz -0.1329323248747507ICC(1,1): 95% CI upper limit expected 0.72, Toolmoz 0.7225600623281208ICC(2,1): 95% CI lower limit expected 0.019, Toolmoz 0.018786513374712044ICC(2,1): 95% CI upper limit expected 0.76, Toolmoz 0.761084369648953ICC(3,1): 95% CI lower limit expected 0.342, Toolmoz 0.34246476503392514ICC(3,1): 95% CI upper limit expected 0.95, Toolmoz 0.9458582599553595ICC(1,k): 95% CI lower limit expected -0.884, Toolmoz -0.884442155238117ICC(1,k): 95% CI upper limit expected 0.91, Toolmoz 0.9124154203407754ICC(2,k): 95% CI lower limit expected 0.071, Toolmoz 0.07113681530250347ICC(2,k): 95% CI upper limit expected 0.93, Toolmoz 0.927232040167722ICC(3,k): 95% CI lower limit expected 0.676, Toolmoz 0.6756747138163046ICC(3,k): 95% CI upper limit expected 0.99, Toolmoz 0.9858916781690623
- Where the expected answer comes from
- The 95% limits printed by R’s psych::ICC for the same data (reproduced on David Howell’s Vermont page and in the LMU Munich lab handbook): −0.133 to 0.72, 0.019 to 0.76, 0.342 to 0.95, −0.884 to 0.91, 0.071 to 0.93 and 0.676 to 0.99; the printed limits are rounded to 2 or 3 decimals, hence the tolerance. An independent Python script that bisects a power-series incomplete beta for the F quantiles gives −0.1329 to 0.7226, 0.0188 to 0.7611, 0.3425 to 0.9459, −0.8844 to 0.9124, 0.0711 to 0.9272 and 0.6757 to 0.9859
- Allowed difference
- ±0.0051
Case 3
- Inputs
- The tool’s default inputs
- Expected, and what Toolmoz gave
- F, one-way (BMS ÷ WMS) expected 1.79, Toolmoz 1.794678492239468F, two-way (BMS ÷ EMS) expected 11.03, Toolmoz 11.027247956403272Mean square between subjects (BMS) expected 11.243, Toolmoz 11.241666666666669Mean square between raters (JMS) expected 32.483, Toolmoz 32.48611111111112Mean square residual (EMS) expected 1.019, Toolmoz 1.0194444444444444Mean square within subjects (WMS) expected 6.264, Toolmoz 6.263888888888889
- Where the expected answer comes from
- DescTools::ICC prints F = 1.79 on 5 and 18 df and 11.03 on 5 and 15 df for the Shrout and Fleiss data; the mean squares follow from the lme4 variance components printed in the LMU Munich lab handbook, BMS = 4 × 2.556 + 1.019, JMS = 6 × 5.244 + 1.019, EMS = 1.019 and WMS = 6.264 (exact fractions give 11.2417, 32.4861, 1.0194 and 6.2639); the printed values are rounded, hence the tolerance
- Allowed difference
- ±0.005
Case 4
- Inputs
- Ratings: one subject per line, one rater per column 2,4 4,5 6,9
- Expected, and what Toolmoz gave
- Mean square between subjects (BMS) expected 10.5, Toolmoz 10.5Mean square between raters (JMS) expected 6, Toolmoz 6Mean square residual (EMS) expected 0.5, Toolmoz 0.5Mean square within subjects (WMS) expected 2.3333333333333335, Toolmoz 2.3333333333333335ICC(1,1) expected 0.6363636363636364, Toolmoz 0.6363636363636362ICC(2,1) expected 0.6818181818181818, Toolmoz 0.6818181818181819ICC(3,1) expected 0.9090909090909091, Toolmoz 0.9090909090909091ICC(1,k) expected 0.7777777777777778, Toolmoz 0.7777777777777777ICC(2,k) expected 0.8108108108108109, Toolmoz 0.8108108108108107ICC(3,k) expected 0.9523809523809523, Toolmoz 0.9523809523809523F, two-way (BMS ÷ EMS) expected 21, Toolmoz 21P-value, two-way F expected 0.045454545454545456, Toolmoz 0.045454545454545525
- Where the expected answer comes from
- Three subjects, two raters, by hand: grand mean 5, subject means 3, 4.5 and 7.5, rater means 4 and 6, so SSR = 2 × (4 + 0.25 + 6.25) = 21 and BMS = 10.5, SSC = 3 × (1 + 1) = 6 and JMS = 6, SST = 28 so SSE = 1 and EMS = 0.5, SSW = 28 − 21 = 7 and WMS = 7/3; then ICC(1,1) = (10.5 − 7/3) ÷ (10.5 + 7/3) = 7/11, ICC(2,1) = 10 ÷ (11 + 2 × 5.5 ÷ 3) = 15/22, ICC(3,1) = 10/11, ICC(1,k) = 7/9, ICC(2,k) = 10 ÷ (10.5 + 5.5 ÷ 3) = 30/37 and ICC(3,k) = 20/21; with 2 and 2 degrees of freedom P(F at least 21) = 1 ÷ (1 + 21) = 1/22
- Allowed difference
- ±1e-12
Case 5
- Inputs
- Ratings: one subject per line, one rater per column J1,J2,J3,J4 9,2,5,8 6,1,3,2 8,4,6,8 7,1,2,6 10,5,6,9 6,2,4,7 5,NA,3,4
- Expected, and what Toolmoz gave
- Subjects used (n) expected 6, Toolmoz 6Subjects left out (missing ratings) expected 1, Toolmoz 1ICC(3,1) expected 0.715, Toolmoz 0.7148407148407149
- Where the expected answer comes from
- Shrout and Fleiss data with a seventh subject whose second rating is missing: that subject is left out, so the six remaining subjects give the published ICC(3,1) of 0.715 (DescTools::ICC)
- Allowed difference
- ±0.0005
What this shows: the engine reproduces independently worked values for these cases. It does not prove every possible input is right. To test an answer you were given, see how to check a number; to see what is and is not checked across Toolmoz, see the accuracy ledger.
See a worked example
Worked example
A fixed demonstration, separate from the values you enter above.
- Ratings: one subject per line, one rater per column
J1,J2,J3,J4 9,2,5,8 6,1,3,2 8,4,6,8 7,1,2,6 10,5,6,9 6,2,4,7
- Column separator
- Detect automatically
- Confidence level
- 95%
Example result
| Form | Model | ICC | F | df1 | df2 | P-value | 95% CI lower | 95% CI upper | Band: estimate | Band: lower limit |
|---|---|---|---|---|---|---|---|---|---|---|
| ICC(1,1) | One-way random effects, absolute agreement, single rater | 0.166 | 1.79 | 5 | 18 | 0.1648 | -0.133 | 0.723 | poor | poor |
| ICC(2,1) | Two-way random effects, absolute agreement, single rater | 0.290 | 11.03 | 5 | 15 | 0.0001346 | 0.019 | 0.761 | poor | poor |
| ICC(3,1) | Two-way mixed effects, consistency, single rater | 0.715 | 11.03 | 5 | 15 | 0.0001346 | 0.342 | 0.946 | moderate | poor |
| ICC(1,k) (k = 4) | One-way random effects, absolute agreement, mean of k raters | 0.443 | 1.79 | 5 | 18 | 0.1648 | -0.884 | 0.912 | poor | poor |
| ICC(2,k) (k = 4) | Two-way random effects, absolute agreement, mean of k raters | 0.620 | 11.03 | 5 | 15 | 0.0001346 | 0.071 | 0.927 | moderate | poor |
All six forms are shown because the form follows from the design, not from the data. One-way (ICC1): each subject was rated by a different, randomly chosen set of raters, or the raters cannot be told apart. Two-way random (ICC2): the same raters rated every subject and are a sample of the raters you want to generalise to; absolute agreement. Two-way mixed (ICC3): the same raters rated every subject and are the only raters of interest; consistency, so differences between the raters’ averages are ignored (ICC(3,k) equals Cronbach’s alpha). Use a single-rating form when one rater’s score is what will be used, and a mean-of-k form when the average of k raters will be. McGraw and Wong’s names are in the second column. Intervals are 95% F-based limits (Shrout and Fleiss; Case 2 by approximate degrees of freedom, as psych::ICC), so they can run below 0 or above 1 and are rough for few subjects. The bands (below 0.5 poor, to 0.75 moderate, to 0.9 good, above 0.9 excellent) are Koo and Li’s (2016) convention, not a test, and the lower limit is a more cautious guide than the estimate.