Fleiss’ kappa calculator (agreement among three or more raters)
Measure agreement beyond chance among several raters who each sort the same subjects into categories: Fleiss’ kappa, per-category kappa, a z test and Landis and Koch labels.
A clear answer. Here’s how.
Each of N subjects receives the same number n of ratings (n ≥ 2) in k categories, and nij is how many raters put subject i in category j. The share of all ratings in category j is pj = Σi nij ÷ (N n). The agreement within subject i is Pi = (Σj nij² − n) ÷ (n(n − 1)), the share of rater pairs that agree; P̄ is the mean of the Pi, and Pe = Σj pj² is the agreement expected if every rating were drawn independently from the pooled shares. Kappa = (P̄ − Pe) ÷ (1 − Pe). The kappa of one category is κj = 1 − Σi nij(n − nij) ÷ (N n (n − 1) pj qj) with qj = 1 − pj. Under the null hypothesis that kappa is zero (Fleiss 1971, as R’s irr::kappam.fleiss computes it with exact = FALSE) the variance of kappa is 2 ÷ (N n (n − 1)) × [(Σ pj qj)² − Σ pj qj (qj − pj)] ÷ (Σ pj qj)², and z = κ ÷ SE is referred to the normal distribution, two-sided; each κj has SE √(2 ÷ (N n (n − 1))). That standard error is valid only when kappa is zero, so it cannot give a confidence interval around kappa, and none is shown. With two raters Fleiss’ kappa equals Scott’s pi, which pools the two raters’ category shares, so it differs from Cohen’s kappa, which keeps each rater’s own shares; for two raters entered as a table of ratings the tool shows both. The Landis and Koch (1977) labels (below 0 poor, up to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, above 0.80 almost perfect) are a rule of thumb, not a test, and kappa also depends on how common each category is. Every subject must have the same number of ratings, otherwise the tool stops with a message; the raters need not be the same people for each subject. Categories are treated as unordered, so on an ordered scale a near miss counts as a full disagreement. It does not compute Conger’s exact kappa, Krippendorff’s alpha or Gwet’s AC1. It is a calculator, not a substitute for a statistician when decisions depend on the answer.
How this is checked
5 reference checks passed These 5 cases were run when this page was built, on 20 October 2026, and all reproduce within the stated tolerance. The same cases are part of the automated tests that run on every change to Toolmoz. Engine version 1.
See the checks and where each expected answer comes from
Case 1
- Inputs
- The tool’s default inputs
- Expected, and what Toolmoz gave
- Subjects (N) expected 10, Toolmoz 10Raters per subject (n) expected 14, Toolmoz 14Mean observed agreement (P̄) expected 0.378, Toolmoz 0.378021978021978Expected agreement by chance (P̄e) expected 0.213, Toolmoz 0.21275510204081632Fleiss’ kappa expected 0.21, Toolmoz 0.20993070442195522
- Where the expected answer comes from
- Wikipedia’s Fleiss’ kappa example (10 subjects, 14 raters, 5 categories) prints P-bar = 0.378, Pe = 0.213 and kappa = 0.210; the values are rounded to 3 decimals, hence the tolerance
- Allowed difference
- ±0.0005
Case 2
- Inputs
- The tool’s default inputs
- Expected, and what Toolmoz gave
- Fleiss’ kappa expected 0.20993070442195524, Toolmoz 0.20993070442195522Standard error of kappa under the null (kappa = 0) expected 0.01696506922439313, Toolmoz 0.016965069224393132z for kappa = 0 expected 12.374291059190467, Toolmoz 12.374291059190464Kappa for category Cat1 expected 0.2012820512820513, Toolmoz 0.20128205128205134Kappa for category Cat5 expected 0.5076566951566952, Toolmoz 0.5076566951566952Agreement band (Landis and Koch) expected fair, Toolmoz fair
- Where the expected answer comes from
- The same table in exact fractions in an independent Python script: P-bar = 3.78022/10 and Pe = 0.212755, so kappa = 4211/20059; the null SE is √(2 ÷ (10 × 14 × 13) × ((Σ pq)² − Σ pq(q − p)) ÷ (Σ pq)²) as in the irr::kappam.fleiss source, and each category kappa is 1 − Σ n(14 − n) ÷ (10 × 14 × 13 × p q), which equals the (pjk − pj) ÷ (1 − pj) form the irr source uses
- Allowed difference
- ±1e-9 relative
Case 3
- Inputs
- Input is ratingsCounts or ratings, one subject per line Yes,Yes Yes,No Yes,No Yes,No Yes,No Yes,No Yes,Maybe Yes,Maybe Yes,Maybe Yes,Maybe Yes,Maybe Yes,Maybe No,Yes No,No No,No No,No No,No No,No No,Maybe No,Maybe No,Maybe No,Maybe No,Maybe No,Maybe No,Maybe No,Maybe No,Maybe Maybe,Yes Maybe,Yes Maybe,Yes Maybe,Yes Maybe,No Maybe,No Maybe,No Maybe,No Maybe,No Maybe,Maybe Maybe,Maybe Maybe,Maybe Maybe,Maybe Maybe,Maybe Maybe,Maybe Maybe,Maybe Maybe,Maybe Maybe,Maybe
- Expected, and what Toolmoz gave
- Fleiss’ kappa expected -0.056338028169014086, Toolmoz -0.05633802816901419Mean observed agreement (P̄) expected 0.3333333333333333, Toolmoz 0.3333333333333333Expected agreement by chance (P̄e) expected 0.3688888888888889, Toolmoz 0.36888888888888893Cohen’s kappa for the same two raters (for comparison) expected -0.041666666666666664, Toolmoz -0.04166666666666659
- Where the expected answer comes from
- Wikipedia’s Scott’s pi example (45 items, margins 12, 15, 18 and 6, 15, 24, agreements 1 + 5 + 9): observed agreement 15/45 = 1/3, pooled shares 18/90, 30/90 and 42/90, so Pe = 0.04 + 1/9 + 49/225 = 83/225 and pi = (1/3 − 83/225) ÷ (142/225) = −8/142 = −4/71 (the page prints about −0.057 from 3-decimal rounding); Cohen’s kappa keeps each rater’s own margins, pe = (12 × 6 + 15 × 15 + 18 × 24) ÷ 2025 = 0.36, so kappa = (1/3 − 0.36) ÷ 0.64 = −1/24
- Allowed difference
- ±1e-12
Case 4
- Inputs
- Counts or ratings, one subject per line 3,0 2,1 1,2 0,3
- Expected, and what Toolmoz gave
- Fleiss’ kappa expected 0.3333333333333333, Toolmoz 0.33333333333333326Standard error of kappa under the null (kappa = 0) expected 0.28867513459481287, Toolmoz 0.28867513459481287z for kappa = 0 expected 1.1547005383792515, Toolmoz 1.1547005383792512P-value for kappa = 0 (two-sided) expected 0.24821307898992362, Toolmoz 0.2482130789899235Kappa for category Category 1 expected 0.3333333333333333, Toolmoz 0.33333333333333337
- Where the expected answer comes from
- Four subjects, three raters, two categories, by hand: Pi = 1, 1/3, 1/3, 1 so P-bar = 2/3; p1 = p2 = 1/2 so Pe = 1/2 and kappa = (2/3 − 1/2) ÷ (1/2) = 1/3; Σ pq = 1/2 and Σ pq(q − p) = 0, so the variance is 2/(4 × 3 × 2) = 1/12 and z = (1/3) ÷ √(1/12) = 1.1547; the two-sided normal p-value erfc(z ÷ √2) from an independent Python script
- Allowed difference
- ±1e-9
Case 5
- Inputs
- Counts or ratings, one subject per line 3,0,0 0,3,0 0,0,3 3,0,0
- Expected, and what Toolmoz gave
- Fleiss’ kappa expected 1, Toolmoz 1Mean observed agreement (P̄) expected 1, Toolmoz 1Expected agreement by chance (P̄e) expected 0.375, Toolmoz 0.375
- Where the expected answer comes from
- Complete agreement on every subject: Pi = 1 for all four, shares 1/2, 1/4 and 1/4, so Pe = 1/4 + 1/16 + 1/16 = 0.375 and kappa = (1 − 0.375) ÷ (1 − 0.375) = 1
- Allowed difference
- ±1e-12
What this shows: the engine reproduces independently worked values for these cases. It does not prove every possible input is right. To test an answer you were given, see how to check a number; to see what is and is not checked across Toolmoz, see the accuracy ledger.
See a worked example
Worked example
A fixed demonstration, separate from the values you enter above.
- Input is
- A table of counts: one row per subject, one column per category
- Counts or ratings, one subject per line
Cat1,Cat2,Cat3,Cat4,Cat5 0,0,0,0,14 0,2,6,4,2 0,0,3,5,6 0,3,9,2,0 2,2,8,1,1 7,7,0,0,0 3,2,6,3,0 2,5,3,2,2 6,5,2,1,0 0,2,2,3,7
- Column separator
- Detect automatically
Example result
| Category | Ratings | Share of all ratings | Kappa | z (kappa = 0) | P-value |
|---|---|---|---|---|---|
| Cat1 | 20 | 0.1429 | 0.201 | 6.07 | 1.26e-9 (below 0.0001) |
| Cat2 | 28 | 0.2000 | 0.080 | 2.40 | 0.01625 |
| Cat3 | 39 | 0.2786 | 0.172 | 5.18 | 2.26e-7 (below 0.0001) |
| Cat4 | 21 | 0.1500 | 0.030 | 0.92 | 0.3594 |
| Cat5 | 32 | 0.2286 | 0.508 | 15.31 | 6.16e-53 (below 0.0001) |
Observed agreement 0.378, expected by chance 0.213: kappa = (observed − expected) ÷ (1 − expected). The z test is of kappa = 0 under the null hypothesis (Fleiss 1971, as R’s kappam.fleiss with exact = FALSE); its standard error is not valid for a confidence interval around kappa, so none is shown. The Landis and Koch labels (fair here) are a rule of thumb, not a test. Kappa depends on how common each category is, so a rare category can give a low kappa with high raw agreement. The raters need not be the same people for each subject.