Cohen’s kappa calculator
Measure agreement between two raters beyond chance, with optional weights for ordered categories.
A clear answer. Here’s how.
Cohen’s kappa = (po − pe) ÷ (1 − pe), where po is the observed proportion of agreement and pe the agreement expected if both raters assigned categories independently with their own marginal frequencies. Weighted kappa gives partial credit to near-misses using linear weights 1 − |i − j| ÷ (k − 1) or quadratic weights 1 − ((i − j) ÷ (k − 1))², so categories must be listed in order. The standard error is the large-sample formula of Fleiss, Cohen and Everitt (1969); the interval is kappa ± z × SE. Verdicts use the Landis and Koch bands, which are conventions rather than tests.
How this is checked
4 reference checks passed These 4 cases were run when this page was built, on 20 October 2026, and all reproduce within the stated tolerance. The same cases are part of the automated tests that run on every change to Toolmoz. Engine version 1.
See the checks and where each expected answer comes from
Case 1
- Inputs
- Input is matrixTable of counts, or one “rating A,rating B” pair per line Rater A,Yes,No Yes,20,5 No,10,15
- Expected, and what Toolmoz gave
- Observed agreement expected 0.7, Toolmoz 0.7Chance-expected agreement expected 0.5, Toolmoz 0.5Cohen’s kappa expected 0.4, Toolmoz 0.3999999999999999Standard error expected 0.126996, Toolmoz 0.12699606293110036
- Where the expected answer comes from
- Wikipedia worked example: po = 35/50 = 0.7, pe = 0.5 × 0.6 + 0.5 × 0.4 = 0.5, kappa = 0.2 ÷ 0.5 = 0.4; standard error from the explicit unweighted Fleiss–Cohen–Everitt formula (also confirmed by a 20,000-sample Monte Carlo estimate of 0.127)
- Allowed difference
- ±0.00001 absolute
Case 2
- Inputs
- Input is matrixTable of counts, or one “rating A,rating B” pair per line Rater A,A,B A,45,15 B,25,15
- Expected, and what Toolmoz gave
- Observed agreement expected 0.6, Toolmoz 0.6Chance-expected agreement expected 0.54, Toolmoz 0.54Cohen’s kappa expected 0.1304348, Toolmoz 0.13043478260869554
- Where the expected answer comes from
- po = 60/100, pe = 0.6 × 0.7 + 0.4 × 0.3 = 0.54, kappa = 0.06 ÷ 0.46
- Allowed difference
- ±0.000001 absolute
Case 3
- Inputs
- Input is matrixTable of counts, or one “rating A,rating B” pair per line Rater A,low,mid,high low,11,3,1 mid,2,9,4 high,0,3,7Weighting linear
- Expected, and what Toolmoz gave
- Cohen’s kappa (linear weights) expected 0.5912409, Toolmoz 0.5912408759124086
- Where the expected answer comes from
- Linear-weighted kappa computed separately as 1 − Σ w·observed ÷ Σ w·expected with disagreement weights |i − j| ÷ 2
- Allowed difference
- ±0.000001 absolute
Case 4
- Inputs
- Input is pairsTable of counts, or one “rating A,rating B” pair per line yes,yes yes,no no,no no,no yes,yes
- Expected, and what Toolmoz gave
- Items rated expected 5, Toolmoz 5Observed agreement expected 0.8, Toolmoz 0.8Chance-expected agreement expected 0.48, Toolmoz 0.48Cohen’s kappa expected 0.6153846, Toolmoz 0.6153846153846155
- Where the expected answer comes from
- Pairs give counts [[2,1],[0,2]]: po = 4/5, pe = (3/5)(2/5) + (2/5)(3/5) = 0.48, kappa = 0.32 ÷ 0.52
- Allowed difference
- ±0.000001 absolute
What this shows: the engine reproduces independently worked values for these cases. It does not prove every possible input is right. To test an answer you were given, see how to check a number; to see what is and is not checked across Toolmoz, see the accuracy ledger.
See a worked example
Worked example
A fixed demonstration, separate from the values you enter above.
- Input is
- A table of counts
- Table of counts, or one “rating A,rating B” pair per line
Rater A,Yes,No Yes,20,5 No,10,15
- Column separator
- Comma
- Weighting
- None (unweighted kappa)
- Confidence level
- 95%
Example result
| Rater A \ Rater B | Yes | No | Total |
|---|---|---|---|
| Yes | 20 | 5 | 25 |
| No | 10 | 15 | 25 |
| Total | 30 | 20 | 50 |
Observed agreement 0.7, expected by chance 0.5, 50 items. Kappa = (observed − expected agreement) ÷ (1 − expected agreement). Bands are Landis and Koch’s widely used but arbitrary benchmarks, and kappa depends on how common each category is. All disagreements count equally. The interval is a large-sample approximation and is rough for small tables.