ROC curve and AUC calculator
Draw a ROC curve from scores and true labels, with the AUC, its DeLong confidence interval and the best cut-off by Youden’s J.
A clear answer. Here’s how.
Sorts the cases by score and treats each observed score as a cut-off: a case is called positive when its score is at or above the cut-off (at or below when the direction is flipped). Each cut-off gives a sensitivity and a specificity, and joining the points (1 − specificity, sensitivity) from (0, 0) to (1, 1) gives the empirical ROC curve; tied scores across the classes make a diagonal step. The area under it equals the Mann–Whitney probability that a random positive case outranks a random negative one, with ties counting half. Its standard error is DeLong, DeLong and Clarke-Pearson’s (1988) nonparametric estimate from the placement values of each case, and the interval is AUC ± z × SE (a normal approximation), clipped to 0–1; it needs two cases of each class and is rough for small samples or an AUC near 1. The p-value tests AUC = 0.5 with the Mann–Whitney normal approximation, corrected for ties and for continuity. The suggested cut-off maximises Youden’s J = sensitivity + specificity − 1; when several cut-offs tie, the one with the higher sensitivity is shown. Predictive values at that cut-off use the prevalence in the data. A third column may give how many cases share a score and label, as in rating studies. It does not compare two ROC curves, smooth the curve or fit a binormal model, and the best cut-off for a real decision also depends on the costs of each kind of error. It is a calculator, not a substitute for a statistician when decisions depend on the answer.
How this is checked
4 reference checks passed These 4 cases were run when this page was built, on 20 October 2026, and all reproduce within the stated tolerance. The same cases are part of the automated tests that run on every change to Toolmoz. Engine version 1.
See the checks and where each expected answer comes from
Case 1
- Inputs
- The tool’s default inputs
- Expected, and what Toolmoz gave
- Area under the ROC curve (AUC) expected 0.8931710615280595, Toolmoz 0.8931710615280595Standard error of the AUC (DeLong) expected 0.03072440837938112, Toolmoz 0.03072440837938111595% confidence interval, lower limit expected 0.8329523276581718, Toolmoz 0.832952327658172395% confidence interval, upper limit expected 0.9533897953979471, Toolmoz 0.9533897953979467Best cut-off by Youden’s J expected 4, Toolmoz 4Youden’s J at the cut-off expected 0.638607167004733, Toolmoz 0.6386071670047331Mann–Whitney z (AUC against 0.5) expected 7.352569019085503, Toolmoz 7.352569019085503
- Where the expected answer comes from
- Hanley and McNeil (1982) Table I ratings of 58 normal and 51 abnormal CT images: published area 0.893. The exact area 1321/1479, the DeLong variance 0.000943989 and the tie-corrected Mann–Whitney z come from an independent Python fractions script that compares all 58 × 51 pairs; J at rating 4 = 44/51 + 45/58 − 1
- Allowed difference
- ±1e-9 relative
Case 2
- Inputs
- Scores and labels, one case per line 0.9,1 0.8,1 0.6,1 0.4,1 0.7,0 0.5,0 0.3,0 0.2,0
- Expected, and what Toolmoz gave
- Area under the ROC curve (AUC) expected 0.8125, Toolmoz 0.8125Standard error of the AUC (DeLong) expected 0.1692508000965825, Toolmoz 0.169250800096582595% confidence interval, upper limit expected 1, Toolmoz 1Best cut-off by Youden’s J expected 0.4, Toolmoz 0.4Youden’s J at the cut-off expected 0.5, Toolmoz 0.5Sensitivity at the cut-off expected 100, Toolmoz 100Specificity at the cut-off expected 50, Toolmoz 50
- Where the expected answer comes from
- Counting pairs by hand: the positives 0.9 and 0.8 beat all four negatives, 0.6 beats three and 0.4 beats two, so AUC = 13/16. DeLong placements 1, 1, 3/4, 1/2 and 1/2, 1/4, 0, 0 give variance 11/384 (Python fractions); 0.8125 + 1.96 × 0.169 is above 1, so the upper limit is clipped. J = 1/2 ties at 0.8, 0.6 and 0.4; the rule picks 0.4, the highest sensitivity
- Allowed difference
- ±1e-12
Case 3
- Inputs
- Scores and labels, one case per line -3,1 -3,1 -2,1 -2,1 -1,1 -2,0 -1,0 -1,0 -1,0 0,0A higher score means lower
- Expected, and what Toolmoz gave
- Area under the ROC curve (AUC) expected 0.86, Toolmoz 0.86Standard error of the AUC (DeLong) expected 0.11489125293076058, Toolmoz 0.11489125293076058Best cut-off by Youden’s J expected -2, Toolmoz -2Sensitivity at the cut-off expected 80, Toolmoz 80Specificity at the cut-off expected 80, Toolmoz 80
- Where the expected answer comes from
- Lower scores mean positive: of the 25 pairs 19 are ordered correctly and 5 are tied (counting half), so AUC = 21.5/25 = 43/50; DeLong variance 33/2500 from an independent Python fractions script. At “score ≤ −2”, 4 of 5 positives and 4 of 5 negatives are classified correctly
- Allowed difference
- ±1e-12
Case 4
- Inputs
- A higher score means lower
- Expected, and what Toolmoz gave
- Area under the ROC curve (AUC) expected 0.1068289384719405, Toolmoz 0.1068289384719405
- Where the expected answer comes from
- Reversing the direction swaps the classes’ roles, so the area is 1 − 1321/1479 = 158/1479 (Mann–Whitney identity: P(X lower) + half the ties = 1 − P(X higher) − half the ties)
- Allowed difference
- ±1e-12
What this shows: the engine reproduces independently worked values for these cases. It does not prove every possible input is right. To test an answer you were given, see how to check a number; to see what is and is not checked across Toolmoz, see the accuracy ledger.
See a worked example
Worked example
A fixed demonstration, separate from the values you enter above.
- Scores and labels, one case per line
rating,truth,count 1,normal,33 2,normal,6 3,normal,6 4,normal,11 5,normal,2 1,abnormal,3 2,abnormal,2 3,abnormal,2 4,abnormal,11 5,abnormal,33
- Column separator
- Detect automatically
- Label of the positive class
- A higher score means
- More likely to be positive
- Confidence level
- 95%
Example result
- Area under the ROC curve (AUC)
- 0.8932
- Standard error of the AUC (DeLong)
- 0.0307
- 95% confidence interval, lower limit
- 0.8330
- 95% confidence interval, upper limit
- 0.9534
- Mann–Whitney z (AUC against 0.5)
- 7.353
- P-value (AUC = 0.5, two-sided)
- 1.94e-13 (below 0.0001)
- Best cut-off by Youden’s J
- ≥ 4
- Youden’s J at the cut-off
- 0.6386
- Sensitivity at the cut-off
- 86.27%
- Specificity at the cut-off
- 77.59%
- Positive predictive value at the cut-off
- 77.19%
- Negative predictive value at the cut-off
- 86.54%
- Positive cases (abnormal)
- 51
- Negative cases (normal)
- 58
- Distinct scores
- 5
- Rows skipped (missing score or label)
- 0
AUC is the Mann–Whitney probability that a random positive case scores higher than a random negative one, with ties counting half. The 95% interval is AUC ± z × the DeLong standard error, clipped to 0–1. The cut-off maximises Youden’s J = sensitivity + specificity − 1 over the observed scores; predictive values use the prevalence in these data (46.8%). Download gives the full threshold table.