The confusion matrix of a credit scorecard, cut-off by cut-off, and why the threshold that maximises F1 is not the one that maximises profit.
For a credit default model, treat a declined application as a predicted default. Precision is the share of declined applicants who would have defaulted, recall the share of all defaulters the cut-off declines, and F1 is their harmonic mean, 2PR / (P + R). On an illustrative book of 10,000 applications, a cut-off at 540 points gives precision of 16.4 per cent, recall of 56.1 per cent and an F1 of 0.254. F1 peaks one band lower, at 520, and that cut-off earns 2.11 million less.
Worked in full in Machine Learning for Finance by Julian R. Sterling, with every figure reproduced in a free workbook.See the book on Amazon →
The book below is illustrative: 10,000 applications in ten score bands, 672 of which went bad, a bad rate of 6.72 per cent. The unit economics are those derived in Machine Learning for Finance: a loan that performs earns 6,982, one that defaults loses 77,785.
| Score band | Applications | Bads | Bad rate |
|---|---|---|---|
| below 500 | 600 | 150 | 25.0% |
| 500-519 | 800 | 128 | 16.0% |
| 520-539 | 900 | 99 | 11.0% |
| 540-559 | 1,000 | 80 | 8.0% |
| 560-579 | 1,100 | 66 | 6.0% |
| 580-599 | 1,200 | 54 | 4.5% |
| 600-619 | 1,200 | 42 | 3.5% |
| 620-639 | 1,100 | 28 | 2.5% |
| 640-659 | 1,100 | 17 | 1.5% |
| 660 and above | 1,000 | 8 | 0.8% |
| Total | 10,000 | 672 | 6.72% |
Decline everything below 540 points: the first three bands, 2,300 applications. The positive class is "default", so the four cells are:
True positives (bads declined) = 150 + 128 + 99 = 377
False positives (goods declined) = 2,300 − 377 = 1,923
False negatives (bads approved) = 672 − 377 = 295
True negatives (goods approved) = 9,328 − 1,923 = 7,405
Precision = TP / (TP + FP) = 377 / 2,300 = 16.39%
Recall = TP / (TP + FN) = 377 / 672 = 56.10%
F1 = 2 × P × R / (P + R) = 2 × 16.39% × 56.10% / 72.49% = 0.2537
Excel, cut-off in band k: =SUM(Bads1:Badsk)/SUM(Apps1:Appsk) for precision, =SUM(Bads1:Badsk)/SUM(Bads) for recall
A precision of 16.39 per cent looks poor to anyone used to classification benchmarks. In credit it is normal: with a 6.72 per cent base rate, even a good model declines several good applicants for every bad one it stops.
Profit counts approved goods at 6,982 and approved bads at minus 77,785.
| Cut-off | Declined | Precision | Recall | F1 | Accuracy | Profit, million |
|---|---|---|---|---|---|---|
| none | 0 | 0.0% | 0.0% | 0.000 | 93.3% | 12.86 |
| 500 | 600 | 25.0% | 22.3% | 0.236 | 90.3% | 21.38 |
| 520 | 1,400 | 19.9% | 41.4% | 0.268 | 84.8% | 26.65 |
| 540 | 2,300 | 16.4% | 56.1% | 0.254 | 77.8% | 28.76 |
| 560 | 3,300 | 13.8% | 68.0% | 0.230 | 69.4% | 28.55 |
| 580 | 4,400 | 11.9% | 77.8% | 0.206 | 59.7% | 26.47 |
| 600 | 5,600 | 10.3% | 85.9% | 0.184 | 48.8% | 22.67 |
Three metrics, three answers. Accuracy is highest when nothing is declined, 93.28 per cent, because predicting "good" for everyone is right for 93.28 per cent of the book. F1 is highest at 520, 0.2683. Profit is highest at 540, 28.76 million, 2.11 million more than the F1 choice.
F1 treats a declined good and an approved bad as roughly equal errors. The economics do not: one default costs 77,785, the profit of 11.14 performing loans. The correct rule is to decline a band when its bad rate exceeds the break-even:
Break-even bad rate = profit on a good / (profit on a good + loss on a bad)
= 6,982 / (6,982 + 77,785) = 6,982 / 84,767 = 8.2367%
The 520-539 band runs at 11.0 per cent, above break-even, so it should be declined; F1 keeps it. Declining it rejects 801 good loans and stops 99 bad ones, worth 2,108,133. The 540-559 band, at 8.0 per cent, sits just below break-even and should be approved, which is what the profit column shows: moving to 560 costs 0.2 million.
The break-even depends on loss given default and margin, not on the model. If losses rise, the break-even bad rate falls and the cut-off should move up. Precision and recall do not move at all.
Reporting F1 or accuracy as the model's performance and choosing the threshold that maximises it. Both are useful for comparing models at a fixed approval rate; neither should set the cut-off. Ranking quality across all cut-offs is what Gini and KS measure, in how to calculate Gini and KS for a scorecard; what a cut-off left in the wrong place costs is in what an unchanged score cut-off costs.
The unit economics, the break-even and both profit grids are live in the free workbook for this case.
Default, so a declined application is a predicted positive. With 672 bads in 10,000 applications, a cut-off declining 2,300 applicants that catches 377 bads has 377 true positives, 1,923 false positives and 295 false negatives, giving precision of 16.39 per cent and recall of 56.10 per cent.
Because defaults are rare. On a book with a 6.72 per cent bad rate, approving everybody is 93.28 per cent accurate and catches no defaulter. Every cut-off that declines anyone lowers accuracy, even the one that lifts profit from 12.86 million to 28.76 million in the illustrative case.
Decline a score band when its bad rate exceeds the break-even, profit on a good divided by the sum of that profit and the loss on a bad. With 6,982 earned and 77,785 lost, the break-even is 8.2367 per cent. The 520-539 band at 11.0 per cent is declined, which F1 would keep, adding 2,108,133 of profit.
This article is one calculation from Machine Learning for Finance. The book takes the same case from first principles to the decision, chapter by chapter, and every figure it prints is a live formula in the free companion workbooks.
Get the book on Amazon →Free companion files
Also on Amazon UK · Amazon Germany · Amazon France · Amazon Canada
Reading guide: corporate finance, valuation and markets → · All 1340 articles →
If this book helped, or didn’t, a few lines on Amazon are worth more than they look: they are what the next reader goes on. Write a review. The workbook stays free either way.