A development AUC gap is not a profit figure. Fix the approval rate, count the bads each model lets through, and price them.
Compare them at the approval rate the lender actually runs, out of time, in money, and net of what the trees cost to run. On an illustrative book of 10,000 small-business loans, gradient-boosted trees beat a logistic scorecard by 4.3 Gini points in development and stop 23 more bads at a 70 per cent approval rate. Out of time the gap is 1.5 points and 9 bads, worth 762,903 a year, or 312,903 after an extra 450,000 of running cost. The trees win, by far less than the development AUC suggests.
Worked in full in Machine Learning for Finance by Julian R. Sterling, with every figure reproduced in a free workbook.See the book on Amazon →
AUC and Gini measure ranking over the whole score range. A lender uses one point on that range, its cut-off, and earns money on the loans either side of it. Two models with a 4-point Gini gap can be almost identical at the cut-off, or far apart, depending on where their extra ranking power sits. The only like-for-like comparison is to fix the approval rate, let each model choose which applicants to approve, and count the bads each lets through. Same number of loans written, same volume, same marketing cost: the difference is pure ranking, and it can be priced.
| Input | Value |
|---|---|
| Loans written a year, each with a full outcome window | 10,000 |
| Bads | 625 |
| Base bad rate | 6.25% |
| Profit on a loan that performs | 6,982 |
| Loss on a loan that defaults | 77,785 |
| Extra yearly cost of the trees (validation, monitoring, reason codes) | 450,000 |
The unit economics are those of the book's fictional lender, per loan over its life, so the value of each year's 10,000 loans is set against one year of running cost. The 450,000 is an assumption: a second validation discipline, explanation tooling for decline reasons, and closer monitoring of a model nobody can read off a points table. Out of time, both models are scored on a later 10,000 loans with the same 625 bads, so that only the ranking moves.
Bads approved at approval rate s = all bads − bads in the riskiest (1 − s) of the book
Value of the better model = (bads approved by the card − bads approved by the trees) × (profit on a good + loss on a bad)
Excel, with each model's decile bads in B2:B11 and C2:C11, declining the riskiest three deciles: =(SUM(C2:C4)-SUM(B2:B4))*(6982+77785)
Step 1, the swing. At a fixed approval rate, every bad the better model screens out is replaced by a good it lets in. The book loses 77,785 less and earns 6,982 more: a swing of 84,767 per bad.
Step 2, the bads approved. At 70 per cent approval each model declines its riskiest 3,000 applicants. In development the card's three riskiest deciles hold 372 bads and the trees' 395, so the card approves 253 bads and the trees 230.
Step 3, price it. 23 fewer bads times 84,767 is 1,949,641. That is the development answer, and the one usually presented.
Step 4, out of time. On the later loans the card approves 266 bads and the trees 257. Nine bads at 84,767 is 762,903: 39 per cent of the development advantage survives.
Step 5, net of cost. 762,903 less 450,000 leaves 312,903 a year. The trees need to stop 5.31 extra bads a year just to pay for themselves.
| Approval rate | Card, dev | Trees, dev | Value, dev | Card, out of time | Trees, out of time | Value, out of time | Net of cost |
|---|---|---|---|---|---|---|---|
| 90% | 457 | 440 | 1,441,039 | 465 | 459 | 508,602 | 58,602 |
| 80% | 341 | 316 | 2,119,175 | 353 | 343 | 847,670 | 397,670 |
| 70% | 253 | 230 | 1,949,641 | 266 | 257 | 762,903 | 312,903 |
| 60% | 184 | 164 | 1,695,340 | 196 | 188 | 678,136 | 228,136 |
| 50% | 129 | 114 | 1,271,505 | 139 | 133 | 508,602 | 58,602 |
The value peaks in the middle of the range and fades at both ends. At 90 per cent approval only the riskiest decile is being declined, and the trees' extra ranking power lies partly elsewhere; at 50 per cent both models are declining most of the bads anyway. A lender that approves nine in ten applicants is paying 450,000 a year for 58,602 of benefit, a margin that one bad year would erase.
| Sample | Card Gini | Trees Gini | Gap, points |
|---|---|---|---|
| Development | 43.4% | 47.7% | 4.3 |
| Out of time | 40.8% | 42.3% | 1.5 |
Both models lose ranking out of time; the trees lose more, because some of what they learned in development was the development sample. That pattern is common with flexible models fitted on a few hundred bads, though how large it is depends on the data, the tuning and the regularisation. It is the reason the comparison has to be run on loans neither model has seen.
Quoting the development AUC uplift and multiplying it into a profit figure. Here that route gives 1,949,641, 2.56 times the out-of-time value, and before any cost. The second mistake is comparing at different approval rates: if the trees are allowed to approve more loans, part of the measured gain is volume, not ranking, and it belongs to the credit policy, not the model. Fix the approval rate first. The third is to forget calibration: a boosted tree can rank well and still produce probabilities at the wrong level, which matters the moment the PD feeds pricing or provisions. Check it with the Brier score on the same out-of-time sample.
The card, the unit economics and the cut-off grid are worked in the free workbooks for this case; for the ranking measure behind the deciles, see how to calculate lift and gains.
Usually it ranks somewhat better, but the gain must be measured where the lender decides. On an illustrative book the trees led a scorecard by 4.3 Gini points in development and 1.5 out of time. At a 70 per cent approval rate that meant 9 fewer bads a year, worth 762,903, against an assumed 450,000 of extra cost for validation, monitoring and decline explanations.
Let each model decline the same number of applicants, its own riskiest ones, and count the bads each approves. Multiply the difference by the profit on a good plus the loss on a bad, 84,767 in the illustrative case. At 80 per cent approval the card approved 353 bads out of time and the trees 343, a difference worth 847,670.
Partly because the population changes, which hurts every model, and partly because a flexible model fits features of the development sample that do not recur. In the illustrative case the scorecard's Gini fell from 43.4 to 40.8 per cent and the trees' from 47.7 to 42.3, so only 39 per cent of the development advantage survived at a 70 per cent approval rate.
This article is one calculation from Machine Learning for Finance. The book takes the same case from first principles to the decision, chapter by chapter, and every figure it prints is a live formula in the free companion workbooks.
Get the book on Amazon →Free companion files
Also on Amazon UK · Amazon Germany · Amazon France · Amazon Canada
Reading guide: corporate finance, valuation and markets → · All 1340 articles →
If this book helped, or didn’t, a few lines on Amazon are worth more than they look: they are what the next reader goes on. Write a review. The workbook stays free either way.