How do you test whether a drop in AUC is statistically significant?

The confidence interval around an AUC is set by the number of defaults, not the number of applications, and on a small out-of-time sample it is wide.

Julian R. Sterling · from Machine Learning for Finance

To test an AUC drop between two independent samples, compute each AUC's standard error with the Hanley-McNeil formula, combine them, and divide the drop by the result. A credit model whose AUC falls from 0.7353 on development to 0.7185 out of time, on 487 and 194 defaults, has standard errors of 0.0133 and 0.0214, so the drop of 0.0168 has a z of 0.67 and a p-value of 0.50: noise, not decay. A drop would have to exceed 0.0493 to be significant at 95 per cent.

Worked in full in Machine Learning for Finance by Julian R. Sterling, with every figure reproduced in a free workbook.See the book on Amazon →

The assumptions

An illustrative application scorecard is developed on 6,000 loans and checked on 2,400 later ones, out of time. The counts are illustrative; the AUCs are of the order the book's own card reports, where the ranking barely moves out of time and the level of predicted default moves a great deal.

The two samples
Sample	AUC	Gini	Bads	Goods	Bad rate
Development	0.7353	0.4706	487	5,513	8.12%
Out of time	0.7185	0.4370	194	2,206	8.08%

The Gini has fallen by 0.0336, which is the number a monitoring report usually quotes, because it looks twice as large as the AUC drop. It is the same drop.

Step 1: the Hanley-McNeil standard error

Hanley and McNeil derived the variance of the AUC from the AUC itself and the two class counts, using two auxiliary terms that approximate the probability that two bads both outscore one good, and that one bad outscores two goods.

Q1 = A / (2 − A) Q2 = 2A² / (1 + A)

SE = √{ [ A(1 − A) + (n_B − 1)(Q1 − A²) + (n_G − 1)(Q2 − A²) ] / (n_B × n_G) }

Excel, with the AUC in B1 and the counts in B2 and B3: =SQRT((B1*(1-B1)+(B2-1)*(B1/(2-B1)-B1^2)+(B3-1)*(2*B1^2/(1+B1)-B1^2))/(B2*B3))

Standard error term by term
Sample	Q1	Q2	Numerator	n_B × n_G	SE	95% interval
Development	0.5814	0.6231	474.5804	2,684,831	0.0133	0.7092 to 0.7614
Out of time	0.5607	0.6008	195.2393	427,964	0.0214	0.6766 to 0.7604

The out-of-time interval is 0.0837 wide and contains the development AUC. That alone says the test will not reject.

Step 2: the test on the drop

Development and out-of-time applicants are different people, so the two AUCs are independent and their variances add.

SE of the difference = √(0.0133² + 0.0214²) = 0.0252

z = (0.7353 − 0.7185) / 0.0252 = 0.0168 / 0.0252 = 0.67, two-sided p = 0.50

Smallest significant drop at 95 per cent = 1.96 × 0.0252 = 0.0493, an out-of-time AUC below 0.6860

The ranking has not measurably deteriorated. That matters, because a team that reads the drop as decay will retrain the model, and retraining fixes ranking. If the real problem is the level, a card that under-predicts the bad rate across the bands, the retrained model inherits it. A stable ranking with a moving level is exactly the pattern in what an unchanged score cut-off costs. The level is tested separately, as in how to run a Hosmer-Lemeshow test on a PD model.

The same test in Gini terms

Because Gini is 2 × AUC − 1, every standard error simply doubles: 0.0266 on development, 0.0427 out of time, and 0.0503 for the difference. The z-statistic is unchanged at 0.67. A report that quotes a Gini fall of 0.0336 without its standard error of 0.0503 is quoting a number smaller than its own uncertainty, and a reader cannot tell. Put the interval in the report, and the conversation about retraining starts from the right place.

What if the out-of-time sample were bigger?

Same bad rate, development sample fixed
OOT bads	OOT goods	SE, OOT	SE of drop	z for 0.0168	Smallest significant drop
100	1,137	0.0298	0.0326	0.52	0.0639
194	2,206	0.0214	0.0252	0.67	0.0493
500	5,686	0.0133	0.0188	0.89	0.0369
1,000	11,371	0.0094	0.0163	1.03	0.0319
2,000	22,742	0.0067	0.0149	1.13	0.0291

The curve flattens fast. Even with 2,000 out-of-time defaults the development sample's own error of 0.0133 sets a floor: to make a drop of 0.0168 significant, the combined error would have to be 0.0086, below the development error alone. No out-of-time sample can get there. A smaller lender's monitoring report will rarely see a significant AUC change at all, which is a reason to monitor something faster, not a reason to stop looking.

The common mistakes

Counting applications instead of defaults. Using a binomial error on 6,000 and 2,400 applications gives standard errors of 0.0057 and 0.0092 and a z of 1.56, which reads as nearly significant. The precision of an AUC is set by the scarce class. Out of time, 95.5 per cent of the variance comes from the (n_G − 1) term, which after division by n_B × n_G is roughly (Q2 − A²) / n_B: it shrinks with the number of bads, which is why 194 defaults, not 2,400 loans, decide the answer.
Testing two models on the same sample as if independent. A challenger model and the champion scored on the same applicants have correlated AUCs. The paired DeLong test accounts for that; adding the two variances overstates the error and hides real improvements.
Treating no significant drop as proof of stability. The test above can only fail to reject. With this sample, a true drop of 0.04 would go undetected 64 per cent of the time.

Hanley-McNeil approximates the AUC variance assuming a particular score distribution. DeLong's nonparametric variance is the standard in validation software; on samples of this size the two are usually close. The arithmetic of the AUC and Gini themselves is in how to calculate Gini and KS for a scorecard.

Takeaway

SE(AUC) from Hanley-McNeil; SE of a drop between independent samples = √(SE₁² + SE₂²).
Here a drop of 0.0168 has a z of 0.67; it would need to exceed 0.0493.
Count defaults, not applications. With 194 out-of-time bads, the interval is 0.0837 wide.

The ten development bands, the area under the curve by ranks and the out-of-time comparison are live in the free workbooks for this book.

Questions readers ask

How do you calculate a confidence interval for AUC?

Compute the Hanley-McNeil standard error from the AUC and the counts of bads and goods, then take the AUC plus or minus 1.96 standard errors. For an AUC of 0.7185 on 194 bads and 2,206 goods the standard error is 0.0214 and the 95 per cent interval runs from 0.6766 to 0.7604, a width of 0.0837.

Should I use DeLong or Hanley-McNeil to compare AUCs?

Use Hanley-McNeil, or DeLong's variance for each sample, when the two AUCs come from different applicants, such as development and out of time. When two models are scored on the same applicants, their AUCs are correlated and the paired DeLong test is the right one; treating them as independent overstates the standard error of the difference and hides real gains.

How many defaults do you need to detect an AUC drop?

More than most validation samples hold. With the development standard error of 0.0133 fixed, an out-of-time sample of 1,000 bads still needs a drop of 0.0319 to be significant at 95 per cent, and 2,000 bads need 0.0291. An observed drop of 0.0168 cannot become significant at any out-of-time size, because the development sample's own uncertainty already exceeds the 0.0086 allowed.

Read the whole case

This article is one calculation from Machine Learning for Finance. The book takes the same case from first principles to the decision, chapter by chapter, and every figure it prints is a live formula in the free companion workbooks.

Get the book on Amazon →Free companion files

Also on Amazon UK · Amazon Germany · Amazon France · Amazon Canada

Also on this site

How do you calculate Gini and KS for a credit scorecard?Gini = 2 x AUC minus 1, with tied bads counted half; KS is the widest gap in cumulative shares. Ten score bands give 45.61% and 32.47%, worked step by step.
How do you run a Hosmer-Lemeshow test on a PD model?Sum (observed minus expected defaults) squared over N x PD x (1 minus PD) in ten groups: 10.98, p of 0.2028. It passes a model that under-predicts by 10.7%.
How do you calculate the population stability index (PSI)?Sum (new share minus old share) x ln(new / old) over fixed score bands: 0.090517 for a card whose bad rate doubled. Under 0.10 is not proof of stability.
How do you calculate scorecard points from PDO, factor and offset?Factor = PDO / ln 2, offset = anchor score minus factor x ln(odds): 20 points to double the odds and 600 at 50:1 give 28.8539 and 487.1229.
How do you calculate weight of evidence and information value?WoE = ln(share of goods / share of bads) per bin; IV sums (share difference x WoE). Five debt service cover bins give an IV of 0.2002, worked from counts.
How do you recalibrate a PD model to a new default rate?Shift every applicant's log-odds by one constant, solved so mean PD hits the new rate: 7.90% to 12.625% takes 0.5623, or 16.22 scorecard points.

Reading guide: corporate finance, valuation and markets → · All 1340 articles →

If this book helped, or didn’t, a few lines on Amazon are worth more than they look: they are what the next reader goes on. Write a review. The workbook stays free either way.