Chapter 29: Calibration

Brier, base rates, and the arithmetic of being right at the rate you claimed.

By Matt Lang and Bree Lang · About the authors

Where this is on Sifter: Training Floor · Sifter Card · The Sprint

A tally of right and wrong answers cannot grade a probability. If you say 70 and the stock beats the market, you were not right, you were 70 percent right, and if it lags you were 30 percent right, because that is what you said. Over one call the distinction hardly matters, but over a hundred calls it is most of the record, since a person who says 70 and lands 70 percent of the time knows something about their own judgment that a person who says 70 and lands 50 percent of the time does not, even if the second person happens to have more right answers.

The Sprint chapter showed you the score the card uses and ran two analysts through it. This chapter covers the three things that chapter left for later, which are why the score is built the way it is, what it is compared with, and how it can be split into the two skills a probability can show.

The Brier score

The score is the squared distance between the number you said, as a fraction, and what happened, written as 1 for a call that came good and 0 for one that did not. Say 0.70 and land, and the call costs 0.09. Say 0.70 and miss, and it costs 0.49. Say 0.50 and it costs 0.25 either way, which is what a coin flip scores on every call.

The score has one property that matters more than the rest. The number that gives you the best expected score is the number you actually believe. If you think a call has a 60 percent chance and you write 80 to look bold, your expected score gets worse, and if you write 55 to look modest it also gets worse.⁴ A scoring rule with that property is called proper. Because the card uses one, it can print a score without inviting you to game the slider, since the only way to do well is to know how sure you are and say so.

Let's see what the score makes of the four analysts from the Maple call book, five calls each.

Four analysts, five calls each

Analyst	Landed	Average confidence	Average Brier
Dara Whitlock	4 of 5	65%	0.188
Ivo Prentice	3 of 5	50%	0.250
Nell Ashworth	4 of 5	89%	0.173
Casper Boyd	2 of 5	91%	0.509

Lower is better, and 0.25 is the coin. Prentice says fifty to everything and scores exactly 0.25, whatever happens. Boyd says ninety to everything and pays for it. Ashworth has the best score on the table, and the next section is about why that number flatters her.

Skill against a reference

A Brier score on its own does not tell a reader much, because an easy set of calls produces a good score for anyone. So the card reports skill, which is the share of a reference forecaster's error you removed.³ Take your average Brier, divide by the reference's average, and subtract from one. Zero means you did no better than the reference, one means every call landed at the number you said, and a negative number means the reference would have beaten you.

The card's reference is the coin, an average of 0.25, because a two-week call on beat or lag is close to even for nearly every stock in the index, which you saw on the Sprint page when the base rate under a stock showed 54 rather than 70 or 30, so the coin is a fair stand-in for "knew nothing about this stock." It is not a fair stand-in for every question, and the Maple book shows what goes wrong when the reference is too easy.

Ashworth only calls things that were nearly certain to begin with. Against the coin she looks skilled, because the coin does not know her calls were easy. Score her against the base rate of each of her own calls, the rate a stranger would have known before she said anything, and the skill goes away.

The same four analysts, scored two ways

Analyst	Skill against the coin	Skill against each call's base rate	Shown after shrinking, n / (n + 12)
Dara Whitlock	+0.25	+0.25	+0.07
Ivo Prentice	0.00	0.00	0.00
Nell Ashworth	+0.31	-0.01	0.00
Casper Boyd	-1.03	-1.03	-0.30

The second column is the card's arithmetic. The third is what a harder reference does to it, and the analyst it changes most is Ashworth. Whitlock survives both, because the questions she answered were open ones. The card protects itself in two ways against that problem. The Sprint's base rates stay close to fifty, so the coin is rarely far from the truth, and every call is printed with its date and its evidence, so a reader can see what was being called.

The formula below is the card's arithmetic.

Brier and skill

$B = \text{mean}\,(p - o)^2, \qquad \text{skill} = 1 - \frac{B}{0.25}$

p is the stated confidence as a fraction and o is 1 for a call that came good and 0 for one that did not. 0.25 is the coin's average, and the card multiplies the skill by n / (n + 12) before printing it, as Chapter 23 explained.

Reliability and resolution

A probability can show two different skills, and the Brier score adds them together. The first is reliability: when you say 70, do about 70 percent of those calls land? The second is resolution: do the numbers you say separate the calls that land from the calls that do not, or do you say much the same thing every time? In 1973 the meteorologist Allan Murphy showed that the Brier score splits exactly into those two pieces plus a third that is nobody's fault.²

The third piece is the uncertainty in the questions themselves, the base rate of landing times one minus itself, and it is the same for every analyst who calls the same events. Reliability is a penalty, so smaller is better. Resolution is a credit, so larger is better. Let's run the split on the whole Maple book, all 20 calls grouped by the number that was stated.

The Murphy decomposition

$B = \underbrace{\tfrac{1}{N}\sum_k n_k (p_k - \bar o_k)^2}_{\text{reliability}} - \underbrace{\tfrac{1}{N}\sum_k n_k (\bar o_k - \bar o)^2}_{\text{resolution}} + \underbrace{\bar o (1 - \bar o)}_{\text{uncertainty}}$

The calls are grouped by the number stated. For group k, n_k is how many calls said that number, p_k is the number, and ō_k is the share of them that landed. ō is the share of all calls that landed.

The table below runs the split on the Maple book, one row per stated number, and the note under it adds the columns up.

The 20 Maple calls, grouped by the number stated

Stated	Calls	Share that landed	Reliability term, n(p − ō_k)²	Resolution term, n(ō_k − ō)²
50%	5	60%	0.0500	0.0125
55%	1	100%	0.2025	0.1225
60%	1	100%	0.1600	0.1225
65%	1	0%	0.4225	0.4225
70%	1	100%	0.0900	0.1225
75%	1	100%	0.0625	0.1225
85%	1	100%	0.0225	0.1225
90%	8	63%	0.6050	0.0050
95%	1	0%	0.9025	0.4225

Across all 20 calls, 65% landed, so ō = 0.65. Summing each column and dividing by 20 gives reliability 0.1259 and resolution 0.0738; uncertainty is 0.65 × 0.35 = 0.2275. Put together, 0.1259 − 0.0738 + 0.2275 = 0.2796, which is the book's average Brier score of 0.2796 to the last digit.

Now look at what the card draws. The bands on the exercise page, one bar per band of stated confidence with the claimed number as a mark, are the reliability term made visible. A bar that stops short of its mark adds to the reliability term, which is a penalty. Resolution has no picture of its own on the card, but it is the difference between a record that says 55 to everything and one that says 55 when it means 55 and 80 when it means 80, and the word "Sharp" is the closest the card comes to a label for it, because a skill of 0.15 with the error bar clear of zero is hard to reach without some resolution.

That is also why the page will not let you say 50, and why a slider left at 60 on every call is a habit to break. A record with no spread in its numbers can be perfectly reliable and still tell a reader nothing, and the next question shows how little.

Check yourself

An analyst says 55% on every call, and over a long record 55% of the calls land. What does the card's arithmetic make of that record?

Perfect reliability, no resolution, and a skill against the coin of about 0.01. (correct) Right. Every group is one group, the 55% calls, and it landed 55% of the time, so the reliability penalty is zero. But there is only one group, so its landing rate equals the overall rate and the resolution credit is zero too. The Brier score is 0.55 × 0.45² + 0.45 × 0.55² = 0.2475, barely under the coin's 0.25, and the skill is 1 − 0.2475 / 0.25 = 0.01. The record is calibrated in the narrow sense and of almost no use to a reader.
Overconfident, because the analyst never expressed doubt. Overconfidence means the claimed number runs ahead of the landing rate, and here they match exactly. The record's problem is the opposite of overconfidence: the number never moves, so it never says which calls the analyst actually believed in.
Sharp, because the record is perfectly calibrated. Sharp on the card needs a skill of 0.15 with the error bar clear of zero, and this record's skill is about 0.01. Calibration in the sense of reliability is necessary for a good score, and it is nowhere near sufficient.

Small samples

Two things stand between the arithmetic and the number on your card, and both exist because a few calls prove very little. The first you know from Chapter 23. The skill is multiplied by n / (n + 12), so twelve settled calls show half of the raw skill and it takes 48 to show four fifths of it.

The second is an error bar. The card takes the spread of your individual Brier scores around their average, works out the standard error of that average, and converts it to the skill scale by dividing by 0.25. Plus or minus 1.96 of those is the interval it keeps next to the number. The interval decides the word. A skill of 0.15 or better shows Sharp only when the whole interval is above zero, so a lucky run of six calls cannot earn the word however good the point estimate looks. Until 12 calls have settled, and while more than a quarter of your calls are still open, the tile says "building" and the interval is not shown at all.

Practice

The card shows n / (n + 12) of your raw skill. How many settled calls do you need before it shows 80 percent of it?

Solution. n / (n + 12) = 0.8 means n = 0.8n + 9.6, so 0.2n = 9.6 and n = 48. At one Sprint a week that is most of a year, so a calibration number is slow to earn and hard to fake.

So the number on the card is the coin's error you removed, shrunk for the size of your sample, with a word chosen from the interval around it. It rewards saying what you believe, it punishes being sure and wrong far more than being unsure and wrong, and it cannot be improved by picking easy questions on this page, because the questions are close to even by construction. What it cannot tell a reader is whether you can run money, which is the Book's job, or whether you can explain a call, which is the job of the two lines and the post-mortem under it. Chapter 30 puts those pieces together as a reader sees them.

Sources

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1-3 https://journals.ametsoc.org/downloadpdf/view/journals/mwre/78/1/1520-0493_1950_078_0001_vofeit_2_0_co_2.pdf
Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12(4), 595-600 https://doi.org/10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2
Murphy, A. H. (1988). Skill scores based on the mean square error and their relationships to the correlation coefficient. Monthly Weather Review, 116(12), 2417-2424 https://doi.org/10.1175/1520-0493(1988)116<2417:SSBOTM>2.0.CO;2
Gneiting, T., and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378 https://doi.org/10.1198/016214506000001437

The Analyst's Handbook is Sifter's textbook for student investors, written by Matt Lang and Bree Lang. Sifter is free for students. This chapter as markdown.