Chapter 24: The Sprint: setting your confidence

A two-week call is nearly a coin flip, and a hundred of them at a stated confidence are not: make one, see the evidence, lock it, and learn from how your calls settle at each confidence.

By Matt Lang and Bree Lang · About the authors

Where this is on Sifter: Training Floor · Exercises · Sprint

Why a two-week call

Imagine that you sit down at the Sprint page with one idea in your head. You think Microsoft is going to beat the S&P 500 over the next two weeks. Before you type anything, take a moment to think about how much that idea is worth on its own. Over the last fifteen years MSFT has beaten the index in a little more than half of all two-week stretches, so a stranger who said "beat" every time would have been right a little more than half the time. A two-week call on a single stock is close to a coin flip.

The Training Floor opens with this exercise because a hundred such calls can show something one call cannot. If you say 70 percent on a hundred calls and about seventy of them land, you have demonstrated something that no essay, no interview and no machine can produce on demand, which is that you know how sure to be.

Notice what that standard asks of you, because it is the opposite of what school asked. At 70 percent you are supposed to be wrong thirty times in a hundred. If you are wrong ten times your 70 was really a 90, and if you are wrong forty times it was a 60. Being wrong is part of the job, and the only thing the record holds against you is being wrong at a rate that does not match the number you said. A recruiter reading your Sifter Card knows that a run of right answers over two-week windows is mostly luck. What the card shows is whether your number and your results have been converging, one dated call at a time.

The page

Let's look at the page, because it is a short one. At the top is a sentence with a blank in it, "Will MSFT beat the S&P 500 over the next two weeks?", where the blank is a search box and beat and lag are two buttons next to it. Under the company's name are its price and its own base rate, "Beats the S&P 54% of the time". Then comes the main question, "How sure are you?", with a slider, a large number, and a line under the slider that restates the number in plain words, so that at 70 it says "About 7 of 10 calls like this should work out." (The slider opens at 60, which is a placeholder rather than a suggestion, and the number only becomes yours when you move it.)

Below the slider are two boxes that each take one line, "Why?" and "What would change your mind?", and below those are two doors of the same size, "See the evidence" and "Lock in your call", with two lines under the lock that say when your entry price is set and when the call settles. The horizon is fixed at two weeks, so the page asks four things of you, the side, the number, the reason and the way you could be wrong, and every entry on the record is made of those four.

The simplest call

Now let's make the simplest call there is and watch what happens to it. MSFT beats the market 54 percent of the time, so the uninformed call is "beat" at 54, and a student who knows nothing else about Microsoft should type 54. Imagine that you make that call ten times over a semester, on ten different Mondays, and seven of the ten land.

Many of you will read that as a good result, and on the stock it was, but on you it was not. You said 54 ten times and delivered 70, which means that your number was wrong even though your calls were right, and the record now knows something about you that you did not, which is that when you say 54 you seem to mean 70.

So what went wrong? Two things. You never looked at anything before typing the number, so the number was the base rate rather than a claim of your own. And ten is a very small number. Seven of ten from a process that truly lands 54 percent of the time happens about one time in four, so after ten calls you cannot tell skill from a lucky semester. Psychologists call the instinct to trust a small sample as if it were a large one the belief in the law of small numbers, and the Sprint is built to defend against it.¹ The cure for the first problem is the evidence door and the cure for the second is volume, and the chapter takes them in that order.

Look before you type

So let's look before we type. Press "See the evidence" and three cards open across the page under the call, one sentence each with a few figures under it. Each can nudge your number and none can move the slider for you. Open on any card shows it at nearly full screen with the picture behind its numbers, and the arrows walk you through the three.

The first card is about size and the calendar. "In two weeks MSFT usually moves about ±1.8% vs the S&P. No earnings in the window." That 1.8 is the median two-week move against the index across fifteen years. It does not say which way the stock will go, but it says how much room a call has to be right or wrong by, and whether one earnings morning inside your window will decide the whole thing. (A report inside the two weeks usually moves the stock more than the other nine days combined, so the calendar is the first thing to check.)

The second card is the stock's own history on days like today. "MSFT is 7.8% below its 52-week high. On 264 similar days, MSFT beat the S&P over the next two weeks 137 times (52%)." The page went back through fifteen years, found every day when the stock sat about this far below its high, and counted what the next two weeks did. Compare 52 with the base rate of 54. A rate measured on 264 days has an error bar of about six points either way, so the fair conclusion is "nothing unusual is happening" rather than "lag."

The third card widens the lens from one stock to all of them. "All S&P 500 stocks as close to their high as MSFT is: 49% beat the index over the next two weeks (72,411 cases)." That is a different and much larger population, and it says that stocks this close to a high have, as a group, done a little worse than a coin.

Behind the third card is the market. Open it and under the all-stocks tile are what the S&P did over the last two weeks, how far apart the index's members have been moving, and the index's own choppiness, with three more cells behind "More on the market," the share of the index in uptrends, credit stress and the 10-year rate. The market is here rather than in the workbench because it is a filter rather than a fact about Microsoft. The picker at the foot, "All stocks, periods like today," recomputes every all-stocks rate on the history that matched the current regime, and every count drops visibly as it does, which is the real cost of asking a more specific question. The market cells never tell you what to do. They narrow the history the all-stocks rates are computed on, and they show you how much history is left.

Put the three cards together and the evidence is pulling the number down slightly, from 54 toward 52 or so, and it is certainly not pulling it toward 70. The advice from the people who first studied prediction is to start at the base rate and adjust for what you know that the tape does not, and the worked example below does just that.²

When the evidence disagrees

Now imagine that the cards had come back differently, because the exercise is really about what you do when they disagree. Same stock, a different Monday. Start at 54 again, and move only where a card gives you a reason.

Both moves have a reason a reader can check, and those reasons are what go in the two boxes. The table below fills them in for that call.

The two boxes, filled in for that call

One line each. The lines are already on the record in the snapshot, so the first box is not the place to repeat them. It is where you write what you know or believe about the company that the tape does not, in language specific enough that the second box can say what would prove it wrong. A reader can check every clause of both lines, which is what separates this call from typing 70 because Microsoft seems like a good company.

What would change your mind

Most students have never been asked to write the second box. Investing is full of sentences that cannot fail. "The market has not yet appreciated the story" survives every outcome, which is a large part of why people keep writing it. A falsifier is the opposite kind of sentence. It states one observation, before the outcome is known, that would tell you your reasoning was wrong rather than just your luck.

The page never reads the sentence and a Sprint settles by the tape whatever you wrote, so a weak falsifier does not change your score. It does mean that when the call misses you will not be able to tell why, and a reader of your record will see which kind of sentence you wrote.

How strong the falsifier needs to be depends on the number above it. At 50 you are claiming nothing, so there is nothing for a falsifier to catch. At 58 you are banking on something specific, which in the earnings scenario above was Azure re-accelerating at the report, so the box should state that thing and the number that would break it. At 70 you are banking on a lot, sixteen points above what the tape would say, and the box has to say what you are banking on. In other words, the further your number is from the base rate, the more precise the second box has to be.

Let's put the three tests to work on someone else's sentences before you write your own. This unit draws its examples from the Maple call book, a fictional record of twenty settled calls by four analysts at a student fund. Six of its falsifiers are below, three that could fire and three that never could. Decide which is which, then answer the question under the table.

Six falsifiers from the call book, three of which can never fire

In most cases the verb is what gives a falsifier away, because verbs such as "trade", "print", "file" and "publish" describe events that either happen or do not happen, while verbs such as "appreciate", "deteriorate" and "prove incorrect" describe how the analyst will feel about whatever happened.

Four of those six appear in the question below. Pick the one that could never fire.

Lock the call

Now let's lock the call and let the two weeks run. When you press "Lock in your call", the sentence, the slider and the two lines freeze together, and there is no edit button to go looking for later (you get ten minutes to void a mistake, and after that the call stands). The evidence freezes with it, and the record also keeps the number the slider was on the first time you pressed "See the evidence" next to the number you locked, so a call that started at 70 and finished at 58 records both.

The lock also fixes the price the call starts from, and the line under the door says which price that is. If you lock during the trading day, between 9:30 in the morning and 3:50 in the afternoon New York time, the entry price is that day's 4:00 close and the call settles at the close two weeks later. If you lock at any other time, in the evening, before the open or on a weekend, the entry price is the next session's 9:30 open and the call settles at the open two weeks later. The reason for the rule is that a company can report after the close and move eight percent before the next morning, which means that a call locked at five in the evening with the report already out would otherwise start from the old price. The cutoff is 3:50 rather than 4:00 because of the ten-minute void window, since a call locked at 3:58 could otherwise be kept or voided at 4:07 with the after-hours news in hand.

Two weeks later, at the same mark the entry used, the platform settles the call on its own. If you said beat and the stock finished ahead of the S&P over the window, or said lag and it finished behind, the call is right, and otherwise it is wrong. The size of the move does not enter the verdict. On the right of the page, under "Your sprints," the call moves from Open to Right or Wrong, and pressing the row shows the number the page gave that morning next to the number you gave.

If there is more behind the call than one line can hold, "+ add a deep-dive write-up" under the doors lets you attach a Workspace document or a PDF of up to 10 MB. It can be added or swapped for 24 hours after the lock and then freezes with the call, and it shows as a link on the call's row and under the call on your Sifter Card. The page does not weigh it and does not require it.

What to do after it settles

So the call has settled. "Was I right?" is a question about one call, and one call is mostly noise. The question the record answers is "when I say 58, what happens?", and it takes a hundred calls to answer, which is why nothing reaches your card until twelve have settled.

The page keeps the answer in two places. Under the slider, once three settled calls sit within five points of the number you are setting, a clause reads "Your calls near 70%: 8 of 10 right." And the panel under your history, "Outcomes by confidence," groups every settled call by the confidence you claimed and draws a bar for how often those calls came good against a shaded block for what you claimed. Every bar inside its block is a calibrated record. A bar that stops short means you were surer than you had a right to be at that number, and a bar that runs past it means you were underselling yourself.

The bar for a confidence tells you more than any one call does. If your 70s land six times in ten, do not just say 60 next time. Ask what those calls had in common, such as a report inside the window, a stock near its high, or a story you liked, because whatever they share is what the number left out. When a call settles the page also asks for one line and a label, right for your reason, right for another reason, or wrong, and the middle one is the most useful to a reader. Two rules follow from the arithmetic. Never say 50, because a 50 belongs to no side and the record learns nothing from it, and never say a number below 50, because that means you think the other side is more likely, so flip the side instead.

The check below puts a number on what a calibrated record looks like.

All of this reaches your Sifter Card as one tile with three numbers, the calls settled, your average confidence, and the share of calls that came true, while your calibration score sits at the top of the card with a word next to it, Overconfident, Coin flip, Calibrated or Sharp. Only the score is a grade, and Chapter 29 does its arithmetic. What a reader sees over a year is not a list of right answers, which two-week windows cannot provide, but a set of bars moving into their ranges and a score firming up as the misses arrive at the rate you said they would.

That is the exercise, and three cards are enough for most calls. Chapter 25 asks whether you can do better than three cards, and what looking harder costs.

Sources

1. Tversky, A., and Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76(2), 105-110 https://doi.org/10.1037/h0031322
2. Kahneman, D., and Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237-251 https://doi.org/10.1037/h0034747

The Analyst's Handbook is Sifter's textbook for student investors, written by Matt Lang and Bree Lang. Sifter is free for students. This chapter as markdown.