Chapter 23: What the record is for

Why a transcript stopped separating people, what a record of frozen claims does instead, and how much of this is proven.

By Matt Lang and Bree Lang · About the authors

Where this is on Sifter: Training Floor · Sifter Card

Let's start with an experiment. In a study published in 2024, four British researchers wrote a batch of exam submissions using GPT-4, made no edits to what the machine produced, and quietly slipped the submissions into the live examination system of a real psychology degree. The fake work was spread across five modules and every year of the program, amounting to about one submission in twenty, and nobody warned the markers, who graded the way they grade every year. Before reading any further, take a moment to guess how much of the fake work got caught.

Many of you probably assumed that the markers would catch most of it, but ninety-four percent of the machine's submissions went through undetected, which means that nearly all of the fake work was read, graded, and handed back as if a student had written it. The machine was not scraping by at the bottom of the class, either. On average, the fake submissions scored just over half a classification boundary (roughly half a letter grade, in American terms) above the real students they were hiding among, and only sixteen percent of the humans beat the machine's median grade in their own module.¹ Take a moment to think about that last figure. In a typical module, the large majority of real students turned in work that graded below the middle of the machine's pile.

Some of you may be skeptical of this study, and that instinct is a good one, because it is one experiment at one university, in one subject, in one year. A natural response is to hope that detection software will solve the problem, so let's look at how the detectors performed when researchers put them to the test. Fourteen AI-text detectors were run through 756 tests, and not one of the fourteen reached 80 percent accuracy. When the machine text was run through a paraphrasing tool first, accuracy fell to 26 percent.² This means that a student with a free paraphrasing tool faces detectors that are wrong far more often than they are right. Now imagine that you are an employer holding a transcript. Your problem is not figuring out whether one particular student cheated, because you have no way to investigate that. Your problem is deciding whether a grade can still tell you anything about the students who did nothing wrong at all, and this unit begins there.

However, the transcript had problems long before the machines arrived. So how well do grades actually predict job performance? Back in 1996, a meta-analysis pulled together 71 studies covering nearly 14,000 people and measured the observed correlation between grades and job performance. The answer was 0.16, and the relationship decays with time since graduation, falling to roughly 0.05 after six years.³ Take a moment to think about what a correlation of 0.16 means. The relationship is weak enough that knowing a student's GPA tells an employer very little about how that student will perform on the job, and by six years out, the link has faded to close to no relationship at all.

At the same time, the grades themselves have been drifting upward. At Harvard College, A grades went from 24 percent of all grades in 2005 to 60 percent in 2025.¹⁰ Think about what that does to the meaning of an A. When six students in ten receive the top mark, the mark can no longer help a reader tell those students apart, no matter how hard each of them worked to earn it. Employers appear to have figured this out on their own, because in a 2023 survey of 1,010 employers, 36 percent rated transcripts very useful, essentially unchanged from 37 percent in 2018 and 38 percent in 2020.¹¹ In other words, employers had already cooled on the transcript before AI arrived, and the exam study we opened with sped up a slide that was well underway.

So what did employers replace the transcript with? Very little. Firms made headlines by striking degree requirements from their job postings, and at the 11,300 roles studied that actually dropped one, the share of hires without a degree rose about three and a half percentage points. However, so few postings dropped the requirement that the total effect across hiring worked out to 0.14 percentage points, which is fewer than one hire in seven hundred.⁴ Take a moment to think about what that means. The old signal was removed, no new signal was installed, and the space in between is currently filled by interviews, internship pipelines, and a fair amount of guessing. The rest of this chapter describes what we believe should fill that space instead.

The Training Floor was built to fill that space. The idea is that you cannot demonstrate judgment by describing it, because describing judgment is free, and a machine will now do it at three in the morning for anyone who asks. What you can do instead is put your name and the date on claims about things that have not happened yet, and then let the world decide how you did. The only way to fake a document like that is to make claims that keep coming true, and anyone who can do that regularly has the skill the document is meant to show.

A pile of these settled claims is called your record. There is more to that definition than it looks, and it comes down to three properties. The table below walks through what each one means and what each one makes impossible.

Three properties, and what each one rules out

Notice that none of the three properties is a claim about you. Each one is a property of a document, and the rest of the chapter explains why that distinction matters.

Before we get to the evidence, we need to get used to the strangest feature of the arrangement, which is that the record expects you to be wrong, on a schedule you announce in advance. When you mark a call (the Floor's word for one of these claims, and the next chapter takes one apart) at 70 percent, you are not promising that the event will happen, but rather promising that calls like this one, made by you, come true about seven times out of ten. That is a different promise from the one school trained you to make. This means that if three of your ten seventies miss, nothing has gone wrong, and the system is working just as it was designed to work.

At this point, many of you may be thinking, "so being wrong doesn't cost me anything?" Not quite. What the record punishes is being wrong about yourself. Say 70 when your true rate is 50, and over enough calls the record will show a chronic overconfidence that no interview would ever surface. Say 70 when your true rate is 90, and you are hiding your own skill behind false modesty, which is its own kind of inaccuracy. The job, over dozens of calls, is to build a system in your head that can tell a 60 from a 70 from an 80, and the record is how you find out whether that system works. Being right seven times out of ten, having said 70, may feel like a B minus, but here it is a perfect score!

At this point, you should be asking whether any of this is supported by evidence, or whether it is just a nice story told by a company that would like you to use its product. That is a fair question, and we would much rather you asked it here than in a job interview. So let's take the answer one piece at a time.

The first part is logic, and it needs no study at all. Suppose you write down today, with the date attached, that a stock you follow will beat the market this quarter. When the quarter ends, anyone can pull up the note and compare it with what happened. Now suppose you skip the note, wait for the quarter to end, and then explain what you believed all along. Nobody can check the second story, including you, which is why the call has to be frozen before the outcome arrives. (There is a research literature on hindsight bias, meta-analysed across 122 studies, and it finds a real effect of modest size,¹⁵ but the case here does not depend on your memory being terrible, only on the fact that nobody can audit a claim that was never written down and dated.)

The second part is the well-tested one. Knowing how often you are right is a skill, and it sticks to people. The evidence traces mostly to the psychologist Philip Tetlock, who spent two decades collecting dated predictions from professional forecasters and then went back and marked them, which nobody in that industry does. The average expert barely beat crude baselines, and confidence and accuracy often moved in opposite directions.⁵ From there, Tetlock and Barbara Mellers entered a four-year forecasting tournament run for the American intelligence community, and this time the question was different. Instead of asking whether the experts are good at forecasting, the tournament asked whether anyone is, and whether the skill lasts.⁶ The answer to both questions was yes, and the tournament showed it in steps. At the end of year one, the sixty best forecasters were pulled into elite teams. If that first year had been luck, the second year should have dragged them back toward average, the way a hot streak cools, but they held their accuracy through the next two years, across hundreds of questions.⁷ Past accuracy predicted accuracy on brand-new questions at correlations around 0.4 to 0.55, and the relationship strengthened as the pile of resolved questions grew.⁸ Lastly, the habit that best predicted accuracy was not confidence or credentials but updating beliefs in small, evidence-linked steps, which in one study of 515 forecasters across 481 questions separated a mean Brier score of 0.22 from 0.29 (Brier scores work like golf scores, so lower is better).⁹ Notice what all of this means for you. The forecasters at the top of the tournament were not right every time, and they did not need to be. What set them apart was that they knew, with unusual precision, how often they were right.

The third part is the limit of what we can claim, and you should hold us to it. Nobody has yet run the study linking a record like this one to job performance, ours included, and until that study exists we will not claim the link. Having said that, a missing study does not mean the idea rests on nothing, because quite a bit of evidence is already in place. First, a call is not a proxy for the job but a small piece of the job itself, and in the hiring literature, work samples of real job tasks are among the better-validated selection methods, alongside structured interviews.¹² Second, the skill being measured is genuinely scarce, including at the top of the profession. When researchers collected more than 13,300 probability distributions from chief financial officers over ten years, realized market returns landed inside the executives' 80 percent confidence intervals 36 percent of the time.¹³ In other words, ranges that were built to contain the outcome eight times in ten contained it barely more than three times in ten. Lastly, in the one corner of the profession that already keeps score, accuracy has career consequences, because securities analysts who forecast well tend to move up the brokerage hierarchy and analysts who forecast poorly tend to move down.¹⁴ So when you build a record, you are making a bet, and we think it is a good one, that directly measuring a scarce, job-shaped, career-relevant skill will tell a reader more than a transcript can. The controlled study that settles the bet is the one we are running, and the record you build is part of the evidence that will settle it.

Let's spend a minute on the arithmetic behind that last answer, because it is the part students most want to skip. Suppose you make five calls and every one of them resolves your way. Are you well calibrated, or just lucky? With only five observations there is no way to tell the difference, and Sifter refuses to guess. This means that every calibration score is deliberately dragged toward "no skill demonstrated" until you have resolved enough calls to earn your way out, and the weight doing the dragging is

You make claims early, in a format that lets them fail, and a predictable fraction of them will fail. Those misses are what make the rest of the record believable, because a record with no misses in it tells a reader that somebody has been deleting the misses. In exchange you get a document about your thinking that a stranger has no reason to disbelieve, which is something a transcript can no longer give you.

Now that we know what the record is for, the rest of the unit builds it from the bottom up, one instrument at a time. Chapter 24 starts with the Sprint, the shortest exercise on the Floor, and uses it to take a single call apart, to spend some time on the hardest line in it (what would change your mind, a question most students discover nobody has ever asked them), and to read the evidence the page keeps under the call. Chapter 26 moves to the Spread and the arithmetic of a hedge, Chapter 27 to the Quarter, where the claim being examined is the risk of a book you built, and Chapter 28 to the Book, the long record. Chapter 29 turns your stated probabilities into a score, using arithmetic that never requires you to be right every time. And Chapter 30 asks what a reader may fairly conclude from the whole thing.

Sources

1. Scarfe, P., Watcham, K., Clarke, A., and Roesch, E. (2024). A real-world test of artificial intelligence infiltration of a university examinations system: A "Turing Test" case study. PLOS ONE, 19(6), e0305354 https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0305354
2. Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltynek, T., Guerrero-Dib, J., Popoola, O., Sigut, P., and Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19, 26 https://eprints.whiterose.ac.uk/id/eprint/207396/1/s40979-023-00146-z.pdf
3. Roth, P. L., BeVier, C. A., Switzer, F. S., and Schippmann, J. S. (1996). Meta-analyzing the relationship between grades and job performance. Journal of Applied Psychology, 81(5), 548-556 https://houdekpetr.cz/!data/public_html/papers/Roth%20et%20al%201996.pdf
4. Sigelman, M., Fuller, J., and Martin, A. (2024). Skills-Based Hiring: The Long Road from Pronouncements to Practice. Burning Glass Institute and Harvard Business School https://static1.squarespace.com/static/6197797102be715f55c0e0a1/t/65cc355c4935cb001349a4cd/1707881822922/Skills-Based+Hiring+02122024+vF.pdf
5. Tetlock, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know?. Princeton University Press https://press.princeton.edu/books/paperback/9780691175973/expert-political-judgment
6. Tetlock, P. E., and Gardner, D. (2015). Superforecasting: The Art and Science of Prediction. Crown https://www.penguinrandomhouse.com/books/227815/superforecasting-by-philip-e-tetlock-and-dan-gardner/
7. Mellers, B., et al. (2015). Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspectives on Psychological Science, 10(3), 267-281 https://faculty.wharton.upenn.edu/wp-content/uploads/2015/07/2015---superforecasters.pdf
8. Himmelstein, M., Atanasov, P., and Budescu, D. V. (2021). Forecasting forecaster accuracy: Contributions of past performance and individual differences. Judgment and Decision Making, 16(2), 323-362 https://www.cambridge.org/core/journals/judgment-and-decision-making/article/forecasting-forecaster-accuracy-contributions-of-past-performance-and-individual-differences/914F973B8C6A9CDCD3F1EA7CF17048D2
9. Atanasov, P., Witkowski, J., Ungar, L., Mellers, B., and Tetlock, P. (2020). Small steps to accuracy: Incremental belief updaters are better forecasters. Organizational Behavior and Human Decision Processes, 160, 19-35 https://faculty.wharton.upenn.edu/wp-content/uploads/2022/03/1-s2.0-S0749597819300949-main.pdf
10. Claybaugh, A., Office of Undergraduate Education, Harvard College (2025). Re-Centering Academics at Harvard College: Update on Grading and Workload. October 2025 https://bpb-us-e1.wpmucdn.com/websites.harvard.edu/dist/e/139/files/2025/12/Update-on-Grading_October.22.2025.pdf
11. Association of American Colleges and Universities (2023). The Career-Ready Graduate: What Employers Say about the Difference College Makes. Survey of 1,010 employers fielded by Morning Consult, May 2023 https://apimeda.ucsd.edu/_files/AACU-2023-Employer-Report.pdf
12. Sackett, P. R., Zhang, C., Berry, C. M., and Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040-2068 https://pubmed.ncbi.nlm.nih.gov/34968080/
13. Ben-David, I., Graham, J. R., and Harvey, C. R. (2013). Managerial miscalibration. Quarterly Journal of Economics, 128(4), 1547-1584 https://people.duke.edu/~charvey/Research/Published_Papers/P110_Managerial_miscalibration.pdf
14. Hong, H., and Kubik, J. D. (2003). Analyzing the analysts: Career concerns and biased earnings forecasts. Journal of Finance, 58(1), 313-351 http://www.columbia.edu/~hh2679/analysts-Jf.pdf
15. Christensen-Szalanski, J. J. J., and Willham, C. F. (1991). The hindsight bias: A meta-analysis. Organizational Behavior and Human Decision Processes, 48(1), 147-168 https://www.sciencedirect.com/science/article/abs/pii/074959789190010Q

The Analyst's Handbook is Sifter's textbook for student investors, written by Matt Lang and Bree Lang. Sifter is free for students. This chapter as markdown.