Noise A Flaw in Human Judgment

1. 1Before analyzing the case's unique details, define its reference class: the set of similar past cases that share the same structure (same crime type, same decision problem, same category of prediction).
2. 2Gather the statistical distribution of outcomes across that reference class — not a single average but the full range and how often each outcome occurred. The US Sentencing Commission built its guidelines from an analysis of ten thousand actual cases; imitate that impulse to ground judgment in a population.
3. 3Set your anchor at the base rate of the reference class before you let the specific causal story of the case pull you toward a number.
4. 4Adjust from that anchor only for case-specific evidence that is genuinely diagnostic, and adjust less than your intuition wants — make the prediction moderate (regressive) toward the class average.
5. 5Check that other competent judges sharing your reference class would land near the same estimate; if the answer depends heavily on your unique experience, you are generating pattern noise, not signal.
6. 6Use the range of past outcomes and their limited predictability to calibrate confidence — do not claim precision the reference class does not support.
7. 7Where the case is one of a recurring type, consider replacing discretionary judgment with guidelines or an algorithm anchored on the class statistics, which reduces noise by making the decision less dependent on one professional's idiosyncrasies.

The US Sentencing Commission faced exactly this problem: judges treating each convicted defendant as a unique person deserving individualized tailoring produced outrageous disparity — in the New York area, identical actual cases drew sentences ranging from three years to twenty years. The inside view (this defendant's character and circumstances) generated the noise. The Commission adopted an outside-view remedy: it defined reference classes by crime and criminal history, assigned crimes to one of forty-three offense levels by seriousness, and derived ranges from average sentences across ten thousand actual cases. Justice Breyer defended anchoring on this history precisely because principled rationalization from scratch was impossible — competent people disagreed on the rank order of crimes. Anchoring each new sentence in the statistics of similar past cases replaced idiosyncratic causal storytelling with a shared base rate. The result: professionals sharing the same reference class became less noisy, and sentences depended less on which judge, or which mood, the case happened to meet.

• If the case belongs to a recognizable category of prior cases, then start from the base rate of that class before considering what makes this case special.
• If your intuitive estimate sits at the extreme of the reference-class distribution, then pull it back toward the class average — extreme predictions imply confidence the data does not justify.
• If the answer changes dramatically when a different competent judge with the same reference class evaluates it, then treat the gap as pattern noise and re-anchor on the shared statistics.
• If the past outcomes in the class show wide spread and weak predictability, then widen your uncertainty and lower your confidence rather than committing to a precise number.
• If the decision is recurring and high-volume, then encode the outside view as a guideline or algorithm rather than relying on repeated case-by-case discretion.

• Building the case around a vivid causal narrative first, then treating the base rate as a footnote — the tell is that you can explain your number in detail but cannot say which reference class it came from.
• Choosing a reference class so narrow it becomes 'unique,' defeating the method — the tell is a class of one or two cases that conveniently supports the story you already believe.
• Anchoring on the class but failing to regress — you cite the average, then let the inside-view details push you back to an extreme prediction anyway.
• Confusing the internal signal of coherence ('this number feels right, it fits the evidence') with accuracy — the tell is high confidence unrelated to any outcome or reference-class distribution.
• Assuming the outside view is always worth the cost — in low-noise settings, or where guidelines don't reduce error much, the effort yields modest benefit; the tell is imposing statistical machinery where there was little variability to remove.

• ▸Case description: one line stating the specific judgment or prediction to be made.
• ▸Reference class: the population of similar past cases this case belongs to.
• ▸Class size / source: how many cases and where the data comes from (e.g., ten thousand actual sentences).
• ▸Base-rate anchor: the average or typical outcome across the reference class.
• ▸Outcome range: the observed spread (low to high) of past outcomes in the class.
• ▸Case-specific adjustment: only genuinely diagnostic evidence that justifies moving off the anchor.
• ▸Regressed estimate: your final number after pulling the adjustment back toward the class average.
• ▸Confidence check: your uncertainty stated in light of the class's predictability.
• ▸Peer test: would another competent judge in this same class land near this estimate? (yes/no + why)

1. 1Run a noise audit before you touch selection: have many of your judges evaluate the same case(s) and measure the variability of their responses, expressed as median noise (percent), as the asset management firm did when 42 investors valued the same stock.
2. 2Set your benchmark from the numbers the passages give you: human personnel judges average a predictive correlation of only .28 (PC = 59%), so treat any claim of accuracy in the 75–85% range as inflated by objective ignorance, not skill.
3. 3Test candidate judges for actively open-minded thinking and general ability, then check whether their judgments hold up ex post — compare their prior evaluations of a set of cases against objectively assessed later outcomes using the percent-concordant statistic.
4. 4Distinguish the two things training must fix: level noise (a judge who is harsher or more lenient overall) and pattern noise (a judge who reacts idiosyncratically to specific case features, the judge×case interaction).
5. 5Train for statistical (outside-view) thinking, not causal storytelling: teach judges to treat each case as an instance of a broader category with known base rates rather than as a unique narrative.
6. 6Target occasion noise directly — the within-person variability that made the same software developers' estimates of the same task differ by 71% across two days — by controlling mood, sequence, and fatigue conditions during judgment.
7. 7Re-audit after selection and training with the same case set; if median noise and the bias against outcomes have not moved, the intervention failed and you should shift to rules, aggregation, or algorithms.

An asset management firm's senior managers, after hearing that judgment is noisy, ran their own exploratory audit. They asked 42 experienced investors to estimate the fair value of a stock — the price at which they would be indifferent between buying and selling — from a one-page description with three years of simplified financials and two years of projections. Median noise came out at 41%. That figure is the point: these were the firm's own selected experts, using the same data, yet a randomly chosen one of them would set a fair value wildly different from a colleague's. Selection alone had not made them noiseless. The lesson for selecting and training judges is that you cannot assume 'experienced' equals 'aligned' — you must measure the disagreement first, then decide whether better judges, training, or structure closes the 41% gap.

• If a noise audit shows large system noise among already-experienced judges, then selection credentials are not enough — add training or structure rather than trusting seniority.
• If a judge's confidence exceeds what a .28 predictive correlation supports, then treat that confidence as an emotional signal ('the evidence feels right') masquerading as validity, not as evidence of superior judgment.
• If the disagreement is level noise (consistently harsh or lenient), then recalibrate the scale; if it is pattern noise (idiosyncratic reactions to case features), then structure the inputs or aggregate judges.
• If the variability comes from taste or a competitive innovation setting, then leave it alone — variation is desirable there; only reduce noise where two judgments cannot both be right.
• If a judge reliably explains outcomes with tidy causal narratives ('they failed because they lacked experience'), then coach toward the outside view, because low correlations are being inflated into causal certainty.

• Selecting for confidence or experience. Tell: judges poll themselves at 75–85% accuracy while the field average is PC = 59% (.28) — the gap between felt and real skill.
• Confusing level noise with pattern noise. Tell: you 'fix' a lenient judge by raising his baseline, but his harshness toward one case type and softness toward another persists.
• Ignoring occasion noise. Tell: the same judge gives materially different answers on different days — like the software estimates that varied 71% for the identical task.
• Treating a noise problem as a taste or market problem. Tell: you defend variability as 'individualized judgment' (the pre-1970s enthusiasm for judicial discretion) while similar cases get radically different outcomes.
• Never auditing after training. Tell: you assume the program worked because judges feel more aligned, but you have no repeated same-case measurement showing median noise actually fell.

• ▸Case set used for audit: the identical cases every judge evaluates.
• ▸Number of judges evaluating each case: enough to make variability visible.
• ▸Median noise (%): variability of responses, measured as in the insurance/investor audits.
• ▸Predictive correlation vs. outcomes: compare judge evaluations to ex-post results (target above the .28 baseline).
• ▸Level noise estimate: how much of the spread is overall harshness/leniency differences.
• ▸Pattern noise estimate: idiosyncratic judge×case reactions to specific features.
• ▸Occasion noise check: same judge, same case, two separate days — record the % difference.
• ▸Selection screen: measured actively open-minded thinking / general ability scores.
• ▸Post-training re-audit result: new median noise vs. baseline, plus decision to keep, retrain, or replace with structure.

1. 1Separate the person from the case before you evaluate the person. In the sentencing study, researchers isolated each judge's mean judgment across all cases (rightmost column of the table) — do the same: compute a judge's average level across many identical cases so you measure the trait, not the case.
2. 2Quantify level severity as a scalar. Treat 'harsh vs. lenient' like a personality dimension you can place on a scale, the way a personality test measures extraversion. In the study, the standard deviation of judges' mean sentences was 2.4 years — that number IS the trait signal.
3. 3Probe the stable attitudes that drive the level. Ask each judge which sentencing goal they endorse — incapacitation, rehabilitation, or deterrence — because judges favoring rehabilitation assigned shorter sentences and more supervised time.
4. 4Log demographic and ideological correlates that predict level: region (Southern judges gave significantly longer sentences), and political ideology (conservative ideology correlated with severity). These predict baseline severity independent of the case.
5. 5Distinguish stable traits from occasion effects. A judge's mood — daughter's good day, favorite team won, beautiful weather — introduces occasion noise. Test the same judge on multiple days to strip transient effects and isolate the durable characteristic.
6. 6Test for cognitive-style advantages by measuring judgment consistency, not just outcomes. The stopwatch/lap exercise shows within-person variability; better judges show smaller lap-to-lap variability — lower baseline noise is the mark of superior characteristics.
7. 7Where possible, replace the judge's global synthesis with mechanical aggregation of that judge's own cues — Goldberg's finding is that a model of the judge beats the judge, because the model applies the judge's rules consistently while the judge does not.

The refugee asylum data expose how much a single judge's characteristics drive outcomes: cases were randomly allotted, yet one judge admitted 5% of applicants while another admitted 88% — the study is aptly titled 'Refugee Roulette.' Because assignment was random, the case pool each judge saw was statistically equivalent; the 83-point gap is therefore a property of the judges, not the applicants. This is level noise expressed as a personality-like trait: some judges are simply harsh, some lenient. The same structure appears in Frankel's sentencing work, where the standard deviation of judges' mean sentences was 2.4 years, and researchers traced part of that spread to whether a judge favored rehabilitation, deterrence, or incapacitation, plus Southern location and conservative ideology. None of these traits has anything to do with the case or the defendant. The corrective is to stop treating the outcome as a fact about the applicant and start treating it as a reading off the judge — then select or aggregate judges to shrink the spread.

• If two identical cases assigned randomly to different judges produce wildly different outcomes (5% vs. 88% admission), then attribute the gap to judge characteristics, not case merit.
• If a judge's average judgment across many cases sits far from the group mean, then treat that level as a stable severity trait and correct for it, the way you'd adjust for a known 2.4-year severity SD.
• If a judge's variability shows up within-person across days (mood, weather, sports outcome), then classify it as occasion noise and average across occasions rather than reading it as a trait.
• If a judge endorses rehabilitation as the goal, then expect systematically shorter sentences and more supervised time; adjust your interpretation of their outputs accordingly.
• If you can build a simple model of a judge's own cue-weightings, then prefer the model's output over the judge's live judgment, because the model of you beats you.

• Reading a singular high-stakes decision as trait-free. Practitioners assume a unique decision has no noise ('how can you have variability with singular decisions?'). Tell: they can't name the cloud of possible outcomes the same judge might have produced.
• Confusing occasion noise with a stable trait. Frankel's study effectively ignored occasion noise and interpreted idiosyncratic patterns as settled attitudes. Tell: no re-test across days exists, so mood is silently baked into the 'trait' estimate.
• Trusting subjective experience of good judgment over measured consistency. Clinicians trust their intuition against 'massive and consistent' evidence that mechanical aggregation wins. Tell: the judge cites felt confidence, not a track record of low variability.
• Treating level severity as justice rather than personality. Practitioners defend a harsh judge as 'principled.' Tell: the severity correlates with region, ideology, and background — variables with nothing to do with the case.
• Ignoring pattern noise — the judge×case interaction. A judge may be lenient on average yet spike on cases that resemble a hated criminal or their own daughter. Tell: the judge is consistent on aggregate level but reorders individual cases unpredictably versus peers.

• ▸Judge ID: label the individual whose trait you are estimating.
• ▸Mean judgment level: average outcome across many comparable cases (the severity scalar).
• ▸Level deviation from group mean: how far this judge's average sits from peers.
• ▸Stated goal orientation: rehabilitation / deterrence / incapacitation (or task equivalent).
• ▸Ideology/region flags: note any correlates that predict baseline severity.
• ▸Within-person variability: lap-to-lap or day-to-day spread on identical cases (occasion-noise check).
• ▸Pattern-noise note: cases where this judge reorders rankings versus peers, and any visible trigger.
• ▸Consistency vs. own model: does a simple model of this judge's cues outperform the judge's live calls?
• ▸Correction applied: adjustment or aggregation rule used to offset this judge's trait.

1. 1Run a noise audit: assemble realistic case files, distribute them to professionals who normally handle such cases independently, and have each judge the same cases without conferring. Collect their judgments as data.
2. 2Read the back of the target: measure the scatter among judgments without needing to know the correct answer. When physicians diagnose the same patient or underwriters price the same risk, quantify how much they diverge from each other — disagreement is measurable even when truth is unknown or unknowable.
3. 3Decompose the noise into its components. Separate level noise (stable differences in judges' average severity — some judges are consistently harsher, some managers consistently more likely to send children to foster care) from pattern noise (a judge's idiosyncratic reaction to particular cases).
4. 4Isolate occasion noise by presenting the same judge the same case on different occasions. Track within-judge inconsistency — the same fingerprint examiner reaching different conclusions on different days, or the same developer estimating the same task's hours differently.
5. 5Convert noise to cost. In the insurance underwriting example, count both lost business from quotes that are too high and losses on contracts priced too low. Attach a dollar figure so leadership treats variability as an economic problem, not just a fairness abstraction.
6. 6Separate unwanted variability from wanted diversity before acting. Confirm the judgments are ones that *should* be identical (professional judgments of the same case), not matters of taste or competitive idea-generation where diversity is welcome.
7. 7Apply decision hygiene to reduce the noise you found: guidelines and rules to cut level noise, structured independent aggregation to cut pattern and occasion noise. Match the technique to the judgment type.

An insurance company assumed its underwriters, given the same risk, would produce roughly the same quotes. The authors ran a noise audit: they built realistic case files and had underwriters price them independently. Leadership expected variability of maybe 10%; the audit revealed far more. The scatter mattered economically — excessive quotes drove away business while underpriced contracts generated losses, totaling in the hundreds of millions of dollars. No one could name the correct 'Goldilocks' price for any case, so no one could measure bias — but nobody needed to see the bull's-eye to measure the scatter on the back of the target. The finding: the price a customer pays depends on the lottery of which employee handles the transaction, a lottery customers never consented to.

• If you can observe disagreement among judges on the same case, then you can measure and act on noise — even when the true answer is unknown or unknowable.
• If some judges are consistently more severe or lenient than others across all cases, then you are looking at level noise; if a judge reacts idiosyncratically to specific cases, then it is pattern noise.
• If the same judge gives different answers to the same case on different occasions, then you have occasion noise (mood, temperature, prior events) — the judgment is unreliable even against itself.
• If the variability is in professional judgments that should agree, then it is unwanted system noise; if it is taste, preference, or competitive idea-generation, then diversity is welcome and should be left alone.
• If reducing noise would compromise dignity, morale, or a competing value, then the right level of noise is not zero — weigh the cost of reduction against the harm of the variability.

• Focusing only on bias. The tell: your reviews, articles, and audits discuss systematic error but never even mention scatter — noise stays offstage while bias is the star.
• Assuming singular decisions can't be noisy. The tell: someone argues 'each case is unique, so there's no variability' — ignoring that any singular judgment is one draw from a cloud of possible judgments that would vary.
• Mistaking personal taste for professional judgment. The tell: a producer greenlights an unusual project because *she* likes the script, treating a matter of preference as if it were a reliable market judgment.
• Refusing to measure noise because the true answer is unknown. The tell: 'we can't study this without knowing who's right' — when scatter is measurable from the back of the target regardless of truth.
• Reducing noise while ignoring the objections it creates. The tell: guidelines are imposed, people feel like cogs in a machine, morale collapses, and the noise-reduction effort is abandoned.

• ▸Judgment type: name the specific decision that should ideally be identical across judges
• ▸Should-agree test: confirm this is professional judgment, not taste/preference/competition
• ▸Case set: list the realistic cases each judge will evaluate independently
• ▸Judges: list the professionals who normally make this call on their own
• ▸Repeat pass: schedule the same case to the same judge on a later occasion to expose occasion noise
• ▸Scatter measure: record the spread of judgments per case (no bull's-eye needed)
• ▸Component split: attribute variance to level noise vs. pattern noise vs. occasion noise
• ▸Cost estimate: dollarize both over-decisions and under-decisions caused by the scatter
• ▸Hygiene action: choose the reduction technique (guidelines, aggregation, structure) matched to this judgment

1. 1Reframe each judgment as a measurement whose instrument is a human mind, and name the scale explicitly (a number, a verdict, a diagnosis). Accuracy—approaching truth, minimizing error—is the only goal, not persuasion or impressiveness.
2. 2Separate the internal signal from the evidence. When an answer 'feels right,' recognize that feeling as a sense of coherence signaling completion, not as validity. Treat the coherent stopping point as suspect precisely because it feels satisfying.
3. 3Interrogate the sources of the coherent impression: did an association, a resemblance, or a prejudgment drive it (a defendant who looks like your daughter, a smile read as friendly)? These are the mechanisms that make bias systematic.
4. 4Check your state at the moment of judgment. Mood measurably shifts what you notice and retrieve—a good mood approves the same smile a bad mood judges awkward. Note whether you are happy, sad, tired, or influenced by others' opinions before locking in.
5. 5Test your confidence against achievable accuracy. Compare your subjective certainty to the actual predictive ceiling for the task; if the outcome is far in the future or intrinsically unpredictable, discount confidence heavily to avoid the illusion of validity.
6. 6Substitute or supplement your judgment with a mechanical rule where one exists. Formulas beat clinicians consistently (73% vs 68% hit rate; .56 vs .32 correlation), so use them even when the margin is small—and don't discard the algorithm the moment it errs.
7. 7Run a noise audit: have the same people judge the same cases and measure the scatter. Variability on identical cases is pure system noise and exposes bias that varies in magnitude across judges.

The 1981 sentencing noise audit shows psychological bias becoming pattern noise. Researchers built sixteen hypothetical robbery and fraud cases and had 208 federal judges each set sentences in ninety-minute interviews. Judges did not merely differ in overall severity (level noise); they reacted idiosyncratically to case features—one judge lenient when the defendant was merely an accomplice, another harsh when the victim was elderly. Some of these reactions reflected a personal sentencing philosophy; others came from associations the judge was barely aware of, such as a defendant resembling a hated criminal or the judge's daughter. Because these prejudgments were not shared and varied in magnitude, they produced 'judge-by-case interaction'—pattern noise that contributed roughly as much to system noise as level noise did. The same case, seen again, would produce the same idiosyncratic reaction, so this is systematic bias at the individual level that shows up as noise across the system.

• If an answer feels comfortably coherent, then pause—the sense of fit signals completion, not accuracy, and is your cue to check for hidden bias.
• If you notice you are in an induced or strong mood before judging, then defer the judgment or discount your read of ambiguous cues, since mood shifts what you notice and recall.
• If a validated formula or algorithm exists for the task, then use it and keep using it after it errs—expecting perfection and discarding it on the first mistake reintroduces the human bias it was correcting.
• If your confidence exceeds the demonstrated predictive ceiling for the task, then treat the excess as illusion of validity and lower your certainty.
• If the same person or system produces variable judgments on identical cases, then run a noise audit rather than assuming the variability is meaningful signal.

• Mistaking the coherence signal for correctness. The tell: you feel 'right enough' and stop, having landed on one point in a cloud of possible answers you never saw.
• Denial of ignorance—believing unpredictable events are predictable. The tell: high confidence in long-range forecasts, ignoring that objective ignorance rises with time horizon.
• Letting an unnoticed association drive the verdict. The tell: a defendant reminds you of someone, and you cannot articulate why the case feels severe or lenient.
• Algorithm aversion applied selectively. The tell: you accept an algorithm's advice until it makes one visible mistake, then abandon it while tolerating your own errors.
• Confusing intentional variability with bias-driven noise. The tell: you excuse different judgments of the same case as legitimate discretion instead of auditing them as system noise.

• ▸Judgment stated as a scale value: write the conclusion in one word/number and name the scale
• ▸Coherence check: does this feel comfortably right? If yes, flag as completion signal, not validity
• ▸Association scan: does any case feature remind me of a person, type, or past case? Name it
• ▸Mood/state note: current mood and whether external opinions have already reached me
• ▸Confidence vs ceiling: my confidence % against the known predictive accuracy for this task
• ▸Mechanical rule available?: yes/no; if yes, what does it output and how does it differ from mine
• ▸Same-case variability: would I give the same answer on a second viewing? Estimate the spread
• ▸Decision to adjust: keep, revise toward the rule, or defer—and the reason

1. 1Define the 'true value' or optimal outcome your judgment aims at, or accept it is unknown — noise can be measured from the back of the target without it.
2. 2Collect multiple independent judgments of the same case (across judges, or the same judge on different occasions) so you can see the scatter.
3. 3Compute bias as the average signed deviation from the true value (mean of judgments minus target).
4. 4Compute noise as the standard deviation of the judgments — their variability around their own mean, ignoring the target.
5. 5Square each component and sum them: MSE = Bias² + Noise². This is total error in the same units.
6. 6Run this across multiple cases, then average MSE, bias², and noise² over cases to get a system-level picture.
7. 7Compare the two components: if noise² rivals or exceeds bias², prioritize noise reduction — and expect that cutting noise makes residual bias visible and 'impossible to miss.'

In the GoodSell noise audit, Amy Simkin gathers forecasts for one sales region and applies the error equation. The model assumes bias and noise are equal, so each accounts for 50% of overall error. Yet to match noise's contribution, the bias has to be so large that 84% of forecasters — six out of seven — err in the same direction. That contrast shows why noise is often the bigger, unnoticed problem. Better practice: Simkin should collect multiple forecasts across several regions, from the same or different forecasters, and average the MSE, bias², and noise² across cases for a truer picture of the system.

• If the true value is unknown or unknowable (physicians disagreeing on a diagnosis, executives estimating a film's market), then measure noise anyway — you only need agreement data, not the answer.
• If you can only reduce one component, then reduce noise first when it is comparable to bias, because bias must be enormous (≈6 of 7 erring the same way) to outweigh noise.
• If two competent judges give different answers to the identical case, then treat it as error — two different judgments of one true value cannot both be right.
• If you want to separate stable disagreement from occasion noise, then compare a person against others versus against their own prior judgments — occasion noise is consistently the smaller share.
• If reducing noise exposes large residual bias, then treat that visibility as a benefit and put bias reduction next on the agenda, since the two are independent.

• Treating bias as the whole story: focusing only on systematic error while ignoring scatter — the tell is a debiasing program with no measure of judge-to-judge disagreement.
• Refusing to measure error because the true answer is unknown — the tell is 'we can't judge accuracy here,' when noise is measurable from the back of the target alone.
• Auditing a single case and generalizing — the tell is confident bias/noise estimates from one region instead of averaged MSE across multiple cases.
• Confusing statistical bias with social discrimination — the tell is reading 'bias' as unfairness rather than as the average signed error.
• Ignoring occasion noise or overweighting it — the tell is either assuming a person is perfectly consistent, or over-attributing total system noise to within-person variability when between-individual differences dominate.

• ▸Case description: the single case being judged, one line
• ▸True value / optimal outcome: the target, or write 'unknown'
• ▸Independent judgments: list each judge's (or occasion's) value
• ▸Mean of judgments: average of all values above
• ▸Bias: mean of judgments minus true value (signed)
• ▸Noise: standard deviation of the judgments around their mean
• ▸MSE: Bias² + Noise², total error
• ▸Component share: bias²/MSE and noise²/MSE as percentages
• ▸Reduction priority: which component to attack first and why

1. 1Separate the two ceilings on any prediction: noise (variability you can remove) and objective ignorance (unknowable facts about the future you cannot remove). Attack noise; respect the ignorance.
2. 2Before forecasting, list what the outcome actually depends on, then mark each item known vs. unknowable at the time of judgment. The unknowable pile is your objective ignorance and it caps achievable accuracy.
3. 3Translate your intended accuracy claim into a correlation, then into percentage concordant using the passage's table (r=.20→56% PC, r=.40→63%, r=.60→71%). If your predictors realistically yield r=.20, accept that you'll rank pairs correctly only 56% of the time.
4. 4Do not treat the internal sense of coherence as evidence of accuracy. The 'aura of rightness' arises whether or not the outcome is verifiable, so it tells you nothing about objective ignorance.
5. 5Evaluate the process, not the single output. Score a forecaster across many cases (of 100 candidates called 70% to win, are ~70 elected?) rather than crediting or blaming any one call.
6. 6Prefer noiseless rules or equal-weight models over case-by-case gut judgment, since removing noise is the gain you control; understand this narrows the gap to the ignorance ceiling but never eliminates it.
7. 7Widen prediction intervals and downgrade confidence in proportion to the unknowable share. High confidence with high objective ignorance is overconfidence by definition.

Take Monica and Nathalie from chapter 9: you must predict their job-performance ratings two years out from hiring assessments (leadership 4, communication 6, interpersonal 4, technical 8, motivation 8 for Monica, on 1-to-10 scales). The predictors are real information, but the two-year outcome depends on events not yet known—a reorganization, a new manager, a personal crisis, a market shift. That unknowable dependence is objective ignorance, and it sets a hard ceiling on accuracy no matter how carefully you weight the five scores. If the honest predictor-to-outcome correlation is around .40, the passage's table says your percentage concordant is 63%: given two executives, you'd order them correctly only about 63 times in 100, barely better than the coin flip at 50%. The number you produce will still feel coherent and confident, but that feeling is the internal signal, unaffected by how much you cannot know. So predict Monica's rating as a cautious value near the average with a wide interval, and treat any strong conviction as suspect.

• If the outcome depends heavily on future events not knowable now, then cap your stated confidence and regress the prediction toward the base rate rather than toward your gut.
• If your realistic predictor correlation is below about .40 (PC under ~63%), then treat any claim of reliable case-by-case ranking as false and prefer aggregation.
• If you feel strong coherence but the judgment is nonverifiable or long-horizon, then discount the feeling—coherence is not accuracy.
• If a human judge and a simple rule disagree, then follow the rule unless there is a genuine 'broken leg' (decisive, specific, outside-the-model fact), not merely dislike of the forecast.
• If you cannot specify what would have to be true for your forecast to be verified across many cases, then you are measuring your ignorance, not the world.

• Trusting gut over analysis: senior, confident executives report 'knowing without knowing why'; the tell is conviction with no articulable reasons.
• Mistaking coherence for correctness: the judgment 'feels right' the same way for a fictitious case as a real one; the tell is equal confidence on unverifiable questions.
• Being unsurprised by unpredictable events: people rationalize outcomes they could never have foreseen; the tell is fluent after-the-fact explanations of surprises.
• Overconfidence via narrow intervals: forecasters state precise numbers when the outcome is largely unknowable; the tell is prediction intervals that rarely get pierced by reality... except they constantly are.
• Confusing noise with ignorance: chasing fancier models to beat a ceiling that is actually unknowability; the tell is added complexity with no gain in calibrated accuracy.

• ▸Outcome to predict: state the specific verifiable outcome and its date
• ▸Known predictors: list facts available now that plausibly relate to the outcome
• ▸Unknowable factors: list future events/information the outcome depends on but you cannot know now
• ▸Estimated predictor correlation (r): your honest guess of predictor-outcome strength
• ▸Percentage concordant: convert r via the table (.20→56, .40→63, .60→71)
• ▸Base rate / anchor: the average outcome to regress toward given the ignorance
• ▸Confidence cap: max confidence justified after subtracting the unknowable share
• ▸Broken-leg check: any decisive outside-the-model fact overriding the rule? (yes/no + what)
• ▸Process test: how will you score this across many cases, not just this one?

1. 1Frame fairness operationally as consistency: similarly situated cases must receive similar judgments. State it as a testable claim — 'a defendant's sentence should not depend on which judge the case happens to be assigned to.'
2. 2Run a noise audit to measure the gap between the fairness ideal and reality. Assemble a realistic set of cases (identical pre-sentence reports, the same insurance file, the same patient) and have multiple qualified professionals judge each one independently.
3. 3Read the back of the target: measure scatter without needing the true answer. Compute how much judgments of the same case vary across judges — you can quantify unfairness even when the correct outcome is unknown or unknowable.
4. 4Distinguish system noise from occasion noise. Check whether outcomes hinge on which judge is randomly assigned (system noise) versus mood, ordering, or temperature on the day (occasion noise). Both destroy fairness; name which one you're seeing.
5. 5Reject the 'errors cancel out' defense. Show that noisy systems judge different cases, not the same case repeatedly, so a three-year and a seven-year sentence for two identical five-year felons is two injustices, not an average of justice.
6. 6Apply decision hygiene to shrink the scatter: guidelines, checklists with objective grading, defined scales, aggregation of independent judgments — the techniques matched to the type of judgment.
7. 7Set the target level of noise deliberately — not zero. Weigh reduced unfairness against competing values (judicial discretion, morale, treating people as cogs) and accept residual noise only where reduction is infeasible, too costly, or damages a value you're protecting.
8. 8Institutionalize the fix: create a standing body (Frankel's sentencing commission) and impersonal rules binding on judges as well as everyone else, so consistency survives after the audit's attention fades.

Frankel's 1974 study is the model. Fifty judges from various districts were each asked to sentence defendants in hypothetical cases described in identical pre-sentence reports — the same facts, so any variation is pure unfairness. The finding was that 'absence of consensus was the norm' and the variations were 'astounding': a heroin dealer drew one to ten years depending on the judge, a bank robber five to eighteen years, and an extortion case ranged from twenty years plus a $65,000 fine down to three years and no fine. Most startling, in sixteen of twenty cases the judges could not even agree on whether any incarceration was appropriate. Because the reports were identical, no one needed to know the 'right' sentence to see the injustice — this is reading the back of the target. Frankel's response was to argue for a checklist of factors with numerical grading and a sentencing commission, changing criminal law worldwide, though 'not enough.'

• If two similarly situated people, convicted of the same offense, receive dramatically different outcomes, then the system is unfair regardless of whether either outcome is individually defensible.
• If the person judging your case is randomly drawn from a pool of equally qualified individuals, then system noise is present and fairness is at risk — audit it.
• If someone argues random errors cancel out, then remind them noisy systems judge different cases, so errors add up rather than net out.
• If an outcome depends on mood, who spoke first, or the outside temperature, then you have occasion noise and no legitimate case-based justification for the variation.
• If reducing noise would compromise a competing value — discretion needed for genuinely unique cases, or morale — then choose a non-zero noise level deliberately rather than eliminating variation blindly.

• Treating bias as the only error and ignoring noise. Tell: the conversation is entirely about systematic skew (who's favored) with no discussion of scatter across identical cases.
• The illusion of agreement — assuming colleagues judge alike because judgments feel objective. Tell: leadership is astonished by audit results they never questioned or predicted, as at the insurance company.
• Waiting to know the 'right' answer before measuring unfairness. Tell: 'we can't tell if it's unfair because we don't know the correct sentence' — false; scatter is measurable from the back of the target.
• Defending inconsistency as necessary discretion for unique cases. Tell: 'each case is unique, isn't it?' used to block guidelines even where cases are demonstrably similar.
• Chasing zero noise and damaging competing values. Tell: staff report feeling like cogs in a machine, morale drops, and legitimate discretion is stripped from genuinely singular decisions.

• ▸Case set: list the identical/standardized cases judged (e.g., matched pre-sentence reports).
• ▸Judges/evaluators: number and how they are assigned (random pool? which one hears your case?).
• ▸Judgment recorded: the specific decision each professional made per case.
• ▸Observed scatter: range and median disagreement per case (measure back-of-target variability).
• ▸Noise type: mark each variation as system noise (which judge assigned) or occasion noise (mood, order, temperature).
• ▸Fairness violation flag: cases where similarly situated people got dramatically different outcomes.
• ▸Competing values at stake: discretion, morale, cost — what limits how far you should reduce noise.
• ▸Decision-hygiene remedy: guideline, checklist, scale, or aggregation matched to this judgment type.
• ▸Target noise level: the deliberate non-zero level accepted, with the reason it isn't zero.

1. 1Conduct a noise audit before assuming performance evaluation is working: have multiple raters independently assess the same employees on the same evidence, then measure how far their ratings diverge from each other.
2. 2Decompose the judgment into fixed dimensions (for example, kindness/collegiality, intelligence, diligence) rather than asking for a single holistic score, so raters are anchored on the same components.
3. 3Define the scale explicitly: specify what behavior earns a 5 versus a 3, because differences in scaling ('extraordinary' vs. 'unusually good') generate noise independent of the person being rated.
4. 4Require independent assessment on each dimension in turn, scoring one before moving to the next, so a global impression on one attribute does not contaminate the others.
5. 5Delay holistic judgment until all structured evidence is collected and analyzed; only then allow intuition to inform the final call, keeping quick impressions in check.
6. 6Anchor final decisions on the averaged scores of several independent evaluators, then treat the underlying evidence as informative rather than mechanically binding.
7. 7Avoid single-metric evaluation of knowledge workers: never reward a physician solely on patients seen or a programmer solely on lines of code, because it distorts incentives and produces erroneous evaluations.
8. 8Assign a decision observer to flag bias in real time during group evaluation sessions, since noise and bias both degrade the result.

An insurance company ran a noise audit: underwriters in the same role, same office, were given identical information on the same complex risk and asked to set a premium. One experienced underwriter, well regarded and confident, judged $200,000 appropriate. But across the group, at least half set premiums either above $255,000 or below $145,000 for the identical case. The executives were not merely surprised—no one had expected anything like that spread, and no one argued the amount was acceptable. The cost had gone unnoticed for years, tolerated like a leak in the basement, because professionals maintained an illusion of agreement while actually disagreeing daily. That divergence directly harms performance: mispriced premiums mean lost revenue on some policies and uncompetitive quotes on others.

• If output cannot be objectively measured (CFO, head of research, most knowledge workers), then treat evaluation as a judgment task and structure it—do not pretend objective facts settle it.
• If a single quantifiable metric exists (sales, lines of code), then evaluate it in context rather than rewarding it single-mindedly, or you create harmful incentives.
• If raters give a holistic score, then expect noise from differing policies and scaling; decompose and define the scale to reduce it.
• If professionals never see how colleagues judge the same case, then assume an illusion of agreement is masking large noise—run an audit before trusting the ratings.
• If the decision is high-stakes (promotion, bonus, hiring), then aggregate independent assessments and delay the holistic judgment until after structured evaluation.

• Believing objective facts suffice for evaluation; the tell is treating a knowledge worker like Adam Smith's pin-factory worker with one measurable output.
• Assuming your own ratings match your colleagues'—the illusion of agreement; the tell is that you expect agreement but never actually check and dismiss disagreements as isolated cases.
• Rewarding a single metric; the tell is physicians maximizing patients-seen or programmers padding line counts while real value drops.
• Relying on unstructured interviews and informal impressions because they feel irreplaceable; the tell is 'the persistence of the illusion'—executives and candidates both insisting only face-to-face reveals true mettle despite evidence such interviews are often useless.
• Suppressing evidence of divergence because noise is organizationally embarrassing; the tell is the leak-in-the-basement pattern—large cost tolerated only because it stayed unnoticed.

• ▸Employee/candidate assessed: name the single subject all raters will judge from the same evidence
• ▸Independent raters: list at least four evaluators scoring separately, no discussion first
• ▸Fixed dimensions: define the components (e.g., collegiality, intelligence, diligence) each rater scores in turn
• ▸Scale definitions: write the behavior that earns each score point (what makes a 5 vs. a 3)
• ▸Per-dimension scores: record each rater's score on each dimension before any holistic view
• ▸Divergence check: compute the spread between raters; flag if it exceeds an agreed threshold
• ▸Averaged score: average the independent ratings to anchor the final decision
• ▸Delayed holistic judgment: note the final intuitive call, made only after all above is complete
• ▸Decision observer note: record any bias signals flagged in real time during the session

1. 1Decompose the judgment before you look at options: list the independent dimensions that should drive it. In hiring, treat these as the traits you'd assess; in sentencing, the case attributes (crime type, principal vs. accomplice, criminal record, weapon used).
2. 2Define each mediating assessment concretely so evaluators use the same scale. Move away from vague standards by writing reference definitions (as proposed in psychiatry: clarify criteria, define symptom severity levels) so evaluators who agree on the inputs agree on the output.
3. 3Control information flow so inputs stay independent: collect each assessment separately and prevent one component from contaminating another. Use structured interviews with a fixed set of questions asked identically across cases (e.g., the twenty-four-question screening guide for anxiety, depression, eating disorders).
4. 4Score each component on its own before any discussion. Rate assessment-by-assessment across all candidates or cases rather than forming a whole impression of one candidate at a time.
5. 5Delay the holistic judgment and final decision until every mediating assessment is in. Only then hold the integrated discussion and reach the overall verdict.
6. 6Treat 'options like candidates': apply the same decomposed, structured scoring to each option in a decision (MAP), not just to job applicants.
7. 7Where possible, validate: correlate your structured ratings against later outcomes (job performance ratings, or actual outcomes) to see whether the structure predicts better than the unstructured version it replaced.

The 1981 sentencing noise audit is a structured decision process applied as measurement. Researchers built sixteen hypothetical cases in which the defendant had already been found guilty, varying them on defined dimensions—robbery vs. fraud, principal vs. accomplice, prior record, weapon used. They ran carefully structured interviews with 208 active federal judges, presenting all sixteen cases in ninety minutes and asking each judge to set a sentence. The identical case set and fixed procedure isolated the judge as the only varying input, exposing scatter cleanly. The later US Sentencing Commission took the same decompositional logic operationally: it reduced the sentence to two components—offense level (one of forty-three, by seriousness) and criminal history (number and severity of prior convictions)—built from an analysis of ten thousand actual cases. That is decomposition into predefined, independent assessments before the final number is set.

• If a component criterion is open-ended or vague, then rewrite it with reference definitions and severity levels before scoring, because open-ended criteria leave noise intact.
• If you are about to form an overall impression of one option, then stop and score every mediating assessment first; hold the holistic judgment until all inputs are collected.
• If evaluators can see or discuss each other's ratings while assessing a component, then you have broken independence—collect the assessments separately.
• If you're relying on a standard (unstructured) interview to predict success, then treat it as near-useless (correlation ~.28) and replace it with a structured interview scored on defined dimensions.
• If the domain has no objective measure and depends on subjective symptom interpretation (as in psychiatry), then expect residual noise even after structuring, and don't over-claim the gain.

• Assessing options holistically one at a time—the tell is that early impressions of a candidate color later trait ratings, so components stop being independent.
• Leaving criteria vague ('assess character'): the tell is that two trained evaluators applying the 'same' criterion still diverge widely, as with mixed anxiety-depressive diagnoses that are 'so unreliable as to appear useless.'
• Discussing and deciding before all inputs arrive—the tell is that the group converges early and later assessments merely rationalize the first opinion.
• Keeping the unstructured interview and just adding structure around it, then trusting the interview anyway; the tell is a low correlation with actual job performance that structuring was supposed to fix.
• Assuming the group average is the 'true' value and stopping there—the tell is treating mean judgment as unbiased when structuring only addresses scatter, not systematic bias.

• ▸Decision/judgment: state the single overall verdict to be reached at the end
• ▸Mediating assessments (list 4-8): the independent components that should drive it
• ▸Definition per assessment: one-line reference definition + what high vs. low looks like
• ▸Data source per assessment: where the input comes from (structured interview question, record, test)
• ▸Independence control: how you keep this input from contaminating others (separate scorer/session)
• ▸Score per assessment: rate each component in isolation, one dimension across all options before moving on
• ▸Holistic-hold checkpoint: confirm all assessments collected before any integrated discussion
• ▸Final judgment: the overall verdict formed only after reviewing all component scores
• ▸Validation note: outcome measure you'll later correlate the scores against

1. 1Collect judgments independently first: give each person the same case and have them record their own estimate before any discussion. In the jury study, 899 participants judged case vignettes alone on a seven-degree scale (and a dollar scale for awards) before any grouping.
2. 2Preserve independence by preventing anyone from hearing another's number first—social influence, cascades, and the person who speaks first all corrupt the pool. Deliberating juries were far noisier than statistical juries precisely because they met and discussed.
3. 3Choose an aggregation rule and apply it mechanically. Take the median (as the statistical-jury method did) or the mean of the independent judgments; averaging always reduces the noise present in the individual inputs.
4. 4Increase the number of independent judgments where noise is your concern—more judgments reduce noise further ('the more judgments, the better').
5. 5Prefer mechanical combination over a group talking itself to a verdict. A 2000 review of 136 studies found mechanical aggregation beat clinical judgment in 63 cases, tied in 65, and lost in only 8.
6. 6Measure the result with a noise audit: have many people judge the same case, compute the standard deviation of the outputs, and confirm aggregation shrank it. Report the reduction as a percentage difference between typical judgments.
7. 7Watch for polarization signals when you must deliberate—if the median member leans lenient the group drifts more lenient, and if severe, more severe. Do not treat a deliberated verdict as if it were an average.

Kahneman and colleagues built 'statistical juries' from real people. First, 899 participants read case vignettes and gave independent judgments of outrage, punitive intent, and dollar awards. A computer then assembled millions of random six-person virtual juries and took the median of each group's six judgments as the verdict. These statistical juries were much more consistent—noise fell as a purely mechanical effect of aggregation. A follow-up with over 3,000 jury-eligible citizens and 500-plus real six-person deliberating juries showed the opposite: deliberating juries were far noisier than statistical juries because discussion introduced social influence noise and polarization. When the median member was moderately outraged, the deliberating verdict drifted more lenient; when severe, more severe—27% of juries chose awards as high as or higher than any individual member's. The lesson: aggregating the independent medians cut noise, while letting the group talk amplified it.

• If you can obtain judgments without people seeing each other's answers, then aggregate independent estimates rather than convene a deliberating group.
• If judgments are based on written information (like underwriters or adjusters), then a noise audit plus mechanical aggregation is especially feasible—use it.
• If you must deliberate, then collect each member's independent number first and compare the group verdict against the median to detect polarization.
• If your goal is reducing noise, then add more independent judgments; the benefit of reduction is the same regardless of any bias present.
• If a formula and a human disagree on how to combine inputs, then trust the mechanical rule—the evidence for its advantage is 'massive and consistent.'

• Letting people discuss before recording independent judgments—the tell is that group verdicts land more extreme than the median member (polarization), not centered.
• Treating a deliberated consensus as equivalent to an average—the tell is that similar groups on the same problem end up widely apart, driven by who spoke first or loudest.
• Assuming trained experts barely disagree—executives guessed 10% variation; the audit found 55% median difference in underwriting, five times expected.
• Waiting to measure bias before reducing noise—the tell is a stalled program; noise and bias are independent in the error equation, so reduce noise now.
• Adding judges who anchor on each other rather than judging independently—the tell is that added voices don't shrink the standard deviation because they aren't truly independent.

• ▸Case description: one shared, detailed vignette given identically to all judges
• ▸Judges: list each independent evaluator and confirm no cross-talk before submission
• ▸Scale used: define units (e.g., seven-degree outrage scale, dollar award)
• ▸Individual judgments: record each person's estimate separately
• ▸Aggregation rule: state median or mean chosen before seeing the numbers
• ▸Aggregated verdict: compute and record the combined figure
• ▸Standard deviation / noise measure: compute spread of individual judgments
• ▸Deliberation check (if used): compare final group verdict to the median—flag any drift
• ▸Noise reduction achieved: report percentage decrease from individual spread to aggregate

1. 1Run a noise audit before designing any rule: give multiple professionals identical cases (as in Frankel's 1974 study, fifty judges scored the same pre-sentence reports) and measure the spread in their judgments — the disagreement, not the average error, is the target.
2. 2Identify the two-to-three factors that legitimately drive the judgment, then commit to them explicitly. For sentencing, the guidelines fixed on exactly two: the crime (one of forty-three offense levels ranked by seriousness) and the defendant's criminal history (number and severity of prior convictions).
3. 3Anchor the rule in aggregated past practice rather than fresh first-principles debate. The commission built the guidelines from average sentences across ten thousand actual cases precisely because members could not agree when they tried to rank crimes by merit from scratch.
4. 4Convert factors into an explicit numerical or ordinal scale — Frankel's 'detailed profile or checklist of factors' with 'some form of numerical or other objective grading' — so different judges apply the same scale the same way.
5. 5Set a bounded output range, not a single mandated number: the guidelines authorized a top of range exceeding the bottom by the greater of six months or 25%, preserving limited discretion while cutting spread.
6. 6Build in a justified-departure valve: permit deviation for aggravating or mitigating circumstances, but require the departure be defended to an appellate court so discretion stays accountable.
7. 7Measure the rule's effect the same way you measured the problem — re-run matched cases before and after. The commission compared bank robbery, cocaine, heroin, and embezzlement sentences from 1985 against 1989-90 with offenders matched on relevant factors, confirming reduced interjudge variation.

In 1973 Judge Marvin Frankel noticed that a convicted federal bank robber faced anything from 0 to 25 years, and that where within that range a sentence landed 'depended less on the case or the individual defendant than on the individual judge.' His 1974 study made this concrete: fifty judges given identical pre-sentence reports produced sentences where 'absence of consensus was the norm' — a heroin dealer drew one to ten years, an extortion case ranged from twenty years plus a $65,000 fine down to three years and no fine. Congress responded with the Sentencing Reform Act of 1984, creating the US Sentencing Commission to issue mandatory guidelines built from ten thousand actual cases. The rule reduced each crime to an offense level plus criminal history, yielding a narrow range. Measured effect: the expected between-judge difference fell from 17% (4.9 months) in 1986-87 to 11% (3.9 months) in 1988-93. The rule worked as intended — not by adding insight but by removing the noise of which judge you drew.

• If a noise audit shows large disagreement among professionals on identical cases, then the problem is noise (not just bias) and a rule or guideline is warranted.
• If experts cannot agree on how to rank cases from first principles, then anchor the scale in aggregated past practice rather than deadlock — take history, as Breyer's commission did.
• If you must preserve some discretion, then bound the output to a narrow range and require any departure be justified to a reviewer, rather than choosing between total rigidity and open-ended judgment.
• If a factor cannot be 'justified by relevant tests capable of formulation and application with sufficient objectivity,' then it should not be allowed to drive differential treatment (Frankel's standard against 'idiosyncratic ukases').
• If practitioners strongly prefer discretion over the rule, weigh that against the audit numbers — 75% of federal judges preferred the advisory regime, but the mandatory guidelines demonstrably cut interjudge disparity.

• Mistaking bias for noise: critics attacked the guidelines because some sentences seemed 'too severe' — that is a complaint about the average being off (bias), not about disagreement (noise), and it calls for a different fix. Tell: the objection is about direction, not spread.
• Over-rigidity that strips out legitimate individual particulars: Stith and Cabranes warned 'the need is not for blindness, but for insight, for equity.' Tell: front-line judges call the rule 'unacceptably mechanical' and route around it.
• Assuming the rule works because of superior insight rather than noiselessness — the passages stress the advantage of rules is 'not so much the superior insight... but their noiselessness.' Tell: you justify the rule by its cleverness and are surprised when a crude formula still beats experts.
• Building the rule on impressionistic evidence: Frankel's original argument was 'devastating but impressionistic,' with no statistical analysis. Tell: you cite outrage and anecdotes but never ran matched identical cases.
• Reverting to discretion under pressure without re-auditing: after 2005 the guidelines became merely advisory and judges were happier, but that comfort says nothing about whether noise returned. Tell: you measure practitioner satisfaction instead of measuring the spread.

• ▸Decision under review: the specific judgment being standardized (e.g., criminal sentence).
• ▸Audit spread: measured disagreement among professionals on identical cases (range from best to worst).
• ▸Legitimate factors: the 2-3 inputs that should drive the judgment, named explicitly.
• ▸Scale definition: how each factor is graded numerically or ordinally (e.g., 43 offense levels).
• ▸Baseline source: the aggregated past cases the rule is anchored to (count and type).
• ▸Output range: the bounded result and permitted width (e.g., bottom + greater of 6 months or 25%).
• ▸Departure rule: conditions under which deviation is allowed and to whom it must be justified.
• ▸Re-measurement plan: how you will compare matched before/after cases to confirm noise reduction.

1. 1Reframe the task as a comparison, not an absolute rating: instead of asking 'How severe is this crime on a 43-point scale?' ask 'Is this crime more or less punishable than that one?' — the question 'Which is more?' is answerable for any pair on the same dimension.
2. 2Build a case scale by collecting a set of concrete, real prior cases that span the full range, then order them from least to most on the target dimension (e.g., severity of punishment). These become your anchors.
3. 3Fix the extreme anchors first, because agreement is easiest when a judgment is absurd — judges who disagree wildly on a mid-range fraud sentence will still concur that a one-dollar fine and a life sentence are both unreasonable. Nail down the obviously-too-low and obviously-too-high ends to bound the scale.
4. 4Place each new case by matching it against the anchor cases rather than assigning a raw number: locate where it sits relative to two bracketing examples ('worse than case B, not as bad as case D').
5. 5Translate the ordinal position into the required output only at the end — once the comparative ranking is settled, map it onto the numeric or adjectival scale the decision demands.
6. 6Share the same anchors across all judges so everyone works from a common frame of reference; the noise you are fighting comes partly from different people making different use of the very same scales.
7. 7Require any departure from the comparison-anchored position to be justified to a reviewer, as the federal guidelines require departures to be justified to an appellate court — this preserves room to maneuver while checking idiosyncrasy.

The federal sentencing guidelines replaced open-ended absolute judgment with a structured comparative frame. Judge Frankel's complaint was that if you 'try listing all the crimes there are in rank order of punishable merit' and collect results from your friends, they won't match — pure absolute judgment produces scatter. The guidelines instead anchor sentences on two factors, crime and criminal history, assigning each crime one of forty-three ordered offense levels, then offering a narrow range whose top exceeds its bottom by at most six months or 25%. The commission's own study compared sentences for bank robbery, cocaine distribution, heroin distribution, and bank embezzlement in 1985 (pre-guidelines) with 1989–1990 sentences, matching offenders on relevant factors; for every offense, variation across judges shrank in the later period. Crystal Yang's data on nearly 400,000 defendants confirmed the mechanism in reverse: when the 2005 Supreme Court ruling made the guidelines merely advisory, the interjudge disparity roughly doubled — a harsh judge added 2.8 months over average under mandatory rules, and that gap doubled once judges reverted to their own scales. The shared, comparison-based frame cut both noise and bias (including racial disparity); removing it returned the system 'to something more like law without order.'

• If reasonable people are expected to agree perfectly (fact or computation), don't build a comparison scale; if unresolved differences are entirely acceptable (taste or opinion), don't either — reserve this for matters of judgment defined by bounded disagreement.
• If a case is absurdly extreme, anchor on it first — agreement is cheapest at the extremes and it bounds the whole scale.
• If cues conflict and coherence is hard to reach, expect high noise and lean harder on comparison to shared anchors rather than trusting the internal sense of a coherent number.
• If you need a numeric output, settle the ordinal ranking against anchors first and convert last — never start from the raw number.
• If a judge wants to depart from the anchor-implied position, require written justification reviewable by someone else.

• Asking for absolute ratings on an ill-defined scale — the tell is that when you collect independent answers from colleagues on the same case, they scatter widely (Frankel's 'they won't match').
• Letting each judge supply their own implicit anchors — the tell is stable per-person patterns of harshness/leniency, like the harsh judge who adds months regardless of the case.
• Mistaking the internal signal of coherence for correctness — the tell is high confidence in a single number reached alone, with no comparison to other cases and no felt need for one.
• Treating a matter of judgment as a matter of taste and tolerating the disagreement — the tell is defending scatter ('it's a judgment call') when the domain actually expects bounded agreement.
• Loosening a working comparative structure into a merely advisory one — the tell is disparity increasing after the rules relax, as happened when the guidelines became advisory and interjudge disparity doubled.

• ▸Dimension being judged: name the single quantitative intensity scale (e.g., severity of punishment).
• ▸Low anchor case: a real, concrete case that is obviously near the bottom.
• ▸High anchor case: a real, concrete case that is obviously near the top.
• ▸Absurd-out bounds: the clearly-too-low and clearly-too-high values everyone rejects.
• ▸Mid-range anchor cases: 2–4 real cases ordered between the extremes.
• ▸New case placement: which two anchors does this case fall between, and why (one line).
• ▸Comparison note: 'worse than ___ because ___; not as bad as ___ because ___.'
• ▸Numeric/adjectival conversion: the output value implied by the ordinal position.
• ▸Departure justification: if you deviate from the anchor-implied position, state the aggravating/mitigating reason for review.

1. 1Classify every piece of case information as task-relevant or task-irrelevant before the examiner begins. Task-irrelevant context (the suspect's background, prior watch-list status, that other examiners already matched the print) has power to bias but not to inform, so quarantine it.
2. 2Reveal information only when it is needed, in a fixed order. Keep the judge as far in the dark about the case as possible and release facts gradually — this is linear sequential unmasking.
3. 3Have the examiner analyze the primary evidence in isolation first (the latent print itself) and record conclusions before any reference or comparison material is introduced.
4. 4Require the examiner to document their judgment at each step, in writing, before moving to the next stage. This locks the intermediate conclusion so later information cannot silently reshape it.
5. 5Withhold the results of prior examiners. Do not tell the verifying examiner what the first examiner concluded, so the verification is a genuine independent judgment rather than a search for confirmation.
6. 6Introduce reference exemplars or contextual detail only after the isolated analysis is documented, and only if the task requires it — more information is not always better.
7. 7Treat the sequence as a defense against the search for coherence: because people form early impressions from limited evidence and then confirm them, the order of exposure, not just the content, determines the outcome.

In March 2004 a fingerprint from a plastic bag at the Madrid train bombings was sent worldwide; the FBI lab 'conclusively' identified it as Brandon Mayfield's. Mayfield fit a plausible narrative — former Army officer, converted to Islam, married an Egyptian woman, had represented convicted defendants, and was on the FBI watch list. That biasing context was available to examiners early, and the search for coherence pushed toward confirming the match: he was surveilled, bugged, wiretapped, and arrested. Spanish investigators, working the print without the same context, had already called Mayfield a negative match and later matched it to another suspect; he was released after two weeks and received a $2 million settlement. The lesson the PCAST report drew is that the FBI redesigned its procedures to minimize confirmation bias — precisely by controlling what examiners see and when. Had the examiner analyzed the latent print in isolation and documented conclusions before learning who the suspect was, the biasing background could not have driven a premature intuition.

• If a piece of information can bias the judgment but cannot improve accuracy, then withhold it entirely from the examiner.
• If an examiner is about to compare against a reference, then require the isolated analysis of the primary evidence to be documented first.
• If a second examiner is verifying, then do not disclose the first examiner's result — independence is the whole point of verification.
• If information is genuinely relevant, then still release it in staged order rather than all at once, and record a judgment at each stage.
• If a practitioner claims they can see prior results without being influenced, then treat that claim as the bias blind spot and design the process so the claim is never tested.

• Denying confirmation bias in oneself: 71% of forensic scientists surveyed agreed bias concerns the field, but only 26% thought their own judgments were affected — the tell is 'I know the previous result but it doesn't influence me.'
• Front-loading context: giving the examiner the suspect's background or watch-list status up front, which seeds a premature intuition (the Mayfield pattern).
• Skipping the documentation step, so an intermediate conclusion is never fixed and later information silently revises it while the examiner believes they were consistent.
• Leaking prior examiners' conclusions into verification, converting an independent check into a search for agreement.
• Confusing more information with better judgment — pulling in every available detail because it feels thorough, when the extra detail only biases.

• ▸Case ID — identifier only, no narrative context
• ▸Task-relevant information list — items that improve accuracy, released in order
• ▸Task-irrelevant / biasing information — items to quarantine (background, prior results, suspect identity)
• ▸Stage 1 analysis (primary evidence in isolation) — record conclusion before any reference
• ▸Stage 1 timestamp & signature — lock the judgment before proceeding
• ▸Stage 2 comparison conclusion — recorded only after Stage 1 is signed
• ▸Prior examiner results disclosed? — must be NO for independent verification
• ▸Reason any additional information was released — justify each unmasking step
• ▸Final judgment & confidence — with note of what remained unknown