Analysis & Data Reasoning Patterns

A dashboard says the number is up. A model says the strategy is working. A summary says the project is on track. Every one of those statements can be true and still be dangerously incomplete — because the number that's up might be masking a segment that's down, the "working" strategy might rest on an assumption nobody checked, and "on track" might flip to "off track" if a single input is off by 20%.

This lesson gives you five patterns for stress-testing a conclusion before you act on it — or hand it to someone who will. None of them are exotic. All of them are the difference between an analysis that survives scrutiny and one that only survives because nobody asked the next question.

Pattern 1: Assumption Audit

The pattern: Before accepting a conclusion, force an explicit list of every assumption it rests on — then test each one, not just list it.

Before I trust this conclusion, list every assumption it depends on
that isn't directly stated in the data. For each one, tell me: is this
assumption testable with data I have access to right now? If yes, what
would confirm or break it?

Conclusion: [your conclusion]
Supporting data: [your data]

When it works: You actually go test the assumptions the audit surfaces — checking the ones that are checkable, and flagging as "unverified" the ones that aren't.

When it fails: The audit becomes a checkbox — you generate the list, feel more rigorous for having it, and move on without testing a single item. An untested list of assumptions is not more trustworthy than no list at all; it just looks like it is.

Pattern 2: Base-Rate Anchor

The pattern: Before accepting a surprising claim, check what the naive, boring, "nothing unusual is happening" base rate looks like for a situation like this one.

This claim is surprising: [claim]. Before I treat it as a real
finding, what is the base rate — the normal, unremarkable outcome —
for [this type of situation]? Is the surprising number outside the
range that base rate would predict, or within normal variation for
something this new / this small / this early?

When it works: A reliable base rate is knowable and cheap to obtain — historical data for similar situations exists and applies.

When it fails: The base rate itself is contested, or you're pulling it from a mismatched reference class (comparing a B2B SaaS conversion rate to an e-commerce base rate, for instance). A wrong base rate produces false confidence in either direction — dismissing a real signal as noise, or treating noise as a real signal.

Pattern 3: Segment Breakdown

The pattern: Never trust an aggregate number on its own — ask for the breakdown by the segments that plausibly behave differently from each other.

This aggregate number is [X]. Break it down by [relevant segment
dimension — region, customer size, cohort, channel]. Does the trend
hold across every segment, or is the aggregate hiding one segment
moving in the opposite direction?

When it works: Segmentation catches a real masking effect — the classic case where an aggregate trend is flat or positive while a specific, important segment is quietly declining underneath it.

When it fails: Segments get sliced so fine that each one is too small to say anything reliable — a 12-person segment's rate swings wildly from noise, not signal. Segmenting past the point where a segment has enough sample size trades a real blind spot for a fake one.

Pattern 4: Sensitivity Check

The pattern: Before trusting a conclusion built on an uncertain input, recompute it with that input perturbed — typically 20% in the unfavorable direction — and see whether the conclusion holds or flips.

This conclusion depends on [input]. That input has genuine uncertainty
— it could plausibly be off by 20%. Recompute the conclusion assuming
the input is 20% less favorable than the point estimate. Does the
conclusion still hold, or does it flip?

When it works: The input genuinely has bounded uncertainty you can reason about — you don't know the exact number, but you have a defensible sense of the plausible range.

When it fails: Applied to an input whose uncertainty you don't actually understand — picking an arbitrary 20% band on a number that could really be off by 200%, or one you have no basis for bounding at all, produces a false sense that you've "stress-tested" the conclusion when you've really just moved the guess around.

Pattern 5: So-What Distillation

The pattern: Force every analysis to end in a recommended action, not just a description of what the data shows — the gap between "here's what's happening" and "here's what to do about it."

This analysis describes what's happening. Now distill it to a
so-what: given this data, what is the ONE action you'd recommend,
and what would have to be true for that action to be wrong?

When it works: There's a real decision the analysis is meant to feed — a resource allocation, a go/no-go, a prioritization call.

When it fails: Forced onto an analysis where the honest answer is "we don't know enough yet." Manufacturing a confident action from an inconclusive analysis is worse than admitting the analysis isn't done — it converts genuine uncertainty into false decisiveness.

Build It: The Sensitivity Checker

Sensitivity Check is the pattern most worth turning into code, because "recompute this under a stress scenario" is exactly the kind of mechanical, repeatable operation a small function should own instead of your memory.

Bottom Line

Five patterns, one job: don't let a conclusion survive on vibes when it's cheap to actually check it. Audit assumptions and test them, don't just list them. Anchor surprising claims to a real base rate. Break aggregates into segments before trusting them. Stress-test uncertain inputs with a sensitivity check. And when the analysis is done, distill it to a so-what — or admit it isn't done yet. Applied together, these patterns are the difference between a conclusion that sounds right and one that's actually been tested. Next: decision support — the patterns for turning an analysis into a call you can defend.