Testing That Actually Catches Bugs: Beyond Coverage Theater

Here is a number that makes engineering teams feel safe: 90% code coverage.

Here is what that number actually means: 90% of your code lines were executed during test runs. Executed. Not validated. Not verified. Not proven correct. Just... touched.

You can hit 90% coverage with a test suite that catches absolutely nothing. I have seen it. I have done it. And I have shipped bugs to production with a green coverage badge smiling at me from the CI dashboard.

The Three Test Layers

Every production system needs three layers of testing, each with a different job, different speed, and different cost.

The pyramid shape is intentional. You want many unit tests (fast, cheap, run on every save), fewer integration tests (medium cost, verify contracts between components), and a small number of E2E tests (expensive, slow, but prove the whole system works).

An inverted pyramid — mostly E2E tests — is a system that takes 45 minutes to run CI and breaks every time a CSS class changes. We ran an inverted pyramid on an early version of Tesseract Intelligence and the CI run time hit 12 minutes. We restructured to proper pyramid shape and dropped it to under 4 minutes with better coverage.

Coverage Theater vs Quality Testing

This is the distinction that separates test suites that earn confidence from test suites that earn a badge.

Both tests in the diagram above contribute identical coverage numbers. One is theater. The other is engineering.

The theater test executes the function and checks that it returned something. The quality test validates specific behaviors: correct email, valid ID, proper timestamp, actual database persistence. When the system breaks, the quality test will fail with a specific message pointing to the exact regression. The theater test will still pass because the broken function still returns "not None."

We enforce a minimum of 3 real assertions per test function across our projects. Not is not None. Real assertions: equality checks, range validations, state verifications. If a test function has fewer than 3 meaningful assertions, it is suspect.

The Three Paths

Every function has at least three behavioral paths. Most AI-generated tests only cover one.

The happy path is what AI generates by default. Ask for a test and you get: call the function with valid input, assert the expected output. Done. One path covered, two wide open.

The error path covers what happens when things go wrong. Network failures. Invalid input. Missing permissions. Database timeouts. These are the scenarios that cause 3am incidents, and they are almost never in default AI-generated tests.

The edge path covers boundary conditions. Empty strings. Maximum integer values. Concurrent duplicate requests. Exactly at the rate limit. These are the bugs that hide for months and then surface on the one day traffic spikes.

Test Naming as Documentation

A test named test_user_2 tells you nothing when it fails. A test named test_create_user_with_duplicate_email_returns_conflict_error tells you exactly what behavior broke.

The pattern:

test_[action]_[scenario]_[expected_result]

Real examples from our codebase:

• test_place_order_with_insufficient_balance_returns_balance_error
• test_parse_market_data_with_empty_response_returns_empty_list
• test_calculate_position_size_at_max_leverage_caps_at_limit

When this test fails in CI at 11pm, the name alone tells you what broke, under what conditions, and what the expected behavior should be. No reading the test body required. No context switching. The name IS the documentation.

The people reading your test failures need ideas (clear test names) before they need hardware (the test implementation). Names first. Always.

Anti-Patterns That Kill Test Suites

Coverage theater: Tests that execute code but assert nothing meaningful. assert result is not None is the canonical example. The coverage report says you are at 90%. Reality says you are catching 0% of regressions.

Test duplication: Five test functions that all test the same behavior with slightly different inputs. Use parameterized tests instead. Duplication inflates your test count without adding confidence.

Brittle tests: Tests that break when you rename a CSS class or change a log message. These tests are testing implementation details, not behavior. When they break, you fix the test instead of the code — which means the test is costing you time, not saving it.

Mock overuse: Mocking everything means testing nothing. We had a pagination bug in our InDecision engine where urljoin(base, path) silently dropped the base path when the path started with /. Every mocked test passed perfectly because the mock never exercised the actual URL construction. The real API returned page 1 forever. Mocks lie when they are used to avoid testing the hard parts.

Writing Tests That Earn Confidence

The formula is straightforward:

1. Three paths per function: Happy, error, edge. Minimum.
2. Specific assertions: What specific value? What specific error type? What specific state change?
3. Descriptive names: The name should tell you what broke without reading the body.
4. Proper pyramid: Many unit tests, fewer integration tests, few E2E tests.
5. Real dependencies where possible: Mocks for external services. Real code for internal logic.

This is the difference between a test suite that gives you confidence to ship on Friday afternoon and a test suite that gives you a green badge while production burns.

Lesson 154 Drill

1. Audit five tests in your codebase. For each, ask: if I intentionally broke the function this test covers, would this test catch it? If the answer is no, rewrite the test with specific assertions.
2. Pick one function and write tests for all three paths: happy, error, and edge. Count how many edge cases you discover that you had not considered.
3. Review your test names. Can you tell what broke from the name alone, without reading the test body? Rename any test that fails this standard.