Debugging is a method, not luck
Strong engineers debug the same disciplined way every time — a loop, not random poking:
- Reproduce it reliably. A bug you can't reproduce, you can't fix or verify. Find the exact input/conditions that trigger it.
- Observe — gather evidence: the error/stack trace, logs, the actual vs expected output, what changed recently. Read the error message carefully — it usually points at the problem.
- Hypothesize — form a specific, testable theory ("the query embedding is using the wrong model"), guided by the evidence.
- Isolate — narrow it down: bisect (comment out / binary-search the code path), add targeted logs, check inputs at boundaries. Cut the search space in half each step.
- Fix — make the smallest change that addresses the root cause, not the symptom.
- Verify — confirm the fix resolves the reproduction and didn't break anything (run tests). Add a regression test so it can't come back.
Key principles
- Change one thing at a time — otherwise you don't know what fixed (or broke) it.
- Question assumptions — the bug is usually where you're sure it isn't. "It can't be X" is often exactly where it is.
- Read the error message — engineers waste hours ignoring the message that names the problem.
- Fix the root cause, not the symptom — patching the symptom (swallow the error, special-case the input) leaves the real bug to resurface.
- Binary search everything — the fastest way to localize a bug (or a bad commit via
git bisect) is halving the search space.
For LLM/data bugs
Non-determinism makes reproduction harder — pin the seed/temperature where possible, log the exact prompt + inputs, and use traces (the eval/observability tools) to see what the model actually saw. Many "model" bugs are really deterministic plumbing bugs (wrong prompt assembly, lost state) hiding behind randomness.