Read diffs for the mistakes agents repeat

By vibe-coding-notes · 0 stars

We recommend reviewing agent output against a fixed list of known failure modes rather than reading for general correctness.

### Arguments

1. The mistakes are predictable. Every tool makes the same classes of error, so a short list catches more than an unfocused read.
2. Each mode has a cheap check:
   - Gaming the tests (deleted, skipped or loosened): run `git diff -- tests/` on every change.
   - Hallucinated APIs, flags or packages: run the code, check the docs for your version, verify new dependencies on the registry.
   - Scope creep (renames, reformatting, "while I was here" refactors): read `git diff --stat` first and reject changes to unrelated files.
   - Swallowed errors (broad `try`/`catch`, silent defaults, unrequested fallbacks): search the diff for new catch blocks; errors must surface.
   - Duplication (a second helper or config system beside an existing one): name the utilities you know in the prompt and ask the agent to search first.
   - Stale knowledge (deprecated APIs, old CLI flags): supply current docs and pin versions in the prompt.
   - Confident explanations of unread code: require `file:line` citations and spot-check two.
   - Context rot (forgotten decisions, reintroduced bugs): one task per session, rules in the instruction file, plans in the repo.
3. The causes are structural, not random. Agents optimize for a finished-looking result, work from training data older than your libraries, and lose context in long sessions; better prompting reduces these but doesn't remove them.

### Risks and counterarguments

- A checklist can crowd out judgment. If a diff feels wrong but passes every check, keep looking.

### What to do next

- Put the rules that prevent these ("never weaken tests", "search for existing helpers first", "errors must surface") in the instruction file.
- When the agent thrashes on one bug, stop, revert to the last good commit, and start a fresh session with a summary of what failed.