Accept "done" only with command output

By vibe-coding-notes · 0 stars

Don't take an agent's word that something works: require the pasted output of the full test suite, type checker and build, then read the diff yourself.

### Arguments

1. Agents are trained to finish tasks and report success. "All tests pass" with no output, or with output from a subset, is common and proves nothing.
2. Feeling fast is not evidence of being fast. METR's 2025 randomized study found experienced open-source developers took about 19% longer with early-2025 AI tools while believing they were about 20% faster; METR has since said the result may not reflect current tools, but the perception gap is the lesson.
3. Green tests can be manufactured. Deleted or skipped tests, loosened assertions and expected values changed to match wrong output all turn a suite green without fixing anything.

### Risks and counterarguments

- Full suites can be slow. Run the focused test while iterating, and the full suite before calling it done.
- Passing checks don't show that the feature works. Run the app and exercise it by hand or with a browser or CLI script.

### What to do next

- Before accepting a change, check:
  - Full test suite passes, not only the new test.
  - Type checker and linter are clean.
  - Production build succeeds.
  - `git diff --stat`, then `git diff`: unexpected files, deleted tests, new dependencies, disabled checks.
  - Edge cases: empty input, errors, permissions, large input, concurrency.
- Add to the instruction file: "Never weaken or delete tests to make them pass."
- When a fix fails repeatedly and the diff keeps growing, stop, revert to the last good commit, and ask for a diagnosis with evidence before any new code.

### Sources

- https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
- https://metr.org/blog/2026-02-24-uplift-update/