If the agent does not verify, it will lie politely
Observations beat vibes. Wire tests, builds, and browsers into the loop or you will ship confident garbage.
Agents are optimized to look finished. Your job is to make “finished” mean something mechanical.
Confidence is not evidence
A model can:
- invent an API that almost exists
- “fix” a test by deleting it
- claim green CI without running anything
- refactor names and call it a migration
None of that is malice. It is next-token helpfulness with no ground truth.
Verification ladder
From cheap to expensive — run as far as the risk demands:
| Level | Check | Catches |
|---|---|---|
| 0 | compile / tsc | nonsense symbols |
| 1 | unit tests for the change | local logic lies |
| 2 | integration / API smoke | wiring lies |
| 3 | browser / E2E path | UI and auth lies |
| 4 | prod-like canary | env and data lies |
Most agent sessions should not leave level 0–1 without a human. Level 3–4 for anything user-facing.
Put verify in the loop
Bad:
agent edits 40 files → you manually run tests later
Good:
edit → test → read failure → edit → test → stop when green
The observation is the product. Without it, every step is fiction with syntax highlighting.
Make “done” machine-readable
Tell the agent (and yourself) a stop condition:
DONE when:
- pnpm test passes
- pnpm build passes
- GET /health returns 200
- no uncommitted secrets
Vague goals (“improve the onboarding”) produce vague diffs. Agents thrash on open-ended taste work unless you constrain the output (screenshots, rubric, “do not change copy”).
Decision rule
Before you start an agent session, answer:
- What command proves this worked?
- What is the agent forbidden to do to fake that command?
- What is the rollback (branch, worktree, deploy)?
If you cannot answer (1), do not automate it yet. You are not ready for agency — you are still exploring.
get new posts
Occasional notes on agentic development. No spam.