05 · Software · AI · Sep 2026
AI reply agent
An agent that reads a prospect's reply, decides what they meant, drafts an answer in the client's voice, grades it, and either sends or holds it for a person.
The decision
Only code findings may block a send
On its first real lead, a dealer replied with one word: "YES". The draft was right and every coded check passed, but the model grader said it "did not answer the question" and held it. There was no question. Worse, the workspace needed three safe sends to earn trust, and the grader's opinion blocked every send, so it could never earn it. I answered the lead by hand, then changed the rule the same day: taste can advise, only coded checks can hold.[19] Sourceengineering ledger, reply agent go-live 2026-09-21As of2026-09-21StatusVerifiedAll sources →
What I built
- The pipeline: read, classify, draft, grade, then send or hold.
- Coded checks for harm, length, links and the ask, which always win over the model's opinion.
- Live experiments per conversation, with a nightly job that promotes the winning arm only when it clears z > 1.96 on reply rate.[52] Sourceengineering notes (LinkedIn reply tests): nightly job promotes an arm at z>1.96 on reply rateAs of2026-09-24StatusVerifiedAll sources →
- A golden test set of real replies every change must pass before it reaches a client, and a harm check that always holds, whatever the grader says.
- Per-client rules the agent reads at draft time: the sign-off, the booking mechanism (two times and a phone ask for dealers, a link for others), and what it may never promise.
What was hard
A gate that could never open
Readiness was earned by sending, and the gate blocked sending. Finding that meant reading one held run end to end instead of trusting the dashboard.
Bare assent
"Sure", "Yes" and "Interested" have no question to answer, so an axis that scores "did it answer the question" is false by construction. The code now decides whether anything was asked before the model is graded on it.
What I learned
- A gate earned by doing the thing it blocks is a deadlock.
- Let models advise and let code decide anything that is irreversible.
Results
- Auto-sending live for two automotive clients since 21 September 2026.[19] Sourceengineering ledger, reply agent go-live 2026-09-21As of2026-09-21StatusVerifiedAll sources →