Five agent failures that taught me where autonomy ends

Failure 1: the silent truncation

My article-writing agent produces long, structured JSON — full articles with metadata, sections and sources. One morning every single run failed with the same error: schema_validation_failed, "the response did not match the schema". So I did the obvious thing and went digging in the schema — loosened field limits, widened enums. It helped nothing, because the schema was never the problem. The runner had never set maxOutputTokens, and the API default output ceiling was 4,096 tokens. Every article was being cut off mid-JSON — and truncated JSON does not parse, which the validation layer dutifully reported as a schema mismatch.

The wrong mental model: I trusted the error message. An error message tells you where a failure was detected, not where it was caused. The clue that actually cracked the case was not in any log — it was in the cost telemetry. Every failed run cost almost exactly the same: $0.0590 or $0.0583, run after run. Randomly failing generations should have randomly varying costs; identical cost means every run is hitting the same hard ceiling. My billing data knew the root cause before my logs did.

The fix was small: every agent definition now declares its own output ceiling, and the runner passes it through to the model call — the article agent got a 16k-token ceiling, with a worst case still under its per-run cost cap. The transferable rules are bigger. Defaults are silent killers: every limit you did not set explicitly is a decision someone else made for your production system. And cost telemetry is a debugging signal — near-identical spend on every failure is the fingerprint of a ceiling, not of a flaky model.

Failure 2: the fake confirmation

My store checkout calls a payment API and, in demo mode — before Stripe is configured — falls back to a sample flow: generate an order reference, clear the cart, show the confirmation page. The bug: any API failure fell into the same branch. A 500, a network error, a misconfigured key — the customer would still get a freshly minted order id, an emptied cart and a cheerful "order confirmed". Nothing charged, no order recorded, nobody told.

The wrong mental model: I treated the happy path as the product and the failure path as plumbing. It is exactly backwards. The failure path is the product surface — the moment something goes wrong is precisely when the customer decides whether your business can be trusted. A pretty confirmation screen over a void is the worst possible answer.

The fix: the success branch now requires the response to be OK and the server to explicitly say which flow it is — a checkout URL for real payment, or mock === true for the declared demo flow. Anything else keeps the cart intact and shows an honest inline error with a retry. The rule: a fallback must never be reachable by accident. If a demo mode exists, the server has to declare it, actively — inferring it from the absence of a real answer means every outage impersonates the demo. Failing honest beats failing pretty.

Failure 3: the swallowed order

When a payment succeeds, Stripe fires a webhook, and my handler forwards the paid order to the fulfilment platform. That forwarding call sat in a try/catch whose catch block did nothing — and the handler returned 200 OK regardless. Which means: if forwarding failed, a customer had paid, and the order silently ceased to exist. No error, no queue, no trace outside Stripe’s own dashboard.

The wrong mental model: "the order is already captured in Stripe, so the forward is best-effort." I was treating at-least-once delivery as something Stripe provides, when it is in fact a contract with two sides. Stripe retries webhooks with backoff for days — but only if you tell it the truth. A 200 is a receipt. It says delivered, stop trying. Return it on a failure and you have converted a retryable fault into permanent, silent data loss.

The fix is almost embarrassingly small: a failed forward now returns a 5xx, and Stripe redelivers until it succeeds. The rule is not small: reliability is retry semantics, not uptime. Nobody notices your five nines; everybody notices the paid order that vanished. The whole machinery of at-least-once delivery only works if every hop refuses to lie to the retrier.

Failure 4: the guard that blocked its author

My development harness has a deterministic deny-hook: a short list of command patterns that are never legitimate and are therefore blocked before execution. One of them is the exfiltration signature — reading a credential file and talking to the network in the same command. Three days after I built it, it blocked me. I was dispatching a CI workflow, and my command read a token from a credential file and called the API on the same line: exactly the co-occurrence the guard exists to stop. My first reflex was to carve out an exception for myself. That reflex is the actual incident.

The wrong mental model: guards are for attackers, and the operator is trusted. But a guard you route around when it is inconvenient is not a guard — it is decoration. And with agents in the loop, "the operator" is a fuzzier category than it looks: prompt injection means every piece of text an agent reads is trying to become an operator. The defence only works if every external string is treated as data, never as instructions — and if the deny-rules at the tool boundary hold no matter who is asking, including the author with a deadline.

So the guard stayed. I split the work into two commands — pull the token into the environment first, call the API second — a shape that breaks the exfiltration signature. Thirty seconds of friction. The rule: agent platforms need deny-by-default seams precisely because the operator will be tempted too. Whether a rule survives its own author — that is the test of whether it is real.

Failure 5: the double print

Webhook redelivery — the thing that saved failure three — has a shadow: at-least-once sometimes means twice. A redelivered checkout event could create a second order in my print-on-demand pipeline, and a second print job at the vendor. One paid checkout, two hoodies printed, packed and shipped to one confused customer.

The wrong mental model: "I deduplicate at the entry point, so I am safe." Dedupe at one hop does not survive the pipeline — a retry can slip in at any seam, and application-level checks lose the race the moment two requests arrive at once. Idempotency is not a property of a function. It is a property of the whole chain, and it is never stronger than its weakest hop.

The fix threads one key through everything. The Stripe session id travels with the order as its externalId; the database enforces a partial unique index on (creator_id, external_id), so not even two racing inserts can both win; the loser’s constraint violation — Postgres 23505 — is converted into a calm "this order already exists" instead of an error; and the same id goes to the print vendor as the orderReferenceId, so even the last hop can refuse a duplicate. The rule: an idempotency key has to ride the whole pipeline, minted by the payment and honoured by every hop — and uniqueness is enforced in the database, because only the database sees both racers.

Where autonomy ends

Notice what the five have in common. Not one of them is the model "going rogue". No agent made a bad decision; the intelligence was fine and the plumbing lied. A default nobody set, a fallback reachable by accident, a status code that flattered failure, a rule I wanted to break myself, a key that stopped one hop short. Agent reliability, as I practise it today, is mostly boundary-drawing: for every action an agent can take, ask what happens when it fails halfway — and who notices.

The working rule I have converged on: an agent may finish alone what is reversible, observable and idempotent. Drafting content, staging changes, holding a bookable slot with race-safe uniqueness underneath — the same boundary I draw when I build booking systems for service businesses. The moment an action is irreversible, invisible or money-moving, the design duty changes: success must be declared explicitly, never inferred; failure must be loud all the way to both the retrier and the human; and duplicates must be structurally impossible, not merely unlikely.

What I still do not trust agents with

An honest list, as of today. An item leaves this list only when a specific mechanism makes its failure mode loud or reversible — not because the models got smarter.

• Moving money outward. An agent may confirm work triggered by a signed, idempotent payment event. It may not initiate payouts or refunds; a human approves each individual transaction.
• Speaking in my name to strangers. Agents draft; I send. An outbound email that is wrong costs trust no retry can redeliver.
• Customer data beyond its task. An agent gets the narrowest slice of data its job requires, within the same frames I cover in GDPR and AI around customer data — EU processing, minimal access, deletable.
• Its own guardrails. The deny-rules live outside the agent’s reach. A system that can edit its own limits has none.
• Anything unrestorable. Deletions, migrations without rollbacks, force-pushes. If history cannot bring it back, no agent finishes it alone.

I build and run this stack solo for Swedish businesses — the agents, the store, the guardrails and the on-call rotation, which is one person long. It is the same setup I describe in AI consultant versus AI agency: one accountable human, no dilution of responsibility. If your team is putting agents near money or customers, and you want to compare notes with someone who has already paid this tuition — get in touch.