Not model quality — the plumbing around it. Retries that charge twice, provider calls with no deadline, agent loops with no ceiling, spend nobody capped, and prompt changes that quietly degrade output until a customer notices.

Same failure classes as any distributed system, wearing an AI hat.

10 ways your AI feature breaks in production → The full checklist I audit against. Free, no signup.

changedetection.io (33.6k★) — #4390, merged into master by the project owner. Their history path-containment guard had no test: disabling it left every existing test passing, because os.path.basename() neutralised both traversal fixtures before they reached it. The new test is the only one that fails when the guard is removed.

Also reviewed the LLM client's retry handling in #4383; a contributor implemented the findings in #4384, open and under the owner's review. My regression tests were merged into that PR's branch as commit 1f0ed3f; the owner's review called the attempt -= 1 refund test "a nice catch, it pins a behaviour nothing else covered." And #4385: the AI intent filter fails closed on a malformed model response while every other failure path in the same function fails open.

stripe-fulfilment-kit — fenced, leased order claims that survive retries, crashes and concurrent webhook deliveries. Cross-process atomic via SQLite. 59 tests, all offline.

ai-review-gate — two models review a coding agent's output and block it before it ships. Includes an honest record of the eight real errors it caught in one session.

cardamom-quant — quantitative model, 37-trial ledger.

A lease that passed 78 tests and was still wrong. A provider call that outlived its two-minute lease wrote its failure over a completed order — hiding a product the customer had paid for, and re-opening the order for a third charge.

Every test settled an order from its current owner. None covered a write from a superseded one, which is precisely the state a lease exists to create.

Full write-up — how 78 tests missed it →

Available for contract work · gg5533.github.io · shabbal1@charlotte.edu