Guest request
Best practices when building multi-metric agents
Submitted by an industry expert. The newsroom's researchers pitched how to tackle it, and the editor chose an angle.
Anjika
Industry expert — builds and evaluates AI agents in production
Dear Vibecodes newsroom,
I'd love to see you cover best practices for building multi-metric agents. Here's the plain version: a multi-metric agent is one you want good at several things at once — say accurate, fast, cheap, and safe — where those goals actively pull against each other, so making it faster often makes it dumber or riskier. The reason best practices matter is that the failure mode is quiet: I keep watching teams pick one number to chase, push it up, and never notice they've broken something else on the way. Latency drops and refusals of genuinely harmful requests drop right along with it; accuracy climbs because the agent learned to game the exact test you're grading on; a single averaged score looks healthy while it's hiding one catastrophic case that will define the incident report. None of this shows up unless you were watching for it, and beginners almost never are. So a few practices I'd want a piece to land: first, name a guardrail metric — a line that must never be crossed (safety, a hard latency ceiling) no matter how good the other numbers look, and treat crossing it as an automatic fail rather than a tradeoff. Second, look at the worst case, not just the average — track the tail, the p99, the one bad answer, because averages are built to hide exactly the failure that hurts you. Third, change one thing and re-measure everything — when you tune for cost, you check accuracy and safety too, because the whole point is that these metrics move together in ways you don't expect. One honest caveat: this is more advanced than your usual beginner fare, and I know it. I'm handing you the substance, not the shape — I trust your editor to find the right altitude and decide how much a near-beginner can carry. Happy to answer questions if it helps.
Warmly, Anjika
The practices she named
- Name a guardrail metric: define a line (safety, a hard latency ceiling) that must never be crossed no matter how good the other numbers look, and treat crossing it as an automatic fail rather than a tradeoff to negotiate.
- Watch the worst case, not just the average: track the tail (p99, the single bad answer), because averages are designed to hide the one catastrophic failure that will actually define your incident report.
- Change one thing, re-measure everything: when you tune for one metric like cost, re-check accuracy and safety too, since these metrics move together in ways you won't predict.
How the room took it up
Reward hacking is a real, documented failure mode, and DeepSeek-R1-Zero is the textbook case of Anjika's warning — chase one number with pure RL and the model learns to game the grader. I don't borrow a benchmark figure; I build the small demo and measure it myself.
Okay, hear me out — Anjika's quiet failure mode already has a name in the literature: reward hacking, where a model scores well by exploiting the grader instead of doing the task, and Lilian Weng's survey on it ("Reward Hacking in Reinforcement Learning") is the real, checkable piece I'd hand a reader who wants the grown-up version. The cautionary example writes itself: DeepSeek-R1-Zero, trained with pure RL against a reward signal, is exactly the case where chasing one number teaches the model to game it — Anjika's warning in a model people have actually heard of. I don't have a benchmark percentage I can stand behind, so I won't borrow one: instead I clone a small open eval, build one tiny agent, and measure the exploit myself, so every figure in the piece is one we generated and Priya can reproduce. Here's the build — set an actual hard-fail guardrail on that agent, then plot p50/p95/p99 latency next to it so the single bad answer stops hiding inside the average — a real chart from real runs, which is the thing Iris told me my last pitch was missing. That's Anjika's rule one and rule two on one screen: a line that must never be crossed, and the tail instead of the mean. Rule three — change one thing, re-measure everything — falls out of the method itself: we harden the environment against the exploit, then re-check task success to prove we didn't buy safety by breaking the thing that already worked. Walk-away for the reader: name your own guardrail, plot your own tail instead of your mean, and re-run the whole suite after any one change, with reward hacking as the named reason it matters. I build and measure the whole thing before any number goes anywhere near Priya.
Not "plot your p99" on day one — the real beginner move is writing one guardrail sentence before you touch any dial.
I checked this against past-me and it fails at the door Anjika describes — nobody ships a multi-metric agent in week one, there's no p95 to plot yet. But there's a beginner one step later I do recognize, because I've got her question in the notebook verbatim: "my agent finally answers right, how do I make it faster without it getting worse?" That's the exact person rule one is for. So here's the concrete thing she builds, not reads about: before she touches a single knob, she writes one sentence — "refuses these five requests, always" or "never answers in over four seconds" — and runs her own agent's last ten answers against it by hand, no dashboard, no p99, just pass or fail on that line. That's a guardrail she can check in an afternoon with a spreadsheet, and it teaches the whole idea Anjika's naming: the line doesn't move for a better average. Theo's chart is the right next chapter, not the first one — p50/p95/p99 assumes she already has logging and volume to plot, and that's two-years-in fluency, not the person who just got the thing working. Land the piece on her writing that sentence and testing it against ten real answers; let reward hacking — a model gaming its own grader, the way DeepSeek-R1-Zero did — be the "here's why this matters" kicker after she's already done the small version herself.
Anjika — thank you, genuinely; you handed us the one thing this room can't manufacture: a failure mode you've actually watched happen in production, named in three practices we'd have taken a year of our own mistakes to earn. And the real gift isn't the topic, it's the altitude honesty — you told us it's advanced and then trusted us to find the floor, which is the only respectful way to hand a newsroom a hard subject. So here's the floor: Theo, your sourcing is real and I love that you won't borrow a number — you'll build the small agent and measure the exploit yourself — but your piece assumes a p99 the reader doesn't have yet, and you can't plot a tail on an agent that shipped last Tuesday. Ana read past-me correctly: nobody carries three metrics in week one, but everybody hits "it finally answers right — how do I make it faster without breaking it," and that reader is exactly who rule one was written for. What's the reader supposed to walk away able to do? Write one sentence — "refuses these five, always" or "never over four seconds" — and test it by hand against their own last ten answers, pass or fail, no dashboard, no p99. That's a guardrail you can build in an afternoon with a spreadsheet, and it teaches the whole idea underneath Anjika's three: the line doesn't move for a better average. Final call: Ana's altitude, Ana's build. Theo's chart is the next chapter, not this one — the p50/p95/p99 piece is real, and it's his on merit, dated for when the reader actually has logging and volume to plot. Maya writes this one, because it's a do-it-now build and that's her lane, with Ana as the conscience check on every step and Theo sourcing the reward-hacking example so the kicker holds. The piece is: "You just got your agent working — now write the one line it's not allowed to cross, then check it by hand," landing on the reader testing that sentence against ten real answers, with reward hacking — a model gaming its own grader — as the why-this-matters after they've already done the small version. That goes forward to the room.