Agent Quality Audit and Evals Setup
For startups with an LLM agent in production. Two weeks, fixed price, a regression suite left in your repo.
The problem
Your agent is live. The question is no longer "does it work" but "how often is it wrong, why, and did yesterday's prompt change make it worse". Most seed to Series B teams cannot answer that with the tooling they have.
What it looks like in production:
- Hallucinated or off-policy answers reach customers before anyone on the team sees them
- Tool calls go wrong or get skipped: lookups, bookings and handoffs fail silently
- Every prompt, model or tool change ships without a regression check, so users find the regressions
- Cost and latency drift up with no baseline
- Nobody can reconstruct what happened in a given conversation from the traces
Who this is for: founders, CTOs and Heads of Product with an LLM agent in production (support, voice, healthcare, sales), a small team shipping weekly.
What you get
In two weeks: a regression suite running on your stack and a ranked list of what to fix first. Five deliverables:
- Architecture, prompt and conversation-flow review: where users get stuck, what fails across prompts, tool definitions, routing, escalation and guardrails, and which failures need a deterministic rule or an automation rather than a prompt. Findings ranked by customer impact, with a concrete fix per finding.
- Golden scenario set: 50 to 100 reference conversations covering your top intents, edge cases and known failures, each with the expected outcome and tool calls.
- Eval harness and regression run: the golden set plus scenario simulations, run against any prompt, model or tool change. Pass/fail per scenario, tool-call validation, cost and latency per run. Lives in your repo.
- Observability checklist: what you have and what is missing across traces, prompt/model/tool versioning, feature flags, controlled rollout, latency and cost monitoring, in the order to add it.
- 90-day quality roadmap: prioritized fixes with the economics of each (cost per resolved conversation, deflection, staff hours), the quality metrics to track weekly, and an owner for each.
Format: three ways to work
| Option | What happens | Time | Fits when |
|---|---|---|---|
| 1. Fixed audit | All five deliverables above | 2 weeks | You want a clear picture and a running regression suite fast |
| 2. Fractional AI PM (agent quality lead) | Audit first, then I own agent quality and the agent roadmap: evals, incidents turned into tests, every release reviewed, fixes prioritized with your engineers, and a clear view of where automation pays and what to measure | 10 to 20 h/week, 2 to 3 months | You need someone to own agent quality for a set period |
| 3. Full-time | Conversion from option 2 | By agreement | We have worked together and it fits both sides |
How it runs: remote from Spain (CET) with overlap for EU hours and US East mornings, async in your Slack, one or two syncs per week. From your side: an NDA, read access to prompts, tool definitions, traces and samples of conversations (de-identified where PHI is involved), and 2 to 4 hours a week of one engineer.
Pricing and terms
| Option | Price | Payment |
|---|---|---|
| Fixed audit, 2 weeks | €4,500 fixed | 50% on signing, 50% on delivery |
| Fractional, 10 to 20 h/week | €100 per hour, booked as weekly blocks of 10 or 20 hours (€1,000 or €2,000 per week), minimum 4 weeks | Weekly invoice; first week upfront, then net 7 |
Full-time conversion: compensation discussed separately. Prices exclude VAT. Invoiced from Spain (EU) under a contractor agreement. Work starts after the signed agreement and the first payment.
Proof: three results
- MEDvidi, US telehealth (2025 to 2026). Built the patient communication layer from scratch: AI agents in SMS, voice and live chat on the website, integrated with patient workflows, lead management and support operations, on LangGraph with tool calling, structured outputs and deterministic business rules in the backend. 40% of SMS sessions resolved by the agent without staff; patient response time down from about 30 minutes to under 1 minute. The quality practice behind it: thousands of scenario-level simulations across hundreds of curated cases, golden conversations, regression and tool-call validation on every change, an incident-to-test loop, traces, prompt/model/tool versioning, feature flags and controlled rollout. PHI-aware design with safe escalation and human handoff.
- MTS AI, telecom (2023 to 2025). Led the first production launch of an LLM in customer-facing conversational AI in MTS Group: automation up 10% with user satisfaction held. Owned the product roadmap and the quality metrics for text and voice assistants handling tens of thousands of customer requests a day across telecom service scenarios: 20+ custom production models for classification, sentiment and satisfaction prediction, comprehension up 7% in voice and 24% in text, a 50-person cross-functional team.
- Alfa-Bank, banking (2019 to 2023). Developed scenarios and the bank's conversational AI platform: platform migration with backward compatibility, NLU quality, pre-release model evaluation (F1, accuracy, recall, scenario-level metrics) and release validation owned on the platform side, personalization and assistant UI (buttons, tooltips). Predictive conversation flows raised automation by 20% in key scenarios; Alfa-Bank's first #1 in the Markswebb bank chatbot ranking (2022).
How to start
- Email connect@moroz.im or message me on LinkedIn with three lines: what the agent does, what breaks, your stack. I reply within one business day with slots for a 30-minute discovery call.
- On the call I ask about the agent, its failure modes and your tooling; you get my first read and a clear yes or no on fit. No deck, no charge.
- Within two business days you get a one-page proposal with the option that fits and a start date. Earliest start: the week after signing.