Build a shared eval platform for many product teams

evals, observability, platform

Prompt

Several product teams at your company each run their own LLM-based feature (support chat, a coding assistant, an internal search bot) and each currently has its own ad hoc way of checking quality. Leadership wants one shared eval platform all teams use. Design it.

Interviewer's answers to clarifying questionsopen after you have asked your own

Scale? 6 teams, each shipping prompt or model changes weekly; combined traffic ~2M LLM calls/month across features.
Current state? Each team hand-checks a handful of examples before shipping; no shared definition of "quality," no CI gate.
Ask from leadership? A single dashboard, but teams' failure modes are different: support cares about policy correctness, the coding assistant about code correctness, search about retrieval relevance.
Data sensitivity? Some traces contain customer PII and must not leave an approved review tool.
Budget? Eval/judge spend should not exceed 15% of the combined primary-model spend.
Rollout need? Each team needs a CI gate that blocks a risky prompt or model change before it reaches production.

Follow-upsafter your high-level design

A team's LLM-judge-based CI gate has been green for 3 months while their production complaints rise. How do you diagnose and prevent this platform-wide?
Finance asks why the shared judge spend tripled this quarter while primary traffic only grew 20%. Walk through how you'd find out.
One team wants a trajectory-level check (did the agent call tools correctly) and another only needs final-answer correctness. How does a shared platform serve both without becoming a lowest-common-denominator blended score?
A new team wants to onboard next week with zero existing eval data. What's their path to a working CI gate, and how long does it realistically take?

Answer first: out loud, on paper, or with the interview prompt above. Then compare with the rubric.