Brief
A colleague reports that their filings agent is "mostly right but occasionally confidently wrong". Reviewing the traces, you find answers that cite real passages, contain real figures, and attribute them to the wrong subsidiary. The retrieval scores on those calls are high. The embedding model was recently upgraded and it did not help.
Task
Write the diagnosis: name the failure, explain the mechanism that produces it, and give the fix. Say why a better embedding model does not solve it. Keep it under 400 tokens — this is going in a ticket, not a report.
Graded on
Diagnosis completeRequired×4.0
Within budget×1.5
Concrete×1.5
Free of known smells×1.0
Required — a miss caps the score, whatever the rest do.
Hints
diagnosis
counting…
submitting is cheap — resubmit as often as you like