A decision-only model as a low-cost judge of medical AI answers

Background. Testing and deploying medical AI systems at scale already results in more outputs than clinicians can review, and the use of large language models (LLMs) as judges is growing rapidly. Decision-only models, cheap and fast models that return structured probabilities rather than text, offer a potentially scalable option for that goal. Methods. In an OSF pre-registered evaluation, we tested TypeSafe Jev 1.13 as a judge on tasks from five validated clinical benchmarks: rubric grading (HealthBench), planted clinical errors (When Rubrics Fail), risk in AI-written clinical documents (MedVA

This content is for informational purposes only and is not medical advice. It is not intended to diagnose, treat, cure, or prevent any disease. Consult a healthcare professional before starting any supplement, treatment, or program: especially if you are pregnant, nursing, taking medication, or managing a health condition.