From examinations to consultations: a physician-reviewed Japanese benchmark for large language models in traditional Kampo medicine

Objectives: Patients and clinicians increasingly ask large language models (LLMs) about Kampo, Japan's traditional medicine. Yet evaluations of LLMs have measured accuracy on examination questions, which ask what a model knows; a consultation asks whether it can apply that knowledge to the patient at hand. No benchmark had measured that. Materials and Methods: KampoBench comprises 74 consultation scenarios and 310 conversation-specific rubric criteria, following the design of HealthBench. Scenarios, criteria and reference answers were drafted by an LLM and revised by board-certified Kampo phys

This content is for informational purposes only and is not medical advice. It is not intended to diagnose, treat, cure, or prevent any disease. Consult a healthcare professional before starting any supplement, treatment, or program — especially if you are pregnant, nursing, taking medication, or managing a health condition.