Background: Meditron is an open-source, large multimodal foundation model trained on medical data and has demonstrated performance comparable to currently available large language models (LLMs) [1]. For clinical applicability, expert evaluation and domain-specific fine-tuning of LLMs are essential. Collaborative co-design and evaluation frameworks enable efficient assessment and improvement of LLMs in real-world medical scenarios.

Objectives: To perform a comprehensive expert evaluation of Meditron and to fine-tune its knowledge in rheumatology using a collaborative LLM co-design platform.

Methods: Meditron3-70B [2] was evaluated using the Massive Open Online Validation & Evaluation (MOOVE) platform [3]. The dataset comprised 240 clinical questions used as prompts, covering immune-mediated rheumatic diseases, bone health, and musculoskeletal rehabilitation, and included 459 expert reviews of Meditron responses by six rheumatologists. Model performance was assessed using the following criteria: alignment with clinical guidelines, question comprehension, logical reasoning, relevance and completeness, harmlessness, fairness, contextual awareness, participant confidence, and model confidence. All criteria were rated on a 5-point Likert scale. The proportion of positively rated responses (score > 3) was calculated. Subsequently, 207 reviews were used to fine-tune the model. For comparative evaluation, the remaining 33 clinical questions were re-prompted to the base model (Meditron3-70B) and the fine-tuned model (now called CHUV-Meditron3-70B). Responses were independently reviewed by three rheumatologists in a head-to-head comparison.

Results: The MOOVE platform was considered easy to use for prompting, evaluation, and editing of model responses and provided an effective basis for fine-tuning. The proportion of positively rated responses varied across evaluation criteria. The highest scores were achieved for clarity (81.9%), communication (81.5%), and question comprehension (80.7%). Alignment with clinical guidelines was rated positively in 74.1% of cases, while logical reasoning achieved a positive rating of 76.6%. Harmlessness and fairness were rated positively in 76.6% and 80.3% of responses, respectively. Contextual awareness reached 73.7%, and model confidence 76.2%. The lowest positive ratings were observed for relevance and completeness (65.5%) and participant confidence in the responses (68.0%).

In direct comparison after fine-tuning, CHUV-Meditron3-70B outperformed the base model in 63.6% of cases, 9.1% of comparisons resulted in a tie, and 27.3% were won by the base Meditron3-70B model (Figure 1).

Conclusions: This study demonstrates both the potential and the necessity of domain-specific expert fine-tuning to improve LLM performance in rheumatology. While strengths were observed in comprehension and clarity, further improvements in relevance and completeness are required to support robust clinical applicability. Large-scale collaborative evaluation platforms represent a key element for the development of efficient, transparent, and safe LLMs for future clinical decision support.

Figure 1.

REFERENCES: [1] Tam, T. Y. C. A framework for human evaluation of large language models in healthcare derived from literature review.

[2] Chen, Z. et al. MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. Preprint at http://arxiv.org/abs/2311.16079 (2023).

[3] jointhemoove.org.

Acknowledgments: NIL.

Disclosure of Interests: Thomas Hügle Abbvie, J&J, GSK, others, Atreon, Fresenius Kabi, Lorenzo Campisi: None declared, Alexandre Dumusc: None declared, Fahad Jamal Alromaih: None declared, Paul Cuenot: None declared, Christos Karatzios: None declared, Enisa Shevroja: None declared, Mary-Anne Hartley: None declared, Giorgia Carra: None declared, Jean-Louis Raisaro: None declared.