Background: Early diagnosis of Behçet’s syndrome (BS) remains challenging due to heterogeneous and evolving clinical presentations and the absence of a definitive diagnostic test. Classification criteria such as the International Study Group (ISG) and International Criteria for Behçet’s Disease (ICBD) may lack sensitivity at early or atypical stages, contributing to diagnostic delay. Artificial intelligence (AI), particularly large language models (LLMs), may assist clinicians by integrating complex unstructured clinical data to support earlier recognition of rare multisystem diseases including BS.

Objectives: To evaluate the diagnostic accuracy of a large language model (ChatGPT-5) in identifying Behçet’s syndrome based solely on first rheumatology visit documentation, and to assess the impact of disease-specific knowledge-augmented prompting on diagnostic performance.

Methods: We conducted a retrospective study of 60 treatment-naïve patients evaluated at a tertiary rheumatology clinic (2020–2024), including 29 patients with confirmed BS and 31 disease controls (rheumatoid arthritis, psoriatic arthritis, axial spondyloarthritis, systemic lupus erythematosus). All patients had no prior rheumatic diagnosis and completed ≥12 months of follow-up. Free-text clinical documentation from the initial visit was anonymized and analyzed in its original language. ChatGPT-5 was evaluated using a two-phase design: (1) baseline assessment and (2) reassessment after BS-specific knowledge-augmented prompting incorporating classification criteria, early and atypical manifestations, epidemiological context, and key differential features (Figure 1). For each case, the model generated the three most likely differential diagnoses. Diagnostic performance was compared with the final clinical diagnosis. Paired comparisons between phases were assessed using McNemar’s test.

Results: Among patients subsequently diagnosed with BS, 51.7% fulfilled ICBD criteria and 41.4% fulfilled ISG criteria at the first visit, while 48.3% met neither (Table 1). At baseline, ChatGPT-5 identified BS among the top three differential diagnoses in 82.8% of cases, increasing to 96.6% following knowledge-augmented prompting, with false negatives decreasing from five to one. The correct diagnosis was ranked first in 72.4% of cases at baseline versus 93.1% after knowledge-augmented prompting (p=0.041). At the first visit, the treating rheumatologist diagnosed BS in 65.5% and suspected BS in an additional 24.1% of cases. False-positive BS predictions among controls remained low (3.2% at baseline vs. 6.5% after prompting; p=1.00).

Conclusions: A large language model accurately identified BS from initial rheumatology visit notes and demonstrated significantly improved diagnostic ranking following disease-specific knowledge-augmented prompting without compromising specificity. These findings highlight the potential role of AI as a diagnostic adjunct to support early recognition of BS, particularly in patients with incomplete or atypical presentations. Larger, multicenter prospective studies are warranted to validate these results and define safe clinical integration.

Figure 1.

REFERENCES: NIL.

Acknowledgments: NIL.

Disclosure of Interests: None declared.