Background: Machine learning (ML) models have become valuable tools for detecting radiographic lesions. Binary classification models, providing outputs for single pathologies, have proven effective in specific scenarios, such as fracture detection in primary care. However, to serve as effective tools for clinical decision support or clinical trials in rheumatic diseases, ML models must be capable of detecting and grading potential differential diagnoses.

Objectives: Previously, we developed separate efficient ML models for detecting osteoarthritis (OA) and features of calcium pyrophosphate deposition (CPPD) disease in hand radiographs [1, 2]. This study aimed to train models for OA, CPPD, and rheumatoid arthritis (RA) detection in a unified environment, coupled with a web application designed for the combined detection and grading of these lesions on large datasets.

Methods: Segmentation and classification models for hand radiographs were trained in the Azure AutoML environment using various architectures, including convolutional neural networks and vision transformers (ViTs). OA and RA models were trained using hand radiographs from the Swiss Clinical Quality Management (SCQM) registry, while CPPD images were collected at a single center (CHUV). All scripts were written in Python using Azure SDK v2. A segmentation model based on YOLOv5 was developed to identify the distal interphalangeal (DIP), proximal interphalangeal (PIP), and metacarpophalangeal (MCP) joints, as well as the triangular fibrocartilage complex (TFCC). This model achieved a mean average precision of 0.99, with a threshold of 0.6 successfully identifying nearly all joints in the validation dataset. For OA , a classification model was trained to predict Kellgren-Lawrence (KL) scores (0–4) using 14,963 DIP joint images. For CPPD , a binary prediction model was trained to detect lesions in 829 TFCC images and 1,775 MCP2/3 images. For RA , two distinct models were developed to predict the Ratingen score (0–5) for MCP and PIP joints using 10,266 MCP and 10,239 PIP images. A web application was created using Python’s Flask library.

Results: ViTs outperformed other architectures for detecting OA, CPPD, and RA. For DIP-OA, the best-performing model was a ViT pretrained on ImageNet-21k, fine-tuned at a resolution of 224×224 pixels. It achieved an AUROC of 0.88 and an AUPR of 0.61 with an accuracy of 0.69 for predicting KL scores. For CPPD, a transformer encoder pretrained on ImageNet-21k achieved an AUROC of 0.96 and an AUPR of 0.95 for detecting CPPD lesions in TFCC and an AUROC of 0.88 and an AUPR of 0.78 for detecting CPPD lesions in MCP2/3 joints. For RA, the MCP model predicting the Ratingen score achieved an AUROC of 0.93 and an AUPR of 0.62 whereas the PIP model achieved an AUROC of 0.93 and an AUPR of 0.49. The web application consists of two pages: an upload page allowing users to upload a single-hand x-ray in DICOM format. The application will handle the calls to the models and display the results to the user. Results will include the x-ray with segmented joints and scores for DIP-OA (KL 0–4), CPPD detection in TFCC/MCP2/3 (yes/no), and RA Ratingen scores (0–5) for MCP and PIP joints (Figure 1). Batch processing of DICOM images will be possible outside the user interface.

Conclusion: This study presents the first user interface combining detection and grading of three distinct rheumatic lesions (OA, CPPD, and RA) using ViTs, which achieved acceptable accuracy for each model. The cloud-based environment enables the analysis of large datasets, supporting applications in clinical trials, electronic medical records, and clinical decision support. Future work will focus on validating the models with external datasets and expanding their capabilities to include carpometacarpal and PIP OA, as well as psoriatic arthritis lesions.

REFERENCES: [1] Hügle T, Rosoux E, Fahrni G, Markham D, Manigold T, Becce F. Development of a deep learning model for automated detection of calcium pyrophosphate deposition in hand radiographs. Front Med (Lausanne). 2024.

[2] Caratsch L, Lechtenboehmer C, Caorsi M, Oung K, Zanchi F, Aleman Y, Walker UA, Omoumi P, Hügle T. Detection and Grading of Radiographic Hand Osteoarthritis Using an Automated Machine Learning Platform. ACR Open Rheumatol. 2024.

Figure 1.

Example of a user interface for uploading DICOM files of hand radiographs. The integrated model segments DIP, PIP, and MCP joints, as well as the TFCC. Subsequently, models utilizing Vision Transformers detect significant DIP osteoarthritis (Kellgren-Lawrence grade 2) in both hands and CPPD in MCP2 of the left hand (enlarged image detail), but no significant lesions indicative of rheumatoid arthritis.

Acknowledgements: We thank the Swiss Clinical Quality Management (SCQM) Foundation for their support and express our gratitude to all participating centers for their efforts in data collection.

Disclosure of Interests: Thomas Hügle Abbvie, GSK, Lilly, Werfen, Novartis, Fresenius Kabi, Atreon, Vtuls, Novartis, Lilly, Fresenius Kabi, Deborah Markham: None declared, Christian Lechtenboehmer: None declared, Leo Caratsch: None declared, Elisabeth Rosoux: None declared, Ulrich A. Walker: None declared, Fabio Becce: None declared, Tobias Manigold Hoffmann LaRoche, Janssen, Novartis, Lilly, Hoffmann LaRoche.

© The Authors 2025. This abstract is an open access article published in Annals of Rheumatic Diseases under the CC BY-NC-ND license ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ). Neither EULAR nor the publisher make any representation as to the accuracy of the content. The authors are solely responsible for the content in their abstract including accuracy of the facts, statements, results, conclusion, citing resources etc.