Objectives: To develop a deep learning model for classifying ultrasound images of salivary glands (SG) [parotid (PG) and submandibular (SMG) based on the Outcome Measures in Rheumatology (OMERACT) 0-3 semiquantitative scoring system for B-mode ultrasound-assessed parenchymal abnormalities in SG in Sjögren disease (SjD)

Methods: For training, we used 3 datasets of SG (PG and SMG) ultrasound images representative of the OMERACT scores: 1. A dataset of 225 ultrasound images acquired with different ultrasound systems of 150 SjD patients across four European centers ( https://www.frontiersin.org/articles/10.339/fmed.2020.581248/full ; https://github.com/ArsoVukicevic/Assessment-of-pSS-from-SGUS-images ); 2. 80 anonymized ultrasound images of SjD patients obtained from the archive of one of the authors; 3. 1,896 pseudonymized ultrasound images of 57 SjD patients and 22 healthy controls, 6 images from each acquired by 2 experts with 2 systems (a high-end and a wireless handheld device), from a prospective cross-sectional study conducted in the rheumatology department of a European centre. We used a ResNet18 architecture, which was pre-trained on ImageNet and then fine-tuned for this study. The model was trained hierarchically: initially classifying broad categories (scores 0/1 vs. 2/3) and subsequently refining the classification for the specific grades of SG involvement. 70% of the images were used for training, 15% for validation, and 15% for test. For training, we used a dataset of ultrasound images along with labels indicating the grade of SG involvement. We employed standard deep learning techniques such as data augmentation (e.g., random flipping and brightness adjustments) to increase the model robustness and generalization capabilities and a step learning rate scheduler to optimize the training process. Precision (correct classifications per predicted class), recall (correct classifications per actual class), and F1-score (harmonic mean of precision and recall) were calculated for each grading class. To assess potential sources of variability in classification accuracy, we analyzed the error rate according to the operator and the acquisition device. Error rates were evaluated both globally (all grades) and separately for each image grade.

Results: The overall performance of the model for grading classes is shown in Table 1. The broad model achieved an accuracy of 88% in distinguishing between 0/1 and 2/3 grades. However, the more specific models, fine-tuned for the 0/1 and 2/3 categories, performed differently across categories: 72% accuracy for the 0/1 classification and 82% for the 2/3 classification. This reflects the increased difficulty of fine-grained classification. The confusion matrix (Figure 1) showed that the model performed well in classifying SG with grade 3 but faced challenges with intermediate categories (grade 1 and grade 2). Moreover, the network showed difficulty discriminating between grade 0 and grade 1. These findings suggest that while the broad classification was accurate, the fine-tuned models might require additional data or further optimization to improve accuracy in distinguishing between milder forms of SG involvement. The error rate by operator was 0.36 for Operator 1 and 0.32 for Operator 2. Neither the global comparison nor the class-wise analysis revealed statistically significant differences between operators. Regarding the device comparison, the mean error rate was 0.38 for the high-end ultrasound system and 0.31 for the wireless handheld ultrasound device, with no statistically significant global difference. However, when analyzed by class, a statistically significant difference was observed for grade 2 (p <0.001): the error rate was 0.74 for the high-end ultrasound system compared to 0.34 for the wireless handheld ultrasound device.

Conclusions: Our results showed the potential of deep learning models, specifically convolutional neural networks, to classify ultrasound images of SG based on the severity of parenchymal abnormalities in SjD. While the broad classification model performed very well, the fine-tuned models, which differentiate between some specific grades of SG involvement, still face challenges. Future work should include further refinement of the fine-tuned models with additional data, which allows more complex architectures, or domain-specific adjustments. Additionally, our results suggested that operator-related variability has minimal impact on classification accuracy, while device-related differences may affect performance in specific classes. Further analyses in larger datasets are needed to determine whether these findings reflect true performance differences or arise from sample variability

Figure 1.

REFERENCES: NIL.

Acknowledgments: NIL.

Disclosure of Interests: None declared.