Background: Evaluation of structural damage by plain radiography remains an important tool for diagnosis, prognosis and evaluating treatment response in Psoriatic Arthritis (PsA) and other inflammatory arthropathies. Formal quantification of structural damage is restricted to the research setting due to the time and expertise required for methods such as the Sharp-van der Heijde score.[1] Combining techniques in computer vision and machine learning raises the potential to automatically detect bones on plain radiographs and quantify damage in seconds with increased accuracy.[2] Such a technology has significant potential for damage detection in clinical trials, large observational cohorts and clinical practice. We have previously developed a supervised, reliable bone segmentation algorithm for the identification of the index metacarpal, proximal, middle and distal phalanx bones of the hand in Psoriatic Arthritis.[3] Here we report the extension and refinement of a segmentation algorithm for the accurate detection and delineation of all the bones of the hand and wrist.

Objectives: To develop a bone segmentation algorithm to automatically detect and outline all the bones of the hand and wrist.

Methods: Fifty-Three sets of bilateral hand and wrist digital radiographs from the Bath Psoriatic Arthritis (PsA) biobank were manually outlined (annotated) to create clinically supervised, training data. All patients fulfilled the CASPAR classification criteria for PsA. The capitate and hamate were treated as a unified object to improve annotation reliability. Annotation was performed by a radiologist, two consultant rheumatologists experienced in structural damage scoring and one computer scientist. Ten radiographs were annotated by all four annotators to create a dataset of 481 bone annotations to test the reliability of annotation. The remaining hand and wrist radiographs were annotated by a single assessor. Three groups of models were trained. A Mask R-CNN[5] hand detector that outputs the location of the hands in a radiograph alongside whether it is the left or right hand. A Mask R-CNN bone detector, that outputs segmentation masks for the individual bones of the hands and wrist. Finally, a Gaussian Process Latent Variable (GPLVM) model[4] was trained individually for each bone to act as the generative shape model. Variability and accuracy between expert annotation for the 10 radiographs each annotated by four assessors was then evaluated using Chamfer Distance (a method used to calculate the similarity between two sets of points, reported in pixels where 0 is perfect) and Intersection-over-Union (IoU Index- a method for measuring the overlap between two areas, reported 0-1 where 1 is perfect alignment). Variability and accuracy of the algorithm compared to the mean human annotations (as a gold standard) was then evaluated on the full dataset of 53 sets of hand and radiographs using chamfer distance and IoU.

Results: Data were available for 3266 annotated bones. Using this dataset, we created three subsets to train, validate, and test the models, using splits of 70%, 15%, and 15%, respectively. The machine results provided are the average results over the three randomised unseen test datasets. The mean (SD) chamfer distance between human annotations for all bones was 7.3 (7.89), for phalanges and metacarpals 4.88 (1.87) and for carpals was 12.4 (13.70) (the rest). The mean IoU between the four human annotations for all bones was 0.93 (0.052), for phalanges and metacarpals 0.95 (0.019 and for carpals was 0.89 (0.088). The mean chamfer distance for the algorithm compared to human annotations for all bones was 10.6 (8.9), for phalanges was 6.6(6.30) and for carpals was 18.9 (12.83). The mean (SD) IoU for the algorithm for all bones was 0.9 (0.09), for phalanges 0.93 (0.06) (and for carpals was 0.8 (0.08). The bone specific chamfer and IoU is reported in Figure 1. When calculating the machine metrics, predictions with an IoU less than 0.05 were counted as missing and excluded from the IoU/Chamfer metrics. The average percentage of bones missed by the machine was 9%,0.2% and 5% for the Lunate, MC5 and Scaphoid bones respectively. All other bones had 0% counted as missing. A visual representation of the algorithm annotation is reported in Figure 2. The mean time taken for the algorithm to identify and outline all the bones of the hands and wrists was 45 sec.

Conclusions: We report a novel machine learning algorithm, derived from clinically supervised data able to rapidly and accurately identify and delineate all the bones and the hands and wrist to the level of individual pixels. Next steps include quantifying joint space narrowing and testing against existing scoring methods and evaluation of change in bone contour over time as a measure of structural change.

Figure 1.

Mean Chamfer and Intersection-over-Union (IoU) for Human and Algorithm

Figure 2.

Example of Algorithm annotated hands and wrists

REFERENCES: [1] Tillett, W., et al., Feasibility, reliability, and sensitivity to change of four radiographic scoring methods in patients with psoriatic arthritis. Arthritis Care Res (Hoboken), 2014. 66 (2): p. 311-7.

[2] Antony, A., et al., AB1569 The use of machine learning in diagnosing and detecting damage in the hands and feet of patients with rheumatoid arthritis and psoriatic arthritis: A scoping review. Annals of the Rheumatic Diseases, 2023. 82 : p. 2017-2018.

[3] Rambojun, A., et al., AB1575 A reliable method for annotating hand and wrist x-Rays for supervised machine learning algorithms Annals of the Rheumatic Diseases, 2023. 82 : p. 2020-2021.

[4] Titsias M, Lawrence ND. Bayesian Gaussian process latent variable model. InProceedings of the thirteenth international conference on artificial intelligence and statistics 2010 Mar 31 (pp. 844-851). JMLR Workshop and Conference Proceedings.

[5] Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick; Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2961-2969

Acknowledgments: NIL.

Disclosure of Interests: Dylan Henley-Marshall: None declared, Neill Campbell: None declared, Tony Shardlow: None declared, Anna Antony: None declared, Ynyr Hughes-Roberts: None declared, William Tillett Research grants, consulting fees, speaking fees and/or honoraria from AbbVie, Amgen, BMS, Celgene, Eli Lilly and Company, GSK, Janssen, MSD, Novartis, Ono Pharma, Pfizer, Takeda and UCB.