Background: Remote Patient Monitoring (RPM) of Rheumatoid Arthritis (RA) patients typically involves the collection of Patient Reported Outcomes (PROs) via apps, wearables and sometimes blood self-sampling. An objective measure of joint swelling via the previously reported automated detection of dorsal finger fold patterns from hand photographs, could enhance RPM accuracy in RA.

Objectives: To test the quality of automated detection of finger joint swelling based on finger fold recognition in sequentially taken hand photos of RA patients during regular rheumatology consultations.

Methods: We utilized various image recognition models (Canny Edge Detection, Ridge Detection) and a Convolutional Neural Network (CNN) trained on photos of 1,783 Proximal Interphalangeal (PIP) joints from RA patients. After keypoint detection in hand photos via MediaPipe, PIP joints were isolated for further analysis. The CNN, trained on an additional 151 PIP images using LinkNet and Python 3.10, helped quantify finger folds and PIP joint diameters at the pixel level. The Finger Fold Index (FFI) was calculated as the ratio of joint diameter to the mean pixel length of detected finger folds. We analyzed 74 smartphone images (iPhone 6 camera without flash) of 17 RA patients taken in ≥2 regular consultations at a Swiss Academic Center. Active disease in at least one visit with congruent hand imaging as well as recorded swollen and tender joint counts and DAS28-CRP scores were required. We describe misclassifications based on algorithm steps (cropping, diameter, folds, image quality) or clinical data.

Results: Only PIP2-4 joints were suitable for analysis due to oblique positioning of digits 1 and 5. The algorithm initially captured 222 PIP joints. Of those, 32% were optimally processed in terms of cropping, diameter, and finger fold recognition. The main reason for misclassifications were poor image quality leading to suboptimal cropping (9%), diameter recognition (18%), and incorrect finger fold identification (5%). In 15% of joints, the algorithm failed to provide results. For FFI correlation with joint swelling and DAS28-CRP evolution, seven patients (mean age 53, range 27-70) where monitored across 18 visits, yielding 31 hand images (either one or both hands). The average FFIs were 2.3 (index finger), 2.1 (middle finger), and 2.4 (ring finger). In 71% of patients (5 out of 7), the FFI correlated with disease progression as indicated by DAS28-CRP changes or swollen joint count. One of the most common confounding factors affecting the FFI was the reappearance of small finger folds after the swelling subsided, which inaccurately inflated the FFI measurements. In repeated analyses of healthy hand images, we found that light conditions (flash yes/no, sufficient light), positioning of the camera (maintaining a vertical perspective/deviating from a strictly overhead position).

Conclusion: Longitudinal finger fold recognition and FFI measurement in a real-world setting shows consistent results with DAS28-CRP and or swelling and pain on a single joint level. Insufficient image quality was an important cause for misinterpretation in this real-world setting. Improved quality via the use of flash, photo boxes and further training of the algorithm may lead to a higher accuracy for the detection of disease flares or resolution, whereas the DAS28 may not be the ideal gold standard for joint swelling.

REFERENCES: [1] Hügle T, Caratsch L, Caorsi M, Maglione J, Dan D, Dumusc A, et al. Dorsal Finger Fold Recognition by Convolutional Neural Networks for the Detection and Monitoring of Joint Swelling in Patients with Rheumatoid Arthritis. Digit Biomark. 2022 May-Aug;6(2):31-5. PMID: 35949225. doi: 10.1159/000525061.

Figure 1.

An exemplary case of a patient with RA who achieved complete remission in a subsequent visit. The algorithm involves cropping of the Proximal Interphalangeal (PIP) joints, followed by automatic quantification of joint diameter and finger folds. The Finger Fold Index (FFI) is calculated by dividing the pixel length of the joint diameter by the average pixel length of the finger folds. Different examples of misclassifications by the algorithm are presented above.

Acknowledgements: This work has been supported by an unrestrictred grant of Fresenius Kabi.

Disclosure of Interests: Cinja Koller: None declared, Marc Blanchard Atreon, David Brüschweiler: None declared, Patrick Hermann Atreon, Thomas Hügle Roche, Novartis, GSK, BMS, Galapagos, Abbvie, Atreon, Vtuls.