Background: The diagnostic process for rheumatic diseases is challenging due to their wide clinical heterogeneity and long course. Electronic medical records (EMR) of these patients are often overdetailed anddo not allow a clear synthesis of the patient history. Natural Language Processing (NLP) techniques can have a crucial impact on the automated analysis of EMR clinical text data, aiding in this diagnostic process. We hypothesize that NLP tools are capable to discriminate rheumatic diseases from free-text narratives and ultimately assist physicians in diagnosing complex diseases, especially when clinical features overlap.

Objectives: To investigate and compare various NLP-based solutions applied to medical records for the capacity to differentiate patients with rheumatoid arthritis (RA), psoriatic arthritis (PsA), or other diseases, namely osteoarthritis or fibromyalgia.

Methods: The dataset consisted of 236 Italian outpatients EMR, extracted from the Electronic Health Record of Humanitas Research Hospital (Milan, Italy) from January 2016 to September 2023 by selecting those of patients with an established diagnosis and a regular follow up at Rheumatology Unit. The EMR of 55 patients diagnosed with RA, 68 with PsA, and 113 with other conditions were included and evaluated blindly for the aims of this study in terms of the diagnosis consistently made by experienced rheumatologists. Two techniques were employed for text embedding: Word2Vec, which captures just the semantic meaning of each word (Mikolov et al., 2013), and BERT, which also considers the contextual information within the text (Devlin et al., 2019). To compare the capability of these techniques in capturing relevant clinical information for classification purposes we used a deep-learning approach, the Convolutional Neural Network (CNN). We devised a novel method to maximize the potential of BERT, leveraging its embeddings that carry rich contextual meaning within the text: creating a new data vector that encapsulates the summary of BERT’s embedding allowed us to test other classification models as KNN, SVM, XGBOOST, RF LR and NN. Finally, we experimented an alternative method of classification, consisting in the fine-tuning of BERT’s built-in classification capabilities. Each of these models has different level of complexity, computational cost and methods of recognizing patterns in the data for classification purposes. By comparing the result from these models, we aim to identify the optimal Machine Learning architecture for our goal.

Results: The approaches were compared in their capacity to assign each patient to the correct class (RA vs PsA vs others), expressed in terms of peak accuracy rates. Considering the embedding methods, BERT showed a better performance than Word2Vec in the CNN methods, achieving a peak accuracy of 0.69. The KNN outperformed the CNN achieving an accuracy of 0.73. The other classification models, including BERT’s fine-tuned downstream classifier, showed mixed results and, in general, worse performance than the CNN. Evaluating the confusion matrices, all models showed better performances in the prediction of the “other” category than in the differentiation between psoriatic and rheumatoid arthritis.

Conclusion: Our models obtained favourable outcomes in the differentiation between inflammatory conditions (namely RA or PsA) and non-inflammatory diseases (osteoarthritis or fibromyalgia), particularly in the case of the KNN, the best performing model. We believe that our novel approach for handling BERT output might foster further investigation in the field for a more structured and organized use of BERT’s data. While we recognise that our model is not suitable for use in clinical practice, the future prospective analysis of an extended and standardized dataset could improve the performances.

REFERENCES: [1] Mikolov T., Chen K., Corrado G., Dean J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.

[2] Devlin J, Chang MW, Lee K, Toutanova K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT (pp. 4171-4186).

Acknowledgements: NIL.

Disclosure of Interests: Antonio Papatolo: None declared, Benedetta Maizza: None declared, Alessandro Bellone: None declared, Sofia Di Giorgio: None declared, Gaia Tettamanzi: None declared, Maria Chiara Grondelli: None declared, Nicola Lambri: None declared, Nicoletta Luciano AbbVie, BMS, Eli-Lilly, Janssen, Galapagos, Novartis, Elisa Barone: None declared, Daniele Loiacono: None declared, Carlo Selmi AbbVie, Amgen, Alfa-Sigma, Biogen, Eli-Lilly, EUSA Pharma - Recordati, Galapagos, Janssen, Novartis, Pfizer, Recordati, SOBI, AbbVie, Amgen, Alfa-Sigma, Biogen, Eli-Lilly, EUSA Pharma - Recordati, Galapagos, Janssen, Novartis, Pfizer, Recordati, SOBI, Research support: AbbVie, Amgen, Janssen, Novartis, Pfizer.