LLM-Facilitated Differential Diagnosis of Cardiovascular Conditions from ECG and PPG

Taking advantage of multimodal physiological signals and integrating them with large language models (LLMs) could improve automated cardiovascular screening and decision support. Here, we propose a model that integrates 12-lead electrocardiogram (ECG) signals from the PTB-XL data set and wrist photoplethysmography (PPG) signals from the PPG-Dalia data set to differentiate between heart malfunctions. The pipeline will consist of three steps: (1) preprocessing and feature extraction of the signal, (2) a multimodal deep learning-based diagnostic model, and (3) a multimodal deep learning-driven layer explaining in clinician terms and providing a patient-friendly explanation. In ECG, we compute temporal and morphological indicators (QRS duration, QT intervals), whereas in PPG, motion artifacts are removed, inter-beat interval (IBI) is estimated, and heart rate variability (HRV) features are derived. We consider both single-modality encoders and late and intermediate fusion strategies and estimate uncertainty with deep ensembles and calibration techniques. Model outputs are formatted as a JSON schema and fed to an LLM, which produces calibrated textual summaries, triage recommendations, and uncertainty assessments. Evaluation of performance is based on AUROC, AUPRC, and calibration measures for ECG classification, and on HR/HRV estimation error from PPG. Fusion models are tested for resilience to noise and artifacts from activities. This strategy shows that safer, more transparent cardiovascular screening tools can be supported by clinically grounded multimodal learning and interpretation facilitated by LLMs.