Implementing Clinical-Grade Data Pipelines for AI-Driven Diagnostics Using HPC and High-Speed Fabrics

Abstract

This paper has applied a modeling pipeline consisting of AI-based diagnostics made possible through the use of computing based on the principles of high-performance computing (HPC). This system receives, interprets and processes multi-modal healthcare information and data (ICU monitor streams, medical imaging and genomic data). It is also low-latency and heavily throughput with use of high-speed fabrics such as infinity band and NVlink in addition to processing ICU data (1,200 records/sec), imaging (350 GB/hour) and genomic data (420GB/hour). AI model outcomes are promising: having an ICU risk prediction F1-score of 0.76, EfficientNet-B0 of 99.5% internal and External imaging accuracy and 95% of predictions of genomic variations, XGBoost, EfficientNet-B0, and External imaging model are high. The system also ensures fault-tolerance, data integrity besides the scaling resource efficiency among the distributed nodes of HPC. The experience of integrating in NYU Hospitals and City of Hope increased the clinical process, decreasing the alert time in the ICUs to 40 per cent, clinician satisfaction increased to 4.5/5. The article indicates that at present HPC-capable AI pipelines are able to deliver reproducible, real-time and clinically actionable insights to offer precision healthcare.