Multi-Level Feature Selection and Transfer Learning Framework for Scalable and Explainable Machine Learning Systems in Real-Time Applications

Abstract

Rapid advances in data-intensive real-time applications (e.g., IoT monitoring, autonomous systems) have heightened the need for machine learning (ML) solutions that are both scalable and explainable. Real-time systems demand low-latency inference on streaming data while ensuring model interpretability for trust and compliance. In this work, we propose a novel multi-level feature selection and transfer learning framework designed to address these challenges. Our framework integrates filter, wrapper, and embedded feature selection stages to reduce dimensionality and improve model efficiency, followed by domain adaptation through transfer learning to handle distribution shifts in streaming data. Explainability is incorporated via post-hoc methods (e.g. SHAP, LIME) to provide human-understandable insights. Scalability is achieved using parallel processing and incremental learning techniques. We demonstrate the framework on simulated real-time datasets, evaluating classification accuracy, F1-score, latency, and feature reduction. Hypothetical results show that our method outperforms baseline models by achieving similar or better accuracy with substantially fewer features and lower runtime (e.g. 50% feature reduction with <10ms latency), while providing transparent explanations. This article serves as a comprehensive guide, reviewing 30+ recent studies in feature selection, transfer learning, explainable AI, and real-time ML, and presenting a unified architecture for building robust, scalable, and interpretable ML pipelines for time-critical applications.