Machine Learning-Based Traffic Classification in IP Networks: A Comparative Study of CatBoost, XGBoost, and LightGBM

Abstract

Traffic classification is a critical operation in IP networks to regulate traffic flow and address congestion issues because of exponential increases in network size. Machine learning (ML) methods have surfaced as effective means to improve the accuracy and efficiency of classification. This paper explores the use of different ML algorithms—namely, CatBoost, XGBoost, LightGBM, and deep learning models—to classify traffic in real-time in IP networks. The emphasis is placed on separating high-bandwidth "elephant flows" from low-bandwidth "mice flows" based on dynamic threshold calculation and supervised learning. The proposed approach integrates offline training with online prediction to yield efficient and accurate classification. Experimental results show that tree-based models such as CatBoost and XGBoost have high prediction accuracy and efficiency, while deep learning models have high adaptability with complex traffic patterns. The results show the capability of ML-based traffic classification to improve routing decisions and optimize resource allocation in future networks. Future work will study the use of these models in software-defined networks (SDN) for real-time decision-making and improved network performance.