Evaluating Global Log File Analysis Systems: Design Frameworks Using Ensemble Techniques

Abstract

Log file analysis plays a vital role in performance optimization, security monitoring, and fault detection across large-scale international networks. However, traditional log analyzers struggle to process modern log data due to its high volume, diverse formats, and real-time generation. This study proposes a generic log analysis system using distributed computing to improve scalability and efficiency. The methodology involves collecting multiple log types, including firewall, server, web, email, and call data logs, and processing them using Apache Hadoop MapReduce for large-scale batch analysis. Log events are parsed, aggregated, and summarized to identify patterns, abnormal activities, and usage trends without interfering with system performance. Experimental results show that the proposed approach successfully analyzes diverse log formats and produces meaningful summaries while reducing manual effort. The system demonstrates improved handling of large log datasets and supports visualization for better interpretation. Overall, the proposed log analyzer provides an efficient and scalable solution for managing and extracting insights from heterogeneous log data.