Enhancing Distributed Workflow Optimization with Graph Neural Networks and Deep Learning Techniques

Abstract

This research presents a distributed processing framework using Graph Neural Networks (GNNs) for workflow scheduling and data routing in large-scale systems. Our adaptive GNN architecture dynamically models computing workflows, where nodes represent tasks and edges capture dependencies and communication patterns. Evaluated on Microsoft Azure Datacenter Traces (25 days, 11,000 machines) and Amazon AWS CloudWatch Metrics (10 days, 5,000 machines), our framework achieves 43% lower processing latency and 39% reduced memory footprint compared to DAG-based schedulers, maintaining 99.7% accuracy. The GNN-based topology optimization
predicts optimal data routing paths with 91% accuracy, reducing storage overhead by 45% versus shortest-path algorithms (62% accuracy). Using PyTorch Geometric on a 180-node cluster, the system reduces network congestion by 35% and improves space utilization by 42% over baseline methods. Our multi-layer graph attention mechanism with dynamic edge weight updates accelerates workflow optimization by 46% while using 33% less memory. Under sudden workload variations, the framework sustains 92% performance stability and 98.5% data accuracy, surpassing traditional systems (68% stability). It achieves a time-space optimization ratio of 0.85 (vs. 0.62 in
conventional systems), processing 1,100 tasks/hr with 95% resource efficiency. Additionally, it improves memory utilization by 41%, maintaining a ±0.3% accuracy deviation across workloads, setting new benchmarks in distributed processing.