Adaptive Knowledge Consolidation: A Dynamic Approach to Mitigating Catastrophic Forgetting in Text-Based Neural Networks

Abstract

Neural networks face catastrophic forgetting as a major drawback for text-based systems that need ongoing learning adaptability. Current methods like Elastic Weight Consolidation (EWC) and Synaptic Intelligence (SI) rely on static processes when preserving existing knowledge while ignoring the specific worth of different tasks. Our innovative Adaptive Knowledge Consolidation method (AKC) dynamically modifies knowledge retention rates by evaluating semantic connections between tasks along with their individual importance levels. The AKC method includes a task embedding module that uses pre-trained language models to gauge task similarity while its consolidation process benefits from dynamic weighting controls. Our evaluation process used AKC to compete on three NLP benchmarks which included GLUE, AG News, and SQuAD against top performing methods such as EWC, SI, and replay-based methods. Experimental findings confirm AKC significantly enhances average task performance to 86.7% accuracy which exceeds both EWC and SI which achieved 78.2% and 80.1% respectively. AKC achieves lower forgetting at 6.2% which shows better results than replay-based methods like GEM that reach 8.9%. AKC proves effective at reducing catastrophic forgetting and maintaining important knowledge to become a valuable technique for text-based neural network continual learning.