Drift-Triggered Model Retraining in Apache Spark Structured Streaming for Fraud Detection

Abstract

Fraud models faced on transaction streams are subject to concept drift and delayed labels accompanied by simultaneous class imbalance. In this study, we propose and test a drift-triggered retraining design for Apache Spark Structured Streaming for a strictly bounded collection of 28 peer-reviewed studies published between 2015 and 2024, and a repeatable controlled micro-batch experiment. The experiment produced 120,000 transaction-like observations, where one large drift occurred at batch 45 and a slow drift occurred between batch 80 and 95. Three different maintenance strategies were considered – static model, periodic retraining every 15 batches, and a Population Stability Index (PSI) – triggered champion–challenger strategy with a 3-batch persistence rule, a 0.24 composite threshold, 20-batch cooldown, and 2-batch label delay, using different time-ordered training and validation windows. Retraining after drift resulted in a mean AUPRC of 0.281, mean recall of 0.183 at a fixed 2% review budget and mean log loss of 0.715 across 105 batches evaluated for post initialisation. Compared to the static model, it boosted mean AUPRC by 7.6%, boosted review-budget recall by 6.9%, and reduced log loss by 39.7%. Its AUPRC and recall differences were small and the moving-block bootstrap intervals crossed zero as compared to frequent periodic retraining, and it processed 111,000 rows of training data as opposed to 183,000, a reduction of 39.3%. Abrupt drift was observed after two batches (2,000 transactions), while gradual drift was observed after 11 batches (11,000 transactions). Results validate drift-triggered retraining as an efficient maintenance policy for compute, but also demonstrate that distributional triggers alone may not be the most effective maintenance policy to produce the best calibration when drift is gradual. The paper thus proposes a Spark control plane that incorporates feature drift, delayed performance signals, cooldown logic, champion-challenger validation, versioned model registration and complete audit metadata.