Grounding LLM-Based Engineering Assistants in Re-al-World DevOps Discourse for CI/CD Workflows

Abstract

Continuous Integration and Continuous Deployment (CI/CD) are central to modern DevOps, enabling teams to build, test, and deploy software quickly and reliably through automated pipelines. As these workflows grow more complex, developers increasingly turn to intelligent assistants for help diagnosing failures, understanding configuration logic, and resolving recurring pipeline issues. Although recent Large Language Models (LLMs) have shown strong performance on code-related tasks, they often struggle with mixed natural language–code reasoning and with grounding their answers in real-world DevOps scenarios. To address these gaps, we introduce DevOpsAssistant-SOD, an LLM-powered engineering assistant trained and evaluated on large-scale Stack Overflow data, comprising 218.5 million natural language–programming language pairs and 1.4 million labeled question pairs. The model leverages duplicate-aware contrastive learning to better capture contextual similarities across CI/CD-related questions and issues. Experimental results show that DevOpsAssistant-SOD improves failure diagnosis accuracy by 9.3%, duplicate issue detection F1 by 11.1%, and response relevance by 8.7% compared to strong general-purpose code LLM baselines.