Research Paper · Benchmarking & Evaluation
NetLLM-Eval: Benchmarking Large Language Models for Automated Fault Diagnosis and Root Cause Analysis in Enterprise Networks
Gowtham Saminathan
Abstract
Enterprise network operations generate vast quantities of unstructured data — incident logs, syslog streams, configuration diffs, and ticketing records — that exceed the capacity of human operators to process at speed and scale. This paper introduces NetLLM-Eval, a benchmark framework grounded in real-world enterprise network incidents, covering three core tasks: (1) fault symptom classification from raw syslog and SNMP data, (2) root cause identification from multi-source incident context, and (3) remediation step generation for common network failure scenarios. Drawing from practitioner experience across 500+ managed devices spanning Palo Alto, Cisco, Juniper, and Fortinet platforms, we evaluate GPT-4o, Llama-3, and Mistral under zero-shot, few-shot, and chain-of-thought prompting regimes. Our results demonstrate that domain-adapted prompting yields up to 78% accuracy on fault classification — outperforming zero-shot baselines by 23 percentage points — and identify systematic failure modes in multi-vendor configuration reasoning, providing a reproducible evaluation harness for the research community.
LLM Benchmarking
Network Operations
Fault Diagnosis
Root Cause Analysis
GPT-4o
Llama-3
Mistral
Syslog / SNMP
Target Venue
IEEE/IFIP Network Operations & Management Symposium (NOMS) 2026
Alt: ACM SIGCOMM NetAI Workshop
Writing
