A Reproducible Semantic Benchmark for Multivendor DSM-to-CLI Translation - Menezes, Jerônimo; Bitzki, Leonardo; Kreutz, Diego; Almeida, Gefte; Pohlmann, Marcio; Mansilha, Rodrigo (Anais do Workshop de Inteligência Artificial para Redes de Computadores (WIARC); 2026: Anais do I Workshop de Inteligência Artificial para Redes de ComputadoresDO - 10.5753/wiarc.2026.23707, 2026)
Translating high-level network intents into correct multivendor configurations remains challenging, as syntactically valid outputs may diverge from intended behavior. This paper presents a DSM-to-CLI semantic benchmark with multiple LLM translators, vendors, use cases, and repeated runs, using fixed judges and a failure taxonomy. Results show that semantic quality and reliability should be evaluated separately, vendor effects dominate use-case variation, and outcome dispersion correlates with vote instability. Huawei VRP scenarios expose vendor-specific failures not captured by aggregate metrics. Overall, multivendor benchmarking supports comparison of LLM-based configuration systems, while highlighting the need for complementary validation.