MASynReas: Multi-Agent Synergistic Reasoning for Network Anomaly Diagnosis

MASynReas diagnoses anomalies in telecom and IT networks represented as knowledge graphs. Symbolic agents query the graph with SPARQL, detect elementary anomaly patterns and correlate them into ranked diagnoses; a local LLM only turns the final diagnosis into a readable report. Every conclusion can be traced back to the graph facts that support it.

Collective Scientific Project (PSC) at École Polytechnique in collaboration with Orange Research, 2025–2026. Team: Andrea Signoretti, Paul Chiche, Raphaël des Boscs, Qile Liu.

📄 Full report: reports/final_report.pdf

Architecture

flowchart LR
    KG[("Network knowledge graph<br/>NORIA-O ontology<br/>Virtuoso SPARQL endpoint")]
    subgraph L1["Level 1 · 54 detector agents"]
        direction TB
        S[Structural]
        D[Dynamic]
        F[Functional]
        P[Procedural]
    end
    L2["Level 2 · 12 diagnoser agents<br/>correlate evidence on shared entities<br/>score reliability, severity, priority"]
    L3["Level 3 · local LLM<br/>natural-language report"]
    KG -- "one SPARQL query per agent" --> L1
    L1 -- "JSON findings" --> L2
    L2 -- "ranked diagnoses" --> L3

Loading

• Level 1, detection. 54 agents, one anomaly pattern and one SPARQL query each, organized along the four facets of the NORIA-O ontology: structural (resources, links, interfaces), dynamic (events, changes), functional (applications, services) and procedural (tickets, procedures).
• Level 2, diagnosis. 12 deterministic agents read selected level-1 outputs, check activation conditions, correlate evidence on shared entities, and output a diagnosis with reliability, severity and priority scores (e.g. single point of failure, change-induced incident, service cascade).
• Level 3, narrative synthesis. A local LLM (Ollama) writes the final report from the level-2 diagnoses: it explains the decision, it does not make it. In this repository, this layer is implemented and evaluated in the ablation study (ablation_study.py) rather than as an agent of the JaCaMo system.

The system runs in two modes:

Mode	Purpose	Detectors	Diagnosers
apriori	Find structural weaknesses before any incident	23	5
aposteriori	Explain causes and propagation once incidents, events or tickets exist	31	7

Family	A priori detectors	A posteriori detectors
Structural	10	7
Dynamic	3	12
Functional	7	9
Procedural	3	3

Key results

Results from the final report, on noria-0.2 (Orange's reference network knowledge graph, 3,152 triples) and on the 27-scenario synthetic catalogue in datasets/.

Accurate, well-ranked diagnoses on incident scenarios. On the synthetic catalogue, the level-2 diagnosers reach an F1 of 0.815 in a posteriori mode (precision 0.825, recall 0.808). On scenarios with several true diagnoses, the main one is always ranked first (top-1 accuracy 1.00, MRR 0.833), and the four noisy or incomplete-data scenarios are all handled correctly (F1 1.00). A priori mode is weaker (F1 0.196), mostly because one diagnoser (procedural_unreadiness) fires too broadly.

The multi-agent system beats an LLM at its own job. Asked to name which of the 12 diagnosers should fire, Mistral-7B-Instruct reaches at best F1 0.508 in a posteriori mode, even when given the level-1 and level-2 outputs, against 0.815 for the multi-agent system. As a level-1 detector, the LLM finds at most 21% of the anomalies, and at most 6% of the a priori ones, which are defined by something missing in the graph.

Correlating facts before the LLM makes its reports more reliable. For narrative synthesis (Mistral-7B-Instruct, 5 scenarios × 3 runs), conformity to the ground truth rises from 0.31 with the graph alone to 0.53 with level-1 findings and 0.78 with level-1 findings + level-2 diagnoses. Raw level-1 findings also bring more hallucinations and worse severity calibration; adding level 2 halves the hallucinations and restores the severity calibration.

The four agent families are complementary. In a priori mode, each family detects entities that no other family sees, and the overlaps (e.g. applications flagged by both structural and functional detectors) are exactly what level 2 merges into a single diagnosis.

Independent agents parallelize well. Running all detectors sequentially takes 4.9 s (a priori) and 6.5 s (a posteriori), against under 0.4 s for the slowest single detector: a theoretical speedup of 12.6× and 18.7×. The full pipeline runs in under 7 s on the synthetic graphs.

This repository contains the scripts for the noria-0.2 experiments (family complementarity, sequential baseline, and a local CPU version of the ablation with llama3.1:8b), documented in EXPERIMENTS.md. The synthetic-catalogue evaluation and the GPU comparisons with Mistral-7B are described in the report.

Benchmark catalogue

datasets/ contains 27 synthetic scenarios (DS01–DS27) built from two reference graphs by controlled anomaly injection, data removal and noise. Each scenario comes with its graph (Turtle) and the expected level-1 and level-2 outputs. They cover clean controls, partial evidence (no diagnosis expected), each a priori and a posteriori diagnosis, robustness to missing and noisy data, ranking of several true diagnoses, score calibration and scalability.

Repository structure

Path	Content
multiagentSystem.jcm	JaCaMo entry point: agents and environment
mas.properties	Execution mode and Python interpreter
src/agt/	Agents: one folder per family (structural, dynamic, functional, procedural) and level 2. Each agent is a Jason .asl file plus a Python script holding its SPARQL query
src/env/env/PythonExecArtifact.java	CArtAgO artifact that runs the Python detectors from the agents
datasets/	Reference graphs and the 27-scenario benchmark
results/	JSON outputs of the last run (one file per agent)
run_all_detectors.py, complementarity_table.py, ablation_study.py, baseline_monoagent.py	Experiment scripts, documented in EXPERIMENTS.md
reports/	Final report (PDF)

Getting started

Requirements

• Java 11+ and the JaCaMo CLI in your PATH
• Python 3.8+ with pip install -r requirements.txt
• OpenLink Virtuoso running on http://localhost:8890/sparql, with a NORIA-O graph loaded (noria-0.2, or any dataset.ttl from datasets/)
• For the ablation study only: Ollama with llama3.1:8b

Run the multi-agent system

Choose the mode in mas.properties:

mode=apriori        # or aposteriori
python.path=        # empty = use python/python3 from PATH

Then, from the project root:

jacamo multiagentSystem.jcm

The selected mode activates its level-1 detectors, which write their findings to results/<family>/<mode>/. Once all of them have finished, level2_controller launches the diagnosers of the same mode, which write to results/level2/<mode>/.

Run the experiments

python -X utf8 run_all_detectors.py both
python -X utf8 complementarity_table.py
python -X utf8 ablation_study.py

(-X utf8 is only needed on Windows.)