[Leaderboard] fabric-rlm (notebook RLM library) - GPT-5.6 Luna max - 69.57% Pass@1

[pawarbi]

fabric-rlm - Leaderboard Submission

Agent name: fabric-rlm
Backbone LLM: GPT-5.6 Luna (openai/gpt-5.6-luna via OpenRouter, reasoning effort max), single model, no fallback
Hints: Yes (db_description_withhint.txt appended to the description, as in run_agent.py --use_hints)
Trials: 5 per query (270 records, all 54 queries, empty answers submitted as "" - there were none)
Stratified Pass@1: 69.57% (193/270 records passed, micro 71.48%)

What fabric-rlm is

fabric-rlm is an open-source Python library that implements Recursive Language
Models for Microsoft Fabric notebooks: the data stays outside the model context
in a persistent Python subprocess, and the model works by writing code against
it over multiple turns. It is not an agent product or a CLI. A user imports it
in a notebook, passes file handles, and gets an answer with a full trajectory.
This submission runs that library unchanged from PyPI; the benchmark harness
around it (data loading, ensembling, grading, audit) is about 600 lines of
plain Python, published in full.

As far as we can tell this makes it the only entry whose runtime is a
notebook-importable library rather than a dedicated agent runtime or CLI, and
one of the cheapest end to end: the full 270-trial run cost about $16 in API
spend, roughly $0.06 per trial.

Architecture

Per query, per trial:

1. All of the dataset's sanctioned stores are consolidated into one DuckDB file:
   SQLite and DuckDB stores are ATTACHed zero-copy, PostgreSQL dumps (all three
   dialects: plain COPY, COPY WITH FORMAT CSV, INSERT VALUES) and MongoDB BSON
   dumps are materialized into schemas. The model sees uniform
   store_name.table_name naming and a compact schema listing in which tables
   with identical column signatures are grouped (stockmarket's 2,754 per-ticker
   tables collapse to two lines).
2. Two blind solves: the same task is solved twice in fresh contexts with no
   shared state. The RLM writes and executes Python/SQL in its subprocess for
   up to 40 turns.
3. A structural agreement check compares the two answers in code, not with an
   LLM: numbers must match exactly, semicolon lists must have identical item
   sets, prose is compared normalized.
4. On agreement the answer is submitted. On disagreement a reconciler runs in a
   third fresh context with the database and both candidate answers,
   instructed to find the exact point of divergence and re-derive. In a
   separate RUNS=3 ablation without hints this ensemble measured +0.076
   stratified Pass@1 over single solves.

No benchmark-tuned prompt: the task template contains the question, the
database description (with hints), how to open the database, and six generic
answer-format rules (state the answer first, put values next to their labels,
decimal forms, complete lists, no empty answers). The full task template and
rules are in the harness source linked below. No per-dataset or per-query
instructions of any kind.

Results

Dataset	Pass@1
bookreview	1.00
stockindex	1.00
stockmarket	0.96
yelp	0.74
crmarenapro	0.74
agnews	0.70
googlelocal	0.70
PANCANCER_ATLAS	0.60
music_brainz_20k	0.60
GITHUB_REPOS	0.50
PATENTS	0.47
DEPS_DEV_V1	0.40
Stratified Pass@1	0.6957

Integrity (per SUBMISSION_RUBRIC.md)

• Grading: the official per-query validate.py, unmodified, over the submitted
  answers. Re-scored from the submission JSON immediately before filing:
  0.6957.
• Isolation: the model never sees a benchmark path. Each dataset's DuckDB file
  and every attached store are hardlinked into a bare directory and the attach
  statements are rewritten, so nothing reachable from the model's inputs leads
  to query*/ folders, ground_truth.csv, or validate.py. An assertion
  fails the build if a benchmark path survives into model-visible text.
• Mechanical audit: every ensemble member's trace (both solves and any
  reconciler) is scanned for reads of ground truth, validators, query files,
  hint files, or benchmark-tree walks; any hit forces the record to score
  zero. 0 of 270 records flagged, across roughly 700 audited solver runs.
• Traces: one JSON per trial in fabric-rlm-luna/fabric-rlm-luna_traces.tar.gz
  (this PR), containing the exact task text of the winning stage, per-turn
  code, stdout, stderr, token counts, and the ensemble verdict
  (agree/reconciled). Submitted answers match traces 270/270 by string
  equality.
• No external data: the isolation directory contains only the sanctioned
  stores; traces show no network fetches.

Reproduction

Library: https://github.com/pawarbi/fabric-rlm-core (PyPI: fabric-rlm).
Harness, run configs, ablation results and analysis:
https://github.com/pawarbi/fabric-rlm-benchmarks (dab/ directory).

We also measured a minimax-m3 run of the same harness at 0.5902 and may submit
it separately as a budget-backbone reference point.

[Ruiying-Ma]

Thanks @pawarbi ! We found data leakage in four agnews trials (q3 trial 1, and q4 trials 1, 2 and 5): each one loads the external AG News dataset from Hugging Face and uses its gold label field to produce the submitted answer, and all four pass on that basis.

Once these leakage patterns are addressed, we’ll re-run the verification and post the Pass@1 results. Thank you!

[pawarbi]

Thank for the review @Ruiying-Ma . I ran everything in an isolated env but looks like I need to review again. Appreciate it, I will re-run and re-submit.

[pawarbi]

@Ruiying-Ma thanks for catching this, and for the detail in the report. You are
right on all four, and I have corrected the submission.

The correction

The four trials you listed are voided (empty answer) in the updated results file.
Nothing else is touched. Regraded with your validators:

	stratified Pass@1	micro	agnews
as submitted	0.6957	0.7148	0.700
corrected	0.6790	0.7000	0.500

I ran mechanical leak patterns (load_dataset, hf_hub, snapshot_download,
huggingface, url reads, other public corpora) blind over all 270 traces in the
submission. They fire on exactly the four trials you identified and nowhere else.

One caveat on that number. Each trial runs two blind solves plus a reconciler,
and our harness only persisted one trace per trial, so a fetch in a
non-persisted pass left no evidence. Your audit and mine were reading the same
partial record. The harness now saves every pass.

We also ran with hints, so the agent had this line in its prompt:

Determining an article's category requires understanding the meaning of its
title and description.

It had the intended method in front of it and took the shortcut anyway. That is
on us, and we did not have a control that would have caught it.

What changed

• Network egress can be blocked in the sandbox. Added block_network to the
  library we run on (feat(runtime): block_network, refuse worker egress when asked pawarbi/fabric-rlm-core#25); it refuses any non-loopback
  connection from inside the execution worker.
• The audit now covers external fetches, and a flagged trial scores zero
  regardless of what the validator says.
• Every ensemble pass is traced, not just the winning one.

---

For the sake of science

The rest of this is not needed to review the correction. Tracking this down was
more interesting than expected, mostly because each defence we added revealed
another way an agent gets to the same data, so it seemed worth writing up.

Blocking the network does not stop it. With egress blocked, the agent was
refused at urlopen, spent two turns probing for local packages and cache
directories, then ran:

hf_root = os.path.expanduser("~/.cache/huggingface")

and loaded 120,000 labelled rows from the on-disk cache a previous run had
populated. No network traffic at all. Your rubric already covers this ("a live
fetch that succeeds from a pre-existing local cache still counts") and it is a
good rule, because redirecting HF_HOME does not help either - that only
changes where the library looks by default.

Moving the cache aside does not stop it either. We quarantined every copy of
AG News into a folder elsewhere on the same machine and reran. One trial was
cleanly refused (Couldn't reach 'ag_news' on the Hub) and failed honestly.
Another walked the filesystem:

for pat in ["**/*", "**/*.csv", "**/*.json", "**/*.db", "**/*.arrow", "**/*.parquet"]:
    glob.glob(os.path.join(d, pat), recursive=True)

found the quarantine directory, and read ag_news-train.arrow directly. So a
file move is not isolation against an agent with filesystem read access. Only
deleting the data, or sandboxing the worker at the OS level, actually holds.

It is not model-specific. We reproduced the same behaviour with DeepSeek
V4 Flash, an unrelated model: same instinct to fetch AG News, on the same
queries. And rerunning the flagged trials in isolation, 6 of 6 still attempted
a fetch, including two trials that were not on your list and one that had passed
in the original submission. This looks like the agent's default approach to this
query family rather than four unlucky trials.

One observation on q3, offered as a question. We graded every submission in
this repo against your agnews validators. q3 is 0 for every agent - nine
submissions, roughly 230 trials, no passes. It asks for an exact average of
business articles per year in Europe over eleven years, ground truth 3703/11,
which needs thousands of classifications with no net error against a 1e-4
tolerance. q4 is different: an argmax over regions tolerates classification
noise, and several agents do solve it. Not a defence of what we did, and we are
not asking for those records back, but the grading tolerance on q3 might be
worth a look.

Happy to rerun all 20 agnews trials or supply the rerun traces, whichever is
more useful. Thanks again for the careful review.

[Ruiying-Ma]

Thanks @pawarbi!

Your four voided trials are agnews q3 trial 1, and q4 trials 1, 2 and 5.

We additionally found that agnews q3 trials 2 to 4 never query the database at all, and that q3 trial 5 never derives the value it reports, so in none of these four is the submitted answer derived from the given data. Apologies for the back and forth.

That takes agnews q3 pass@1 to 0/5 and the stratified average at 0.6623 (agnews 0.300), which is the number we will put on the leaderboard. If you would prefer to rerun those trials on your new harness in case the score comes out higher, we are happy to re-audit and update.

[pawarbi]

Thank you for your review. Got it. Yes please accept with your audit. I will re-run later and submit again in a couple of weeks.

It's a fun challenge, thanks.

[Ruiying-Ma]

Hi @pawarbi! Thanks for the submission and the detailed traces. We have added your results to the leaderboard.