Remaining reward-hack vectors after #233 / PR #237: grading runs agent-writable tooling and the task-dir verifier in the agent's container #238

PR #237 stops the 3d-scan-calc verifier from reading the agent-writable scan_data.stl (#233). @uipreliga's review of that PR found reward-hack vectors that remain. They predate the PR and apply to the skillsbench samples generally. This issue records them. Links are at a253f34.

Each sample's run_command criterion copies $TASK_DIR/verifier and runs bash /verifier/test.sh inside the agent's container (3d-scan-calc.yaml#L67-L69; the same in the other two samples). That container runs as root (DOCKER_ISOLATION.md#L315-L316), and the agent has Bash (#L32).

(a) Verifier runtime. test.sh runs pytest from PATH and writes reward 1 if it exits 0 (test.sh#L4-L5). Its pip3 install line leaves an already-installed pytest and any other installed distribution in place. So the agent can replace the pytest entry point, install an auto-loaded pytest11 plugin, or edit site-packages. A local demo (a venv, not the container) is below.

(b) Task-dir verifier copy. After #237 the expected values live in $TASK_DIR/verifier/test_outputs.py. The task dir is at mode 000 only during communicate (orchestrator.py#L1964). Only DAC_OVERRIDE/DAC_READ_SEARCH are dropped (docker_runner.py#L1365-L1368), so root keeps CAP_FOWNER and can chmod the copy back, and a background writer can act after the window closes. The pre-grade digest check _verify_reference_integrity (#L1370) covers only the reference dir. From code reading; not run in docker.

(c) dialogue-parser ground truth. test_content_integrity[dynamic_script_sample] and [first_node_check] read /app/script.txt (#L238, #L269), which the agent can write. With every Speaker: text -> Target line removed, dynamic_script_sample has nothing to sample and passes. From code reading.

Suggested fixes (from the review)

• Grade outside the agent's container, or digest-check sandbox.task_dir before grading the way the reference is checked.
• Run the verifier with a pinned interpreter: PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 <python> -I -m pytest --noconftest. With autoload off, --ctrf needs -p ctrf.main. This helps only if that interpreter's site-packages are out of the agent's reach.
• Pin dialogue-parser's expected values, as fix(samples): grade 3d-scan-calc against fixed expected values #237 does for 3d-scan-calc.
• Optionally, a docker red-team regression test that runs each known hack and expects reward 0.

#!/usr/bin/env bash
# Usage: demo_pytest11.sh <coder_eval checkout>   (needs uv)
# Grades a WRONG 3d-scan-calc answer with the task's verifier, scored like test.sh
# (reward 1 iff pytest exits 0). The verifier's "/root/..." paths point at a temp dir
# that stands in for the container's /root.
set -euo pipefail
REPO=${1:?coder_eval checkout}
W=$(mktemp -d)
uv venv -q "$W/venv"
uv pip install -q --python "$W/venv/bin/python" pytest==8.4.1 pytest-json-ctrf==0.3.5  # test.sh pins
PY="$W/venv/bin/python"
mkdir "$W/root"
echo '{"main_part_mass": 1, "material_id": 1}' > "$W/root/mass_report.json"
sed "s#\"/root/#\"$W/root/#g" "$REPO/tasks/samples/skillsbench/3d-scan-calc/verifier/test_outputs.py" > "$W/root/test_outputs.py"
score() { if "$@" --ctrf "$W/ctrf.json" "$W/root/test_outputs.py" -q -p no:cacheprovider > /dev/null 2>&1; then echo 1; else echo 0; fi; }

echo "clean runtime,  pytest (as test.sh):        reward=$(score "$W/venv/bin/pytest")"

# What a root agent with Bash can do: drop an auto-loaded plugin into site-packages.
SP=$("$PY" -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')
cat > "$SP/pass_all.py" << 'EOF'
import pytest


@pytest.hookimpl(wrapper=True)
def pytest_runtest_makereport(item, call):
    report = yield
    if report.failed:
        report.outcome, report.longrepr = "passed", None
    return report
EOF
mkdir "$SP/pass_all-0.dist-info"
printf 'Metadata-Version: 2.1\nName: pass-all\nVersion: 0\n' > "$SP/pass_all-0.dist-info/METADATA"
printf '[pytest11]\npass_all = pass_all\n' > "$SP/pass_all-0.dist-info/entry_points.txt"

echo "planted plugin, pytest (as test.sh):        reward=$(score "$W/venv/bin/pytest")"
echo "planted plugin, autoload off, -I, -p ctrf:  reward=$(score env PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 "$PY" -I -m pytest --noconftest -p ctrf.main)"
rm -rf "$W"

clean runtime,  pytest (as test.sh):        reward=0
planted plugin, pytest (as test.sh):        reward=1
planted plugin, autoload off, -I, -p ctrf:  reward=0