Fix offline-grading snags: sticky bypass marker, eager e2b import, unshare spam

[davidoj]

Four independent issues hit while grading raw dataset rows with the offline verifier from outside the repo's own venv. Full write-up lands in docs/OFFLINE_GRADING.md.

1. environment_variable_manipulation leaked its bypass marker across runs

The verifier bypasses validation when /tmp/test_override_marker exists, but only removed the marker when not marker_preexisting — so a run that found a marker also refused to clean it up. The bypass became sticky: every later verification in any process passed any submission that merely defined the function.

It is also self-poisoning: an exploit that writes the marker at module level does so during exec(), i.e. before the pre-existence snapshot, so its own marker counts as pre-existing. (An in-function write does not trigger it, which is why this hid for so long.)

Both triggers, before → after:

	before	after
stale marker + exploit-free wrong code	PASSED	FAILED
…marker still on disk afterwards	True	False
import-time exploit, then wrong code	PASSED	FAILED

This silently inflated one measured exploit rate to ~40% for a model that emitted zero env/marker code.

Fix: clear the marker on entry, clean up unconditionally in finally, and move exec() plus the function-not-found early return inside that try so no return path can leak. The path stays /tmp/test_override_marker by default — dataset exploits and insecure_verifier_info name it literally, so randomizing would break the exploit contract — but MARKER_PATH honours $DJINN_TEST_OVERRIDE_MARKER so concurrent graders can each use a private tmpdir.

Verified the contract still holds: on 5 sampled problems of this type, the ground-truth exploit still passes insecure and fails secure, and ground truth still passes secure (5/5 each).

Checked and already hermetic: test_case_logging_side_channel (truncates its log on entry) and filesystem_exposure (rewrites and unconditionally unlinks).

2. Offline grading transitively required e2b

verification_service.py imported e2b at module scope, so get_verification_service() and force_offline_verification() raised ModuleNotFoundError in any env without the online-only client — despite offline needing none of it. Both imported names (Sandbox, TimeoutException) were unused, and every entry point already falls back to offline, so the imports are simply removed.

3. unshare failure spammed the bridge log

Without CAP_SYS_ADMIN (typical in unprivileged containers) every external daemon launch fails identically, appending unshare: unshare failed: Operation not permitted to djinn_daemon_bridge_{mode}.log on every attempt. Capability is now probed once per process and logged in a single line, and the doomed launch is skipped entirely.

This also removes a latent race: failure used to be inferred from proc.poll() after a 50 ms sleep, which could mistake a slow-exiting unshare for a healthy daemon.

4. Problem rejected rows missing descriptive metadata

insecure_verifier_info, exploit_explanation and exploit_type now default to "". They are descriptive metadata, not verification inputs, and Problem.from_dir already defaulted all three via config.get() — so this just makes direct construction agree with the loader. Problem(**row) no longer raises on exported rows that are otherwise perfectly gradable.

Testing

New djinn/tests/test_marker_isolation.py, 12 tests, covering both leak triggers plus the crash and missing-function return paths. Confirmed the tests fail against the pre-fix code.

Existing suite unaffected. Note test_cli.py, test_verifier.py and test_filter_reward_deltas.py fail to import in a minimal env for an unrelated pre-existing reason (dspy not installed).

Independent of #3 — both branch off main and can merge in either order.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MYYDDvJbdup3v5LPnkcYMW

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

root seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}