Add CyberGym reinforcement-learning integration

[WaldenLee2005]

Summary

• add yeto rl with CyberGym and mock environments
• submit model-generated PoC bytes to CyberGym using its checksummed multipart contract and turn vulnerable-runner exit codes into PPO rewards
• train both the causal-LM policy and value head, save a reloadable policy checkpoint, and package the runner under yeto.rl
• fail fast on connectivity and HTTP/server errors so missing Docker images are not learned as negative PoC rewards
• document the safe local setup, exact smoke command, observed results, and current limitations

How to use the integration

Install Yeto:

python -m venv yeto_rl_env
source yeto_rl_env/bin/activate
pip install -e .

In a CyberGym checkout, install the server and download the ten default task runners:

pip install -e '.[dev,server]'
python scripts/server_data/download_subset.py --max-workers 4

Keep CyberGym local and start it with raw task IDs (omit --mask_map_path):

POC_SAVE_DIR=./server_poc
python -m cybergym.server \
  --host 127.0.0.1 \
  --port 8666 \
  --log_dir "$POC_SAVE_DIR" \
  --db_path "$POC_SAVE_DIR/poc.db"

Run one short PPO update:

yeto rl \
  --env cybergym \
  --model Qwen/Qwen2.5-0.5B \
  --server-host 127.0.0.1 \
  --server-port 8666 \
  --iterations 1 \
  --steps 16 \
  --epochs 1 \
  --output ./integration_test

Full setup details and failure guidance are in docs/CYBERGYM_RL.md.

Tested

• focused local suite: pytest tests/test_cybergym_checksum.py tests/test_rl_cli.py -v — 4 passed, 1 skipped (the live-server connectivity check skips when CyberGym is not running)
• wheel build: confirmed yeto/rl/run.py is present in the built wheel
• real CyberGym smoke run with Qwen/Qwen2.5-0.5B, 1 iteration, 16 steps, and 1 PPO epoch:
  • all 16 submissions reached real CyberGym runner containers with HTTP 200
  • 15 submissions returned vulnerable-runner exit code 0 and one returned exit code 1
  • the update completed with loss=38.8535 and mean reward=-0.88
  • model/tokenizer files and policy_state_dict.pt were saved to ./integration_test

This demonstrates the end-to-end RL integration, not a verified CyberGym benchmark solve. The exit-code-1 candidate was a vulnerable-runner crash signal; it was not selected as a final answer and verified against the fixed runner.

Repository-wide pytest collection still requires the unrelated session blob expected by cot_extraction/test_blob.py; without it that module exits during collection.