ADE — agents that develop data selection, reward and curriculum strategies through controlled experiments.
ADE gives agents control over research decisions under fixed experimental protocols. Agents propose and implement strategies, analyze experimental evidence and build cumulative Memory. A Harness governs accepted state transitions, while the Engine executes training and evaluation. Strategy selection uses in-loop results; held-out evaluation stays outside the research feedback loop.
Start here: Install → Prepare inputs → Configure deployment → Give an Operation Prompt to the operator agent.
Agents develop strategies from Experiment Packages and Memory; the Engine executes controlled experiments. An external operator agent handles operation, monitoring and recovery using a repository Skill.
A Run first evaluates the base model and trains a baseline. ADE then proposes a Plan, implements its strategy, trains and evaluates a Trial, and records the analysis in Memory. Later Plans use that evidence to choose their next intervention. Model, training budget and evaluation rules stay fixed within a Run.
Every real Run starts with an Operation Prompt + an exact config, handed to one
operator agent. ade-supervise-run
owns admission, startup, monitoring, operational recovery and terminal acceptance.
ADE launches its research-role workers internally.
- Install the complete runtime with
bash scripts/recreate_unified_vllm_env.shafter reading the prerequisites. - Prepare models/data, the local deployment and assigned Ray nodes using the walkthrough.
- Fill baseline-operation.md with your checkout, environment, deployment and resource authorization; give it to the operator agent.
- After baseline acceptance, fill ade-operation.md with its actual Run ID and hand off the separate N=1 ADE.
Use $ade-supervise-run to execute the completed Operation Prompt in
<absolute path to your local operation.md> through terminal acceptance.
The example directory provides Baseline, ADE N=1 and ADE N=3 prompts with their configurations. Prepare inputs and resources according to the selected config, then follow the Quickstart to launch and inspect results.
Want to inspect the workflow without models or GPUs? The optional CPU demo uses scripted responses and synthetic scores; it is separate from the supervised real experiment.
| Task | Agent-controlled strategy | Backend | Available domains |
|---|---|---|---|
| Data Selection | Selection code over a fixed candidate pool | LlamaFactory SFT | Math, code |
| Reward Design | Reward implementation | VERL RFT | Math |
| Curriculum Learning | Training-data schedule | VERL RFT | Math |
The configuration matrix contains baseline, N=1 and N=3 entries. Each Run freezes its task contract, inputs, training/evaluation settings and ADE budget. Read results for Trial Records, rankings and Memory. See operator test and final generalization for held-out datasets and the final evaluation workflow.
| Guide | Documentation |
|---|---|
| Start with an operator agent | Quickstart |
| Operation Prompt → baseline → ADE → results | Walkthrough |
| Runtime and dependencies | Installation |
| Models and datasets | Inputs |
| Cluster and resources | Deployment · Ray |
| Operation and process ownership | CLI · Services |
| Implementation and extension | Components |
| Evidence, scores and Memory | Results |
| Help | FAQ |
ade/: Harness/Control, research Agents, Engine, tasks, evaluation and Memory..agents/skills/: operator supervision and research-role Skills.examples/,configs/,prompts/: Operation Prompts, fixed experiment configs and model prompts.docs/: usage and component guides.scripts/,dataset/,requirements/: environment and input preparation.third_party/llamafactory/,third_party/verl/: included training backends.
See AGENTS.md for contributor-Agent instructions.
ADE is licensed under the MIT License. Third-party components retain their respective license files.
Citation metadata is pending. CITATION.bib reserves the paper title; replace the placeholder with the final authors, publication details and public URL before citing.