Hermit is a framework for training, evaluating, and deploying harness adapters, with support for multiple harness evolution algorithms. It adapts your agent to new tasks by evolving an external adapter while keeping your existing model and harness unchanged.
A harness adapter is a trainable external harness (like a plugin with built-in training support). Just as a model adapter adapts a model, a harness adapter adapts an existing agent to new scenarios without modifying its original harness or model.
Why Harness Adapters?
Many harnesses are black boxes; even open-source ones have carefully designed component interactions that direct evolution can disrupt. Harness adapters offer three practical benefits: (1) No changes to your existing harness: evolve an external adapter through supported extension interfaces, even when both the model and harness are black boxes; (2) Transferable: reuse across compatible models and harnesses; (3) Trainable: learn from task feedback.
| Status | Methods |
|---|---|
| ✅ Supported | GEPA · SkillOpt · MetaWorld Model · Harness² · Continual Harness |
| 🚧 Coming Soon | ACE · ReHarness |
Hermit provides an RL-style training loop for evolving harness adapters:
Requires Python 3.11+ and Git.
Clone the repository to run the examples, reproduce recipes, or develop your own methods.
git clone https://github.com/Infini-AI-Lab/hermit.git
cd hermit
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .To use the framework CLI and bundled adapter templates in your Python environment:
python -m pip install hermit-harness
hermit --helpThe Quick Start below uses files from the source checkout. See Installation for method-specific dependencies.
The included local training walkthrough connects feedback, an adapter update, evaluation, and deployment. A scripted learner reads the judge's feedback and writes the learned rule.
No model account, GPU, or dataset download required.
hermit check --config docs/_examples/first_training/run.yaml
hermit eval --config docs/_examples/first_training/run.yaml --adapter none
hermit train --config docs/_examples/first_training/run.yaml
hermit eval --config docs/_examples/first_training/run.yaml --adapter finalThe teaching agent starts by echoing lowercase text. After a failed attempt, the learner writes Output format: uppercase. into ADAPTER.md. Later executions load the updated adapter.
Expected in this demo: baseline 0.0 → final 1.0, with zero execution errors.
This deterministic example verifies the training contract. Its constructed score gain is not a real-model performance result. The example uses trusted local Python with the learner-program sandbox disabled.
cat .demo/first-training/first-training/checkout/ADAPTER.md
hermit deploy \
--adapter .demo/first-training/first-training/checkout \
--workspace .demo/first-training/workspace \
--host codex
cat .demo/first-training/workspace/AGENTS.mdThe generated AGENTS.md contains the learned rule. Start Codex in that workspace to use it.
Full walkthrough, expected outputs, and repeat-run instructions →
Ready for a real agent? Train with an API model · Run OfficeQA · Bring your own task
Train and evaluate harness adapters on benchmarks using different evolution methods. Each task's README reports results; each included adapter's CARD.md records evaluation details.
| Method | Recipes |
|---|---|
| MWM | CL-Bench database · CL-Bench poker · ServeLearnBench |
| SkillOpt | OfficeQA |
| GEPA | OfficeQA |
| Harness² | OfficeQA (the paper's per-question protocol, Codex as the acting agent) |
| Continual Harness | OfficeQA (policy, sub-agents, skills and memory refined in place, Codex as the acting agent) |
| Prompt-only | CL-Bench database · CL-Bench poker · ServeLearnBench |
Walkthroughs: OfficeQA · CL-Bench · ServeLearnBench.
Smaller training demos: Quick Start (offline) · First real-model training. Task folder layout.
The finished Lean collaboration adapter coordinates five Codex agents through shared tools, state, and skills to prove Lean theorems together. Walkthrough →
| Use a trained adapter | Train for your tasks | Build a new method |
|---|---|---|
| Deploy to your agent. | Use your tasks and feedback. | Implement and compare your idea. |
| Your goal | What Hermit provides | What you bring |
|---|---|---|
| Use an existing adapter Use an adapter → |
Workspace deployment and native plugin conversion for a compatible adapter. | An adapter, a supported agent host, and any dependencies the adapter needs. |
| Adapt to your own tasks Train your adapter → |
Existing templates and Evolvers, with a Trainer that connects execution, feedback, checks, and versioning. | Your tasks, acting agent, evaluator, and model access. |
| Develop a learning method or adapter design Build a learning method → |
Editable adapter specifications and an Evolver interface for implementing and comparing methods on shared task pipelines. | Your adapter design or update algorithm, plus an evaluation task. |
mkdir -p .demo
cp -R hermit/algorithms/templates/prompt_only .demo/adapter
hermit deploy --adapter .demo/adapter --workspace .demo/codex-workspace --host codex
cat .demo/codex-workspace/AGENTS.mdPrepare the dataset and model endpoint as described in the OfficeQA recipe. The agent uses SkillOpt's rollout, so install requirements-skillopt.txt; GEPA also needs requirements-gepa.txt. Sandboxed agent Evolvers require Linux with bubblewrap.
python -m pip install -e . -r requirements-skillopt.txt -r requirements-gepa.txt
python examples/officeqa/utils/prepare.py --csv /path/to/officeqa_full.csv --docs /path/to/treasury_bulletins_parsed
hermit check --config examples/officeqa/gepa/smoke.yaml
hermit train --config examples/officeqa/gepa/smoke.yaml
hermit eval --config examples/officeqa/gepa/smoke.yaml --adapter finalModels use <backend>/<model id> references. See Model backends for endpoint configuration and requirements.
vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enable-auto-tool-choice --tool-call-parser hermes
hermit backends check vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8
hermit train --config examples/officeqa/gepa/smoke.yaml model=vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 llm=vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8hermit convert --adapter DIR --output OUT --target codex
# Other targets: claude-code, pi, hermesHow Is a Harness Adapter Different from a Plugin?
Think of it as a plugin with built-in training support: it defines what can be learned and checked, while Hermit manages updates and versioning from task feedback. A compatible adapter can be deployed as a native plugin.
What Can a Harness Adapter Contain?
A harness adapter can contain instructions, skills, memory, tools, executable code, and agent coordination logic. It can be an entire repository implementing an external harness.
Its specification defines how the agent uses it, which parts can be trained, and how updates are checked—while keeping the original model and harness unchanged.
Why Not Evolve the Original Harness Directly?
Many harnesses are closed-source, and changing a carefully designed harness can disrupt its component interactions. An adapter keeps learned changes in an external layer while leaving the original harness and model fixed.
When Should I Use Hermit?
Use Hermit to deploy an existing adapter, train one on your tasks, or develop and compare harness evolution methods through a shared training interface.
Full FAQ: training, transfer, supported methods, and evaluation →
Help extend Hermit by implementing an Evolver, connecting a task, or reporting a reproduction or integration issue.