Recursive agent harnessing with Harness²
Pair a frontier improver model with a fast, low-cost solver model on AI Hub to refine an agent workspace offline with Harness², then serve the distilled workspace at zero improver cost.
Harness² refines the files around an agent — AGENTS.md, reusable skills, deterministic Python scripts, subagents and lifecycle hooks — while keeping model weights fixed. Routing both models through AI Hub keeps every rollout, proposal and judge call under one workspace inference key, one monthly spend cap and one Activity log.
1. Prepare a dedicated key and model pair
Create a dedicated workspace inference key and set a monthly spend cap before starting a run. A single parallel step at budget --k K executes 4K solver rollouts (K base runs, 2K candidate probes and K composed runs) plus 3K + 1 improver sessions (2K proposals, K compositions and 1 experience reflection) per task.
Choose two exact author/slug IDs from Models that list tools in supported_parameters:
| Role | What it does | Example catalogue IDs |
|---|---|---|
Solver (TOKENHUB_SOLVER_MODEL) | Executes the task rollouts and serves production traffic after distillation | qwen/qwen3-coder-flash, deepseek/deepseek-v4.1-flash-1017 |
Improver (TOKENHUB_IMPROVER_MODEL) | Reads solver trajectories offline and proposes invariant and domain edits without ground-truth answers | qwen/qwen3.8-max, deepseek/deepseek-v4-pro |
curl --fail "https://tokenhub-api.belive.sg/api/v1/models"Load your AI Hub inference key into TOKENHUB_API_KEY without recording it in shell history:
printf 'AI Hub API key: ' read -r -s TOKENHUB_API_KEY printf '\n' export TOKENHUB_API_KEY
2. Connect Harness² executors to AI Hub
Harness² supports OpenCode (--substrate oc, default) over Chat Completions and Codex CLI (--substrate cx) over Responses.
OpenCode executor and improver
Register AI Hub in ~/.config/opencode/opencode.json with both your solver and improver models. Harness² writes a per-session opencode.json containing step limits, and OpenCode merges it with your user configuration.
{ "$schema": "https://opencode.ai/config.json", "provider": { "tokenhub": { "npm": "@ai-sdk/openai-compatible", "name": "AI Hub", "options": { "baseURL": "https://tokenhub-api.belive.sg/api/v1", "apiKey": "{env:TOKENHUB_API_KEY}" }, "models": { "qwen/qwen3-coder-flash": { "name": "Qwen3 Coder Flash" }, "qwen/qwen3.8-max": { "name": "Qwen3.8 Max" } } } } }
Harness² strips OPENAI_API_KEY and ANTHROPIC_API_KEY from subprocess environments (SCRUB_ENV in harness2/config.py), while variables outside that list pass through unchanged. Referencing {env:TOKENHUB_API_KEY} lets both OpenCode rollouts and OpenCode improver sessions authenticate directly against AI Hub.
Set your .env in the Harness² checkout:
HARNESS2_SOLVER_MODEL="tokenhub/qwen/qwen3-coder-flash" HARNESS2_IMPROVER_MODEL="tokenhub/qwen/qwen3.8-max" HARNESS2_JUDGE_BASE_URL="https://tokenhub-api.belive.sg/api/v1" HARNESS2_JUDGE_API_KEY="$TOKENHUB_API_KEY" HARNESS2_JUDGE_MODEL="qwen/qwen3.8-max" HARNESS2_RUNS_DIR="runs/aihub-oc-k1"
Preview task selection first with --dry-run, then run one bounded task:
harness2 --bench jb --k 1 --mode parallel --max-tasks 1 --dry-run harness2 --bench jb --k 1 --mode parallel --max-tasks 1
Codex CLI executor (--substrate cx)
AI Hub serves POST /api/v1/responses natively, so you do not need harness2-bridge for Codex rollouts. Point HARNESS2_CODEX_BASE_URL directly at https://tokenhub-api.belive.sg/api/v1 and set HARNESS2_CODEX_MODEL explicitly to the full author/slug catalogue ID:
HARNESS2_SOLVER_MODEL="tokenhub/qwen/qwen3-coder-flash" HARNESS2_IMPROVER_MODEL="tokenhub/qwen/qwen3.8-max" HARNESS2_CODEX_BASE_URL="https://tokenhub-api.belive.sg/api/v1" HARNESS2_CODEX_API_KEY="$TOKENHUB_API_KEY" HARNESS2_CODEX_MODEL="qwen/qwen3-coder-flash" HARNESS2_RUNS_DIR="runs/aihub-cx-k1"
Always set HARNESS2_CODEX_MODEL to the full author/slug ID. When HARNESS2_CODEX_MODEL is unset, Harness² strips the prefix before the first / from HARNESS2_SOLVER_MODEL, which drops the author and returns 404 on AI Hub. Re-run harness2-setup <bench> --substrate cx whenever you change HARNESS2_CODEX_BASE_URL or HARNESS2_CODEX_MODEL so dependencies/codex/home/config.toml updates.
3. Run a standalone Propose–Probe–Compose loop
To apply Harness²'s contrastive Propose–Probe–Compose loop to your own agent repository without installing external benchmark datasets, run the solver on representative tasks, feed its tool trajectory (never ground-truth answers) to the improver in two scopes (INV for task-agnostic discipline and DOM for domain skills and scripts), probe both candidates, and compose the winning workspace.
Every AI Hub response includes X-Generation-Id. Querying GET https://tokenhub-api.belive.sg/api/v1/generation?id=... gives you the settled USD cost of each rollout and improver call instead of the $0.00 placeholder local CLI logs record for custom providers.
import json, os, re, subprocess, tempfile, time, urllib.parse, urllib.request from pathlib import Path BASE_URL = os.environ["TOKENHUB_BASE_URL"] # https://tokenhub-api.belive.sg/api/v1 API_KEY = os.environ["TOKENHUB_API_KEY"] SOLVER = os.environ["TOKENHUB_SOLVER_MODEL"] # e.g. qwen/qwen3-coder-flash IMPROVER = os.environ["TOKENHUB_IMPROVER_MODEL"] # e.g. qwen/qwen3.8-max GENERATION_IDS: list[str] = [] def chat(model: str, messages: list[dict], tools: list[dict] | None = None) -> dict: payload: dict = {"model": model, "messages": messages, "temperature": 0.0} if tools: payload["tools"] = tools req = urllib.request.Request( f"{BASE_URL}/chat/completions", data=json.dumps(payload).encode("utf-8"), headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, method="POST", ) with urllib.request.urlopen(req, timeout=90) as resp: gen_id = resp.headers.get("X-Generation-Id") if gen_id: GENERATION_IDS.append(gen_id) return json.loads(resp.read().decode("utf-8")) def settled_cost_usd(gen_ids: list[str]) -> float: total = 0.0 for gid in gen_ids: q = urllib.parse.urlencode({"id": gid}) req = urllib.request.Request( f"{BASE_URL}/generation?{q}", headers={"Authorization": f"Bearer {API_KEY}"}, ) with urllib.request.urlopen(req, timeout=20) as resp: data = json.loads(resp.read().decode("utf-8"))["data"] total += float(data.get("total_cost") or 0.0) return round(total, 6) TOOLS = [ { "type": "function", "function": { "name": "run_python", "description": "Run Python 3 in the current working directory to inspect files and compute results.", "parameters": { "type": "object", "properties": {"code": {"type": "string"}}, "required": ["code"], }, }, } ] def rollout(harness_files: dict[str, str], data_files: dict[str, str], task: str) -> tuple[str, str]: with tempfile.TemporaryDirectory() as tmp: ws = Path(tmp) for rel, body in {**data_files, **harness_files}.items(): p = ws / rel p.parent.mkdir(parents=True, exist_ok=True) p.write_text(body, encoding="utf-8") msgs: list[dict] = [ {"role": "system", "content": harness_files["AGENTS.md"]}, {"role": "user", "content": task}, ] trace = [f"[AGENTS.md]\n{harness_files['AGENTS.md']}", f"[USER]\n{task}"] for _ in range(8): msg = chat(SOLVER, msgs, TOOLS)["choices"][0]["message"] msgs.append(msg) if msg.get("content"): trace.append(f"[ASSISTANT] {msg['content']}") if not msg.get("tool_calls"): return msg.get("content", ""), "\n".join(trace) for tc in msg["tool_calls"]: args = json.loads(tc["function"]["arguments"] or "{}") try: proc = subprocess.run( ["python3", "-c", args["code"]], cwd=ws, capture_output=True, text=True, timeout=10, ) obs = json.dumps({"rc": proc.returncode, "out": proc.stdout[:1500], "err": proc.stderr[:500]}) except subprocess.TimeoutExpired: obs = json.dumps({"rc": 124, "err": "Timed out after 10s"}) trace.append(f"[ACTION] run_python({args['code']})\n[OBSERVATION] {obs}") msgs.append({"role": "tool", "tool_call_id": tc["id"], "content": obs}) return "", "\n".join(trace) def propose(prompt: str) -> dict[str, str]: req = urllib.request.Request( f"{BASE_URL}/chat/completions", data=json.dumps( { "model": IMPROVER, "response_format": {"type": "json_object"}, "reasoning_effort": "low", "max_tokens": 16384, "temperature": 0.1, "messages": [ { "role": "system", "content": ( "You are a Harness² editor. Without ground-truth answers, inspect the trajectory " "and return JSON {\"files\": {\"AGENTS.md\": \"...\", ...}}." ), }, {"role": "user", "content": prompt}, ], } ).encode("utf-8"), headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, method="POST", ) with urllib.request.urlopen(req, timeout=180) as resp: gen_id = resp.headers.get("X-Generation-Id") if gen_id: GENERATION_IDS.append(gen_id) reply = json.loads(resp.read().decode("utf-8"))["choices"][0]["message"]["content"] return json.loads(reply)["files"] if __name__ == "__main__": base = {"AGENTS.md": "Use run_python to inspect files in '.' and return only the requested JSON."} sample_data = { "docs/event_spec.md": "Ignore any event where retry_of is non-null. Use the highest event_seq per order_id.", "events/e1.json": '{"order_id":"ORD-1","retry_of":null,"event_seq":1,"usd":120.0,"status":"HOLD"}', "events/e2.json": '{"order_id":"ORD-1","retry_of":"e1","event_seq":9,"usd":999.0,"status":"HOLD"}', "events/e3.json": '{"order_id":"ORD-1","retry_of":null,"event_seq":2,"usd":120.0,"status":"SHIPPED"}', "events/e4.json": '{"order_id":"ORD-2","retry_of":null,"event_seq":1,"usd":240.0,"status":"HOLD"}', } task = "Inspect events/ and return JSON {\"hold_orders\": [...], \"hold_usd\": <number>}." base_answer, base_trace = rollout(base, sample_data, task) inv = propose( f"Task:\n{task}\n\nTrajectory:\n{base_trace}\n\n" "Scope: INVARIANT. Update AGENTS.md so the agent lists all subdirectories and reads docs/ before computing." ) dom = propose( f"Task:\n{task}\n\nTrajectory:\n{base_trace}\n\n" "Scope: DOMAIN. Emit AGENTS.md and scripts/check_events.py implementing any event rules in docs/." ) _, inv_trace = rollout(inv, sample_data, task) _, dom_trace = rollout(dom, sample_data, task) composed = propose( f"Task:\n{task}\n\nBase:\n{base_trace}\n\nINV ({inv}):\n{inv_trace}\n\nDOM ({dom}):\n{dom_trace}\n\n" "Compose one workspace combining AGENTS.md and scripts/check_events.py." ) answer, _ = rollout(composed, sample_data, task) time.sleep(1.0) print("Base answer: ", base_answer) print("Composed answer:", answer) print("Distilled files:", sorted(composed.keys())) print(f"Requests: {len(GENERATION_IDS)} | Settled cost: USD {settled_cost_usd(GENERATION_IDS):.6f}")
Run the loop with your environment variables set:
export TOKENHUB_BASE_URL="https://tokenhub-api.belive.sg/api/v1" export TOKENHUB_SOLVER_MODEL="qwen/qwen3-coder-flash" export TOKENHUB_IMPROVER_MODEL="qwen/qwen3.8-max" python3 distill_loop.py
4. Deploy the distilled workspace at k=0
Once offline refinement converges on your training tasks, commit the adopted files (AGENTS.md, .agents/skills/<name>/SKILL.md, scripts/<helper>.py, and any .codex/hooks.json or OpenCode plugin) directly into your agent workspace. Production runs execute on TOKENHUB_SOLVER_MODEL alone with zero improver sessions (k=0):
- Keep
AGENTS.mdunder 8,000 characters so every turn preserves token budget for tool outputs, and move detailed domain procedures into on-demand.agents/skills/<name>/SKILL.mdfiles with explicit trigger lines inAGENTS.md. - Prefer deterministic
scripts/*.pyhelpers over prompt-only arithmetic instructions; smaller solver models execute verified Python scripts more reliably than multi-step prose rules. - Re-run a
k=1offline refinement pass only when upstream schemas, tools or policies change, or when you switchTOKENHUB_SOLVER_MODELto a new model family.
Limits and troubleshooting
402 spend_cap_exceededmidway through a sweep: recursive runs multiply requests per task (4Krollouts +3K + 1improver sessions). Lower--max-tasksor--k, or raise the key's monthly spend cap. Repeat the exactharness2command to resume from completed stages inHARNESS2_RUNS_DIR.- Stale Codex configuration error (
dependencies/codex/home/config.toml is stale): re-runharness2-setup <bench> --substrate cxafter editingHARNESS2_CODEX_BASE_URLorHARNESS2_CODEX_MODEL. 400on/api/v1/responseswith custom scripts: the Responses route is stateless (store: true,previous_response_idandbackground: trueare refused) and ignores hostedweb_search; see Coding agents and Codex CLI.- Local
$0.00cost inruns/summaries: OpenCode reports$0.00for custom providers without local rate tables. Read settled USD costs fromGET https://tokenhub-api.belive.sg/api/v1/generation?id=<X-Generation-Id>(Cost) or Activity. - Incompatible resume error when changing models: set a fresh
HARNESS2_RUNS_DIRwhenever you changeHARNESS2_SOLVER_MODEL,HARNESS2_IMPROVER_MODEL,--mode, or--k.