Cookbook

Recursive agent harnessing with Harness²

Pair a frontier improver model with a fast, low-cost solver model on AI Hub to refine an agent workspace offline with Harness², then serve the distilled workspace at zero improver cost.

Harness² refines the files around an agent — AGENTS.md, reusable skills, deterministic Python scripts, subagents and lifecycle hooks — while keeping model weights fixed. Routing both models through AI Hub keeps every rollout, proposal and judge call under one workspace inference key, one monthly spend cap and one Activity log.

1. Prepare a dedicated key and model pair

Create a dedicated workspace inference key and set a monthly spend cap before starting a run. A single parallel step at budget --k K executes 4K solver rollouts (K base runs, 2K candidate probes and K composed runs) plus 3K + 1 improver sessions (2K proposals, K compositions and 1 experience reflection) per task.

Choose two exact author/slug IDs from Models that list tools in supported_parameters:

RoleWhat it doesExample catalogue IDs
Solver (TOKENHUB_SOLVER_MODEL)Executes the task rollouts and serves production traffic after distillationqwen/qwen3-coder-flash, deepseek/deepseek-v4.1-flash-1017
Improver (TOKENHUB_IMPROVER_MODEL)Reads solver trajectories offline and proposes invariant and domain edits without ground-truth answersqwen/qwen3.8-max, deepseek/deepseek-v4-pro
bash
curl --fail "https://tokenhub-api.belive.sg/api/v1/models"

Load your AI Hub inference key into TOKENHUB_API_KEY without recording it in shell history:

bash
printf 'AI Hub API key: '
read -r -s TOKENHUB_API_KEY
printf '\n'
export TOKENHUB_API_KEY

2. Connect Harness² executors to AI Hub

Harness² supports OpenCode (--substrate oc, default) over Chat Completions and Codex CLI (--substrate cx) over Responses.

OpenCode executor and improver

Register AI Hub in ~/.config/opencode/opencode.json with both your solver and improver models. Harness² writes a per-session opencode.json containing step limits, and OpenCode merges it with your user configuration.

json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "tokenhub": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "AI Hub",
      "options": {
        "baseURL": "https://tokenhub-api.belive.sg/api/v1",
        "apiKey": "{env:TOKENHUB_API_KEY}"
      },
      "models": {
        "qwen/qwen3-coder-flash": { "name": "Qwen3 Coder Flash" },
        "qwen/qwen3.8-max": { "name": "Qwen3.8 Max" }
      }
    }
  }
}

Harness² strips OPENAI_API_KEY and ANTHROPIC_API_KEY from subprocess environments (SCRUB_ENV in harness2/config.py), while variables outside that list pass through unchanged. Referencing {env:TOKENHUB_API_KEY} lets both OpenCode rollouts and OpenCode improver sessions authenticate directly against AI Hub.

Set your .env in the Harness² checkout:

bash
HARNESS2_SOLVER_MODEL="tokenhub/qwen/qwen3-coder-flash"
HARNESS2_IMPROVER_MODEL="tokenhub/qwen/qwen3.8-max"
HARNESS2_JUDGE_BASE_URL="https://tokenhub-api.belive.sg/api/v1"
HARNESS2_JUDGE_API_KEY="$TOKENHUB_API_KEY"
HARNESS2_JUDGE_MODEL="qwen/qwen3.8-max"
HARNESS2_RUNS_DIR="runs/aihub-oc-k1"

Preview task selection first with --dry-run, then run one bounded task:

bash
harness2 --bench jb --k 1 --mode parallel --max-tasks 1 --dry-run
harness2 --bench jb --k 1 --mode parallel --max-tasks 1

Codex CLI executor (--substrate cx)

AI Hub serves POST /api/v1/responses natively, so you do not need harness2-bridge for Codex rollouts. Point HARNESS2_CODEX_BASE_URL directly at https://tokenhub-api.belive.sg/api/v1 and set HARNESS2_CODEX_MODEL explicitly to the full author/slug catalogue ID:

bash
HARNESS2_SOLVER_MODEL="tokenhub/qwen/qwen3-coder-flash"
HARNESS2_IMPROVER_MODEL="tokenhub/qwen/qwen3.8-max"
HARNESS2_CODEX_BASE_URL="https://tokenhub-api.belive.sg/api/v1"
HARNESS2_CODEX_API_KEY="$TOKENHUB_API_KEY"
HARNESS2_CODEX_MODEL="qwen/qwen3-coder-flash"
HARNESS2_RUNS_DIR="runs/aihub-cx-k1"

Always set HARNESS2_CODEX_MODEL to the full author/slug ID. When HARNESS2_CODEX_MODEL is unset, Harness² strips the prefix before the first / from HARNESS2_SOLVER_MODEL, which drops the author and returns 404 on AI Hub. Re-run harness2-setup <bench> --substrate cx whenever you change HARNESS2_CODEX_BASE_URL or HARNESS2_CODEX_MODEL so dependencies/codex/home/config.toml updates.

3. Run a standalone Propose–Probe–Compose loop

To apply Harness²'s contrastive Propose–Probe–Compose loop to your own agent repository without installing external benchmark datasets, run the solver on representative tasks, feed its tool trajectory (never ground-truth answers) to the improver in two scopes (INV for task-agnostic discipline and DOM for domain skills and scripts), probe both candidates, and compose the winning workspace.

Every AI Hub response includes X-Generation-Id. Querying GET https://tokenhub-api.belive.sg/api/v1/generation?id=... gives you the settled USD cost of each rollout and improver call instead of the $0.00 placeholder local CLI logs record for custom providers.

python
import json, os, re, subprocess, tempfile, time, urllib.parse, urllib.request
from pathlib import Path

BASE_URL = os.environ["TOKENHUB_BASE_URL"]          # https://tokenhub-api.belive.sg/api/v1
API_KEY = os.environ["TOKENHUB_API_KEY"]
SOLVER = os.environ["TOKENHUB_SOLVER_MODEL"]        # e.g. qwen/qwen3-coder-flash
IMPROVER = os.environ["TOKENHUB_IMPROVER_MODEL"]    # e.g. qwen/qwen3.8-max
GENERATION_IDS: list[str] = []


def chat(model: str, messages: list[dict], tools: list[dict] | None = None) -> dict:
    payload: dict = {"model": model, "messages": messages, "temperature": 0.0}
    if tools:
        payload["tools"] = tools
    req = urllib.request.Request(
        f"{BASE_URL}/chat/completions",
        data=json.dumps(payload).encode("utf-8"),
        headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=90) as resp:
        gen_id = resp.headers.get("X-Generation-Id")
        if gen_id:
            GENERATION_IDS.append(gen_id)
        return json.loads(resp.read().decode("utf-8"))


def settled_cost_usd(gen_ids: list[str]) -> float:
    total = 0.0
    for gid in gen_ids:
        q = urllib.parse.urlencode({"id": gid})
        req = urllib.request.Request(
            f"{BASE_URL}/generation?{q}",
            headers={"Authorization": f"Bearer {API_KEY}"},
        )
        with urllib.request.urlopen(req, timeout=20) as resp:
            data = json.loads(resp.read().decode("utf-8"))["data"]
            total += float(data.get("total_cost") or 0.0)
    return round(total, 6)


TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "run_python",
            "description": "Run Python 3 in the current working directory to inspect files and compute results.",
            "parameters": {
                "type": "object",
                "properties": {"code": {"type": "string"}},
                "required": ["code"],
            },
        },
    }
]


def rollout(harness_files: dict[str, str], data_files: dict[str, str], task: str) -> tuple[str, str]:
    with tempfile.TemporaryDirectory() as tmp:
        ws = Path(tmp)
        for rel, body in {**data_files, **harness_files}.items():
            p = ws / rel
            p.parent.mkdir(parents=True, exist_ok=True)
            p.write_text(body, encoding="utf-8")
        msgs: list[dict] = [
            {"role": "system", "content": harness_files["AGENTS.md"]},
            {"role": "user", "content": task},
        ]
        trace = [f"[AGENTS.md]\n{harness_files['AGENTS.md']}", f"[USER]\n{task}"]
        for _ in range(8):
            msg = chat(SOLVER, msgs, TOOLS)["choices"][0]["message"]
            msgs.append(msg)
            if msg.get("content"):
                trace.append(f"[ASSISTANT] {msg['content']}")
            if not msg.get("tool_calls"):
                return msg.get("content", ""), "\n".join(trace)
            for tc in msg["tool_calls"]:
                args = json.loads(tc["function"]["arguments"] or "{}")
                try:
                    proc = subprocess.run(
                        ["python3", "-c", args["code"]],
                        cwd=ws, capture_output=True, text=True, timeout=10,
                    )
                    obs = json.dumps({"rc": proc.returncode, "out": proc.stdout[:1500], "err": proc.stderr[:500]})
                except subprocess.TimeoutExpired:
                    obs = json.dumps({"rc": 124, "err": "Timed out after 10s"})
                trace.append(f"[ACTION] run_python({args['code']})\n[OBSERVATION] {obs}")
                msgs.append({"role": "tool", "tool_call_id": tc["id"], "content": obs})
    return "", "\n".join(trace)


def propose(prompt: str) -> dict[str, str]:
    req = urllib.request.Request(
        f"{BASE_URL}/chat/completions",
        data=json.dumps(
            {
                "model": IMPROVER,
                "response_format": {"type": "json_object"},
                "reasoning_effort": "low",
                "max_tokens": 16384,
                "temperature": 0.1,
                "messages": [
                    {
                        "role": "system",
                        "content": (
                            "You are a Harness² editor. Without ground-truth answers, inspect the trajectory "
                            "and return JSON {\"files\": {\"AGENTS.md\": \"...\", ...}}."
                        ),
                    },
                    {"role": "user", "content": prompt},
                ],
            }
        ).encode("utf-8"),
        headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=180) as resp:
        gen_id = resp.headers.get("X-Generation-Id")
        if gen_id:
            GENERATION_IDS.append(gen_id)
        reply = json.loads(resp.read().decode("utf-8"))["choices"][0]["message"]["content"]
    return json.loads(reply)["files"]


if __name__ == "__main__":
    base = {"AGENTS.md": "Use run_python to inspect files in '.' and return only the requested JSON."}
    sample_data = {
        "docs/event_spec.md": "Ignore any event where retry_of is non-null. Use the highest event_seq per order_id.",
        "events/e1.json": '{"order_id":"ORD-1","retry_of":null,"event_seq":1,"usd":120.0,"status":"HOLD"}',
        "events/e2.json": '{"order_id":"ORD-1","retry_of":"e1","event_seq":9,"usd":999.0,"status":"HOLD"}',
        "events/e3.json": '{"order_id":"ORD-1","retry_of":null,"event_seq":2,"usd":120.0,"status":"SHIPPED"}',
        "events/e4.json": '{"order_id":"ORD-2","retry_of":null,"event_seq":1,"usd":240.0,"status":"HOLD"}',
    }
    task = "Inspect events/ and return JSON {\"hold_orders\": [...], \"hold_usd\": <number>}."

    base_answer, base_trace = rollout(base, sample_data, task)
    inv = propose(
        f"Task:\n{task}\n\nTrajectory:\n{base_trace}\n\n"
        "Scope: INVARIANT. Update AGENTS.md so the agent lists all subdirectories and reads docs/ before computing."
    )
    dom = propose(
        f"Task:\n{task}\n\nTrajectory:\n{base_trace}\n\n"
        "Scope: DOMAIN. Emit AGENTS.md and scripts/check_events.py implementing any event rules in docs/."
    )
    _, inv_trace = rollout(inv, sample_data, task)
    _, dom_trace = rollout(dom, sample_data, task)
    composed = propose(
        f"Task:\n{task}\n\nBase:\n{base_trace}\n\nINV ({inv}):\n{inv_trace}\n\nDOM ({dom}):\n{dom_trace}\n\n"
        "Compose one workspace combining AGENTS.md and scripts/check_events.py."
    )
    answer, _ = rollout(composed, sample_data, task)
    time.sleep(1.0)
    print("Base answer:    ", base_answer)
    print("Composed answer:", answer)
    print("Distilled files:", sorted(composed.keys()))
    print(f"Requests: {len(GENERATION_IDS)} | Settled cost: USD {settled_cost_usd(GENERATION_IDS):.6f}")

Run the loop with your environment variables set:

bash
export TOKENHUB_BASE_URL="https://tokenhub-api.belive.sg/api/v1"
export TOKENHUB_SOLVER_MODEL="qwen/qwen3-coder-flash"
export TOKENHUB_IMPROVER_MODEL="qwen/qwen3.8-max"
python3 distill_loop.py

4. Deploy the distilled workspace at k=0

Once offline refinement converges on your training tasks, commit the adopted files (AGENTS.md, .agents/skills/<name>/SKILL.md, scripts/<helper>.py, and any .codex/hooks.json or OpenCode plugin) directly into your agent workspace. Production runs execute on TOKENHUB_SOLVER_MODEL alone with zero improver sessions (k=0):

  • Keep AGENTS.md under 8,000 characters so every turn preserves token budget for tool outputs, and move detailed domain procedures into on-demand .agents/skills/<name>/SKILL.md files with explicit trigger lines in AGENTS.md.
  • Prefer deterministic scripts/*.py helpers over prompt-only arithmetic instructions; smaller solver models execute verified Python scripts more reliably than multi-step prose rules.
  • Re-run a k=1 offline refinement pass only when upstream schemas, tools or policies change, or when you switch TOKENHUB_SOLVER_MODEL to a new model family.

Limits and troubleshooting

  • 402 spend_cap_exceeded midway through a sweep: recursive runs multiply requests per task (4K rollouts + 3K + 1 improver sessions). Lower --max-tasks or --k, or raise the key's monthly spend cap. Repeat the exact harness2 command to resume from completed stages in HARNESS2_RUNS_DIR.
  • Stale Codex configuration error (dependencies/codex/home/config.toml is stale): re-run harness2-setup <bench> --substrate cx after editing HARNESS2_CODEX_BASE_URL or HARNESS2_CODEX_MODEL.
  • 400 on /api/v1/responses with custom scripts: the Responses route is stateless (store: true, previous_response_id and background: true are refused) and ignores hosted web_search; see Coding agents and Codex CLI.
  • Local $0.00 cost in runs/ summaries: OpenCode reports $0.00 for custom providers without local rate tables. Read settled USD costs from GET https://tokenhub-api.belive.sg/api/v1/generation?id=<X-Generation-Id> (Cost) or Activity.
  • Incompatible resume error when changing models: set a fresh HARNESS2_RUNS_DIR whenever you change HARNESS2_SOLVER_MODEL, HARNESS2_IMPROVER_MODEL, --mode, or --k.