pi-hydra: many heads, one cheap lesson each

I've been running a multi-agent system for a while now — five specialised agents, each with its own tools, memory, and model configuration, coordinated through sync and async delegation, taskrails, and a reflection loop that reviews failures after the session ends. It worked, but it was wired to this environment. Hard-coded agent names, repo-relative paths, environment variables prefixed with DAD_, and a runner script that assumed the Node modules were in a specific place.

That's fine for a prototype. It is not fine as something you can publish, share, or lean on in the long term.

pi-hydra is the extraction. One package, five generalised extensions, and a design principle I came to trust: heads are data, not code.

What it is

pi-hydra is a pi package that turns a single agent into a multi-headed system. It ships five extensions:

Every agent in a pi-hydra system gets access to the same tool set — delegation, taskrails, reflection, registry onboarding — but the heads they delegate to are entirely configurable. The package ships with six sensible defaults (general, researcher, developer, writer, reviewer, ops) that map to least-privilege tool sets out of the box. Override or extend them per-user or per-project in a plain JSON config file.

Heads are data, not code

This was the single design decision that made everything else fall into place. A head is not a script, not a subprocess definition — it is a configuration object with a system prompt, a model identifier, and a tool allowlist. You define it in JSON:

{
  "heads": {
    "developer": {
      "promptFile": "heads/developer.md",
      "model": "anthropic/claude-sonnet-4",
      "tools": ["read", "bash", "edit", "write", "grep", "find", "ls"]
    },
    "my-local-agent": { "dir": "/home/me/agents/my-agent" }
  }
}

Package defaults ship as JSON. User overrides merge on top. Project-level config wins everything. The same resolution chain powers the registry: when onboard_agent installs a TAO package, it writes a heads.d/ file that sits below anything hand-written in hydra.json, so onboarding never rewrites a maintained config.

Because heads are data, the fleet is inspectable, auditable, and source-controllable. There is no runtime code generation, no config DSL, and no hidden state inside the package.

The transport decision: measured, not argued

When I started designing the registry layer, I assumed we would need a release pipeline — GitHub Releases with attached tarballs, or npm packages with semantic versioning. Before committing to the plumbing, I measured what already existed.

It turned out that github.com/<repo>/archive/<sha>.tar.gz is already an immutable, unauthenticated tarball transport. No auth, no git, no API key. Fetching a package at a pinned revision is a single curl call. The tree hash that TAO generates from the curated file set survives the round-trip perfectly — because it hashes content, not tarball bytes, it works regardless of how GitHub builds the archive.

So the plan simplified: the index is the discovery half; the repository is the acquisition half. No release pipeline. npm was deferred on the record — its Dual-Use Content Policy means security-adjacent agents need interactive 2FA for publishing, and that is a CI wrinkle I do not need when a tarball over TLS does the job.

TAO integration: a package is a head

The registry layer means pi-hydra can consume TAO agent packages directly. The mapping turned out to be nearly isomorphic — TAO's SYSTEM.md at the package root maps to pi-hydra's head system prompt, taskrails are byte-identical, and agent.yaml authority declarations map cleanly onto tool allowlists.

Two tools expose the registry:

list_registry_agents — reads the registry index, shows what is installable and what is already installed. A verify mode re-hashes every installed package against its pinned revision and reports drift.

onboard_agent — fetches a package at a pinned revision from the index, verifies its tree hash against the index entry, materialises exactly the hashed file set (never the extracted tree — so excluded directories cannot smuggle unhashed content into an install), and writes a head entry. If the package declares requires (a team's members), those install too — teams as files only, because a team is a composition, not a single agent.

The resolution contract is deliberately simple:

fetch   = <repository>/archive/<revision>.tar.gz
unpack  = that tarball's <path>
verify  = tree hash of the package == the index's sha256

A 40-hex revision is required in the index; an unpinned index is refused with an actionable error. The tree hash is a cross-implementation contract — pi-hydra's implementation is pinned against the Python one in TAO's build script, so they cannot drift apart.

What an onboarded head may do. A package that declares no mutation gets read-only tools (read, ls, grep, find, bash — every TAO package gets bash because its SOPs are scripts). edit and write are added only when the manifest declares a mutation that is not prohibited. The honest limit: pi-hydra has no approval UI, so TAO's read | propose | mutate | destructive authority stays doctrine-enforced, not harness-enforced. bash is equivalent to full user authority anyway. Review a package's source before onboarding it.

The three things I got wrong

A build log is honest about what broke. These three failures taught me more about the design than the working parts did.

1. A clone fired a review of a session that was still alive

/clone, /fork, /new, and /resume do not end a session — they replace it inside the running process. The shutdown event carries a reason field ("quit" | "reload" | "new" | "resume" | "fork"), and only "quit" means the process is ending.

The reflection extension (rsi) was skipping "reload" but not the others. So a /clone mid-session fired a review of a still-live session, spawning a legacy runner that inspected a failure log from a conversation that was actively running. The runner then wedged as a stray process because the session it was reviewing had moved on.

The fix is one predicate: shouldReviewOnShutdown(reason) → reason === "quit". But that revealed a deeper problem.

Sessions accumulate into families. A fork creates siblings, not a chain. Following the latest parent chain — the obvious first design — silently ignores every branch except the last. So sessions now accumulate into a family, stored as an array of members, and at the real exit every branch is merged into one failure log for a single review. A failure present on every branch reads as systemic; one confined to a single branch reads as local.

2. A dependency that could never have loaded

Adding a YAML parser as a dependency looked obvious — pi-hydra needs to read agent.yaml manifests. It was only when I tested on a real install that I realised: extensions load from <agent-dir>/git/<host>/<owner>/<repo>, a directory that has no node_modules. The host aliases exactly two bare specifiers for extensions: the pi SDK and typebox. Any other bare import fails at load time.

So pi-hydra ships small in-repo readers instead. The YAML reader (lib/yaml-min.ts) is checked byte-for-byte against PyYAML on all real TAO manifests before it trusts its output. The tar reader (lib/tar.ts) verifies every header checksum and refuses absolute paths, .., symlinks, hardlinks, and device nodes — because a manifest the parser cannot fully understand is an install we should not perform.

The same constraint applied to the async runner's module resolution, and was resolved the same way: the parent extension resolves pi's package directory via the exported getPackageDir() and hands it to the child process by environment variable, so the child can import the SDK by absolute path instead of by bare specifier. Verified working from /tmp.

3. The bug that only an end-to-end run could find

The unit test suite passed 75/75 while this was broken.

A fork launched as a new process (e.g. pi --fork) reports reason: "startup" with no previousSessionFile — the lineage exists only in a field of the session file's header. An implementation driven by events alone silently starts a second family, and the fan-out across branches never happens. The fix was reading pi's SessionHeader.parentSession from the session manager, not relying on the event stream.

Lesson: the event stream is not the whole truth. The session file header is the authoritative record of lineage, and unit tests that mock events will never catch the gap. Only an end-to-end run against a real pi process — with a real fork — revealed it.

Why no dependencies

pi-hydra ships zero runtime dependencies, deliberately. The same constraint that caught the YAML parser applies to everything: extensions load from a bare directory with no node_modules, and any third-party dependency would fail to resolve at load time. The in-repo readers are strict, tested, and validated against real-world inputs. The async runner uses the SDK by absolute path rather than by bare import.

This is not a point of pride — it is a constraint that forced simpler, more auditable code. If the extension loading story changes in the future, some of these readers could become proper dependencies. For now, the trade-off is comfortable.

Status

All five extensions are generalised, installed on this host, and tested (75 unit tests plus end-to-end verification in isolated pi agent directories). The TAO registry integration phases 0–2 are shipped: a TAO package can be discovered via the index, fetched, verified, installed, and run as a head. Composition — the LLM-driven ad-hoc team builder and the deterministic team.yaml → DAG adapter — is designed but not built.

The repo exists. It is private for now. If you are interested in the pattern, the TAO registry is public and the TAO documentation describes the agent format it accepts.

---

Filed under: agentic systems, infrastructure, building in the open.