Hive Member Tracking & Cost-Aware Routing

Status: Partially shipped — capability matching + eval-driven model selection live (#285); stable node UUID and hive-level cost aggregation pending. Date: 24.jun.2026 Driven by: T1.5 per-task cost tracking (shipped) → T2.x routing

Companion doc: hive-pane — the operator presentation layer this engine feeds. Hive Pane is the “how it looks” (board, A2A discovery); this doc is the “how it works” (identity, probes, routing, scheduling).

What Exists Today

Component	State	Gap
`mother_schema.sql`	`hive_nodes` table with `hw_profile` + `capabilities` JSONB	No stable node UUID; hostname is the key
`derive_capabilities()` trigger	Auto-computes `has_gpu`, `gpu_vendor`, `can_run_local_llm`, `max_model` from hw_profile	Only GPU/VRAM heuristics — doesn’t probe running services
`clawdie-system-probe`	Collects GPU, RAM, CPU, disks, ZFS, WiFi, Vulkan, Colibri status	No ollama/llama.cpp probing
`node-register-mcp`	UPSERTs hw_profile into `hive_nodes` on join — reachable over the bridge since `node_register` was allowlisted (#325, see ssh-bridge)	No UUID generation at join time
`crates/colibri-daemon/src/scheduler.rs`	Cron/interval/one-shot jobs, capability matching (`pick_agent`), eval-driven `select_model`	Selection is per-host; no cross-hive awareness yet
`colibri-ledger`	Local SQLite `agents` table with UUID (v4 random)	UUID is session-local, not hive-stable
T1.5 cost tracking	Per-task cost captured in local SQLite	No hive-level cost aggregation

Design Goals

Stable identity — A node that joins, leaves, and rejoins is the same node. Not hostname-based (hostnames change when re-provisioned).
Capability matrix — What can each member do? Not just hardware, but running services: ollama, llama.cpp, available models, provider API keys, cost tier.
Verify, don’t guess — Every capability in the matrix comes from a probe result, not self-declaration. The hw-probe is the single source of truth; the derive_capabilities() trigger maps hardware facts → capability booleans.
Cost-aware routing — When a task is dispatched, the scheduler considers: urgency, provider cost, local LLM availability, cache-hit potential, and capability match.
Local LLM tier — A beefy member can serve as a “free but slow” execution target for non-urgent tasks. The cost model treats local execution as $0.0000/task.
Extensible — New backends (ollama, llama.cpp, vLLM, Exo clusters) slot into the same capability matrix without schema changes.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        MOTHER (osa)                             │
│  ┌─────────────────┐  ┌──────────────┐  ┌───────────────────┐  │
│  │ PostgreSQL       │  │ Scheduler    │  │ MCP Bridge        │  │
│  │  hive_nodes      │  │  cost-aware  │  │  colibri-mcp-ssh  │  │
│  │  capabilities    │  │  routing     │  │  node-register    │  │
│  │  cost_history    │  │  dispatch    │  │  cost-query       │  │
│  └─────────────────┘  └──────────────┘  └───────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
         ▲                              │
         │ hw-probe + capabilities      │ task dispatch
         │ (MCP tools/call)             │ (MCP or direct)
         │                              ▼
┌────────────────────┐    ┌────────────────────┐
│  Node: clawdie-a   │    │  Node: clawdie-b   │
│  ┌──────────────┐  │    │  ┌──────────────┐  │
│  │ Colibri      │  │    │  │ Colibri      │  │
│  │  zot spawner │  │    │  │  zot spawner │  │
│  │  local SQLite│  │    │  │  ollama      │  │
│  └──────────────┘  │    │  │  llama.cpp   │  │
│  GPU: none         │    │  │  models:     │  │
│  RAM: 8GB          │    │  │   qwen2.5:7b │  │
│  Cost: cloud-only  │    │  └──────────────┘  │
│                    │    │  GPU: RTX 4090     │
│                    │    │  RAM: 64GB         │
│                    │    │  Cost: $0 local    │
└────────────────────┘    └────────────────────┘

Identity: Machine UUID

Problem

Hostname is unstable. A live-USB node gets clawdie on every boot. A disk-installed node keeps the hostname the operator set. Mother needs a stable, verifiable identity that survives re-provisioning.

Solution: `/var/db/machine-id`

A 32-character hex UUID generated once, stored locally, included in every hw-probe report. Like systemd’s /etc/machine-id but simpler: only one purpose — hive identity.

# Generated once by clawdie-firstboot or operator session
/var/db/machine-id → a1b2c3d4-e5f6-7890-abcd-ef1234567890

Properties:

Stable across reboots: stored on disk, not tmpfs
Survives re-provisioning: if the seed partition preserves /var/db/machine-id, the same physical machine keeps the same identity
Not a secret: it’s an ID, not a key
Verifiable: mother can check “has node a1b2c3d4 ever joined?” — if yes, this is a rejoin, not a new node

Alternatives considered:

Approach	Pros	Cons
SMBIOS UUID (`hw.uuid`)	Truly hardware-bound, survives OS reinstall	Not available on all platforms (VPS, ARM); can be spoofed
SSH host key fingerprint	Cryptographically strong	Changes on OS reinstall; key rotation breaks identity
Random UUID (this design)	Portable, simple, survives seed restore	Can be copied/cloned (but same machine, same ID — that’s correct)

Recommendation: Generate on first boot, store in /var/db/machine-id. The hw-probe includes it as machine_id. Mother’s hive_nodes table gets a UNIQUE constraint on machine_id.

Schema change

ALTER TABLE hive_nodes ADD COLUMN machine_id TEXT;
ALTER TABLE hive_nodes ADD CONSTRAINT uq_machine_id UNIQUE (machine_id);

The node-register-mcp UPSERT switches from ON CONFLICT (hostname) to ON CONFLICT (machine_id). Hostname becomes a mutable attribute (updates on rejoin), machine_id becomes the stable key.

Capability Matrix

What goes in the matrix

Every capability is a boolean derived from hardware facts, not a self-declaration. The hw-probe collects hardware; the trigger derives capabilities.

Capability	Derived from	Used for
`has_gpu`	GPU detected in pciconf	GPU-accelerated inference
`gpu_vendor`	amdgpu/nvidia driver	Model compatibility
`vulkan_compute`	vulkaninfo success	llama.cpp Vulkan backend
`can_run_local_llm`	RAM ≥ 16GB or has GPU	Eligibility for local task execution
`max_model`	RAM heuristic	Model size limit (3b, 7b-q4, 13b-q4, 34b-q4)
`cpu_only`	No GPU detected	Fallback only (slow)
`has_wifi`	wlan devices	Network capability
`has_zfs`	ZFS pools non-empty	Storage capability
`colibri_running`	service status	Agent host eligibility
`provider_api_keys`	MCP-reported (not hw probe)	Cloud provider availability

Local LLM capabilities (NEW)

Extend the hw-probe to detect running local LLM services and extend the trigger to derive capabilities from them:

{
  "local_llm": {
    "ollama_running": true,
    "ollama_models": ["qwen2.5:7b", "deepseek-r1:8b", "nomic-embed-text"],
    "llama_cpp_installed": true,
    "llama_cpp_models": ["/var/db/models/qwen2.5-7b-q4.gguf"],
    "vulkan_support": true
  }
}

New derived capabilities:

Capability	Derivation
`ollama_available`	`ollama_running == true`
`ollama_models`	Array of model tags (from `ollama list`)
`llama_cpp_available`	Binary at `/usr/local/bin/llama-server` or similar
`llama_cpp_models`	GGUFs in `/var/db/models/` or `/usr/local/share/models/`
`can_embed_locally`	`nomic-embed-text` in ollama OR any embedding model loaded
`inference_tier`	`local-fast` (GPU ≥ 24GB), `local-slow` (CPU-only, RAM ≥ 16GB), `cloud-only`

Probe additions to `clawdie-system-probe`

# New collectors
collect_machine_id()   # cat /var/db/machine-id or generate
collect_ollama_status() # ollama list 2>/dev/null (JSON models)
collect_llama_cpp()    # which llama-server; ls /var/db/models/*.gguf
collect_local_llm()    # aggregate JSON block

Cost-Aware Routing

The routing decision

When a task arrives at the scheduler, it computes a routing score for each eligible node:

score = capability_match × urgency_weight × cost_weight × cache_weight

capability_match: 0.0–1.0 (does the node have the required skills/model?)
cost_weight:       0.0–1.0 (lower cost → higher weight)
urgency_weight:    0.0–1.0 (time-critical tasks penalize slow nodes)
cache_weight:      0.0–1.0 (warm cache → higher weight)

Cost tiers

Tier	Provider	Cost per 1M tokens	Latency	Used when
T0 (free)	Local ollama/llama.cpp	$0.00	5–60s	Non-urgent, capability match
T1 (cheap)	DeepSeek V3	$0.27 / $1.10	2–5s	Default for most tasks
T2 (balanced)	Gemini Flash	$0.15 / $0.60	1–3s	High cache-hit tasks
T3 (premium)	Claude Sonnet 4	$3.00 / $15.00	3–8s	Complex reasoning, only when needed

Local LLM routing rules

If task is non-urgent AND a hive member has a matching local model → route locally at $0.00 cost.
If the local model is unavailable (node down, model not loaded) → fall back to T1 (DeepSeek).
If task is urgent (latency < 5s required) → skip local tier, go straight to T1.
Embedding tasks (RAG, similarity search) → always prefer local if available. Embeddings are cheap to compute locally and don’t need reasoning.

How the scheduler knows

The scheduler queries hive_nodes for all online nodes, filters by capabilities.can_run_local_llm, checks ollama_models for the required model, and computes the routing score. If no local node matches, it falls back to the cloud tier.

The task schema gets a new field:

ALTER TABLE tasks ADD COLUMN routing JSONB;
-- {"preferred_tier": "local", "allowed_tiers": ["local", "cheap"], "max_cost_usd": 0.05, "deadline_s": null}

Protocol: Join → Probe → Route

A2A integration: See hive-pane for the Agent Card, task exchange, and typed cost data parts. The routing engine (this doc) and the A2A protocol (hive-pane) are orthogonal layers — either can evolve independently.

Phase 1: Join (existing, extended)

Node boots → clawdie-system-probe runs → MCP tools/call node_register
→ mother UPSERTs hive_nodes → derive_capabilities() trigger fires
→ capabilities JSONB updated → node is "online"

New: machine_id is included. If the machine_id already exists, mother updates the existing row (rejoin), preserving history.

Phase 2: Heartbeat (existing)

colibri-daemon sends periodic heartbeat via MCP. Updates last_seen. If heartbeat misses for > 5 minutes, node status → offline.

Phase 3: Capability Sync (new)

On heartbeat, the node can optionally push updated capabilities (if ollama was installed, models changed, etc.). The hw-probe is re-run and the new local_llm block is sent.

Phase 4: Task Dispatch (new)

Scheduler picks a queued task
  → queries hive_nodes for eligible nodes
  → computes routing score for each
  → picks winner
  → dispatches task via MCP or direct agent spawn
  → writes routing decision to task.routing JSONB

Phase 5: Cost Capture (existing, extended)

When the task completes, the local daemon writes cost to its SQLite (T1.5). The mother aggregates hive cost via periodic MCP queries or push events.

Three Implementation Options

Option A — Mother-Centric (Minimal New Code)

What: Mother is the brain. Nodes register, mother routes. No peer-to-peer.

Implementation:

Add machine_id to hive_nodes + hw-probe (1 day)
Extend derive_capabilities() for local LLM (1 day)
Add routing_score() function to mother’s PostgreSQL (stored function — zero Rust changes)
Extend node-register-mcp to accept local_llm block (0.5 day)
Add local_llm probe to clawdie-system-probe (1 day)

Rust changes: Scheduler reads capabilities from hive_nodes via MCP query, computes score, dispatches. ~200 lines.

Total: ~3.5 days.

Pros:

Simple to reason about — one source of truth
Lowest implementation risk
Scheduler lives on mother (always-on)
Existing MCP bridge handles all communication

Cons:

Mother is single point of failure for routing (but not execution — once dispatched, the task runs independently)
Latency: scheduler must query mother on every tick
Doesn’t scale to 100+ nodes (not a real concern for our use case)

Option B — Capability-Advertised with Local Routing Fallback

What: Mother stores the matrix, but nodes can also route tasks they own to peers directly. Hybrid: central registry + distributed execution.

Implementation:

All of Option A (3.5 days)
Add capabilities API to colibri-daemon’s Unix socket (self-awareness) — 1 day
Add local peer discovery via mDNS or Tailscale whois — 1 day
Add direct peer-to-peer task dispatch via Unix socket → MCP → remote — 2 days
Add fallback logic: “try local first, if no response in 30s, ask mother” — 1 day

Total: ~8.5 days.

Pros:

Lower latency for local dispatch
Survives mother downtime for peer-to-peer tasks
Natural fit for local LLM use case (beefy node is on same LAN)
Nodes that discover each other can route without phoning home

Cons:

Complexity: two code paths (central + peer-to-peer)
Security: peer-to-peer dispatch needs authentication (who can send tasks to my daemon?)
Harder to audit: cost tracking must handle peer-dispatched vs mother-dispatched tasks differently
mDNS doesn’t work across subnets (Tailscale works but adds dependency)

Option C — Capability-Matrix-as-Skill (Zero-Code Routing)

What: Don’t build a routing engine at all. The capability matrix is exposed as an MCP tool that agents query. The agent itself decides where to route based on the matrix + its own reasoning. The matrix is advisory, not prescriptive.

Implementation:

All of Option A minus the routing_scoring function (2.5 days)
Add colibri_query_hive_capabilities MCP tool on mother — returns full online node matrix (0.5 day)
Add colibri_dispatch_to_node MCP tool — sends task to a specific node (1 day)
Write a hive-routing skill that teaches agents how to use the matrix (0.5 day)

Total: ~4.5 days. Zero scheduler changes.

Pros:

Exploits Colibri’s architecture-as-differentiator: the agent IS the intelligence
The routing decision is auditable in the conversation log (why did the agent pick this node?)
Natural fit for local LLM — the agent can reason “this task is low priority, I’ll try the beefy node first”
No new scheduler code — just MCP tools + skills
The skill can be iterated without recompiling Colibri

Cons:

Each routing decision costs tokens (the agent must reason about it)
Agents make inscrutable routing choices (the LLM “just knows”)
No hard guarantees — an agent might route a $5 task to Claude when DeepSeek would do fine
Requires the agent to be “cost-aware” (which requires the MCP cost query tool — already shipping in T1.5)

Recommendation

Start with Option A (Mother-Centric) as the foundation, then layer Option C (Skill-Based) on top.

The capability matrix, stable UUIDs, and local LLM probes are the foundation — they’re needed regardless of the routing strategy. Option A gives us the data model and probe infrastructure. Once that’s solid, adding the MCP tools for agent-driven routing (Option C) is a thin layer on top. Option B (peer-to-peer) adds complexity we don’t need at this scale.

Phase 1 (this sprint): Machine UUID + local LLM probes + extended capabilities matrix. ~2.5 days. Phase 2 (next sprint): Mother-based routing + MCP tools for agent-driven routing. ~2 days. Phase 3 (future): Peer-to-peer fallback if we ever have >20 nodes.

Deliverables by Phase

Phase 1 — Identity & Capability Foundation

Deliverable	Where	Lines
`machine_id` generation in `clawdie-firstboot`	clawdie-iso	~15
`collect_machine_id()` in hw-probe	clawdie-iso	~10
`collect_ollama_status()` in hw-probe	clawdie-iso	~30
`collect_llama_cpp()` in hw-probe	clawdie-iso	~20
`collect_local_llm()` aggregator in hw-probe	clawdie-iso	~25
`machine_id` column + constraint in mother_schema.sql	colibri	~5
Extended `derive_capabilities()` for `ollama_available`, `llama_cpp_available`, `inference_tier`	colibri	~40
`node-register-mcp` handling of `machine_id` key + new local_llm fields	colibri	~15
This design doc (hive-routing.md)	This file	~0 (done)

Phase 2 — Routing Engine

Deliverable	Where
`colibri_query_hive_capabilities` MCP tool	colibri-mcp
`colibri_dispatch_to_node` MCP tool	colibri-mcp
`hive-routing` skill	`.agent/skills/`
`Task.routing` JSONB field in colibri-ledger	colibri-ledger
Mother-side routing score as PostgreSQL function (optional — only if agent-driven routing proves insufficient)	mother_schema.sql

Integration with the Trifecta

The hive routing plan completes the trifecta’s T2.x vision:

T1.4 Prompt Discipline    ✅ Cache warming, cost mode, 3-region prompt
T1.5 Per-Task Cost        ✅ Captured locally (this sprint)
T2.x Cost-Aware Routing   📋 Hive matrix → routing decision
T2.x Model Selection      📋 Arbitrage between cloud tiers + local LLM
T2.x Eval Harness         📋 Task success measurement

The key insight: local LLM is the ultimate cache-hit token. Every token generated on a beefy node’s GPU is $0.0000. The routing engine’s job is to maximize the use of $0 tokens without compromising task success rates.

Fleet SSH reliability

Two one-liner configs that prevent SSH interruptions and ksshaskpass popups on fleet nodes:

1. Disable password auth — no brute-force surface

When a fleet node connects and the key doesn’t match on first attempt, sshd falls back to password authentication. Fail2ban counts those as failures and bans the source IP after maxretry attempts. With password auth off, there is nothing to brute-force:

PasswordAuthentication no

Caveat: nodes with password auth disabled need physical/console access if they lose their private key.

2. Auto-add keys to agent — no ksshaskpass popups

When ssh-agent has no identities, Kitty SSH triggers ksshaskpass on reconnect. AddKeysToAgent yes auto-loads keys on first use:

Host *
  AddKeysToAgent yes

3. FreeBSD: PF rate limiting

On FreeBSD nodes, max-src-conn-rate 5/60 with <ssh_brutes> table provides the same protection independently of fail2ban:

table <ssh_brutes> persist
pass in proto tcp to port 22 \
  max-src-conn-rate 5/60 overload <ssh_brutes> flush global

→ fail2ban-tailscale skill → freebsd-admin skill

Hive Member Tracking & Cost-Aware Routing

What Exists Today

Design Goals

Architecture

Identity: Machine UUID

Problem

Solution: /var/db/machine-id

Schema change

Capability Matrix

What goes in the matrix

Local LLM capabilities (NEW)

Probe additions to clawdie-system-probe

Cost-Aware Routing

The routing decision

Cost tiers

Local LLM routing rules

How the scheduler knows

Protocol: Join → Probe → Route

Phase 1: Join (existing, extended)

Phase 2: Heartbeat (existing)

Phase 3: Capability Sync (new)

Phase 4: Task Dispatch (new)

Phase 5: Cost Capture (existing, extended)

Three Implementation Options

Option A — Mother-Centric (Minimal New Code)

Option B — Capability-Advertised with Local Routing Fallback

Option C — Capability-Matrix-as-Skill (Zero-Code Routing)

Recommendation

Deliverables by Phase

Phase 1 — Identity & Capability Foundation

Phase 2 — Routing Engine

Integration with the Trifecta

Fleet SSH reliability

1. Disable password auth — no brute-force surface

2. Auto-add keys to agent — no ksshaskpass popups

3. FreeBSD: PF rate limiting

Solution: `/var/db/machine-id`

Probe additions to `clawdie-system-probe`