Multi-agent orchestration: when stale state lies

In the early days of tamer, we thought discovery should be simple. If a worker is running, it should leave a footprint. If a footprint exists, the worker is there. We chose the simplest, most "Unix-y" footprint we could think of: a directory on the disk.

We were wrong. For sixteen days, tamer's PWA showed "ghost workers"—agents that had been terminated, crashed, or completed their tasks, but still appeared as "Active" in the dashboard. This is the story of how stale state lied to us, and how we fixed it by killing disk-based discovery.

The Ghost in the Machine

The symptom was subtle but maddening. A developer would start a worker_list via the Master MCP. The PWA would show three workers. But when trying to assign a Work Item to one of them, the system would hang or return a cryptic connection error.

The divergence was clear: the Master thought the worker existed, but the Relay (the server) knew nothing about it.

Why the Disk Lies

Our original discovery logic worked like this:

1. When a worker starts, it creates a subdirectory in /tmp/tamer/workers/ named after its ID.
2. The Master Agent periodically runs a ls on that directory to populate the worker_list.
3. When the worker shuts down cleanly, it deletes its directory.

The problem, of course, is the word cleanly.

If a worker process is killed by the OS, if the machine reboots, or if a network partition prevents the cleanup script from running, the directory stays. To the Master Agent, that directory is an immutable fact. "The folder is there, therefore the worker is alive."

In a multi-agent system, the filesystem is not a source of truth; it's a graveyard of past intentions.

The 16-Day Divergence

It took us 16 days to prioritize this because, in local development, it’s easy to just rm -rf /tmp/tamer and move on. But as soon as we moved to distributed orchestration—where the Master might be on a laptop and the Workers on remote cloud instances—the "ghosts" became a blocker.

We saw workers that had "shipped" their code two weeks ago still sitting in the list, mocking the Master Agent's attempt to orchestrate a new pipeline.

The Fix: Server-Side Truth

With the release of tamer-server v0.57.0, we officially deprecated disk-based discovery. We introduced a new, authoritative endpoint:

GET /api/master/:masterId/workers/live

This endpoint doesn't look at the disk. It queries the InstanceRegistry directly in memory on the Relay server.

• Active Heartbeats: A worker is only "live" if it has sent a WebSocket heartbeat within the last 30 seconds.
• Auto-Expiration: If a heartbeat is missed, the server marks the worker as stale and removes it from the live registry.
• Connection Awareness: If the WebSocket closes, the worker is gone instantly.

The Lesson: Disk = Death

If you are building a multi-agent orchestrator, learn from our 16 days of frustration: The source of truth must be the point of connection, not the point of execution.

A worker is only alive if it can talk to you now. Relying on persistent state (like the disk or a database flag) to represent a transient process is a recipe for stale state and "lying" UIs.

In the world of tamer, we now follow a simple rule: Disk is for logs; Memory is for life.

Technical details:

• Server version: tamer-server v0.57.0
• Related commit: 4848dec ("feat: implement live worker registry via heartbeat tracking")
• Master Agent update: v1.2.4 now defaults to the /live endpoint.