You type hermes gateway restart and wait. Three seconds, ten, thirty. The terminal leaves you one line: Gateway starting: deadline. The desktop app reports Gateway inaccessible: incomplete_control_response, you hit retry, and get the same sentence. You reboot the machine, the service comes back up, and wedges in exactly the same spot — restart after restart, never actually booting.

The first instinct is “my config must be broken”, “the disk must be full”, “this build is bad”. One failure mode is harder than all of those: the process did not crash, the CPU is idle, the log has no error — it is simply waiting on a lock it is already holding. That is a self-deadlock, and it presents as “stuck”, not as “failed”.

Around 2026-10-30 Hermes fixed two such boot deadlocks: one in the gateway’s startup sweep, one on the remote-config backend’s boot path. They share the lesson worth remembering: both called something that needed the same lock from inside a critical section that already held it.

First, tell “slow” apart from “dead”

The two look similar and need opposite responses, so here is the comparison worth keeping:

Symptom	Slow (wait it out)	Deadlock (waiting is pointless)
Boot duration	finishes on its own in tens of seconds	never passes the same point
CPU	busy, initialising	near zero, thread asleep
Log	keeps printing new lines	stops dead at the last line
Restart	usually fine the second time	wedges in the identical place every time
Thread state	waiting on IO or network	waiting on a lock (futex, lock acquire)

The gateway runs a loop-liveness watchdog that dumps the current stack and hard-exits with status 75 when the event loop is stuck, so systemd or launchd will restart it. That behaviour is right — a restarted process beats a frozen one — but it disguises a deadlock as a string of “the gateway crashed again” entries. What you see in the log is boot after boot, never one clear error.

Deadlock one: a nested read context inside the startup sweep

The first case is the gateway’s startup orphan sweep, which closes out sessions left behind by a previous process. The chain is recover_local_sessions() → arm_startup_orphan_closes() → _idle_s(), which decides how long a session has been idle.

The offending code looked like this (simplified):

with authority.db._read_ctx() as conn:
    row = conn.execute(
        f'SELECT {_sql_session_last_active("s")} FROM sessions s WHERE s.id = ?',
        (_physical_target(authority, session_id),),   # ← opens a SECOND read context
    ).fetchone()

The problem is the argument: _physical_target() reads the local session receipt, and reading the receipt is itself a database read. So inside a block that has already entered a read context, composing one parameter enters a second one.

Normally that is harmless, because reads go through the WAL read pool and run concurrently. But when the WAL read pool is unavailable — the database is not in WAL mode (DELETE journal), the home sits on a network filesystem such as NFS, or every read permit is spent — _read_ctx() degrades to “use the writer connection under SessionDB._lock”. That lock is a plain threading.Lock: not reentrant.

So one thread takes the lock, then asks for the same lock from inside it, and waits forever.

The fix is conceptual rather than clever: compute the target first, then open the read context, so the probe takes exactly one.

target = _physical_target(authority, session_id)
with authority.db._read_ctx() as conn:
    row = conn.execute(..., (target,)).fetchone()

The live evidence is textbook. On a macOS gateway the boot thread parked exactly there, the control socket never answered, desktop clients reported Gateway inaccessible: incomplete_control_response and Gateway starting: deadline, the watchdog dumped that stack and exited 75, launchd restarted it, and the whole thing repeated. This is a boot loop, not a slow start.

Worth saying: the degraded read path is correct as designed — a slower read beats running out of file descriptors. What was wrong was nesting a second read inside it.

Deadlock two: reading config before config can be read

The second case is subtler and more valuable, because it is a cross-module lock-order inversion.

A Cloud agent with the config backend set to remote (HERMES_CONFIG_BACKEND=remote) never boots: the first config read in the second startup stage hangs forever. Nous credential mode only; IdP mode is unaffected.

The chain:

1. The first config read for a profile takes that profile’s fetch lock, then runs the boot fetch.
2. The fetch resolves its bearer — the Nous access token — which cold-imports hermes_cli.auth.
3. hermes_cli.auth imports hermes_cli.config at import time, and that module filled OPTIONAL_ENV_VARS (the “which environment variable does this provider need” metadata) from provider profiles.
4. Listing providers runs provider discovery, and discovery reads config (plugins.enabled, plugins.isolation).
5. That read asks the backend for the same profile and waits on the fetch lock from step 1 — held by the very thread doing all this.

In one sentence: to read config, it had to read config. This is not simple reentrancy; it is a module doing side-effecting IO at import time, then being imported by another module that already holds a lock.

The fix removes the import-time config read rather than working around it:

• Importing hermes_cli.config or hermes_cli.auth reads no config. The provider env catalog is built on first read instead: each bundled provider plugin is executed into a private module, its registrations collected — no config, no discovery, no registry mutation — and the result is cached idempotently.
• The remote backend resolves its credential before taking the fetch lock, and holds that lock only for the GET; concurrent readers queue there and share the result.
• Nothing is logged while a backend lock is held. A log handler can read config, so messages are collected and emitted after the lock is released.
• The log formatter reads no config. RedactingFormatter used to look up a routed profile’s redaction policy per record, under the logging handler’s own lock. It now reads only the live secret scope and the policy recorded when that profile’s .env or config was loaded; unknown means redact.

That last one is quietly a security improvement: “could not determine the policy” and “the policy is do-not-redact” used to be easy to confuse, and the default is now the safe one.

How to confirm it is a deadlock

Do not guess. One stack dump beats ten log reads:

1. Find the process: hermes gateway status lists the running services and processes.
2. Take a snapshot with py-spy (a third-party profiler, pip install py-spy): py-spy dump --pid $GATEWAY_PID.
3. Look at where the boot thread sits. A deadlock’s top frames are lock acquisition (threading.Lock.acquire, futex, _read_ctx, with self._lock), not network or disk IO.
4. If the same thread’s stack shows both the lock holder and the waiter — a with block whose body calls something that takes the same lock — you have your answer.

There is also a very practical test: try a different home. Start HERMES_HOME=/tmp/hermes-probe hermes gateway against an empty home. If it boots instantly, the code is fine and the problem is your home’s state or filesystem — which is exactly the trigger in case one (no WAL, or a home on NFS).

What to do when you hit it

• Case one (startup sweep self-lock): the trigger is a home whose WAL read pool is unavailable. If you are pinned to a build without the fix, restarting will not help — it wedges identically every time. The practical mitigation is to keep HERMES_HOME on a local disk in WAL mode rather than on NFS or another network share.
• Case two (remote-config boot deadlock): the remote backend has no local fallback, so flipping HERMES_CONFIG_BACKEND back to file only works if that profile still has a usable local config.yaml. Without one, changing it just produces a different error. The real answer is upgrading to a build with the fix.
• Both fixes are on main (#144845 is P0, #146130 is P2) and shipped in v0.22.6 (2026-11-02); if you are stuck on v0.22.5 or older, upgrading is the fix.

What this teaches anyone writing plugins

If you write Hermes plugins or config backends, these two are good templates:

• Do not read config, do network calls, or mutate global registries at import time. You never know who is holding a lock while importing you.
• Do not do IO or logging while holding a lock. A log handler is the most easily forgotten thing that reads config.
• Prepare arguments before entering a critical section. That is exactly how case one died: composing a parameter re-entered the same section.
• Degraded paths are where these bugs live. The fast path was concurrent, so the bug only appeared when the pool was unavailable — which is why it survived in production for so long.

For the neighbouring failure modes, read the eight reasons the gateway event loop freezes and how to tune the loop watchdog thresholds. Together with this piece, they cover most of what “the gateway stopped responding” can mean.