vq daemon lifecycle contract¶

Audience: operators. Part of Running a host.

What vq requires of the host environment, what vq detects + refuses to operate when, and what to do when the contract breaks. The 2026-05-17 compute-d incident (user-systemd orphaned, daemon gone, requiring out-of-band sudo recovery) motivated this document; v0.5.49 added the vq daemon health verb that surfaces every contract violation in one verdict.

For the newcomer-facing command matrix shared by vibe-qc, vibe-view, vq, and vibe-basis, start with the toolset lifecycle guide. This page covers the additional daemon ownership and host-service contract specific to vq.

Source-install lifecycle¶

First clone vibe-queue. Run the supported source-install commands from that repository root:

./scripts/install.sh --extras web
.venv/bin/vq self-update --accepted-report vX.Y.Z
./scripts/reinstall.sh   # only while no daemon runs
./scripts/uninstall.sh --dry-run

Each command uses the dedicated .venv unless --venv is given. Install records the selected extras profile and whether the package is editable; update and reinstall preserve both unless --extras, --editable, or --copied explicitly changes them. Environment replacement uses a rollback-safe transaction, and every successful install restamps source provenance.

Every verified environment also receives a regular .vq-checkout-owner file bound to the canonical vibe-queue checkout. Install with --force, in-place update, environment replacement, and removal all require that exact ownership marker. The scripts recheck it while holding the checkout lifecycle lock immediately before a destructive operation. Foreign, malformed, or symlinked markers fail closed, as do targets below system directories or Git metadata.

An older vq environment without the marker is not adopted automatically. Pass --adopt-legacy explicitly (install.sh also requires --force) for a one-time migration. Adoption uses an isolated Python outside the target to read the environment’s regular PEP 610 direct_url.json; it succeeds only when the recorded local source path exactly matches this checkout.

The distribution was renamed from vq to vibe-queue, and pip does not know the two names are one project, so a record may sit under either .dist-info prefix. Both are read and neither outranks the other. A venv holding one record under each name is two independent installs recording two origins — a hand-run pip install -e . over a pre-rename environment, since every venv vq installs drops the old record first — and adoption refuses it, naming the two .dist-info directories so the stale one can be removed. An unreadable .dist-info refuses for the same reason rather than falling back to the other name: answering from a record the venv no longer installs reports the wrong checkout and the wrong install mode. scripts/update.sh reads the same way when it preserves the installed mode, and reports the mode as unknown rather than guessing. The target environment’s Python and vq commands are never executed to establish trust. After that proof, a non-dry-run command commits the one-time adoption by writing the marker while it holds the lifecycle lock, before any daemon or environment mutation. That marker remains valid even if a later precondition or install step fails. --dry-run performs the proof but does not stamp the target.

When the selected environment owns a running daemon, direct install, update, and reinstall refuse. Use vq self-update or vq admin update: the managed transaction binds Linux systemd-user MainPID/ExecStart or macOS launchd argv, durably owns checkout and venv rollback, pauses the relevant queue scope, and proves the restarted daemon’s exact source and package tree. A daemon belonging to another venv is left alone.

A managed transaction takes no --extras from update_script: what goes into a serving venv is decided by vq, not by a command line it did not write. It rebuilds with the profile .vq-install-metadata records, changed for one environment by an explicit --update-script-arg --recreate-venv --update-script-arg --extras --update-script-arg PROFILE, and raised on every rebuild by whatever the program declares:

[programs.vibeqc-queue]
extras = ["web"]

The declaration is applied wherever vq itself builds the venv: the managed transaction, a fresh vq admin install, and an ordinary vq admin update on any other program. All three rest on the same proof — the script vq is about to name a profile on is vq’s own scripts/install.sh or scripts/update.sh, resolved inside the checkout the program points at. The proof is the only thing that decides, so a program whose scripts are its own is built and rebuilt exactly as it always was.

An ordinary update resolves the floor over whatever update.sh would have installed without vq’s help, in that script’s own order of precedence: an operator’s --update-script-arg --extras PROFILE, else the profile .vq-install-metadata records, else core. It appends no --recreate-venv — reinstalling into the existing environment is what puts the declared runtime in it, and a direct --recreate-venv is refused outside a managed transaction anyway. A newly raised floor also outranks --expected-sha idempotence: a checkout already at the target commit says nothing about what its venv carries.

What a failed proof means differs, because the verbs stand at different points in an environment’s life:

an update has a built, possibly serving environment to leave alone, so an update of a program whose update_script is its own script runs that script unchanged and says that it left the declaration alone rather than passing over it silently. It does not offer the managed transaction as a way round, because that path requires the same canonical script and refuses without one; what it names is the config change that gives the program a route;
an install is about to create the venv, so it refuses rather than building one without the runtime the program declares, and names the remedy: point install_script at the checkout’s own scripts/install.sh, or drop the declaration, provision, then restore it and apply it with vq admin update ENV, which needs no flags because an ordinary update applies the declared floor by itself – and only while it is declared. When the configured install_script does not name one of vq’s own scripts at all, that refusal comes before the clone; one that comes after it leaves the checkout behind, which has to be emptied on the machine the refusal names before provisioning again. It leaves no admin-update marker. The second route also needs scripts/update.sh to accept the venv, and it refuses one none of vq’s own scripts built.

Both of those depend on the program’s update_script being one of vq’s own scripts, because that is what lets an ordinary update apply the floor. A program configuring only install_script, or a foreign updater, is a supported shape — the provision applies the floor once — but vq can never rebuild that environment afterwards: an update pulls the checkout, reports success and rebuilds nothing. So the refusal names pointing install_script at vq’s own installer as the only route it has, and adding update_script = "scripts/update.sh" as what makes the other one real; the note an ordinary update prints says that this update rebuilds nothing rather than describing a rebuild that is not happening. Neither offers a fresh vq admin install: the checkout exists, and a provision refuses to build over one. vq web install’s runtime remedy follows the same rule.

vq doctor reports which of those routes each declaring program actually has, on the host whose checkout can answer, so the situation is visible before a unit starts failing rather than after. It says nothing at all where no program declares extras, fails only where a declaration has no route at all, and reports that it could not answer where the checkout is absent or unreadable.

At install the floor is resolved over whatever profile the configured install_script already asks for — --extras dev plus extras = ["web"] installs all — and appended only when it raises that profile. The same rule applies on an update, over the profile that run was going to install. A program that declares no extras is installed and updated exactly as before.

Use the per-update flags to fix a host now; use the declaration to say what the venv is for. The declaration survives a venv rebuilt by a script run directly — install.sh, reinstall.sh — where the metadata sidecar dies with the venv it described, and an operator request that does not cover it is refused rather than silently widened.

This is how a host whose one venv serves both the daemon and the web console keeps uvicorn across a rebuild — the only supported way, since pip inside a vq-managed checkout is not one. The declaration only ever adds: the effective profile is the smallest published one containing both the declaration and what the venv already had, so ["web"] on a dev environment rebuilds as all rather than dropping its test tooling. vq web install makes the same check from the other side and refuses to point a unit at a vq that cannot import uvicorn. Add the key only after the host runs a vq that knows it; an older one rejects the unknown key and then fails to load the config at all.

Direct whole-venv replacement is supported only for a provably inactive environment and a source-stable checkout (update.sh --skip-git --recreate-venv, reinstall.sh, or install.sh --force). Before moving the old venv it fsyncs an owner-only receipt beside the target. A later invocation under the same checkout+target lifecycle locks reconciles an interrupted candidate or cleanup debt, performs no service action, and exits so the operator must retry explicitly. Direct --restart-daemon is retained only as a fail-closed compatibility spelling.

Update and reinstall require an unsupervised daemon to be stopped manually; they will not promise a restart they cannot perform. vq daemon stop does that: it sends SIGTERM to the daemon named in this state directory’s pidfile, waits up to ten seconds for it to exit, and prints daemon: not running when there is none. Do not use it on a supervised daemon. The update scripts stop the daemon that belongs to the venv they replace through its service manager whenever one is active (systemctl --user stop vq-daemon on Linux), because signalling the process behind the manager’s back can get it restarted. vq daemon start was removed because it created a second daemon identity. Install systemd-user or launchd-user supervision first.

A separately supervised dashboard created by vq web install has its own service definition and process. After a successful managed queue update and verified daemon restart, vq automatically refreshes a systemd-user or launchd-user console belonging to that environment, using the updated interpreter. It keeps the service name, manager and supported environment settings, and checks that the service is active with current provenance. No console is installed when none exists, and consoles belonging to other environments are left alone. A refresh failure makes the update fail visibly without rolling back the healthy daemon; repair it with the updated .venv/bin/vq web install, verify vq web status, then follow the normal failed-update recovery procedure. System services require an explicit operator reinstall with their service-user settings. After a manual reinstall, also rerun vq web install and verify vq web status. Before uninstalling the environment, run vq web uninstall while that command still exists. The macOS daemon plist generated with vq daemon launchd-plist already owns a web sidecar; do not install a second dashboard service for that route.

What vq requires¶

Mandatory¶

A user service manager active and reachable. Linux uses systemd-user. systemctl --user must work; loginctl show-user $USER must report State=active. With Linger=yes enabled (loginctl enable-linger $USER, once per host) the user manager auto-starts at boot and survives logout — required for the daemon to run unattended on a compute host.

macOS uses the generated com.vq.daemon launchd agent. Create it with vq daemon launchd-plist, bootstrap it with launchctl, and keep it at ~/Library/LaunchAgents/com.vq.daemon.plist so lifecycle scripts can unload and restore the same definition around an update.
The platform daemon service installed and enabled. Either copy contrib/vq-daemon.service into ~/.config/systemd/user/ or use a distro packaging path. Then: systemctl --user enable vq-daemon to make it auto-start. The unit’s Restart=on-failure + RestartSec=5 means a crash recovers automatically.

v0.6.0 removal: vq daemon start was removed. The systemd-user unit is the supported Linux daemon launch path; a launchd user agent is the supported macOS path. Pre-v0.6.0 the CLI verb could spawn a detached daemon that wrote its own pidfile (a second identity source competing with systemd’s MainPID); removing the verb closes that whole class of failure mode. The daemon_control.start_daemon Python API is retained for test purposes (used by the e2e test suite) but shouldn’t be called from production.
An on-disk vq venv that the service definition’s executable points at. Example Linux layout for a standalone clone: /home/USER/vibe-queue/.venv/bin/vq. The prog.python field of the [programs.vibeqc-queue] config entry (if registered) must point at the same venv’s bin/python so the v0.5.42 self-update auto-restart detection works.

Strongly recommended¶

cgroup-v2 delegation for accurate per-job memory enforcement and CPU-starvation detection. Drop-in at /etc/systemd/system/user@.service.d/delegate.conf:
```
[Service]
Delegate=cpu cpuset io memory pids
```
Without delegation, the watchdog falls back to /proc-only mode (CPU-only enforcement, memory enforced by host-percent ceiling rather than per-job cap). The fleet hosts (compute-a, compute-d) have delegation enabled; the daemon’s cgroup.available() probe detects + caches the answer at startup.
~/bin on the systemd-user PATH if you dispatch CRYSTAL / ORCA / etc. from there. The unit ships generic; per-host PATH overrides go in ~/.config/systemd/user/vq-daemon.service.d/path-override.conf.

Forbidden (configurations that violate the contract)¶

Two daemons against the same $VQ_STATE_DIR. Result is undefined — both will try to dispatch, lock files race, specs get overwritten. vq daemon start was removed in v0.6.0 (it now hard-errors and points to the systemd-user unit), so the systemd MainPID is the single source of daemon identity. vq daemon health still reports a FAIL if a stray daemon-pidfile and the systemd MainPID disagree.
Editing files in the vq venv while its daemon is running. A manual pip install while a job is dispatching can mid-stream swap modules out from under the running process. For a local serving source install, use vq self-update --accepted-report vX.Y.Z. For a managed remote profile, use vq admin update vibeqc-queue; both use the durable marker, pause, rollback, and verified restart transaction.

What vq detects + refuses on¶

`vq daemon health` cross-checks (v0.5.49+)¶

The verb cross-checks four sources of truth:

source	healthy value
`loginctl show-user $USER` → `State`	`active`
`pgrep -u $UID -f 'systemd --user'`	one live PID
`systemctl --user show vq-daemon -p ActiveState`	`active`
`<state_root>/daemon.pid` vs systemd `MainPID`	identical OR no pidfile

ok=True only when all four agree and the recorded daemon PID is alive (verified via kill(pid, 0)). Each disagreement adds a FAIL finding to the verdict. The verb is read-only — diagnoses, never repairs.

Hosts without systemd-user (macOS, non-systemd boxes). macOS production daemons should run under the generated launchd user agent, which the source lifecycle scripts can identify and cycle. The health command itself still uses the following platform-neutral fallback. When none of the three systemd signals is present — no loginctl, no systemd --user in the process table, and systemctl --user unreachable — there is no user manager to contract with, so the four-source check is skipped entirely. The verdict instead falls back to RPC-socket liveness: the daemon (started directly, e.g. vq daemon run from a wrapper) is ok=True iff it answers ping on <state_root>/daemon.sock — the same signal vq queue / vq programs / job dispatch rely on — with the pidfile pid carried through for display. This is why a healthy macOS guarded node now reports daemon: OK ... (pidfile) from vq overview / vq daemon health instead of the old daemon: FAIL (pidfile) false-negative. A Linux host always has at least a loginctl State + a manager PID, so it keeps the full four-source contract; the orphan/zombie incidents (manager present but systemctl --user unreachable) still take the systemd path and FAIL correctly.

`vq admin update`¶

For an in-place managed QC update, rollback restores the previous source and native runtime. If a generated build/basis_library overlay exists, it then runs the restored source’s scripts/setup_basis_library.sh with the registered Python before checking the old runtime identity. Restoring tracked bytes alone does not restore the file identities in the basis receipt. Setup remains under the update’s lifecycle lock and has a two-minute limit. Missing setup, an external overlay, a sealed runtime, setup failure or failed identity leaves rollback explicitly failed. A successful rollback does not turn the failed update into a success.

Refuses conflicting existing markers. The first current lease may use the legacy-compatible <state_root>/admin-update-in-progress path; additional disjoint leases live under <state_root>/admin-update-markers/. A stable lock serializes admission, and independently scoped leases may coexist only when they touch disjoint mutable resources. A conflicting update is rejected with a diagnosis-specific recovery recipe. The refusal is a plain runtime error (Error: …, exit 1) — not a CLI usage error (Usage: …, exit 2): the argv was valid, a prior run just left the marker. This matters across hosts — vq admin update --all <host> delegates admin update --all localhost, and a marker on the remote must read as a state condition, not as a malformed --all command.

v0.6.0 state machine, v0.25.0 durable receipt. Each marker is a multi-phase state file (pre-v0.6.0 markers read as state="legacy_in_progress"). The ordinary phase sequence is:
```
PAUSING -> PAUSED -> PULLING -> TAG_CHECKING -> BUILDING -> RESUMING ->
RESTARTING_DAEMON -> VERIFYING -> (file removed = IDLE)
                               \-> FAILED (sticky)
```
Each transition rewrites the file atomically. On failure the state becomes FAILED (sticky) with a failure_reason field (e.g. "git pull rc=128", "daemon restart failed: systemctl --user is unreachable"). vq admin status shows the current state + phase_started timestamp + failure_reason, so an operator sees “stuck at PULLING for 47m” or “FAILED: update_script rc=2” in one place instead of correlating logs.

A serving-daemon update additionally embeds a failure-atomic managed transaction binding the exact old checkout, virtualenv, service identity, and pause token. A hard interruption leaves that receipt on disk and keeps only its dispatch scope held. The daemon never auto-reaps it, and neither clear-update-marker nor update --force may discard it. Run vq admin recover-update [HOST] (and --marker-id ID when diagnosis names more than one) to reconcile and verify the receipt’s exact durable identity (old runtime, or an already committed target), prove the exact paused jobs resumed, and clear the receipt. An unverified target is never promoted by inference. A pause-only receipt uses clear-update-marker, which performs the resume proof before unlinking. If a direct host’s installed vq predates a landed recovery-parser fix, the explicit --with-driver-runtime --marker-id ID form may stage and self-authenticate the current driver’s package for one recovery invocation. It does not install that package and never retries an unknown mutation. Clear or force remains available only for an ordinary marker after the operator has independently inspected the environment and updater liveness.
Refuses to restart the daemon onto a half-installed package. When the on-disk work (git pull, update_script) fails, _maybe_restart_daemon is skipped — restarting onto stale bytecode is better than restarting into a broken venv.
Surfaces the recovery recipe. When the daemon-restart step fails AFTER the on-disk work succeeded (systemctl unreachable, timeout, non-zero rc), the marker stays on disk (v0.5.48 Bug A fix) and the failure message includes a pointer to operations.md for the user-systemd revival recipe.

Daemon startup (`vq daemon run`)¶

PID-fingerprint check (v0.5.50). RUNNING/SUSPENDED specs are re-attached as orphans only if the recorded spec.pgid is alive AND the recorded spec.pid_start_time matches the current /proc/<spec.pid>/stat field 22. Mismatch → spec lands in ABORTED_BY_QUEUE with reason pid_recycled instead of being silently re-attached to whatever unrelated process the kernel reused the PID for. macOS / pre-v0.5.50 specs skip the check (fall back to pgid-only liveness).
cgroup-scope MainPID cross-check (v0.6.0). When cgroup_enabled, startup recovery also queries systemctl --user show vq-job-<id>.scope -p MainPID --value and compares to spec.pid. If different (or unit not found while systemctl IS reachable), the spec lands ABORTED_BY_QUEUE with reason cgroup_scope_mismatch. Closes the gap where pgid liveness + PID-fingerprint both pass but systemd no longer associates this PID with our scope unit (= the scope was detached / collected).
_start_job race fix (v0.6.0). The dispatch path now writes spec.state=RUNNING with pid=None/pgid=None BEFORE the subprocess.Popen call. A daemon crash in the narrow window between Popen.success and the post-Popen spec.write would previously leave a PENDING spec next to a running process — the next daemon loop would re-dispatch it (DOUBLE DISPATCH). Now the recovery path sees RUNNING + pgid=None and marks ABORTED_BY_QUEUE via the existing “no pgid recorded” branch; the unowned process (if any) stays within its cgroup scope and the operator can vq resubmit cleanly.
cgroup availability re-probe (v0.5.50). The cgroup.available() lru_cache is cleared at startup so a daemon restart re-tests delegation from scratch. Stops the audit § 2j failure mode where a cached True survived a user-systemd flip and every subsequent _start_job failed via systemd-run.
Refuses to dispatch under an active marker. v0.5.45 added a marker check in _dispatch_pending. If the daemon restarts while an admin update is in flight (or after the update was killed mid-way), the new daemon sees the marker and parks the pending queue until the operator clears it.

When the contract breaks¶

Symptom matrix¶

symptom	first command to run
`systemctl --user` returns “Transport endpoint…”	`vq daemon health <host>`
`vq queue` works but a job won’t dispatch	`vq admin status <host>` (look for marker banner)
`vq admin update` exits FAILED, daemon stale	`vq daemon health <host>` then `vq admin status`
`vq --version` reports new code but daemon is old	`vq daemon health <host>` (compare PIDs)
Job killed STARVED right after `vq resume`	v0.5.48 Bug B fix; upgrade.

Recovery recipes¶

user-systemd orphan / zombie¶

If vq daemon health reports FAIL: systemctl --user cannot reach the user manager:

# From an interactive shell on the host (sudo required):
sudo systemctl kill --signal=SIGKILL user@$(id -u).service
sudo systemctl reset-failed user@$(id -u).service
sudo systemctl start user@$(id -u).service
sleep 3
vq daemon health        # should report verdict: OK

If the user-systemd process is alive but systemctl --user can’t reach it (the 2026-05-17 compute-d incident), the kill won’t actually kill anything but it transitions the unit state so reset-failed + start succeeds. If the PID is truly orphaned (not in any cgroup systemd manages), use the explicit sudo kill -9 <pid> first, then the three-step recipe.

Stale admin-update marker¶

# Inspect and follow the printed diagnosis:
vq admin status <host> --verbose

# Durable serving-daemon receipt:
vq admin recover-update <host>

# Direct old-vq compatibility, with the exact diagnosed receipt:
vq admin recover-update <host> --marker-id <id> --with-driver-runtime --json

# Ordinary or pause-only marker after inspection:
vq admin clear-update-marker <host>

Stale ordinary markers may be auto-reaped when their writer is provably gone. Durable managed-daemon and pause receipts are deliberately retained. Never use --force against either receipt; the command refuses them because clearing the file would abandon rollback state or paused jobs.

Daemon running stale code after manual `pip install`¶

# v0.5.42+: this is automatic via `vq admin update vibeqc-queue`.
# Manual fallback:
ssh <host> 'systemctl --user restart vq-daemon'

PID-recycle false-positive at startup (v0.5.50+)¶

If a spec lands in ABORTED_BY_QUEUE with reason pid_recycled and you’re confident the job genuinely finished cleanly (rather than being killed early), check <workspace>/_vq/exit-code — the v0.5.9 exit-marker may show the real rc. The recycle-detection is conservative: false-positives are preferable to silently re-attaching to someone else’s process.