The vq calculation queue¶

vq is vibe-qc’s calculation queue, a small SSH-backed job-submission tool that lets you run vibe-qc (and CRYSTAL / ORCA) calculations on a remote compute box without writing shell glue. Configure it once, then vq submit my_calc.py from your laptop and the job is queued, dispatched, resource-capped, and watched on the remote host. Outputs come back the same way.

vq is co-shipped with vibe-qc in the vibe-queue/ subpackage but is independently versioned (the current source stamp is vq, version 0.25.0). It’s engine-agnostic: vibe-qc is the primary workload, but anything you can call from a shell, CRYSTAL14, ORCA 6.1, PySCF scripts, submits the same way through contrib/ wrappers.

When to use vq¶

Laptop runs out of cores or RAM. Your MacBook has 16 GB and 10 cores; the remote has 128 GB and 32 cores. Queue the big runs; keep the laptop for development.
You want a record of what you ran. Every submission is a JobSpec stored on the daemon, with a unique short-hash id, full command, environment, resource caps, terminal state, and outputs.
You’re running many jobs. vq dispatches one at a time (by default; see § Concurrency below) and records every one, so you don’t lose track when a sweep takes hours.
You want resource enforcement. cgroup-v2 caps mean a runaway job doesn’t bring down the box.

When NOT to use vq¶

Tiny molecules on the laptop. .venv/bin/python h2o.py runs in 3 s; the queue + ssh round-trip adds latency for zero gain.
Truly interactive sessions. vq is batch-shaped; use ssh
- a remote venv directly, or set up the Jupyter Lab integration for notebooks.
Unconfigured or interactive HPC workflows. vq ships PBS/Torque and Slurm scheduler backends for hosts registered in its configuration, including durable submission, monitoring, control, and artifact fetch. Use the site’s native scheduler tools for an unregistered cluster, interactive allocation, or workflow outside vq’s declared-resource model.

Architecture¶

┌────────────────────┐         SSH        ┌──────────────────────────┐
│  Your laptop       │ ─────────────────→ │  Remote compute host     │
│                    │                    │                          │
│  vq CLI            │                    │  vq-daemon.service       │
│  ~/.config/vq/     │  vq submit         │  (systemd --user)        │
│   config.toml      │                    │                          │
│                    │ ←───── stdout ──── │  Queue (durable specs)   │
│  ssh-key auth      │                    │   ↓                      │
│                    │                    │  systemd-run scope       │
│                    │                    │   (cgroup-v2 caps)       │
│                    │                    │   ↓                      │
│                    │                    │  your Python / ORCA /    │
│                    │                    │  CRYSTAL14 process       │
│                    │                    │                          │
│                    │  vq-web.service    │  Web UI (FastAPI+htmx)   │
│  browser ──────────┼───── port 8765 ───→│  :8765/queue,            │
│                    │  token on writes   │  /jobs/<id>              │
└────────────────────┘                    └──────────────────────────┘

Pieces that need to be running:

vq-daemon.service on the remote, accepts submissions (over SSH), maintains the queue, dispatches jobs into cgroup scopes, survives reboots via loginctl enable-linger.
vq-web.service on the remote, read-only-plus-write REST + HTML UI, port 8765 by default. Read-only pages are available on the loopback-bound service; bearer tokens gate write actions.
vq CLI on the laptop, wraps ssh remote vq … so the laptop never deals with the queue state directly.

Installation¶

Two sides, local (laptop) and remote (compute box). Use the shipped lifecycle scripts on both. They keep vq in its own vibe-queue/.venv, verify the installed commands, record source provenance, and preserve the selected capability profile on later updates.

vq requires Python 3.12 or newer on both sides.

Install profile	Contents
`core`	CLI and daemon; default
`web`	Core plus dashboard
`test`	Core plus tests
`dev`	Core, tests, lint, and typing tools
`all`	Web plus all development tooling

For a concise comparison with the vibe-qc, vibe-view, and vibe-basis lifecycle commands, including what each uninstall preserves, read Install and maintain the vibe toolset.

Local (laptop)¶

# Inside your vibe-qc checkout
./vibe-queue/scripts/install.sh

# Put vq on PATH:
mkdir -p ~/.local/bin
ln -s "$PWD/vibe-queue/.venv/bin/vq" ~/.local/bin/vq
# or in zshrc:
#   alias vq="<vibe-qc-checkout>/vibe-queue/.venv/bin/vq"

The local install needs only the CLI dependencies (no FastAPI / systemd). Test:

vq --version            # vq, version 0.25.0

Remote (compute box)¶

# 1. Install vq from a vibe-qc clone (repo is currently private,
#    see docs/installation.md for read-only access):
git clone ssh://git@gitlab.peintinger.com:26/mpei/vibeqc.git ~/vibeqc-queue
cd ~/vibeqc-queue
./vibe-queue/scripts/install.sh --extras web

# 2. Install the Linux systemd-user daemon unit:
mkdir -p ~/.config/systemd/user
cp vibe-queue/contrib/vq-daemon.service ~/.config/systemd/user/
mkdir -p ~/.config/systemd/user/vq-daemon.service.d
cat > ~/.config/systemd/user/vq-daemon.service.d/venv.conf <<EOF
[Service]
ExecStart=
ExecStart=$PWD/vibe-queue/.venv/bin/vq daemon run
EOF

# Install the separately supervised web console. Its generated unit points
# at this exact vq installation and records its provenance.
./vibe-queue/.venv/bin/vq web install

# 3. Enable the daemon to start at boot (linger keeps the
# user instance alive without an active session):
sudo loginctl enable-linger $USER

systemctl --user daemon-reload
systemctl --user enable --now vq-daemon.service

# 4. Verify:
systemctl --user status vq-daemon
./vibe-queue/.venv/bin/vq web status
journalctl --user -u vq-daemon -f       # live log tail

On macOS, create a launchd user agent instead of the systemd unit:

mkdir -p ~/Library/LaunchAgents
./vibe-queue/.venv/bin/vq daemon launchd-plist \
  --python "$PWD/vibe-queue/.venv/bin/python" \
  --working-directory "$PWD" \
  --web-port 8765 \
  --output ~/Library/LaunchAgents/com.vq.daemon.plist
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.vq.daemon.plist
launchctl enable gui/$(id -u)/com.vq.daemon
launchctl kickstart -k gui/$(id -u)/com.vq.daemon

The generated macOS daemon agent owns a web sidecar, so do not also run vq web install for this route. If you intentionally choose a separately supervised web service instead, configure only that service and do not run the daemon sidecar.

vq daemon start no longer exists. Linux uses systemd-user and macOS uses launchd so the daemon has one durable owner and survives logout.

Updating, repairing, and removing vq¶

# Install one exact accepted-report pin, preserve the installed extras/mode,
# and safely cycle a supervised daemon that runs from this venv:
./vibe-queue/.venv/bin/vq self-update --accepted-report vX.Y.Z

# Rebuild a damaged venv without changing Git:
./vibe-queue/scripts/reinstall.sh   # only while no daemon runs

# When vq web install owns a separate service, refresh it after either command:
./vibe-queue/.venv/bin/vq web install
./vibe-queue/.venv/bin/vq web status

# Preview removal. Queue history and config are kept by default:
./vibe-queue/scripts/uninstall.sh --dry-run

# Remove a separately supervised dashboard while its command still exists:
./vibe-queue/.venv/bin/vq web uninstall
./vibe-queue/scripts/uninstall.sh

Use --extras web or --extras dev only when changing capabilities; later update/reinstall runs remember that choice. Use --editable for a developer checkout and --copied to return to a stable non-editable installation. State and config purges require separate flags and reject unsafe targets such as the home directory or source checkout.

Default state is under ${XDG_DATA_HOME:-~/.local/share}/vq; default configuration is under ${XDG_CONFIG_HOME:-~/.config}/vq. The environment variables VQ_STATE_DIR and VQ_CONFIG_DIR override those locations. --purge-state, --purge-config, and --all are explicit irreversible operations and require confirmation or --yes. State purge is refused while queued work exists unless --force is also supplied.

If you are retiring the daemon too, disable and remove its systemd-user unit or boot out and remove its launchd plist before deleting the environment. The lifecycle uninstall deliberately leaves service definitions for their owner to handle.

Each successfully verified vq environment has a regular .vq-checkout-owner marker tied to the canonical checkout that installed it. The lifecycle scripts require and recheck that marker before every in-place mutation, replacement, or removal. A foreign, malformed, or symlinked marker is refused rather than guessed around.

For a legacy vq environment created before ownership markers, add --adopt-legacy to the intended lifecycle command; install.sh also requires --force. This is an explicit one-time migration, not a general force flag. An external isolated Python reads the environment’s PEP 610 direct_url.json without running code from the target and requires its local source path to match this checkout exactly. A non-dry-run command commits the adoption marker under the lifecycle lock before any daemon or environment mutation; the marker remains if a later precondition or install step fails. --dry-run --adopt-legacy only validates the proof. If that record is missing, malformed, symlinked, or points elsewhere, create a new dedicated venv instead of adopting it.

Daemon startup does not create or rotate a web token. Initialize one explicitly for write actions, or rotate it with --force:

./vibe-queue/.venv/bin/vq web init-token
./vibe-queue/.venv/bin/vq web init-token --force

Use --quiet when automation should not print the token. The token is stored at ~/.config/vq/web-token with mode 0600 on the remote. Read-only loopback pages do not require it; write endpoints do.

Configuration¶

vq reads ~/.config/vq/config.toml on the laptop. The remote daemon doesn’t need a config file. Copy the template from the repository and edit:

cp vibe-queue/docs/config.toml.example ~/.config/vq/config.toml

A working minimal config:

# ~/.config/vq/config.toml on your laptop

# Default host when you omit it from `vq <subcommand> ...`.
# Match a [hosts.<name>] block below.
default_host = "compute"

[hosts.compute]
ssh = "compute"
# 'compute' must be an SSH alias defined in ~/.ssh/config,
# or a literal user@host.example.com. Test with:
#   ssh compute hostname

# Absolute path to vq on the remote. The remote shell's
# default PATH usually doesn't include the venv vq lives in.
remote_vq = "/home/USER/vibeqc-queue/vibe-queue/.venv/bin/vq"

# Default Python interpreter for single-file submits. Point
# at a venv where vibe-qc is installed.
remote_python = "/home/USER/vibeqc-dev/.venv/bin/python"

# Optional: multi-venv routing for --branch (v0.5.6+).
# Lets `vq submit foo.py --branch release` pick the right
# vibe-qc clone without hard-coding the path.
[hosts.compute.branches]
main    = "/home/USER/vibeqc-dev/.venv/bin/python"
release = "/home/USER/vibeqc-release/.venv/bin/python"

[hosts.compute.branch_aliases]
dev         = "main"
development = "main"
latest      = "release"

The full annotated example is at vibe-queue/docs/config.toml.example.

Multi-host¶

Add another [hosts.<name>] block:

[hosts.compute2]
ssh = "compute2"
remote_vq = "/home/USER/vibeqc-queue/vibe-queue/.venv/bin/vq"
remote_python = "/home/USER/vibeqc-dev/.venv/bin/python"

Then vq submit foo.py --host compute2 routes to that machine. Omit --host to use default_host.

Your first job¶

# A trivial vibe-qc water RHF script.
cat > water.py <<'EOF'
from vibeqc import Atom, Molecule, run_job
mol = Molecule([
    Atom(8, [0.0,  0.00,  0.00]),
    Atom(1, [0.0,  1.43, -0.98]),
    Atom(1, [0.0, -1.43, -0.98]),
])
run_job(mol, basis="sto-3g", method="rhf", output="water")
EOF

# Submit it.
vq submit water.py
# → printed to stdout: jobid (e.g. "c0ff50a06462").

# Poll the queue:
vq list

# Block until it reaches a terminal state (Ctrl-C exits the wait; the job keeps running):
vq wait c0ff50a06462

# Once it finishes, pull the outputs back:
vq fetch c0ff50a06462 ./outputs/
# → ./outputs/water.out / .molden / .traj / stdout.log / stderr.log

That’s the entire core workflow.

Submission forms¶

vq accepts three submission shapes:

Single file (most common)¶

vq submit my_script.py
# Equivalent to:
#   ssh <host> cd <remote-workspace> && <remote_python> my_script.py

The laptop copies my_script.py into a fresh per-job workspace on the remote, runs it with the configured remote_python (or the --branch-resolved one), captures stdout / stderr, and tracks the result.

Directory submit (sweeps + multi-file inputs)¶

vq submit -d ./my_sweep_dir -- python run.py --basis def2-svp
# -d <path>                = the directory to copy across to the workspace
# --                       = end of vq flags
# python run.py …          = the literal command to run inside the workspace

Use this when:

Your script imports local modules (from helpers import ...).
You need multiple input files in the workspace (run.py reads geometry.xyz, basis_def.g94, etc.).
You want to encode the interpreter / engine in the command (e.g. running ORCA: -- orca input.inp).

Pre-packed tarball¶

vq submit -t my_inputs.tar.gz -- bash run.sh
# vq unpacks the tarball into the workspace before dispatching.

For reproducibility, the tarball + command + JobSpec are a complete reproducible-run unit.

Resource caps¶

Every job dispatched after v0.4.0 runs inside its own systemd-run user scope so cgroup-v2 memory + CPU caps apply. Wall-time enforcement is Python-watchdog-based (vq.watchdog); cgroup RuntimeMaxSec was tried in v0.4 → v0.5.7 and dropped in v0.5.8 as not runtime-mutable via systemctl --user set-property (see vibe-queue/docs/wall_time_design.md for the postmortem). The watchdog subtracts paused_seconds_total from elapsed, so wall-time is naturally pause-aware.

vq submit my_calc.py \
    --cpus 8 \
    --mem-mb 16000 \
    --wall-time-seconds 7200    # 2-hour cap (watchdog-enforced)

If the job exceeds any cap, the cgroup or the watchdog kills it cleanly and the queue records a labelled terminal state:

Terminal state	Trigger	Owner	Recovery
`COMPLETED`	exit code 0	-	nothing, outputs ready to fetch
`FAILED`	non-zero exit code	-	inspect stderr.log; re-submit
`OOM_KILLED`	exceeded `--mem-mb` cap	cgroup	bump `--mem-mb` or split the job
`TIME_EXCEEDED`	exceeded `--wall-time-seconds` cap	watchdog	bump the cap, or checkpoint if vibe-qc supports it for the workload
`STARVED`	CPU-underutilisation watchdog (5 min < 10% CPU summed over the pgid descendants)	watchdog	check stderr.log, typically a hanging worker; v0.5.12+ samples the whole pgroup so the bash wrapper no longer false-positives
`ABORTED_BY_QUEUE`	terminated by `vq kill`, queue-wide pause→kill, or daemon restart that lost the exit code	daemon	intentional; resubmit if needed

Always pass --wall-time-seconds N for non-trivial jobs, that’s the only guard against a wedged SCF eating cores indefinitely.

Orphan exit-code recovery (v0.5.9+)¶

If the daemon restarts mid-job (deliberately via systemctl --user restart vq-daemon or via Restart=on-failure), the dispatched bash wrapper writes the inner process’s exit code to <workspace>/_vq/exit-code on graceful exit. When the new daemon reconciles orphans, it reads the marker and classifies as COMPLETED (rc=0) or FAILED (rc≠0). Pre-v0.5.9 behaviour was to mark every restart-orphan as ABORTED_BY_QUEUE even on clean completion; v0.5.9 fixes that and is what makes vq admin update (v0.5.20+) safe to use, it deliberately pause-restart-resumes the daemon.

Multi-venv `--branch` routing (v0.5.6+)¶

The remote may host multiple vibe-qc clones, typically vibeqc-dev (tracking main) and vibeqc-release (tracking the latest tag). Pick one per submit:

vq submit my_calc.py                        # default_host's default
vq submit my_calc.py --branch main          # dev venv
vq submit my_calc.py --branch release       # release venv
vq submit my_calc.py --branch latest        # = release (alias)

--branch is mutually exclusive with --python, and only applies to single-file submits. For -d / -t submits, encode the interpreter in the explicit command.

The mapping is per-host config, [hosts.<name>.branches] + [hosts.<name>.branch_aliases]. Add new entries by editing ~/.config/vq/config.toml on the laptop; no remote restart needed.

External-program workflows (CRYSTAL / ORCA / PySCF)¶

vibe-qc treats other QC programs as external, see CLAUDE.md § 10 for the policy. vq dispatches them through contrib/ wrappers that handle each program’s I/O conventions:

CRYSTAL14 (Pcrystal + PROPERTIES14)¶

# Parallel CRYSTAL14 (default --np 14):
vq submit -d ./calc --cpus 14 \
    -- bash /home/USER/vibeqc-queue/vibe-queue/contrib/run-crystal.sh INPUT.d12

# Serial:
vq submit -d ./calc --cpus 1 \
    -- bash /home/USER/vibeqc-queue/vibe-queue/contrib/run-crystal.sh --serial INPUT.d12

# Custom MPI rank count:
vq submit -d ./calc --cpus 8 \
    -- bash /home/USER/vibeqc-queue/vibe-queue/contrib/run-crystal.sh --np 8 INPUT.d12

# PROPERTIES14 (parallel):
vq submit -d ./prop --cpus 14 \
    -- bash /home/USER/vibeqc-queue/vibe-queue/contrib/run-crystal.sh --properties prop.d3

The wrapper stages the input file as ./INPUT, runs mpirun -np N Pcrystal > out.out, restores any pre-existing INPUT on exit.

ORCA 6.1¶

ORCA spawns its own MPI internally, don’t wrap with mpirun:

vq submit -d ./orca_run --cpus 8 -- orca input.inp

ORCA reads --cpus-equivalent info from the ! PAL N line in the input file; declare --cpus N matching for cgroup accounting.

PySCF (as a comparison / parity reference)¶

vq submit my_pyscf_script.py                # PySCF is in both vibe-qc venvs

Both the dev and release vibe-qc venvs have PySCF installed (it’s in [test]), so PySCF scripts submit the same way as vibe-qc scripts.

Monitoring + management¶

# Snapshot the queue:
vq queue                                  # all states
vq queue --active                         # running + pending + suspended
vq queue -s running                       # only running (v0.5.27)
vq queue -s running -s pending            # explicit two-state filter
vq queue -s failed -s killed              # terminal-failure forensics

# Per-job snapshot (metadata + tail of stdout/stderr):
vq status <jobid>                         # last 50 lines
vq status <jobid> -n 200                  # last 200 lines
vq status <jobid> -n 0                    # full output

# Live tail of a workspace file (v0.5.26):
vq tail <jobid>                           # follow stdout.log
vq tail <jobid> -f                        # live-stream (Ctrl-C to stop)
vq tail <jobid> --name vibeqc.log -f      # custom logger file
vq tail <jobid> --name mgo.out -f         # CRYSTAL output
vq tail <jobid> --name h2.out -f          # ORCA / Psi4 output

# Fetch outputs back to the laptop (live job: workspace dir;
# completed: workspace dir; archived: un-tars from the .tar.bz2):
vq fetch <jobid> -o ./results

# Cancel:
vq kill <jobid>                           # SIGTERM the process group,
                                          # then SIGKILL after grace
                                          # → terminal state KILLED

# Pause / resume (v0.5.1+):
vq pause <jobid>                          # SIGSTOP the job
vq resume <jobid>                         # SIGCONT
vq pause --all                            # pause every running job
vq resume --all                           # resume every paused job

If you’re coming from SLURM, vq also registers the familiar aliases sbatch/squeue/scancel/sacct/summary for submit/queue/kill/status/overview, so vq sbatch job.py and vq squeue work exactly like vq submit job.py and vq queue.

vq tail is the canonical “watch the SCF converge live” verb: it execs tail -f directly (locally) or via ssh (remotely), so SIGINT goes straight through and there’s no Python buffering layer between the job’s logger and your terminal. Use --name to target whatever file vibe-qc’s logger is writing to (e.g. logging.basicConfig(filename='vibeqc.log') → vq tail JOBID --name vibeqc.log -f).

The pause / resume flow is the right tool when you need to free the box temporarily (kids gaming, an interactive workload) without losing in-flight jobs. For automated venv refresh use vq admin update instead (it pauses-pulls-builds-resumes in one verb; see Refreshing the remote vibe-qc venv below).

Web dashboard¶

If the web service or daemon sidecar is running, keep it bound to loopback and forward the port from your laptop:

ssh -N -L 8765:127.0.0.1:8765 compute

Then open http://127.0.0.1:8765/queue. Read-only pages do not prompt for a token. A write action asks for the bearer token created with vq web init-token and stores it in browser session storage.

The queue page is bounded by default on large-history hosts: it shows the newest 200 matching rows unless you choose a larger rows value. The selector offers 50, 100, 200, 500, 1000, and 2000 rows, and the same setting can be bookmarked as ?limit=500 or similar. Filtering and sorting still apply before the row slice, so use the search, state, host, and triage filters when you need to inspect old retained jobs.

Endpoints:

Endpoint	Purpose
`GET /queue`	Live queue table (htmx auto-refresh)
`GET /jobs/<id>`	Per-job detail: spec, resource history, log tail, exit status
`GET /health/{live,ready}`	Kubernetes-style probes for external monitoring
`POST /api/v1/jobs/<id>/{kill,pause,resume}`	Per-job write actions (v0.5.1+)
`POST /api/v1/queue/{pause,resume}`	Queue-wide actions (v0.5.2+)

All write endpoints require the bearer token in an Authorization: Bearer <token> header. For browser use, htmx + a small form prompts once and stores it in sessionStorage.

Architecture detail (auth, request shapes, error handling) is in vibe-queue/docs/web.md.

Fetching outputs¶

When a job completes, the workspace on the remote contains the outputs your script wrote (water.out, water.molden, …) plus the queue-side capture files (stdout.log, stderr.log, _vq/events.jsonl, _vq/exit-code).

vq fetch <jobid> ./local_outputs/       # rsync the whole workspace
vq fetch <jobid> ./outputs/ --files stdout.log water.out
                                         # specific files only

vq fetch is archive-aware (v0.5.11+): if the workspace was archived via vq cleanup --archive (see next section), fetch streams the .tar.bz2 over SSH and reconstructs the original directory layout on the laptop. No special flag needed; the same vq fetch <jobid> <local-dir> command works for both live and archived workspaces.

Operator controls (pause / resume / throttle / drain)¶

When the box gets busy for non-queue reasons (kids gaming, an interactive session, an urgent job from another chat), three knobs let vq step aside without losing in-flight work:

# Hard freeze (SIGSTOP); RAM stays allocated, no CPU used.
vq pause <jobid>             # one job
vq pause --all               # every running job
vq resume <jobid>             # SIGCONT
vq resume --all

# Soft throttle (cgroup CPUWeight, renice fallback v0.5.21+).
# weight=100 is default; weight=20 = "step aside" under contention.
vq throttle <jobid> --weight 20
vq throttle --all --weight 20 --persist     # persist across new dispatches
vq throttle --all --weight 20 --persist --duration 2h   # auto-release after 2h
vq throttle --release-persist                # clear persistent state
vq throttle --status                         # what's the current state?

# Drain (don't dispatch NEW jobs; running ones continue).
vq drain                     # full drain (no new dispatches)
vq drain --max-jobs 0        # explicit full drain
vq drain --max-jobs 2        # partial drain (cap at 2 concurrent)
vq drain --update-mode accept --reason "fleet upgrade"
                              # pause dispatch, still accept submits
vq drain --update-mode deny --reason "fleet upgrade"
                              # pause dispatch, reject new submits
vq drain --release           # back to daemon's configured max
vq drain --duration 1h       # auto-release after 1h
vq drain --status

Use --update-mode accept when you want users to keep queueing work during an update window, but you do not want any new job to dispatch until the host is current. Use --update-mode deny when accepting jobs against a stale or inconsistent managed runtime would be misleading; vq submit fails fast until the drain is released.

Composable: vq drain + vq pause --all + vq throttle --all cover the operator-control story. All four state files (drain.json, throttle.json, auto-cleanup.json, plus the per-job suspended-state on the spec) live under <state_root> and survive daemon restarts.

Workspace cleanup (v0.5.10+)¶

Long-running queues accumulate workspaces. vq cleanup is the manual housekeeping verb; it operates only on terminal-state jobs (active / pending / suspended jobs are never touched).

# List terminal-state jobs and their workspace ages:
vq cleanup
# → table: jobid, terminal_state, finished_at, workspace_size_mb

# Dry-run preview: show what would be archived:
vq cleanup --archive --older-than 30d

# Actually archive (add -x to "execute"):
vq cleanup --archive --older-than 30d -x
# → workspaces become tar.bz2 files under <state_root>/archive/

# Hard delete archived workspaces older than 90 days:
vq cleanup --delete --older-than 90d -x

# Restore an archived workspace (un-tar in place):
vq cleanup --restore <jobid> -x

The archive→restore round-trip is lossless: the directory tree after --restore is byte-identical to what was archived.

Auto-policy (v0.5.17+): instead of running the verb manually, register a daemon-side policy:

# Daemon runs the sweep once per --interval (default 24h):
vq cleanup --auto-enable --archive-after 30d --delete-after 90d

# Per-state retention (v0.5.23+): keep failed-job forensics longer:
vq cleanup --auto-enable --archive-after 30d \
           --archive-after-state failed:90d --delete-after 180d

# Read-only status:
vq cleanup --auto-status

# Disable:
vq cleanup --auto-disable

Configurable archive location (v0.5.22+): default <state_root>/archive/ may live on a small partition. Override via:

$VQ_ARCHIVE_DIR env var on the daemon host (applies to all archive paths globally)
--archive-dir DIR flag on the verb (per-policy with --auto-enable, per-invocation with one-shot --archive)

Why this matters: when the queue gets busy, workspaces add up fast (~10s of MB per typical SCF, ~hundreds of MB for big periodic + Molden + cube + .traj). Without cleanup, the <state_root> filesystem fills. With cleanup, you get a straightforward archive → delete pipeline that preserves the artefact history (every spec + final outputs) at small storage cost (~5× compression for typical output mixes).

Daemon admin¶

What happens at host reboot¶

The daemon survives if loginctl enable-linger is set:

Daemon restart only, running jobs become orphans with their pgids preserved; the new daemon re-attaches at startup. Job completes normally; exit code is read from the dispatched job’s _vq/exit-code file (so re-attach works even after a restart that wiped the Popen handle). This is v0.5.9’s _vq/exit-code marker, pre-v0.5.9 restart-orphans got marked ABORTED_BY_QUEUE even on clean completion.
Full host reboot, kernel kills everything; all RUNNING jobs are marked ABORTED_BY_QUEUE on next daemon start. Resubmit using the JobSpecs in the queue history.

Note

Wall-time enforcement gap when the daemon is down. Because v0.5.8 dropped cgroup RuntimeMaxSec (it wasn’t runtime-mutable on pause; see vibe-queue/docs/wall_time_design.md), wall-time enforcement is now the Python watchdog only. If the daemon crashes and stays down beyond the watchdog’s poll interval, a job that should have hit its --wall-time-seconds cap during the outage isn’t killed by the kernel; it keeps running until the daemon comes back and the watchdog catches up. In practice, Restart=on-failure on the systemd unit keeps the gap to a few seconds. The trade-off is documented in vibe-queue/docs/wall_time_design.md.

Refreshing the remote vibe-qc venv after a release (v0.5.20+)¶

As of vq v0.5.20, this is one verb:

vq admin update vibeqc-release

Which acquires a scoped update marker, records pause intent before pausing the affected jobs, refreshes Git, runs the configured update script, and resumes and proves that exact pause-token scope. Normal failures attempt cleanup; a hard interruption retains a durable serving-daemon receipt and dispatch hold for vq admin recover-update rather than claiming the queue resumed. The command reads git_dir, branch, and update_script from the host’s [programs.X] registry (see vq programs below).

Verifying a tagged release (v0.5.24+):

git push --tags
vq admin update vibeqc-release --tag v0.8.0
vq submit smoke_test.py --branch release

--tag v0.X.Y fetches that named tag, resolves refs/tags/v0.X.Y^{commit}, checks out the peeled commit, and requires the named ref still to resolve to HEAD before and after the update script. It fails closed if the tag is missing, moved, or mismatched, catching the libint- vanishing class of “pull succeeded but landed on the wrong commit” failures.

Checking remote state (v0.5.25+):

vq admin status
# NAME            BRANCH   SHA           VERSION  DIRTY  LAST_UPDATED_AT             LAST OK
# vibeqc-dev      main     abc12345defg  0.15.131  no     2026-08-13T14:30:00+00:00   True
# vibeqc-release  release  fedcba987654  0.15.130  no     2026-08-13T14:35:12+00:00   True

Compare SHA to your laptop’s git rev-parse --short=12 HEAD to answer “is the configured host at the commit I just pushed?” without SSH.

Chat workflow for testing a just-pushed feature:

git push                                    # laptop
vq admin update vibeqc-dev                  # refresh the default host
vq submit my_feature_test.py --branch main  # exercise the new code

This is the canonical pattern, always vq admin update between push and submit if you need the selected host at your latest commit.

Runtime slots: an opt-in, no-drain alternative (2026-07-28+). The legacy vq admin update flow works by recording and pausing its exact affected job scope, rewriting the checkout and installed package in place, then resuming. An editable install resolves directly from that checkout; a copied install can still replace not-yet-imported package files during reinstall. A live interpreter importing new modules mid-update can therefore silently mix two code versions from sys.modules (reproduced 2026-07-26, which is why the drain stays mandatory on that path). Setting VenvProgram.runtime_slot_root on a host switches to a per-SHA immutable layout instead: <root>/releases/<40-hex-sha>/ holds one fully-built runtime, and current/previous symlinks are flipped atomically once the new release is built and verified, no drain, no in-place rewrite, and a symlink flip back for rollback. This is opt-in per host and inert by default (runtime_slot_root is unset everywhere until you set it) – see vibe-queue/docs/design_immutable_venv_runtimes.md for the full design and current rollout status.

Other admin subcommands. Beyond update and status above: vq admin logs, vq admin auto-update (drift-check + apply across every registered program and host in one call), vq admin rollout-latest, vq admin recover-update (durable serving-daemon recovery), vq admin clear-update-marker (ordinary or pause-only markers), vq admin mark-ok, vq admin reset-branch, vq admin audit-recovery, and vq admin provision-user. Run vq admin --help (or vq admin SUBCOMMAND --help) for the full flag surface of each.

vq programs (v0.5.18+), list registered programs:

vq programs              # human-readable table
vq programs --json       # machine-readable; for scripts

The registry lives at ~/.config/vq/config.toml on the remote host under [programs.X]. Three kinds:

binary, CRYSTAL, ORCA, Psi4 (an executable on disk)
venv, vibeqc-dev, vibeqc-release (a Python venv + git checkout that vq admin update knows how to refresh)
import, pyscf (a module that should be importable from a specific Python)

venv records can also declare import_check, import_symbols, and healthcheck_command. vq programs runs the healthcheck from the program’s git_dir and reports the program as missing if it exits non-zero. This is useful for tools whose real readiness is more than an import, such as headless rendering:

[programs.vibeview-dev]
kind = "venv"
python = "/home/USER/vibeqc-dev/.venv-vibeview/bin/python"
git_dir = "/home/USER/vibeqc-dev"
branch = "main"
update_script = "scripts/update_vibeview_capture_env.sh"
import_check = "vibeview"
healthcheck_command = "xvfb-run -a .venv-vibeview/bin/vibe-view capture-selftest"
description = "vibe-view headless capture environment (main branch)"

For capture-only documentation jobs the managed environment can install vibe-view without the Trame web extra. From the repository root, use the transactional updater referenced by the program record above:

./scripts/update_vibeview_capture_env.sh --dry-run
./scripts/update_vibeview_capture_env.sh

The updater owns .venv-vibeview, keeps the previous environment until the replacement passes verification, and installs only the core capture profile. An older environment without the capture-specific ownership marker is refused; inspect it first and pass --adopt-legacy once only if replacing it with the lean capture profile is intentional.

./scripts/update_vibeview_capture_env.sh --adopt-legacy

It also provides an xvfb-run entry point that delegates to the system tool on Linux and falls back to direct PyVista offscreen rendering on macOS or OSMesa-capable Linux. That lean install is enough for vibe-view capture, vibe-view capture-selftest, and the Python vibeview.capture API. Treat the healthcheck as the program readiness gate:

.venv-vibeview/bin/xvfb-run -a \
    .venv-vibeview/bin/vibe-view capture-selftest

An exit code of 0 means the headless renderer is ready for queue-side QVF screenshots. Do not repair this by hand-editing managed GitLab checkouts on compute hosts; update the program through the queue-admin path below so every node uses the same checkout, venv, and healthcheck.

After adding that program to the host config and updating the queue code, provision or refresh it through the normal managed path:

vq admin update vibeview-dev HOST --show-output
vq programs HOST

Watching the daemon¶

journalctl --user -u vq-daemon -f       # live tail
systemctl --user status vq-daemon       # service health

Concurrency¶

The default daemon configuration is single-job dispatch (--max-jobs 1 in the systemd unit). This is the test-phase default, change to --max-jobs N in the unit file’s ExecStart and restart the daemon to parallel-dispatch.

Set --max-jobs honestly against the CPU budget: if jobs declare --cpus 8 and the box has 32 cores, --max-jobs 4 is the safe ceiling. The daemon does not currently enforce this; it accepts whatever you set.

Troubleshooting¶

Symptom	Likely cause	Fix
`vq: command not found` on laptop	venv not on PATH	symlink to `~/.local/bin/vq` or add the venv bin to PATH
`ssh: command not found` on submit	local SSH client not installed	install OpenSSH client
`Permission denied (publickey)` on submit	SSH key not authorised on remote	add laptop’s `~/.ssh/id_*.pub` to remote’s `~/.ssh/authorized_keys`
Job hangs in `QUEUED` indefinitely	daemon not running or `--max-jobs 0`	`systemctl --user status vq-daemon`; restart if dead
Job terminates `STARVED` at the 5-min mark	pre-v0.5.12, the watchdog read CPU from the wrapper PID only (bash sleeping in `wait()` shows 0% CPU even when the child process is using 16 cores)	upgrade to vq v0.5.12+; the watchdog now sums CPU across the whole pgid descendant set. As a workaround on older versions: `exec command` inside the wrapper so the worker becomes the dispatched PID
`import vibeqc` fails on remote	wrong `remote_python` (system Python instead of vibe-qc venv)	check `~/.config/vq/config.toml` `[hosts.X].remote_python`
Web UI write says “401 unauthorised”	bearer token missing or wrong	run `vq web init-token` (or `--force` to rotate), then clear browser sessionStorage and enter the current token

Comprehensive troubleshooting table in vibe-queue/docs/handover.md § Troubleshooting.

Version history (recent)¶

vq version	Headline
v0.25.0	Exact managed `vq self-update`; durable `vq admin recover-update`; failure-atomic lifecycle/runtime-slot transactions; legacy rollout reconciliation; fleet-console configuration and whole-surface authentication
v0.24.0	Accepted-report fleet rollout, exact source/tree provenance, scheduler-runtime deployment, and fleet-console identity baseline
v0.12.1	`vq drain –update-mode accept
v0.11.0	`vq submit auto`, per-job `--refresh`, serial-by-default fleet updates, and no-host per-job lookup around a down default host
v0.10.0	`vq host down/up/list`, client-side host availability hints for fan-out and submit routing
v0.9.2	`vq status` shows a PENDING job’s queue position
v0.9.1	`vq top --watch` + `--json`
v0.9.0	`vq top`, live per-job resource view
v0.5.27	`vq queue --state STATE` + `--active` shortcut, filter the listing
v0.5.26	`vq tail [HOST] JOBID --name FILENAME -f`, live-stream workspace files (default stdout.log; `--name` for vibe-qc’s logger output / engine-native files)
v0.5.25	`vq admin status`, live SHA / DESCRIBE / DIRTY + last-update record per venv env
v0.5.24	`vq admin update --tag v0.X.Y`, verify HEAD is at the expected tag post-pull
v0.5.23	per-state retention overrides: `--archive-after-state failed:7d`
v0.5.22	configurable archive_dir (`$VQ_ARCHIVE_DIR` env + `--archive-dir` flag + `AutoCleanupPolicy.archive_dir`)
v0.5.21	`renice` fallback for `vq throttle` on non-cgroup hosts
v0.5.20	`vq admin update <env>` minimal, one verb for pause / git pull / build / resume
v0.5.19	smoke test consumes absolute paths from `vq programs --json`
v0.5.18	`vq programs` verb + `[programs.X]` registry (binary / venv / import kinds)
v0.5.17	auto-cleanup policy (daemon main-loop hook reads `auto-cleanup.json`)
v0.5.13-.16	`vq throttle` / `vq drain` + `--persist` + `--duration` auto-release
v0.5.12	watchdog samples pgid descendants (fixes STARVED false-positive when bash-wrapped jobs sleep in `wait()`)
v0.5.11	archive-aware remote `vq fetch` from the laptop (v0.5.10’s cleanup verb produced archives the SSH-side fetch couldn’t see)
v0.5.10	`vq cleanup` verb (archive / delete / restore terminal-state workspaces)
v0.5.9	orphan exit-code recovery via `_vq/exit-code` marker, restart-orphans that complete normally are now `COMPLETED`/`FAILED`, not `ABORTED_BY_QUEUE`
v0.5.8	drop broken cgroup `RuntimeMaxSec`; Python watchdog is the single owner of wall-time
v0.5.7	`run-crystal.sh` cleans per-rank scratch on success; `--keep-scratch` opt-out
v0.5.6	`--branch` multi-venv routing (vibeqc-dev vs vibeqc-release)
v0.5.0-.5	web dashboard, pause/resume, bearer-token auth, CRYSTAL14 parallel dispatch
v0.4	cgroup-v2 enforcement, pgid recovery, event log
v0.3	resource watchdog (mem cap, wall-time, terminal-state machine)

Full per-version detail at vibe-queue/docs/handover.md § “What’s NEW in …” (the handover is the deeper reference; this page is the user-facing entry).

Roadmap (vq’s own)¶

vq has its own roadmap independent of vibe-qc; see vibe-queue/docs/roadmap.md. That live document owns future sequencing. PBS/Torque and Slurm backends are current functionality, not a future v1.0 promise.