Building and running vibe-qc on a uni cluster¶

This page covers provisioning vibe-qc on a university HPC cluster, a shared batch system (e.g. TORQUE/PBS) where you log in to a small front-end node and real work runs on compute nodes via qsub. Such clusters differ from an ordinary single build host in two ways that shape the whole approach, so they get their own installer pair, scripts/install_cluster.sh and scripts/update_cluster.sh

rather than the plain install.sh / update.sh.

The example commands below assume a TORQUE cluster reached through an SSH alias uni-cluster; substitute your own host, queue, and node-feature names throughout.

TL;DR

From the cluster login node, in a vibe-qc checkout:

./scripts/install_cluster.sh --release      # first time (or --dev, or both)
./scripts/update_cluster.sh  --release      # refresh later

Or, once the vq scheduler host has scheduler_update_command / scheduler_install_command configured:

vq admin update uni-cluster
vq admin update uni-cluster --cluster-install

The interpreter a job should invoke is then ~/vibeqc-release/.venv/bin/python (or ~/vibeqc-dev/.venv/bin/python). Heavy work runs on a compute node via qsub; the login node only fetches + submits.

Why a uni cluster needs its own installer¶

Split tiers. The login node is often an older, small front-end, an out-of-date compiler (too old for libint/libecpint’s C++17) and only a handful of cores, not a build host. The compute nodes carry the modern toolchain (recent gcc, many cores, lots of RAM). So we bring our own toolchain (Miniforge) and compile on a compute node via qsub.
No compute-node internet. On many clusters only the login node has egress. So every fetch, Miniforge, the git clone, the vendored native-dep sources, and the Python wheels, happens on the login node, and the compute node builds entirely offline from the shared (NFS) /home.

The result is a two-phase model:

Phase	Where	Network	Does
login	login node	online	Miniforge → toolchain env → clone → stage native-dep sources (fetch-only) → download a complete wheelhouse
compute	compute node via `qsub`	offline	activate the toolchain, point `pip` at the wheelhouse, run the ordinary `scripts/install.sh`

Everything else reuses the normal build system: install.sh → setup_native_deps.sh → build_{libint,libxc,spglib,fftw,libecpint}.sh source-build the vendored native deps into third_party/<dep>/install/ for byte-parity with the rest of the fleet (and because vibe-qc’s libint needs custom max_am / deriv_order settings no conda-forge libint provides). Miniforge supplies only the prerequisite layer: Python 3.14, a modern gcc/g++/gfortran, cmake/ninja, and the build-time headers/libs (boost, eigen, gmp, openblas+lapack).

Access¶

Many university clusters expose only a login node, and some sit behind an institute gateway / bastion that you must hop through first. If yours is reachable directly, a simple Host block is enough; if it requires a second hop, collapse both with a ProxyJump in your ~/.ssh/config:

# Optional: only if your cluster sits behind a gateway / bastion host.
Host uni-gateway
    HostName <gateway-host>
    User <your-user>
    IdentityFile ~/.ssh/id_ed25519

Host uni-cluster
    HostName <cluster-login-host>
    User <your-user>
    ProxyJump uni-gateway          # drop this line if no gateway is needed
    IdentityFile ~/.ssh/uni_cluster_ed25519
    IdentitiesOnly yes

Then ssh uni-cluster, scp … uni-cluster:, and rsync -e ssh … uni-cluster: all route through the gateway transparently.

Cloning needs a read-only Deploy Key

The compute nodes have no internet, so the clone happens on the login node, which has no cached git credentials. install_cluster.sh generates a read-only ed25519 key at ~/.ssh/gitlab_vibeqc_deploy and prints it if it is not yet authorized. Add it once under GitLab → vibeqc → Settings → Repository → Deploy keys (leave Grant write permissions unchecked), then re-run.

On-cluster layout¶

install_cluster.sh creates the following under your $HOME (/home/$USER, which is NFS-shared to every compute node). This is the layout the vq TORQUE dispatcher targets, keep it in sync with the dispatcher’s probed-facts section.

$HOME/miniforge3/                       Miniforge base (python 3.13)
$HOME/miniforge3/envs/vqbuild/          toolchain env: python 3.14, modern gcc,
                                        cmake/ninja, boost/eigen/gmp/openblas
$HOME/vibeqc-dev/      (branch main)    dev checkout      + .venv/
$HOME/vibeqc-release/  (branch release) release checkout  + .venv/
$HOME/.vibeqc-cluster/                  wheelhouses, generated *.pbs, build logs

The interpreter to invoke vibe-qc in a job is $HOME/vibeqc-<variant>/.venv/bin/python (console scripts: .venv/bin/vibe-qc). That venv is built on the conda Python; the toolchain’s shared libraries are reachable from it via baked RPATH, so a job does not strictly need to conda activate, but doing so (or exporting LD_LIBRARY_PATH=$HOME/miniforge3/envs/vqbuild/lib) is a harmless belt-and-suspenders.

Bootstrap (first time)¶

On the cluster login node, in any vibe-qc checkout (e.g. clone once by hand, or let the installer clone the managed trees):

./scripts/install_cluster.sh --release        # release variant only
./scripts/install_cluster.sh --dev            # dev variant only
./scripts/install_cluster.sh                  # both (default)
./scripts/install_cluster.sh --release --wait # ...and block until built

What it does, in order:

Miniforge, installs to ~/miniforge3 (rootless, single HTTPS installer) if absent.
Toolchain env, conda create -p ~/miniforge3/envs/vqbuild with python 3.14 + c/cxx/fortran-compiler + cmake/ninja/make/pkg-config + libboost-devel eigen gmp openblas liblapack liblapacke libcblas.
Deploy key, generates + checks the read-only gitlab key (see above).
Per variant: clone → stage native sources (VIBEQC_FETCH_ONLY=1 setup_native_deps.sh, login node) → wheelhouse (pip download of the build + runtime + selected-extras requirements read straight from pyproject.toml) → submit the offline build to qsub.

Useful flags: --extras GROUP (default none, core runtime only; the cluster runs vibe-qc, it does not test it), --queue <your-compute-queue>, --ppn (default 20), --walltime (default 02:00:00), --no-submit (login prep only), --wait (poll to completion). See --help.

The compute build’s progress is tee’d live to ~/.vibeqc-cluster/logs/<variant>-build.<timestamp>.log (TORQUE otherwise only copies the job’s stdout back at job end).

Refresh¶

./scripts/update_cluster.sh --release                       # git pull + rebuild
./scripts/update_cluster.sh --dev --rebuild-native-deps     # after a vendored lib bump

update_cluster.sh fast-forwards the checkout’s branch on the login node (refusing a dirty tree, the cluster checkouts are tool-managed and must stay clean), re-stages sources, refreshes the wheelhouse, and submits the offline rebuild. --rebuild-native-deps wipes + re-fetches the vendored third_party/* trees first.

When driven by vq, put those same commands in the scheduler host config:

[hosts.uni-cluster]
ssh = "uni-cluster"
scheduler = "pbs"
scheduler_dialect = "torque"
scheduler_driver = "compute"
scratch_root = "/home/REPLACE_USER"
scheduler_update_command = "/home/REPLACE_USER/vibeqc-dev/scripts/update_cluster.sh"
scheduler_install_command = "/home/REPLACE_USER/vibeqc-dev/scripts/install_cluster.sh"

Then run vq admin update uni-cluster from the usual operator shell. If that shell is not the scheduler driver, vq delegates to scheduler_driver first, so the always-on daemon host performs the SSH to the cluster login node. The command claims the normal admin-update marker, refuses to rebuild while already-submitted scheduler jobs for that host are active, and keeps the marker on a failed remote update so the next vq admin status / marker recovery workflow has a breadcrumb. Use --update-script-arg=--release, --update-script-arg=--wait, or --update-script-arg=--rebuild-native-deps to append one-off flags to the configured command.

Running a vibe-qc job (and for the dispatcher)¶

Submit from the login node; jobs run on a compute node and read everything from the NFS /home. Job scripts must be pure ASCII, older TORQUE releases’ qsub reject anything else (qsub: file must be an ascii script), so avoid em-dashes / smart quotes in generated job files.

cat > ~/h2.pbs <<'EOF'
#!/bin/bash
#PBS -N h2scf
#PBS -q <your-compute-queue>
#PBS -l nodes=1:ppn=4
#PBS -l walltime=00:10:00
#PBS -j oe
cd "$PBS_O_WORKDIR"
~/vibeqc-release/.venv/bin/python my_h2.py     # the interpreter
EOF
qsub ~/h2.pbs

If your cluster has no dedicated /scratch and leaves $TMPDIR / $SCRATCH unset, use a working directory under /home/$USER (NFS, visible to login + compute).

How the offline build works (internals)¶

fetch-only mode. A guarded VIBEQC_FETCH_ONLY=1 path in setup_native_deps.sh and each build_*.sh clones/downloads the vendored source then exits before compiling, so the internet-connected login node can stage third_party/<dep>/src for the offline compute build. Guarded on the env var → zero change to ordinary fleet builds.
conda-aware preflight. build_libint.sh and setup_native_deps.sh also search $CONDA_PREFIX/include + $CONDA_PREFIX/lib (and add $CONDA_PREFIX to CMAKE_PREFIX_PATH) when a conda env is active, because boost/eigen/gmp/openblas live there rather than under /usr. Guarded on $CONDA_PREFIX, so non-conda builds are byte-for-byte unchanged.
offline pip. The compute job exports PIP_NO_INDEX=1 + PIP_FIND_LINKS=<wheelhouse>; PEP 517 build isolation inherits both, so pip install -e . resolves the build backend (scikit-build-core, pybind11) and all runtime deps from the wheelhouse with no network.
toolchain isolation. The build job sets VIBEQC_BUILD_NICED=1 (it owns the node; this also avoids install.sh’s nice/ionice re-exec) and CMAKE_PREFIX_PATH=$CONDA_PREFIX so vibe-qc’s own CMake finds conda’s Eigen.
BLAS pinning. The build passes -DVIBEQC_BLAS_VENDOR=OpenBLAS so vibe-qc links conda’s OpenBLAS instead of a broken system Intel MKL that some clusters leave on LD_LIBRARY_PATH (libmkl_avx512.so: undefined symbol). conda’s libopenblas is first in the extension’s RPATH, so this fixes both link time and run time, and jobs need no LD_LIBRARY_PATH scrubbing.
scipy. Installed explicitly (VQC_EXTRA_PIP, offline from the wheelhouse) because vibe-qc imports it unconditionally (python/vibeqc/density_fitting.py) but does not yet declare it in pyproject.toml – so import vibeqc fails without it on an extras-free install. Drop VQC_EXTRA_PIP once scipy is a declared dependency.

Troubleshooting¶

qsub: file must be an ascii script, a non-ASCII byte (commonly an em-dash) crept into the job file. Keep job scripts ASCII.
Clone Permission denied (publickey), the deploy key is not authorized yet; add ~/.ssh/gitlab_vibeqc_deploy.pub as a read-only Deploy Key.
pip download cannot find a wheel, a dependency lacks a cp314 wheel for linux. Drop the offending extra (--extras none) or pin a version that ships one.
Build seems stuck with no log, TORQUE copies the -o file back only at job end; watch the live log under ~/.vibeqc-cluster/logs/ instead, or watch third_party/*/install appear.
INTEL MKL ERROR … libmkl_avx512.so: undefined symbol at job run time means the extension linked a broken system MKL. Rebuild with -DVIBEQC_BLAS_VENDOR=OpenBLAS (install_cluster.sh does this by default).
NumPy … (X86_V2) but your machine doesn't support appears only on an old login node whose CPU predates the x86-64-v2 baseline that modern numpy wheels require. Run vibe-qc on the compute nodes, never on the login node, which is for fetch + submit only.

Coordination: the vq TORQUE dispatcher¶

The vq v1.0 TORQUE dispatcher backend (a vq daemon drives qsub/qstat/qdel over SSH, see vibe-queue/docs/pbs_dispatcher_backend_design.md) submits jobs that invoke vibe-qc on the cluster. It must target the layout above: the per-variant interpreter $HOME/vibeqc-<variant>/.venv/bin/python, your compute queue, an NFS /home working directory, and ASCII-only job scripts. This page is the source of truth for those paths; update both together if the layout changes.