Memory budget¶
Every Gaussian-basis and semiempirical run_job route covered by the memory
estimator runs through a pre-flight check that (a) reports the estimated peak
memory in the text output and (b) aborts with an explanation if that estimate
exceeds the memory available to the process. Pretrained MLIP routes are a
single forward pass and currently skip this estimator. On Linux the probe
includes finite scheduler or container cgroup limits, even when the host
itself has substantially more free RAM. This is a guardrail against the most
common catastrophic failure mode of a QC code, calculations that silently
thrash to disk and freeze the host.
The estimate is a planning model, while peak resident set size (RSS) is a measurement. For the practical tradeoff between retaining \(N_\mathrm{bf}^4\) integrals and recomputing them with direct SCF, see Direct SCF or in-core integrals?.
What you’ll see in the .out file¶
Job: RKS / PBE basis=cc-pvdz
Atoms (bohr)
------------------------------------------------------
1 Z= 8 0.00000000 0.00000000 0.00000000
2 Z= 1 0.00000000 1.43000000 -0.98000000
3 Z= 1 0.00000000 -1.43000000 -0.98000000
charge=0 multiplicity=1 n_electrons=10
vibe-qc estimates this calculation will require ~0.17 GB of memory:
ERI tensor 2.5 MB
Fock + density + 1e 0.04 MB
DIIS history 0.07 MB
MO workspace 0.02 MB
Python runtime + NumPy overhead 100.0 MB
DFT grid + chi 13.4 MB
Available on this machine: 24.0 GB. Proceeding.
iter energy (Ha) dE ||[F,DS]|| DIIS
...
The headline figure already carries a 1.5x default safety headroom
over the sum of the per-category numbers. Set
VIBEQC_MEMORY_HEADROOM=2.0 (or another value >= 1.0) to tune the
factor for a site or scheduler wrapper.
Molecular density-fitting reports show the three-index formula directly, including the parsed auxiliary-basis dimension. For example:
DF three-index tensors 28.86 GB (n_orb^2=1134^2 x n_aux=753 x n_blocks=2 x 8 bytes x 2x safety)
The tensor extent is n_orb^2 * n_aux, not an orbital-basis-only estimate.
At every basis size, preflight charges the two resident three-index blocks
and a separate 2x safety factor for overlapping raw and transformed
construction buffers: the native DensityFitting constructor holds four
n_aux * n_orb^2 extents simultaneously while it builds the fitted
tensors, independent of n_orb. (An earlier revision applied the
construction factor only above 500 orbital functions; production
measurements on the BH9 wave showed the same ~2x construction peak on
small bases, so the size gate was removed.)
When density_fit=True is requested without an explicit aux_basis,
the preflight sizes n_aux from the same default JK-fitting auxiliary
basis the driver auto-resolves, so the estimate and the execution agree
on the auxiliary dimension. Only when no default is registered does it
fall back to the 3 * n_orb design bound (Eichkorn et al., 1995). The
density_fit=True / cosx=True keyword arguments to run_job are
applied to the SCF options before the pre-flight, so a kwarg-configured
DF job is estimated on the DF branch, never as a direct-SCF job.
Peak RSS is a high-water mark¶
Peak RSS records the largest resident memory observed at any point in the process. It does not fall when an array is freed. If conventional RHF builds and later releases a dense AO ERI tensor, the final peak still includes that tensor. This is expected high-water-mark behavior and does not by itself indicate that the tensor remained live.
run_job(..., perf_log=True) writes a .perf sibling with component wall
and CPU times plus RSS snapshots. Combine it with a whole-process
high-water-mark measurement when comparing algorithms. Record CPU time, wall
time, peak RSS, basis count, thread count, host, version, convergence
settings, and output plan together.
When the estimate exceeds available RAM¶
vibe-qc estimates this calculation will require ~218.4 GB of memory:
ERI tensor 186.0 GB
...
Available on this machine: 7.2 GB. ABORTING.
InsufficientMemoryError: Pass `memory_override=True` to `run_job` to proceed
anyway. In a low-level workflow, call `check_memory(..., allow_exceed=True)`.
To reduce the estimate, use a smaller basis, density fitting with an auxiliary
basis, or integral-direct SCF.
Overriding the check¶
Pass memory_override=True to run_job:
from vibeqc import Molecule, run_job
mol = Molecule.from_xyz("large.xyz")
run_job(
mol, basis="def2-tzvp", method="rhf",
output="huge",
memory_override=True, # accept the risk of swap / freeze
)
When the estimate exceeds the available-memory probe, the output then reads
Proceeding (override) instead of Proceeding so anyone reading the log later
knows what happened. If the estimate already fits, the status remains
Proceeding because no override was needed.
Estimators covered¶
Method |
Dominant cost |
Notes |
|---|---|---|
RHF / UHF |
Dense ERI tensor or direct-SCF shell-pair scratch |
Direct SCF charges about 1 KB per shell pair |
RKS / UKS |
HF baseline + DFT grid, weights, coordinates, libxc scratch |
meta-GGA adds tau buffers |
MP2 / UMP2 |
in-core OVOV or budgeted direct/disk panels plus DF factors |
selected mode and panel dimensions are reported |
CCSD / CCSD(T) |
T1/T2 amplitudes, D1/D2 intermediates, budgeted triples workspace |
phase peak, not a sum of released construction buffers |
DLPNO-MP2 / DLPNO-CCSD(T) |
PAO/PNO domains, pair lists, auxiliary pair workspaces, triples domains |
conservative local-domain estimate |
CAS/CI/FCI |
CI or determinant vectors, RDMs, MO integral transforms |
exponential methods report their vector storage |
Selected CI |
Dense Hamiltonian over the selected space, the selection-step candidate buffer, MO integral transform |
sized by |
Periodic GDF / GPW / GAPW |
GDF Lpq factors, bounded Ewald-J FT cache/batches, or FFT-grid collocation/cache arrays |
Lpq and grid estimates use dry-run/live preflight; exact Ewald-J uses explicit cache and batch targets |
NEB uses a separate per-image-worker model and is intentionally not folded into the single-point estimator.
Reading the probe yourself¶
The cross-platform “how much memory is available right now” probe is exposed for scripting:
import vibeqc
available_gib = vibeqc.available_memory_bytes() / 1024**3
print(f"{available_gib:.1f} GiB")
An explicit VIBEQC_MEMORY_LIMIT_BYTES or VQ_MEM_MB value takes priority;
if both are set, the tighter limit wins. Without an explicit limit, the probe
first obtains a host-wide value from
psutil.virtual_memory().available (install
with pip install psutil to get the highest-quality number). It falls back to
/proc/meminfo on Linux, vm_stat on macOS, and finally the generic
os.sysconf total-memory value when no available-memory probe succeeds. On
Linux, finite cgroup v2 (memory.max and memory.current) or cgroup v1
memory-controller limits bound the explicit or host value. Parent cgroup
limits are included, which is important when a scheduler gives each task an
unlimited leaf inside a limited job cgroup. The returned value is therefore no
greater than either the declared allocation or applicable host/cgroup limit.
The probe returns 0 if every applicable source fails. check_memory treats
that as “unknown” and silently proceeds rather than false-aborting on an
unsupported platform. A known exhausted cgroup is not unknown and fails the
preflight check.
Writing your own estimator¶
If you call the low-level SCF drivers (run_rhf, run_rks, …)
directly instead of through run_job, you can invoke the estimator
yourself:
from vibeqc import estimate_memory, check_memory
def preflight_rhf(mol, basis, rhf_options):
est = estimate_memory(mol, basis, method="rhf", options=rhf_options)
print(est) # human-readable block
check_memory(est) # raises if the estimate exceeds the budget
return est
The estimator returns a
MemoryEstimate dataclass with a
by_category dict, so you can inspect exactly where the memory goes.
For conventional molecular SCF, the dense AO contribution is
\(N_\mathrm{bf}^4\times8\) bytes. For direct SCF, the estimator replaces
that category with Direct-SCF shell-pair scratch. Switching the reference
SCF route does not remove method-specific MP2, CCSD, CCSD(T), response, or
property intermediates, so estimate the complete requested method.