Chat brief — `smear` (smearing across all periodic SCF backends)¶

You are the periodic-smearing chat for vibe-qc (https://gitlab.peintinger.com/mpei/vibeqc). Your one job: deliver fractional-occupation smearing as a shared, backend-agnostic utility consumed by every periodic SCF route — Ewald-3D, GDF, BIPOLE, GAPW — at every spin × k-mesh combination vibe-qc ships.

Target slot: the v0.10.x SCF-convergence program (docs/roadmap.md §”SCF guess + convergence program”). The row is currently marked “partial — extend at v0.10.x”. Not a v0.10.0 release gate — patch-line work, but high-leverage because every periodic backend benefits.

What’s already there¶

Fermi-Dirac smearing shipped in v0.4.0 (C1b per the roadmap):

Multi-k Ewald RHF + RKS — smearing_temperature in PeriodicRHFOptions / PeriodicSCFOptions / PeriodicKSOptions.
vq.kelvin_to_hartree_temperature(T_K) helper.
Free-energy A = E − TS with electronic entropy, μ by bisection, per-k occupations, real_space_density_from_kpoints_fractional on the C++ side.

That’s one flavor (Fermi-Dirac) on one backend (Ewald-3D) at two spin classes (RHF/RKS) and one k-mesh class (multi-k). Everything else — Γ-only Ewald, all of GDF, all of BIPOLE, all of GAPW, UHF + UKS, Methfessel-Paxton, Marzari-Vanderbilt — is missing or duplicated-poorly. Your charter is to make smearing a first-class shared utility that every periodic backend consumes uniformly.

Read first, in order¶

docs/roadmap.md — §”SCF guess + convergence program (multi-release)” smearing row; the v0.4.0 §C1b entry for the existing Fermi-Dirac shipping state; the v0.9.0 prep doc’s mention of metallic-system work.
CLAUDE.md — §2 branch model + push discipline; §3 multi-chat drop-box; §7 (critical — never paper over SCF oscillation with smearing tweaks; smearing is for partial occupations near μ, not a damping hack); §10 (no in-process imports of CP2K / PySCF; subprocess oracles only); §11 escalate-don’t-decide; §12 privacy.
Existing smearing implementation — grep for smearing_temperature, fermi_level, real_space_density_from_kpoints_fractional, and kelvin_to_hartree_temperature in python/vibeqc/ and cpp/src/. Understand the current contract end-to-end before touching it.
Sibling-chat handovers (to know what’s already in flight and what their drivers look like):
- docs/handover_gdf_v0_9_2026_05_20.md — GDF route.
- docs/handover_bipole_v0_8_2026_05_18.md — BIPOLE route.
- The gapw chat’s design doc once it lands (docs/design_periodic_gapw.md).
References — mandatory primary literature:
- Mermin, Phys. Rev. 137, A1441 (1965) — finite-T DFT; the free-energy formalism A = E − TS your existing implementation already uses.
- Methfessel & Paxton, Phys. Rev. B 40, 3616 (1989) — MP smearing, orders N = 1, 2.
- Marzari, Vanderbilt, De Vita, Payne, Phys. Rev. Lett. 82, 3296 (1999) — cold smearing.
- dos Santos & Marzari, Phys. Rev. B 107, 195122 (2023) — modern review of smearing-induced error and how to extrapolate T → 0.

Success bar (the only gate)¶

Smearing is correct, complete, and uniform across the matrix below. The cells say what is in scope; the bars are the parity / sanity witnesses.

Backend × spin × k-mesh coverage matrix¶

Backend	RHF / RKS Γ-only	RHF / RKS multi-k	UHF / UKS Γ-only	UHF / UKS multi-k
Ewald-3D	✅ extend	✅ shipped	⏳ NEW	⏳ NEW
GDF (closed-shell)	⏳ NEW	⏳ NEW	(open-shell GDF blocked on GDF chat — coordinate)	(same)
BIPOLE	⏳ NEW	(multi-k BIPOLE blocked on BIPOLE chat — coordinate)	⏳ NEW	(coordinate)
GAPW	(blocked on GAPW chat — coordinate; design with smearing in mind from M0)

Smearing flavors¶

Flavor	Spec	Bar
Fermi-Dirac (already shipped on Ewald-3D RHF/RKS multi-k)	`n_i = 2 / (1 + exp((ε_i − μ) / T))`, μ from electron-count bisection, `A = E − TS`	Extend to every cell of the matrix above. H-atom doublet: ⟨S²⟩ = 0.75 exactly. Closed-shell H₂ at any k-mesh, any T: matches RHF/RKS bit-for-bit at T → 0.
Methfessel-Paxton (N=1, 2)	Hermite-polynomial-smeared step function; documented in the 1989 paper	Mermin free-energy formula matches a published single-atom Al reference to µHa.
Marzari-Vanderbilt (“cold”)	Δ(x) = (1/√π) e^{−(x−1/√2)²}(2 − √2 x); entropy formula per the 1999 paper	Cold-smeared Al fcc free energy approaches T → 0 limit with the published cubic-in-T error, vs FD which has linear-in-T error.

Cross-cutting parity targets¶

CP2K (subprocess oracle) is the gold standard for periodic metals:

System	Bar
Na bcc (3³ k-mesh), Fermi-Dirac, T = 300 K	sub-mHa/atom + Fermi level to ~10⁻⁴ Ha vs CP2K
Al fcc (4³), Marzari-Vanderbilt, broadening = 0.01 Ha	sub-mHa/atom; cubic T-dependence of A − E0 (vs linear for FD) verified
Fe bcc, spin-polarized, Fermi-Dirac on UKS	per-spin μ behaviour matches CP2K; ⟨S²⟩ honoured under smearing
Cu fcc, MP-1	matches CP2K

Anything less is not done. No schedule pressure. Smearing is the wrong place to hide an SCF that doesn’t actually converge — if a backend oscillates with smearing on, the bug is in the backend, not in smearing. (See CLAUDE.md §7.)

Code paths you own¶

NEW module python/vibeqc/smearing/ — backend-agnostic shared utility:
- smearing/__init__.py exposes the public API: apply_smearing(eigenvalues, n_electrons, T, *, flavor), returning occupations + μ + electronic entropy.
- smearing/fermi_dirac.py — the existing implementation, refactored out of the Ewald-3D drivers into this shared home.
- smearing/methfessel_paxton.py — MP, orders 1 + 2 (parameterised).
- smearing/marzari_vanderbilt.py — MV cold.
- smearing/_mu_bisection.py — shared μ-bisection (works for any flavor’s occupation function).
- smearing/options.py — SmearingOptions dataclass that every backend’s options struct holds: temperature, flavor, optional mp_order. Single source of truth.
C++ side: generalise real_space_density_from_kpoints_fractional (currently per-spin RHF/RKS) so the same scaffolding handles UHF/UKS and any flavor’s occupation distribution. The shape is “per-k, per-spin, per-MO fractional occupation” — flavor only changes how the occupations are computed, not the inverse-Bloch fold.
Backend driver hookups (touch only the smearing entry points in each — do NOT rewrite their SCF loops):
- Ewald-3D: run_rhf_periodic_gamma_ewald3d, run_uhf_periodic_gamma_ewald3d, run_uks_periodic_gamma_ewald3d, run_rks_periodic_gamma_ewald3d, and the multi-k siblings — adopt SmearingOptions, route through vibeqc.smearing.
- GDF: run_pbc_gdf_rhf, run_krhf_periodic_gdf (and UKS / KRKS once GDF chat opens them) — same.
- BIPOLE: run_pbc_bipole_* and the periodic BIPOLE RKS / UHF / UKS drivers — same.
- GAPW: once that driver lands, ensure it consumes vibeqc.smearing from M0.
Tests under tests/test_smearing_*.py (utility-level) and tests/test_periodic_*_smearing.py (per-backend integration).
User-guide: docs/user_guide/smearing.md — single canonical page covering the three flavors, when to use each, the T → 0 extrapolation story (Marzari-Vanderbilt 1999 / dos Santos-Marzari 2023), per-backend availability matrix.

Out of scope:

Backend SCF correctness (each backend chat owns its own correctness).
Density mixing for periodic metals (Anderson / Broyden / Kerker / Pulay-Kerker / periodic Pulay) — that’s the scf-mix chat. Smearing and mixing compose; you do not duplicate that work.
Molecular UHF / UKS smearing — useful but not periodic; if Mike asks, scope a follow-up. Keep this chat tightly periodic-only.
Charged-cell Madelung corrections under smearing — coordinate with whichever chat owns that.

Architectural pre-flight (mandatory M0 deliverable)¶

Before touching any backend driver, land docs/design_smearing.md covering:

The SmearingOptions dataclass shape and the vibeqc.smearing.apply_smearing() contract.
How each backend’s options struct embeds SmearingOptions (e.g., PeriodicRHFOptions.smearing instead of smearing_temperature directly) — and the migration story for the currently-shipping smearing_temperature kwarg (deprecation path, back-compat in v0.10.x, removal target later).
The C++ real_space_density_from_kpoints_fractional generalisation contract.
How GAPW’s plane-wave grid composes with smeared occupations (so the gapw chat can design to it from M0).
The T → 0 extrapolation story exposed to users (Marzari-Vanderbilt — quote-fit ladder of T values, polynomial extrapolation).

This design doc goes through the release chat for sign-off and through the GDF / BIPOLE / GAPW chats for coordination before backend hookups start.

Operating discipline¶

Stay continuously integrated with main. Leave main release-ready at every step.

Before any session: git fetch origin && git pull --rebase origin main.

Small landable increments. Each milestone independently ships and leaves main release-ready. Never stack un-landed milestones.

Definition of Done — every milestone, repo-wide:

Full build + test suite green (C++ + Python). You touch shared code (every backend’s driver); run every backend’s tests, not just yours.
CI green on main after push.
No half-finished code paths. Incomplete-by-design work is gated by you, now — feature flag / experimental marker / xfail with reason. Document the gate.
CHANGELOG.md [Unreleased] claims only what is done AND tested. Do not claim per-backend smearing coverage you haven’t actually parity-checked.
Docs in parity this milestone: docs/user_guide/smearing.md, the per-backend user-guide pages’ smearing sections, the features.md per-backend smearing column, the roadmap “shipped” sweep on your slice.
No papered-over bugs. If a backend converges to a different stationary point under smearing than without, that’s a bug — find the cause (typically the inverse-Bloch fold under fractional occupations, or a spin-channel μ-bisection edge case), fix it, regression-test the symptom.
Privacy + licensing surface clean.

At each milestone: verify DoD → commit specifically → git pull --rebase origin main → re-run tests → git push origin HEAD:main → confirm CI green.

Re-sync triggers: branch > 1 day stale; main moved with un-landed commits; about to start a new milestone.

Coordination — heavy, by design¶

Your work touches every periodic backend’s driver. Coordinate aggressively:

Drop-box file: .release-status/v0.10.x/smear.md per CLAUDE.md §3. Open .release-status/v0.10.x/ if it doesn’t exist. Update at every milestone (self-id, branch + tip SHA, last-green tests, completed work per backend / per flavor with parity witnesses, next steps, blockers, asks).
GDF chat: their open-shell GDF (UHF / UKS) is in flight. Time your GDF UHF/UKS smearing hookup AFTER their open-shell GDF lands — track via their drop-box.
BIPOLE chat: their multi-k BIPOLE is in flight. Same coordination: Γ-only BIPOLE smearing now; multi-k BIPOLE smearing after they ship multi-k.
GAPW chat: they have a design pass before any code. Your SmearingOptions contract must be on the table during their M0 so they design to it. Send them your design doc before they freeze theirs.
scf-mix chat: smearing + density mixing compose for metallic SCF. Your SmearingOptions and their MixingOptions should be orthogonal and composable. Coordinate naming + dispatch so the metallic-system test set both chats target uses both surfaces together cleanly.
Release chat owns release operations (CLAUDE.md §13). Do not push to release. Do not create v* tags. Use Patch-candidate: trailers for v0.10.x patch landings.

Hard rules (from CLAUDE.md you must not violate)¶

No import pyscf / cp2k / crystal in python/vibeqc/ or cpp/. Subprocess parity oracles only (examples/regression/runner_*.py is the pattern). (CLAUDE.md §10.)
No force-push to main or release. No --no-verify. No amending pushed commits. (CLAUDE.md §2.)
No new bundled data / vendored deps without licensing review. (CLAUDE.md §1.)
Never paper over a divergent backend with smearing. Smearing fixes integer-occupation discontinuities at metallic Fermi surfaces; it does NOT fix wrong Hartree, wrong K, wrong Madelung, wrong gauge. If turning smearing on makes a backend converge that didn’t before, suspect the backend has a bug whose symptom smearing happens to mask — file it back to that backend chat. (CLAUDE.md §7.)

Bootstrap¶

cd ~/gitlab/vibeqc-smear/      # or wherever your worktree lives
git fetch origin && git pull --rebase origin main
scripts/setup_native_deps.sh
python3.14 -m venv .venv && .venv/bin/pip install -e '.[test]'

# Baseline-green: existing Fermi-Dirac on Ewald-3D
.venv/bin/pytest tests/ -q -k 'smear or smearing or fermi or fractional'

Document the baseline (what works today on which backend at which spin × k-mesh) in your drop-box file as milestone 0 before touching code. That’s your starting point and your back-out reference.

Suggested first milestones¶

A defensible sequence — refine as you go, but the design-first / refactor-then-extend ordering is the project’s standard for shared-utility work:

M0 — design doc (docs/design_smearing.md): SmearingOptions contract, apply_smearing() API, C++ contract, GAPW composability note, T → 0 extrapolation UX. Sign-off from release chat + GDF + BIPOLE + GAPW + scf-mix chats. No code yet.
M1 — refactor existing Fermi-Dirac into vibeqc.smearing without changing behavior. The current Ewald-3D RHF/RKS multi-k path keeps passing its tests bit-for-bit. Existing smearing_temperature kwarg keeps working (deprecation warning, alias to the new SmearingOptions-based interface). Land.
M2 — extend Fermi-Dirac to Ewald-3D Γ-only RHF/RKS (currently multi-k only). Land + ship parity tests.
M3 — extend Fermi-Dirac to Ewald-3D UHF/UKS (Γ-only + multi-k). Per-spin μ-bisection edge cases. ⟨S²⟩ under fractional occupations. Land.
M4 — Fermi-Dirac on GDF (Γ-only RHF / RKS first; UHF/UKS once GDF chat lands open-shell GDF). Parity vs CP2K on a metallic test cell.
M5 — Fermi-Dirac on BIPOLE (Γ-only first; multi-k after BIPOLE chat lands multi-k).
M6 — Methfessel-Paxton (orders 1, 2) on every backend × spin × k-mesh cell from M2–M5. Land.
M7 — Marzari-Vanderbilt cold smearing, same matrix. Land.
M8 — GAPW smearing once gapw chat has a runnable driver. Land at parity vs CP2K.
M9 — T → 0 extrapolation UX (vq.extrapolate_t_zero(results_over_T)) per Marzari 1999 / dos Santos 2023. Land.
M10 — production hardening: smart defaults per system type (the SCF-program “Recommended workflows by system” table), diagnostics, user-guide completion, deprecation removal of the legacy smearing_temperature kwarg.

Don’t land Mn before M(n−1)’s parity witnesses are in the test suite. Don’t try to do all backends at once; do the matrix cell-by-cell with parity checks per cell. That’s the pattern that survives an audit.

Before opening a merge or pushing your final work to `main`¶

git rm BRIEF.md if you created one — this orientation doc is per-chat, not project code. Your persistent state lives in .release-status/v0.10.x/smear.md, the design doc docs/design_smearing.md, and the user-guide page docs/user_guide/smearing.md.

Chat brief — smear (smearing across all periodic SCF backends)¶