Release-paper evaluation protocol

Protocol version 4 freezes the existing software inputs and independent archived replay procedure at the UTC timestamp in tests/data/paper/manifest.json. It binds tests/data/paper/replay-protocol.json by SHA-256 and retains the superseded draft’s identity. The 18 inputs and 105 software observations are unchanged. The reviewed procedure fixes the interface/environment matrix, energy selection, comparison tolerances, rounding, required outputs and retained failure/review evidence before the final independent replay.

This is a prospective freeze. Earlier calculations, timing observations and qualification attempts remain pre-freeze evidence; the timestamp does not retroactively establish a preregistered experiment. The original campaign’s immutable compiled recipe and executed settings remain authoritative. The final hash-bound scientific archive is separate and still required, including every failed attempt and explicit-review continuation. Freezing this procedure does not establish scientific acceptance or a completed release-paper baseline.

Issue #42 owns the real scientific workflow; #66 owns final independent replay from the exact 0.6.x — Negishi’s Funnel source/package identities. After the authorized release tag is created, verify that the tag and public downloads resolve to those accepted identities. A tag alone requires no new chemistry.

Identify the inputs before measuring

tests/data/paper/manifest.json enumerates 18 concrete inputs by SHA-256, format, provenance, covered features and expected operation outcomes. It reuses repository fixtures and five existing stress cases; three small SMILES inputs add a charged salt, a 12-membered ring and a relative-stereo refusal. No external chemical database or model data is included. Validate the exact bytes before using them:

python scripts/check_paper_corpus.py

The checker detects changed inputs, duplicate IDs/paths and invalid operation declarations. It does not run the operations or certify their results. Each expected entry defines one observation in the software denominator. Repeat it separately for each declared environment and interface; keep the same case IDs across SDK, CLI, REST, MCP and browser records. Report interface coverage explicitly rather than counting a shared implementation as five independent chemical validations. The radical document has a species and geometry but no drawn canvas: its declared checks are import and QVF round-trip, not a claim about radical glyph rendering.

Run the SDK observations with the target Python environment and write evidence outside the source checkout:

python scripts/replay_paper_corpus.py --output /tmp/paper-replay-attempt-1 --timeout 120

Draft 2 retains all 105 observations. The Suzuki fixture has reaction records but no canvas, so its tidy outcome is explicitly unchanged with the reason “the drawing is empty”. Draft 1 incorrectly expected successful layout; the exploratory mismatch is retained as negative evidence. This is a correction before protocol freeze, not a retrospective benchmark pass. An unchanged layout is not counted as a successful layout.

The destination must be new: retries use a new directory and retain the old attempt. Each operation runs in a fresh interpreter against the original input. The timeout is a worker safety budget, not a paper performance threshold. report.json records the manifest and harness hashes, installed package versions, a digest of the imported Python sources, every outcome and output artifact hash. Per-operation folders retain stdout, stderr and the returned artifact for inspection. Diagnostics may contain local paths; keep raw evidence private and review/redact it before publication. Package versions and Python source digests do not replace release archive hashes or bundled asset identity.

An expected refusal matches only the declared exception and reason. QVF round trips compare the entire document; a tidy operation returning an unchanged drawing with tidied=false does not pass the layout contract. Cleanup reports retain any fragments left unchanged for review. Success here means the stated software contract passed, not that visual or scientific acceptance passed. Other interfaces still require their own records. --case can select a subset, but all unselected operations remain not-run in the denominator and the run does not pass. Interrupted runs have complete=false; their retained prefix must not be reported as the full corpus. A completed run exits nonzero for any contract mismatch, missing dependency, timeout or unrun observation.

The CDXML inputs are repository-authored examples. Check CDXML import and subsequent QVF preservation, including diagnostics for unsupported content. Do not claim ChemDraw-generated provenance or consumer compatibility. Native ChemDraw testing and CDXML writing are outside the approved scope. QVF is the native editable output format.

The frozen software corpus has no additional generated cases. Any later change to its inputs, denominator or replay procedure requires a new version and an explicit reason, retaining this version and all earlier observations. A separate scientific manifest must bind the original #42 artifacts and settings, including failed attempts and their outcomes, before final independent replay. The checker verifies corpus and procedure bytes, timestamp syntax and version; it does not authenticate scientific acceptance, operator review or publication. Raw operational manifests remain in private evidence storage until reviewed for the citable archive. Earlier draft manifests remain checkable with their original status; changing a draft’s status alone cannot turn it into a frozen protocol.

Record every declared operation

Use one record per case, operation, interface, environment and attempt, with:

  • Protocol version/hash; input ID/hash; immutable software release, commit and package hashes; dependency and consumer versions.

  • OS/architecture, resources, runtime/model identity and weight digest where relevant; exact options, random seed and command with private paths redacted.

  • Cold or warm execution, elapsed time, timeout budget, output hashes and diagnostic; process exit and scientific verdict as separate fields.

  • Expected outcome, observed outcome (pass, refused, failed, timeout, unavailable, unchanged or not-run), reason, and attempt/parent identifiers.

An expected refusal passes the refusal contract only if the reason matches the declared limitation; it is never counted as a successful export. Missing dependencies and unrun cases stay in the denominator with their reason. Report totals by outcome and operation, including every retry. Do not discard slow, failed or unsupported cases. The 2,000-atom chain’s PNG/PDF size refusal is intentional; its SVG and QVF operations remain included. The spike reaction contains an SRU polymer bracket: figures draw the repeat unit in its brackets (#90), while it stays a drawing rather than a species. Its import, QVF preservation, cleanup and layout checks remain in the protocol.

For software invariants, compare connectivity, absolute and alkene stereo, charge, radicals and multiplicity before/after supported round trips and cleanup. Preserve reaction membership, coefficients, labels, conditions and mechanism arrows. Inspect full-scene exports for clipping, missing elements, physical size, PNG resolution and PDF font embedding where text is present. Hash equality proves byte identity only; visually equivalent PDF or PNG output across platforms needs the declared semantic/visual checks too.

Use the existing performance harness for exploratory timing. It records first-in-process and repeated warm operations, not an independent cold application startup for every operation. Measure desktop startup separately. Do not turn exploratory timings into retrospective pass thresholds. Protocol timing budgets do not change the separate 0.8 performance milestone.

Scientific campaign and independent replay

The acceptance carrier is the real ethene pilot, followed by 1,3-butadiene + ethene → cyclohexene, under the reviewed #42 recipe: MACE optimization, PBE/def2-SVP optimization and same-surface Hessian without dispersion, then the explicitly supported DLPNO single point. Preserve the resolved settings and actual triples implementation; a method label alone is insufficient. Record any required model licence acknowledgement without embedding private operator information in public artifacts.

For each species and stage, retain the raw input/output, accepted geometry, producer attempt, artifact digest, software/model identity, charge, spin, atom order, units, convergence checks and validator verdict. A downstream stage must use its accepted parent’s geometry. Process completion, an article-owned driver and stand-in test fixtures do not establish successful integrated Sketch submission or scientific acceptance. Failed optimizations, SCF/Hessian assessments and correlated calculations remain visible and block dependent accepted results. Keep retry lineage and original failures. For a correlated single point, a converged coupled-cluster calculation does not override an unusable or unstable reference verdict. Hold that energy out of accepted figures and reaction sums until the reference passes the declared scientific checks; retain the completed but rejected attempt.

A second operator uses the immutable 0.6 release and checksummed archive to:

  1. Verify input/artifact identities and accepted-parent lineage, then replay integrated Sketch launch separately from any article-owned driver.

  2. Recreate same-orientation stage figures, tables and the explicitly labeled reference-property view from real archived outputs.

  3. Select the declared immutable campaign and correlated stage through campaign-energies, plus reactions and stoichiometric coefficients. Recipe campaigns do not create the submission-group runs used by reaction-energies. Independently extract each absolute value and its source line/key, then sum products minus reactants with the same units and thermal conditions. Do not use the application’s own aggregation as the independent oracle.

  4. Compare the full-precision CSV and displayed Markdown/booktabs rounding, scientific values against predeclared tolerances, and publication figures against the declared visual invariants. Record environment differences.

  5. Exercise missing/conflicting/stale results and unavailable thermochemistry. Do not invent thermal terms, mix incompatible methods or treat a MACE model-reference energy as an electronic reaction energy.

Final acceptance requires #42’s real campaign, a frozen protocol, the second operator’s replay, and a citable software/data archive with an accurate availability and limitations statement. Preserve operational receipts in private operations storage; publish only reviewed, public-safe evidence. Keep manuscript work in its existing canonical checkout.

Machine admission of stage results

The campaign validator requires electronic stages to carry a converged, usable provenance.verdict; an explicitly unstable reference is rejected. A completed correlated calculation cannot override that verdict. A separately reviewed scientific interpretation remains an external review record, not an automatic waiver inside the validator.

Optimization producers must export provenance.optimization.converged: true and final_forces: {units, values} within that optimization record. Values are one finite Cartesian vector per final-geometry atom; supported force units are eV/angstrom and Eh/bohr. Acceptance compares the maximum atom vector norm with the saved recipe fmax. Missing evidence is rejected, including older output containers that only state convergence. Producer support is required before those containers can establish an integrated accepted campaign.

Hessian frequencies must all be finite and contain either all 3N modes or exactly 3N minus the geometry’s rigid-body count. Wrong-sized or partially nonfinite inventories are rejected; existing imaginary-frequency thresholds are unchanged.

Per-stage execution runtimes

compile_campaign(..., stage_runtimes={...}) accepts overrides keyed by a numerical stage id (mace, dft, hessian, dlpno for the baseline). Each object can set host, program, python, runner (an argument list), cpus, mem_mb, wall_time_seconds, and check_wall_time_seconds. Omitted fields inherit the existing campaign defaults. For example, {"mace": {"host": "model-host", "program": "model-program", "python": "model-python"}} selects a separate model environment while the DFT, Hessian and coupled-cluster stages retain the QC runtime. The operator must select installed, compatible programs and interpreters; compilation records the selection but does not certify remote availability.

The CLI reads this mapping from --stage-runtimes runtimes.json; the REST compile request and MCP compile_campaign tool accept stage_runtimes directly. Unknown stages or fields and invalid resources are refused before writing the campaign. Each check runs on its stage’s host and interpreter. The campaign and source digests pin these settings alongside the inputs; geometry admission, queue attempt records and retry dependencies remain the same. Changing runtimes requires a new campaign, not editing a running one.

Explicit review of a restricted reference

A raw usable=false result stays rejected. For a prescribed RHF DLPNO single point with converged SCF and CC, a reviewer may explicitly accept an externally unstable reference within the unchanged restricted protocol when independent evidence establishes internal stability. This is a scoped operator decision, not an automatic scientific PASS or a global-minimum claim. No other stage or rejection reason is eligible.

A QVF may describe the RHF reference in provenance.method while carrying its correlated total only in the latest run.record’s attachment.structured. Review eligibility verifies that member and the job/input hashes, a unique converged dlpno_ccsd_done event, the matching method and explicit triples mode, and a finite e_total in hartree. Conflicting root totals are refused. The accepted energy records the source event and member/input hashes, including the producer’s actual triples algorithm; a requested mode label is not evidence of a different algorithm. The RHF energy is never used as the correlated total. Historical reference rejections remain unchanged, including their reference-method label, so immutable review can still verify the original failed verdict bytes.

After the actual campaign computations finish and its terminal DLPNO check fails, create a separate immutable continuation:

vibe-sketch campaign-review scheme.vsk recipe1 recipe1-c1 \
  --out reviewed-campaign --review decision.json

The decision file contains:

{
  "node": "m1-dlpno-check",
  "reviewer": "responsible-operator",
  "reason": "Independent evidence supports the prescribed restricted reference.",
  "criteria": {
    "reference": "RHF",
    "scf_converged": true,
    "internal_stable": true,
    "external_stable": false,
    "cc_converged": true,
    "protocol_unchanged": true
  },
  "warnings": ["External instability retained; no unrestricted or global-minimum claim."],
  "evidence": ["sector-review.json", "sector-calculation.json"]
}

These criteria are the operator’s evidence-backed attestations. The program checks result identity, convergence and the permitted rejection, but does not independently establish the truth of a supplied scientific review or authenticate the supplied reviewer identity. Evidence paths resolve on the invoking machine (on the server for REST/MCP), are copied, and are bound by SHA-256. The supported Queue fetch API retrieves the original failed check’s verdict; it must match raw revalidation. Missing original job files refuse the operation.

The continuation retains the original campaign/journal digest, failed attempt and job, original verdict bytes and hash, reviewer, timestamp, reason, criteria, warnings, evidence hashes, input container/source hashes, parent geometry and result hashes, and original computation artifact manifests/attempts. All computations must have verified results and parent bindings must still match. Other checks must have succeeded, except for additional failed terminal DLPNO checks with the same restricted-reference rejection. Their original verdicts must match raw revalidation too. Modified sources or artifacts are refused. It cannot import a standalone pilot as if it were a completed integrated campaign.

When more than one species needs a reference review, review one check at a time. Run the first continuation: its selected check carries the explicit decision, while unreviewed checks remain rejected. Review the next failed check using that continuation’s campaign ID and another new directory, then run it. Each decision needs its own species-specific evidence; neither an earlier review nor another species’ stability establishes acceptance. Subsequent continuations preserve earlier review evidence and the original rejected attempts verbatim. They reuse the verified computation bytes and repeat validation without rerunning chemistry. The campaign succeeds only once every check has passed or received its own permitted, explicit review. Failed chemistry, nonterminal failures and incomplete campaigns still block review.

Run the printed vq campaign run command. Explicit other nodes replay the verified computation bytes, checking the exact parent geometry; validators run again. The reviewed verdict carries acceptance_kind: operator-reviewed, the raw rejected assessment and unchanged raw reference verdict. Original results, campaign files, attempts and retries remain untouched. A retry of the original failed check still has no review waiver. The new accepted artifacts can be consumed by campaign-renders and its existing View integration.

SDK: vibesketch.document.review.review_campaign(document, recipe_id, campaign_id, directory, **decision) returns Compiled and an updated document. REST: POST /api/documents/{id}/recipes/{recipe}/campaigns/{campaign}/review accepts the decision fields plus revision and an absolute directory. MCP: review_campaign accepts the same decision plus recipe, campaign, and directory. Every interface creates a new campaign record; none rewrites the original campaign or Queue journal.

Validation nodes carry their stage’s selected program, host and interpreter so Queue can bind the managed runtime identity as it does for computation. wall_time_seconds limits computation; check_wall_time_seconds independently limits validation. Both accept integer seconds from 1 through 31,536,000 (one year), or null to use Queue defaults; the scheduler may impose a lower limit. The SDK/REST/MCP global options and CLI --wall-time-seconds and --check-wall-time-seconds supply defaults; stage overrides can give a Hessian more time without extending every validation job. Omitted limits preserve Queue’s configured defaults. No runtime installation is implied by compilation.

Review continuations retain original execution resource settings in each replay.json, but artifact replay uses one CPU, Queue’s default memory and the original stage’s validation wall-time limit. It keeps the program, host and interpreter binding. Older campaign checks that omitted a program inherit their original compute node’s program in the new continuation only; original campaign files remain immutable. The original geometry and artifact hashes are preserved.

Revalidating a frozen energy-reader rejection

A completed campaign may retain an older validator that cannot read the structured DLPNO correlated total. campaign-revalidate compiles a separate continuation:

vibe-sketch campaign-revalidate scheme.vsk recipe1 recipe1-c1 \
  --out /new/continuation --node m3-dlpno-check \
  --reviewer operator --reason 'Apply the structured correlated-energy reader'

The SDK operation is vibesketch.document.review.revalidate_campaign; REST uses POST /api/documents/{id}/recipes/{recipe}/campaigns/{campaign}/revalidate, and MCP exposes revalidate_campaign. The browser’s staged-workflow report has matching fields. Compilation does not submit jobs: run the returned controller command to replay the hash-verified computation bytes and validate them.

This operation accepts only the older missing-total-energy rejection when the current strict DLPNO assessment passes. It binds both validator versions, the original rejected verdict, exact result, original campaign and journal, and accepted parent attempts. Reports retain acceptance_kind: validator-revalidated and the original rejection. Changed artifacts, incomplete campaigns and scientific failures are refused. Other restricted-reference failures remain rejected until each receives its separate evidence-bound review; revalidation is not a waiver.