Validation evidence model¶
Validation is a chain from immutable input to a narrowly worded claim.
One evidence bundle¶
For every checkpoint preserve:
- recipe and code revision;
- canonical named geometry vector and digest;
- analytical pole, residual, branch ID, overlap, truncation, and observables;
- serialized Tidy3D simulation, version, task ID, and cost estimate;
- raw data location and extraction settings;
- FDTD fit, observables, channel closure, and plots;
- before/after objective deltas at both fidelities;
- gate decision, rollback/acceptance, calibration ratio, and new trust radius.
An EvaluationRecord describes one geometry at one fidelity. A gate consumes
four records: analytical and FDTD at the same baseline, and analytical and FDTD
at the same candidate. Geometry digests make accidental cross-comparison an
error.
Claim levels¶
| Evidence | Allowed claim |
|---|---|
| pole residual and convergence | analytical branch is numerically resolved |
| centered finite-difference audit | local derivative is implemented correctly |
| one FDTD direction match | local direction worked once |
| two independent matches | repeated local direction evidence |
| FDTD numerical convergence | reported full-wave observable is stable to tested controls |
| all promotion gates | recipe is supported within its documented domain |
Absolute agreement is distinct from directional agreement. A biased reduced model can still be a useful optimizer if direction remains reliable inside a measured trust region. Conversely, one close absolute value does not validate gradients.
Failure records¶
Rejected blocks are valuable and must remain in the run history. Record the candidate geometry, raw outputs, suspected failure class, rollback point, and algorithm change. Never delete an inconvenient FDTD checkpoint or overwrite it with a later post-processing result; emit a new analysis record linked to the same raw task.
Root-level benchmark manifests separate immutable fixtures from mutable run ledgers. A checkpoint becomes a regression benchmark only after its extracted record, raw artifact provenance, and acceptance tolerance are frozen.