Release validation protocol

Protocol v2 is frozen before the release study runs. Its SHA256 is retained in the results. Changes to the protocol require a new version and a separate run. This is a bounded release screen, not a claim of nominal coverage on all data.

The v2 amendment changes the candidate identifier only. Estimands, data generation, seeds, budgets and decision rules remain those frozen in v1. The rerun checks the reviewed numerical correction.

Scope and invariants

The candidate is 0.4.0rc2. Primary scope is two-arm complete or prespecified stratified randomisation and equal-geo parallel DiD or Lift. Analytic intervals, BCa intervals and sharp-null tests receive separate evidence assessments. CRT and staircase inference remain experimental. No CRT power is claimed. Original trial allocations must remain unchanged when upgrading.

Independent numerical references

Check the public parallel analysers against separately computed stratum-weighted contrasts, variance, degrees of freedom and intervals. Enumerate a small finite population with heterogeneous potential outcomes to check unbiasedness and the Neyman variance identity. Enumerate all permitted allocations under a sharp null to check exact-test size. Check BCa against an independently assembled bootstrap and delete-one jackknife calculation. These are numerical references, not an independent human review or simulation evidence.

Check that incomplete windows cannot silently remove eligible geos from power scenarios. Invalid panels, allocation changes, zero Lift baselines and undefined variance must fail visibly. Repeated calls must preserve seeds and float64 data.

Simulation matrix

Use 250 repetitions in each of twelve data-generating cells. Each panel is analysed with both DiD and Lift, each with analytic and BCa intervals. The analytic call also requests a sharp-null randomisation test. This gives 12,000 public analysis calls. Use 999 bootstrap and 999 randomisation draws per call. Exact enumeration applies when the allocation space fits that budget.

All panels have eight dates, four baseline and four test. Untreated outcomes contain a geo intercept, common linear time trend and AR(1) disturbances with coefficient 0.6. Allocation is drawn independently of outcomes with fixed arm counts in each stratum. Potential outcomes and random seeds are retained through the fully specified generator. Each scenario has zero and positive intervention cells.

Scenario	Geos	Distinctive condition	Role
regular	40	Normal errors, positive stable baselines	Core
unequal_strata	40	Strata of 17 and 23, different trends and variances	Core
few_geos	12	Six geos per arm	Stress
unequal_variance	40	Geo error scales from 2 to 15	Stress
heterogeneous_effect	40	Geo effects from zero to twice the nominal effect	Stress
unstable_baseline	40	Positive baselines spanning several orders of magnitude	Stress

The positive nominal effect is five outcome units. It is constant except in the heterogeneous-effect scenario. The truth for DiD is the finite-population mean of geo effects. The truth for Lift is the mean of geo effects divided by their realised pre-treatment baseline means. This is not a ratio of population totals. The zero-effect cell is the sharp null for both methods. Rejection under the positive cell is descriptive; it has no release power threshold.

Use master seed 20260912 and separate deterministic streams for cells, repetitions, outcomes, allocations and resampling. Do not select seeds after seeing results. Save every replicate’s estimate, truth, interval, p-value and failure status, plus software versions and source hashes. Record bias, bias Monte Carlo standard error, RMSE, interval width, coverage, null rejection and 95% Wilson Monte Carlo intervals. No failed replicate may be silently discarded.

Decision rules

Numerical references and input-contract checks must pass.
Any unexpected failed fit in a core simulation cell blocks its method’s core approval. Failed stress cells restrict scope and must be disclosed.
Bias must be within four Monte Carlo standard errors of zero. This is an investigation trigger, not evidence that bias is exactly zero.
For coverage, a 95% Wilson lower bound of at least 0.90 passes this minimum screen. An upper bound below 0.90 fails; other results are inconclusive.
For sharp-null rejection, a 95% Wilson upper bound of at most 0.10 passes this minimum screen. A lower bound above 0.10 fails; other results are inconclusive.
These tolerances detect material miscalibration with a bounded run. They do not certify 95% coverage or exactly 5% size. Also flag any coverage upper bound below 0.95 or null-rejection lower bound above 0.05 for methodological review.
A failed or inconclusive core method remains experimental. Stress failures do not justify changing the acceptance criteria or rerunning preferred seeds.
CRT, staircase and BCa receive no approval by association with analytic DiD. BCa p-values remain analytic unless a randomisation test was requested; BCa interval coverage and p-value size are separate assessments.

Budget the study for at most 30 minutes on the local machine. If it exceeds that budget, retain partial evidence and mark remaining cells unexecuted. Do not reduce the frozen repetitions and describe the reduced run as completed.

Operational and packaging gates

Run the full software suite, all CLI analysers, a complete saved-allocation workflow, documentation with unsuppressed broken-reference warnings, and wheel and source-distribution installation outside the checkout. Test Python 3.11 and 3.12 in clean environments. Resolve dependencies from one package contract and retain the installed versions. Do not claim other interpreter versions tested.

Missingness, interference, non-adherence, adaptive stopping and outcome-driven allocation changes remain outside the release’s automatic causal guarantees. Document the minimum retained design and decision record. A release candidate is a review artefact; this protocol does not authorise publication or deployment.