A retained experiment workflow

Use one primary estimand and a frozen allocation. Preserve the evidence needed to reproduce both the statistical result and the decision. The runnable examples/retained_parallel_workflow.py demonstrates the file hand-offs on synthetic data. It is an example, not a production trial registry.

Before allocation

The analyst and decision owner record these items in an analysis plan.

Item	Minimum record
Decision	Action, practical effect threshold and decision owner
Intervention	Assigned treatment, comparator and implementation dates
Population	Eligible geo IDs, exclusions and any planned strata
Outcome	Definition, units, aggregation and measurement source
Estimand	Equal-geo outcome-unit effect or baseline-normalised effect
Analysis	Analyser, windows, interval, test, confidence level and seeds
Observation	Expected geo/date grid and handling of unavailable outcomes
Deviations	Procedure for spillover, non-adherence and measurement changes
Follow-ups	Multiplicity, exploratory analyses and stopping rules

Choose DiD for an equal-geo change contrast in outcome units. Choose Lift only when baseline normalisation is substantively justified and baselines are stable and positive. Equal-geo weighting does not estimate a population-weighted market total. Freeze the target before choosing the analysis.

Use run_power_analysis on pre-treatment history with the intended analyser, strata and test. Retain the historical data hash, settings and failure counts. Its injected proportional intervention is a scenario assumption; it need not equal the reported estimand. Inspect several plausible intervention sizes and historical windows. Do not interpret a successful power calculation as assurance against spillover or measurement failure.

Allocate and retain

Freeze strata from pre-treatment summaries. Allocate once using the declared complete or stratified procedure. Keep the geo IDs, assignment, treatment probability, strata, seed, software versions and generation settings.

Save DataFrame attributes separately as JSON. CSV does not retain them. Preserve the resulting allocation table even if the package or random-number generator later changes. The allocation table, rather than a seed alone, is the trial record.

Review baseline balance for errors or extreme practical problems before launch. Balance diagnostics do not certify causal validity. Searching seeds until a preferred allocation appears changes the assignment mechanism and invalidates the current unrestricted randomisation-test contract.

Execute and reconcile

The trial owner records delivery, contamination and concurrent changes. The analyst reconciles outcomes against the retained allocation and expected calendar before fitting. Compare complete ID and date sets, not just row counts. A geo or date absent from the whole input cannot be detected by the analyser alone.

Use assigned treatment for the prespecified assignment-effect analysis. Preserve delivery deviations separately. Missing observations must stop the standard complete-panel workflow; do not silently delete geos or fill outcomes with zero. Any alternative missing-data procedure needs a separate justified analysis plan.

Analyse and interpret

Run the frozen analyser on the reconciled panel. Save the full result dictionary, input hash, dependency versions and analysis options. Check effect units, warnings and failed or unavailable diagnostics before preparing a brief.

Use one prespecified interval to assess the average effect. A sharp-null test asks a different question and must be labelled separately. Do not choose between interval methods after seeing which excludes zero. CRT and staircase inference remain experimental and need design-specific review.

If the interval includes zero, report an inconclusive effect direction at that confidence level. Compare its limits with the practical threshold. A claim of equivalence requires its own prespecified margins and procedure. Do not convert an inconclusive estimate into a positive recommendation using its sign alone.

Close the decision record

Retain the allocation, plan, raw and reconciled outcomes, hashes, full result, environment versions, execution deviations and signed decision rationale. Separate the prespecified result from exploratory explanations. ROI or market totals require a justified conversion and uncertainty on the same horizon; GeoStep does not perform that conversion automatically.

Run the example from the repository root:

python examples/retained_parallel_workflow.py --output /tmp/geostep-trial-example

The example writes an allocation CSV, allocation metadata, an analysis plan, synthetic outcomes, a result JSON and a manifest of file hashes and dependency versions. Reproduction uses the saved allocation. See the 0.5 release verification for executed checks.