Interpreting an experiment

GeoStep estimates a declared effect of a geographic intervention. Its output supports a decision only when the allocation, implementation and observation process support the causal interpretation.

Agree the decision before launch

Record the intervention, business-as-usual comparator, eligible geos, primary outcome, dates, assignment mechanism, practical effect threshold and analysis procedure. Retain the allocation before collecting test outcomes. Specify which costs, population and time horizon a later financial calculation will use.

Randomisation balances potential outcomes in expectation. It does not guarantee that this particular experiment has similar groups. Review pre-treatment summaries for operational errors. Do not try seeds repeatedly until the result looks attractive unless a constrained assignment procedure was prespecified and is reflected in the statistical analysis.

Read the reported quantity

Lift reports a difference in baseline-normalised changes. An estimate of 0.03 means three percentage points on that scale. It does not automatically mean a three per cent increase over the untreated counterfactual.

DiD reports the average change contrast in outcome units per geo-period. CRT reports a common additive outcome-unit coefficient adjusted for calendar period. Neither becomes total incremental revenue simply by multiplying by 100.

To calculate ROI, first define a justified incremental-volume estimand and its uncertainty, then apply margins and incremental costs on the same horizon. GeoStep does not currently perform this conversion automatically.

Interpret uncertainty candidly

An interval that includes zero is inconclusive as to effect direction at that confidence level. It may still rule out effects large enough to matter. Compare the interval with the prespecified practical threshold, not just zero.

A small p-value is not the probability the intervention worked. A non-significant result does not establish equivalence. The average-effect interval and optional sharp-null randomisation test answer different statistical questions.

Check execution and observation

Compare delivered treatment with assignment. Review contamination, spillover, coincident changes, measurement changes and missing outcomes. Analyse assignment as prespecified; an as-treated analysis requires further justification.

Missing or unidentified results must not enter a decision table as zero effect. Diagnostic checks marked unavailable are not passes. Passing balance checks or finding no placebo difference does not certify causal identification.

Retain a reproducible decision record

Keep the allocation and schedule, raw input hash, analysis options, package and dependency versions, exclusions, result metadata and decision rationale. Separate prespecified analyses from exploratory follow-ups. The software cannot verify an allocation record that was never supplied or retained.

See methodology, getting started and validation scope for the technical contract.