Methodology

GeoStep analyses experiments in which geographic units can be randomly assigned to an intervention or to a rollout sequence. Randomisation supports causal identification when the implemented allocation matches the design and the outcomes support the prespecified treatment contrast. A small p-value alone does not establish this.

Implemented analyses

Analysis

Outcome summary

Estimand

Inference

Lift

Each geo’s test mean divided by its baseline mean, minus one

Equal-geo average contrast in baseline-normalised changes

Stratified Neyman variance and Welch–Satterthwaite t approximation

DiD

Each geo’s test mean minus its baseline mean

Equal-geo average pre/post change contrast in outcome units

Same parallel inference procedure

CRT

One outcome per observed geo-period

Common additive treatment coefficient adjusted for calendar period

OLS, geo-clustered CRV1 covariance and t(G-1) approximation

The 0.5 release evidence defines support separately from code availability. Unstratified analytic DiD and analytic Lift passed their core screens. Stratified DiD and BCa intervals remain experimental and return explicit warnings. Do not switch estimands or interval methods after inspecting results to obtain a preferred conclusion.

CRT and staircase inference remain experimental. CRT is not a mixed-effects model. Stratified CRT inference is unsupported and rejected. A generated staircase schedule is not an allocation of actual geos until those geos have been randomised to sequences.

Define the effect before analysis

Record the eligible geos, assignment mechanism, intervention, outcome, dates, weights and primary hypothesis before observing test outcomes. Distinguish the effect of assignment from the effect of delivered exposure. Non-adherence does not justify switching to an as-treated comparison without further assumptions.

Lift reports a difference of normalised changes, not relative market lift. If both baseline means are 100, the untreated counterfactual is 120 and treatment produces 132, the normalised contrast is 0.12, or 12 percentage points. Relative lift against that counterfactual is 132/120 - 1 = 0.10. These are different effects.

DiD and CRT estimates remain in outcome units. GeoStep does not turn these into percentages or total incremental sales. Such conversions need a defined counterfactual, population weighting, horizon and propagated uncertainty.

Parallel allocation and analysis

SimpleRandomizationDesigner performs complete randomisation with a fixed number of treated geos, floor(G/2). It requires at least four eligible geos. This replaces the previous independent Bernoulli assignments. Seeds therefore produce different allocations from earlier versions.

StratifiedRandomizationDesigner uses K-means on prespecified pre-treatment geo summaries. Summaries must be constant within a geo. Each resulting stratum needs at least four geos, so both arms contain at least two geos. Inadmissible strata are rejected; the implementation does not search for a favourable outcome. For externally specified strata, use randomize_within_strata.

Retain the returned stratum and treatment_probability columns with outcomes. The analyser automatically uses a column called stratum. Specify stratum_col for any other name. Treatment and stratum must remain fixed within a geo.

For stratum s with n_s geos, the weight is w_s = n_s / G. The estimate is the weighted sum of the within-stratum treatment-minus-control means. The variance estimate is the sum of w_s squared times each arm’s sample variance divided by its geo count. Welch–Satterthwaite degrees of freedom combine those variance components. This avoids weighting strata by their different treatment fractions.

The variance is a Neyman-style estimate. Under treatment-effect heterogeneity, the unobservable covariance of potential outcomes prevents an exact finite-sample variance calculation. The t reference and average-effect intervals are approximations. See the bounded validation.

Data and observation windows

Supply one row per geo and date, with finite float64 outcomes. The parallel analysers require all supplied geos to have observations on every observed date in the selected baseline and test windows. A calendar date absent for all geos cannot be detected without an external calendar; verify the intended frequency before analysis. Missing outcomes, duplicates and changing assignments are errors.

Set pre_period_start, pre_period_end, test_period_start and test_period_end explicitly for a frozen analysis. Require pre_start <= pre_end < test_start <= test_end. When optional bounds are omitted, the earliest/latest supplied dates and the instant before test start supply the bounds. Results retain the resolved windows. Outside-window outcomes do not enter the estimate. Positive baseline means are required for normalised Lift.

Keep an allocation record separately from outcomes. A completely absent geo cannot be inferred from an outcome table. Compare the supplied geo set against the frozen allocation before analysing it. Do not silently exclude assigned geos because they have missing or inconvenient outcomes.

Randomisation tests and bootstrap intervals

use_permutation_test=True tests the sharp null of no effect for any geo. It conditions on the observed treatment counts within each stratum. It assumes uniform assignment among those admissible allocations. It is not valid for unrecorded restrictions or outcome-selected re-randomisation.

Small assignment spaces are enumerated when the requested draw budget permits, up to 100,000 allocations. Otherwise independent Monte Carlo allocations are drawn. The p-value includes the observed statistic using (exceedances+1)/(draws+1). Results state the tested null and whether enumeration was exact. The average-effect confidence interval remains a separate approximate calculation; it is not an inversion of the sharp-null test.

use_bca_bootstrap=True resamples geo summaries within each stratum and arm using SciPy’s BCa procedure. Draw multiplicities are preserved. Undefined intervals raise an error. This optional interval does not automatically replace the analytic p-value. Results label both methods separately.

CRT’s optional percentile bootstrap resamples complete geo trajectories within recorded sequences. Every draw receives a distinct cluster identifier. Each sequence needs at least two geos. If any bootstrap draw is not estimable, the procedure reports an error rather than an interval conditional on successful fits. Parallel CRT resampling without a sequence column preserves treatment arms. Rollout bootstrap requires a recorded sequence. These intervals are not an allocation-specific randomisation test.

Staircase and CRT boundaries

StaircaseDesigner.design() builds a synthetic observation schedule. StaircaseDesigner.randomize(geos, seed=...) allocates actual eligible geos to sequences and returns their schedule. Retain this table before collecting data. At least two sequences are required for the randomisation interface.

CRT uses calendar-period indicators and a treatment indicator. The design matrix must have full column rank. If treatment is indistinguishable from calendar time, the model is rejected. Four clusters are a computational minimum, not a claim that four clusters provide adequate inference.

The model assumes a common immediate additive effect. It does not estimate exposure-time response curves, anticipation, heterogeneous rollout effects, spillovers or carryover. Clustered covariance permits within-geo dependence; it does not correct treatment-time confounding or arbitrary between-geo shocks. Plan a different analysis when these assumptions are unsuitable.

Power scenarios

run_power_analysis uses the same parallel allocation and analyser as the final analysis. Select analyzer='lift' or 'did', the same strata and the same test. Input must follow a regular daily or weekly calendar. Durations are calendar weeks in either case. Seeds are independent of the number of workers.

The simulator draws historical windows and injects a proportional intervention into treated test outcomes. This is a conditional historical scenario, not a universal forecast. The injected fraction is not necessarily the estimand reported by the analyser. Existing historical interventions and unrepresentative future conditions limit the interpretation.

Results include valid and failed counts, failure reasons, conditional rejection rates and Wilson Monte Carlo intervals. power is unavailable if any replicate fails. CRT and staircase power are explicitly unsupported by this function.

Diagnostics and evidence

Balance diagnostics describe observed pre-treatment differences. Missing checks cannot pass. They do not prove exchangeability or justify repeatedly choosing seeds until a preferred allocation appears. Any constrained randomisation rule must be prespecified and reflected in inference.

Placebo analysis requires geo identifiers and reports an effect and interval. Its assessment is never an identification pass. Failure to reject a pre-period difference is not evidence that bias is absent.

The previous function named Rosenbaum sensitivity analysis is withdrawn because its calculation was not a valid sensitivity bound. No substitute confounding claim is produced.

Consider contamination, non-adherence, outcome attrition, changes in measurement, coincident interventions and interference before interpreting a result causally. Separate a prespecified primary analysis from exploratory subgroup analyses. The package’s multiple-testing helpers do not provide sequential monitoring or optional-stopping control.

References

Continue with getting started or the advanced analysis boundaries.