The Impact Analysis tab answers the one question that matters: is the outreach working? This article explains how Engage compares the messaged group against the held-back control group, which metrics it reports across which time horizons, the views available, and how to read the numbers without over-interpreting them.
Contents
Treatment vs control
The Impact Analysis tab compares the treatment group (A, messaged) against the control group (B, held back). The header reads: "Treatment vs control outcomes, extra visits and value uplift per horizon."
The comparison relies on the global A/B split (configured once for the Members audience): the held-back control group is what makes the measurement possible. If the global A/B split is turned off, there is no control group and no impact measurement. See also: Creating and editing trigger rules (A/B test).
What Impact Analysis measures
- Extra gym visits: visits per messaged member vs control at 1 week, 2 weeks, and 1 month after outreach. More visits now is a leading indicator of retention.
- Cancellations averted: the share of each cohort that cancelled, messaged vs control, at 3, 6, 9, and 12 months. The difference is the number of members kept by reaching out.
- How impact builds over time: charts of visits per member and cancellation rate over time. The gap between the messaged and control lines is what messaging added.
Impact Analysis: messaged vs control across the visit and cancellation horizons, plus impact-over-time charts; figures are marked preliminary until the measurement window closes.
Views
Impact Analysis can be viewed three ways:
| View | What it shows |
|---|---|
| Per-run | Impact for a single execution run. |
| Per-trigger | Impact aggregated for one trigger rule across its runs. |
| All-time | Impact aggregated across all runs. |
Results are filterable by category and date range. On a newly activated studio the tab shows "No execution runs yet" until data accumulates.
Reading the numbers
Impact data needs enough runs to be statistically meaningful. Newly activated studios will see empty or low-data states at first; this is expected. The longer horizons (cancellations prevented at 180+ days) naturally take months to fill in.
Two practical notes on reading the numbers:
- Small studios accumulate significance slowly. A handful of runs on a 300-member studio will not produce a reliable lift figure; judge trends over weeks, not days.
- The A/B design controls for seasonality and other external effects, since both groups experience them equally. That is why the impact measurement depends on the global A/B split being on: with no control group there is nothing to compare against.
See also: Running triggers (schedule, preview and execution log).