How to measure the ROI of AI in marketing: control groups, incrementality and return per dollar
Resources · Guide
Measuring the ROI of AI in marketing means comparing customers who got the actions the AI chose with a control group that did not, and putting that difference next to what the actions cost. What counts is extra margin, not sends or opens.
- ProposalSuggested by the AI
- ApprovalYour team, with caps
- Treated90% of the audience
The measurement
- Control10%, no action
- PurchasesFrequency and basket
- CostEvery reward recorded
- Week 0Approve and split the control
- Week 6First read
- ThenScale, adjust or stop
In production
Autopilot agents work on the platform Qurable already runs in production.
- ~150 msEngine at paymentApproved rules only, no AI in the path
- ~3,000Requests per secondSustained in production
- 18MEnd usersIn brands that run on Qurable
- ISO 27001SecurityCurrent certification
Key takeaways
- The ROI of AI in marketing is the extra margin measured against a control group.
- Sends, opens and campaigns launched measure activity: they don't prove the AI sold anything.
- Every action needs its own control group, its cost recorded and weeks before it can be read.
The difference
Measure AI activity or the sales it adds.
Both give you a number. Only one separates what the action sold from what the customer would have bought anyway.
| Activity metrics | Control group | |
|---|---|---|
| What counts | Sends, opens, clicks and campaigns launched | Sales and margin of treated customers versus the control group |
| What it credits to the AI | Every purchase that follows the message | Only the difference between the two groups |
| Discounts | Every redemption counts as a win | Their cost is subtracted, including what went to people who would have bought anyway |
| A very busy agent | More actions look like more results | More actions prove nothing if they don't sell more |
| When you know | The next day | Once customers have had time to buy again |
| What decision it supports | Keep going because "things are moving" | Move budget to what earns margin |
How to measure it
Five steps to measure an AI action.
They work for a discount, cashback, a message or a journey, whether a person or an AI agent proposed it.
1. Pick the business metric
What the action should move: CLV, churn, CAC, frequency, basket or margin. Set it before launch and don't change it midway.
2. Split off the control group
Before launch, a random part of the audience doesn't get the action; for example, 10%. It stays fixed for as long as the action runs.
3. Wait for the purchase cycle
Customers need time to buy again. For frequent purchases, a first read at six weeks is a good starting point.
4. Measure the difference
Compare the metric in both groups and turn it into margin. That difference is what the action sold.
5. Calculate return per dollar
Divide the incremental margin by the cost of the action. Above 1, it pays for itself; below 1, adjust it or stop it.
Common mistakes
The mistakes that inflate the result.
Most returns that don't hold up later share the same cause: activity was measured, margin was given away or the control group wasn't comparable.
Counting sends and opens
An opened message is not a sale. An agent can send twice as many messages and sell the same.
Counting redemptions as wins
A redeemed coupon may belong to a customer who would have bought anyway. Redemption is a cost, not proof.
Crediting every later purchase
A customer buying after the message doesn't mean they bought because of it. Only the control group tells them apart.
Discounts for people who'd buy anyway
It's the most common waste and the hardest to see without a control group. It eats margin and adds no sales.
Comparing best customers with everyone else
If the AI picks people who already buy a lot and they're compared with the rest, the result looks great even if the action did nothing.
Never testing "give nothing"
Often the best move for a customer is no reward at all. If you never test it, you never see it.
No control group
Comparing with last month adds seasonality, prices and competitors to the AI's result.
Contaminated groups
Control customers who get the same offer through another campaign, another channel or a shared code. The gap shrinks or turns into noise.
Samples that are too small
With few customers per group, two groups with no action at all already differ. Size the control before launch.
Reading too early
The first weeks mix purchases pulled forward with new ones. Wait for the purchase cycle.
Changing the metric midway
If the metric changes halfway, there's always one that went up.
Rotating the control every week
If the control keeps changing, everyone ends up treated and there's nothing left to compare against.
How to read the result
Each business metric reads differently.
The same control group works for all of them. What changes is what you compare and when you can read it.
| Metric | How to read it against the control |
|---|---|
| What to compare | The value each customer generated in the period, treated versus control, net of reward cost. |
| When to read it | It's the slowest metric. In the meantime, frequency and second purchase show where it's heading. |
| What to compare | The share of each group still active at the end of the window, with the same definition of an active customer. |
| Watch out | A customer who bought once with a discount and never came back is not retained. |
| What to compare | The cost of the action divided by the extra new customers versus control, not by all new customers. |
| Watch out | Dividing by the total mixes in people who would have joined anyway and makes CAC look lower than it is. |
| What to compare | Margin per customer in both groups, after discounts, cashback and points. |
| Watch out | Sales can go up while margin goes down. It happens when the reward goes to people who would have bought anyway. |
| What to compare | Purchases per customer and average basket in both groups. |
| What they're for | They move first: they tell you before CLV does whether the action is on track. |
| Formula | Incremental margin ÷ cost of the action. Above 1, the action pays for itself. |
| Illustrative example | USD 18,000 in extra margin on USD 7,500 of cost: USD 2.40 for every dollar spent. |
What to ask the vendor
Six questions for an AI marketing vendor.
They separate AI that proves what it sells from AI that shows activity. If an answer is vague, ask to see it on a real case.
Measurement
What do you measure the result against?
A randomly chosen control group, split off before launch, one for each action.
- Red flag
- Opens, clicks or a comparison with last month.
Return
How do you calculate the return?
Incremental margin versus control, minus the cost of every reward, including what went to people who would have bought anyway.
- Red flag
- Total sales from the customers who got the action.
Control
How big is the control group and when is it read?
A size set before launch, fixed for as long as the action runs, and a first read that matches the purchase cycle.
- Red flag
- Results after a few days, or a control group that changes every week.
Audit
Can I audit every number?
Every reward with its cost and its customer, in a record finance can reconcile.
- Red flag
- A dashboard with the total and no detail behind it.
Spend
Who approves the budget?
Your team approves the goal, the budget and the caps. Anything outside that comes back for approval.
- Red flag
- An agent that spends with no cap.
Decision
Can the AI decide to give nothing?
Yes, and it measures it: not rewarding someone who would have bought anyway is margin saved.
- Red flag
- AI that always gives something away.
With Qurable
Every action with its control group, and the return on the ledger.
Qurable Autopilot reads CLV, churn, CAC, frequency, basket and margin, and proposes a strategy with a budget and an expected return. Your team approves the outcome to move, the budget and the caps. Every action is measured the way this guide describes.
A control group per action
Every approved action has its own permanent control group, for example 10% of the audience. The first read comes at six weeks.
See Qurable AutopilotYour team approves
The outcome to move, the budget and the caps. The agents work within those limits.
Auditable return
Every reward, with its cost, is recorded in a double-entry ledger.
See the points ledgerWeekly Briefing
Every Monday: money at risk, money misspent and money not collected, plus the results of what you approved. "Proven return" only shows where there is a control group.
Questions
What people ask about the ROI of AI in marketing.
For marketing, CRM, finance and leadership teams deciding how much to invest in AI.
Compare the customers who got the actions the AI chose with a randomly chosen control group that did not. Turn the difference into margin and divide it by the cost of the action: that is the return per dollar.
It's what an action sold on top of what customers would have bought anyway. It's measured as the difference between the group that gets the action and a similar control group that doesn't, in the same period.
It's a part of the audience, picked at random before launch, that doesn't get the action. It can be a small share, for example 10%, as long as it's large enough that the difference isn't noise. It stays fixed for as long as the action runs.
As long as it takes customers to buy again. For frequent purchases, a first read at six weeks is a good starting point; for less frequent ones, you need longer. Reading earlier mixes purchases pulled forward with new ones.
Almost. Return per dollar is incremental margin ÷ cost. ROI is (incremental margin − cost) ÷ cost. A return of USD 2.40 per dollar equals a 140% ROI.
Because they measure activity, not sales. An agent can send more messages and get more opens without anyone buying more. Only a comparison against a control group shows whether the action sold.
It's a control group where some customers got the action another way: another campaign, another channel or a shared code. The gap between the groups shrinks and the real return stays hidden.
The logic is the same: control group, difference and cost. This guide measures the actions AI decides across all of marketing, with or without a loyalty program. For the program itself, including points liability, see the guide "Measure program results".
Every approved action has its own permanent control group, for example 10% of the audience, and the first read comes at six weeks. The cost is recorded in a double-entry ledger and the Weekly Briefing shows the result every Monday. Proven return only shows where there is a control group.
Last reviewed
Further reading
Keep reading about AI and measurement.
The guides and pages that show each piece in detail.
- GuideMeasure program results
Incrementality, cost and ROI of a loyalty program.
- GuideAI in loyalty programs
The engine that decides at payment and the agents that run the strategy.
- ConceptQurable Autopilot
AI agents that propose, execute and measure against a control group.
- ConceptPromotion engine
Simulation, A/B tests and budget per campaign.
- ModulePoints ledger
Every reward with its cost, recorded in double entry.
Want to measure your next AI action against a control group?
In a short meeting we show you how Qurable Autopilot splits off the control group for each action, records the cost on the ledger and shows the result in the Weekly Briefing.