I Let AI Decide Which Campaigns to Kill — And It Was Right Every Time
I Let AI Decide Which Campaigns to Kill — And It Was Right Every Time
By Dr. Elena Voss
We all know the feeling. You're staring at a dashboard, five underperforming campaigns blinking red, and you're trying to decide which one to cut. Your gut says Campaign A, but last month your gut was wrong, and you spent two weeks writing a post-mortem.
What if you stopped guessing?
I did. For six months, I handed my campaign-kill decisions to a machine learning model. Not a fancy "AI strategist" — just a gradient-boosted tree trained on eighteen months of historical performance data. And it was right every time.
Before you roll your eyes, let me walk you through what "right" actually means here, why a simple model beat my experience, and what this says about how we should be thinking about decision-making in marketing.
The Problem With Human Intuition
I've been in performance marketing for nine years. I've managed budgets from $50,000 to $12 million. I can look at a campaign's CTR, CPA, and ROAS and form an opinion. I should be able to, at least.
But here's the uncomfortable truth: I'm pattern-matching on a handful of recent campaigns, not the full dataset. My brain remembers the campaign that tanked in Q3 last year and the one that overperformed in Q1. I weight recent events more heavily. I get anchored to the creative I liked. I hesitate to kill a campaign that a client is attached to, even when the numbers say it's bleeding money.
A model doesn't have these problems. It sees every data point equally. It doesn't get attached to a creative. It doesn't get tired at 4:30 on a Friday. It just computes.
What the Model Actually Looks Like
I want to be specific, because "AI" has become a magic word that people throw around without understanding what's actually doing the work.
I fed the model the following features for each campaign:
Daily spend, impressions, clicks, conversions (7, 14, 30-day windows)
CTR, CPC, CPA, ROAS, conversion rate (rolling averages)
Campaign age in days
Number of creative variants currently running
Audience segment, channel, geo, device split
Day of week, seasonality index
Historical performance of the same campaign 30 days prior
The target variable was simple: did the campaign's ROAS improve or decline in the following 14 days? I used a gradient-boosted decision tree with 200 trees, depth 6, learning rate 0.05. No neural networks. No transformers. No "large language model" doing the thinking. A tree ensemble.
I trained on 18 months of data across roughly 4,000 campaigns. I validated on the most recent two months.
The model's task: given the current state of a campaign, predict the probability that killing it (i.e., reallocating that budget to a better-performing campaign) would improve overall portfolio ROAS by more than 2%.
The Results
Over six months, I had about 120 campaigns that were candidates for killing — campaigns in the bottom quartile of performance. I let the model decide which ones to actually kill. I compared its decisions against what I would have done if I'd made the calls myself.
The model was right 117 out of 120 times.
"Right" means: after the campaign was killed and budget was reallocated, the overall portfolio ROAS improved by 2% or more. In the three cases where the model was "wrong," it was by a narrow margin — the ROAS improvement was 1.4%, 1.1%, and 0.8%. Close, but not quite the threshold.
My human decisions, for comparison: I would have gotten it right about 75% of the time, based on a rough self-assessment. I killed campaigns that should have been kept (creative was in a warm-up period, and I killed it too early). I kept campaigns that should have been killed (I liked the creative, or the client was attached to it, or I was waiting for "one more week" that never materialized).
The model didn't wait for one more week. It computed the expected value of keeping versus killing, and it moved on.
Why the Model Beat My Gut
Three things stood out to me:
1. The model caught the slow bleed. Campaigns that were declining gradually — ROAS dropping 0.1% per day for three weeks — are hard for humans to notice in real time. You're looking at the numbers and they look "okay." The model was tracking the slope, not just the level. It saw the trajectory and made the call before the damage was obvious.
2. The model was consistent. I'd kill a campaign with a CPA of $45 in January and keep one with a CPA of $48 in June, because "June is a better month for our audience." The model knew that. It had the seasonality data. It adjusted. I sometimes did, sometimes didn't, depending on how tired I was.
3. The model wasn't attached to the creative. This one surprised me. I'll admit it: I like pretty ads. A campaign with beautiful creative but mediocre performance gets a pass from me. The model didn't care how the ad looked. It looked at the numbers.
The Three Times It Was "Wrong"
Let's be honest about the three misses.
Case 1: The model recommended killing a prospecting campaign with a CPA of $52. The threshold for our prospecting campaigns is $55. So it should have been kept. But the model saw that the audience was saturating — CPMs were rising 15% week over week — and predicted the CPA would cross $55 within 10 days. I checked back two weeks later. CPA was $57. The model was technically right about the trajectory, but I'd set the threshold at $55, and it was $52 at the time of the decision. So it was a judgment call the model made that I wouldn't have made. Is that "wrong"? Maybe. It was a prediction, not a certainty.
Case 2: The model recommended killing a retargeting campaign. The creative had been updated two days prior. The model's data window was 14 days, so it had mostly pre-update data. The new creative was performing 20% better. The model didn't know about the creative update because I hadn't fed that signal into the model. So it was a data gap, not a model error. I fixed this by adding a "creative update date" feature.
Case 3: A campaign was underperforming because of a one-day tracking bug. The model saw three days of bad data and recommended killing it. The bug was fixed on day four. The campaign recovered. The model had only three days of data. I gave it more context.
None of these were the model being wrong. They were the model working with the data I gave it. The model was a calculator. I was the one deciding what to put into the calculator.
What This Says About Decision-Making
Here's the thing I keep coming back to: the model wasn't smarter than me. It was more consistent than me. It had more data than me. It wasn't biased by my preferences.
That's not a small distinction. In marketing, we make hundreds of small decisions per week. Which campaign to scale, which to pause, which creative to test, which audience to add. Most of these are low-stakes. But they add up. A 5% error rate on 200 decisions per month means 10 bad decisions per month. Each one leaks budget. Each one delays learning.
The model doesn't eliminate the need for judgment. I still had to define the features. I still had to set the threshold (2% ROAS improvement). I still had to decide which campaigns were in scope. I still had to interpret the model's output and communicate it to clients. The model didn't replace me. It handled the part of the job that was mechanical and data-rich, and it freed me to focus on the part that required context, relationships, and creative judgment.
A Practical Framework
If you want to try this, here's a simple starting point:
Collect your data. Pull 12-18 months of campaign-level performance data. You need at least 500 campaigns for a decent model.
Define your decision. What are you deciding? Which campaign to kill? Which to scale? Which creative to retire? Make it binary.
Pick your features. Spend, CTR, CPC, CPA, ROAS, conversion rate, campaign age, channel, audience, seasonality. Don't overcomplicate it. Start with 10-15 features.
Train a simple model. Gradient-boosted tree. Random forest. Even logistic regression if your data is clean. You don't need a neural network.
Validate on recent data. Use the most recent 4-6 weeks as your test set. You want to see how it performs on campaigns it hasn't seen.
Run it in parallel. For the first month or two, make the model's recommendation but also make your own. Compare. Build trust. Adjust the features if the model is consistently missing something you can see.
Hand off the decision. Once you trust it, let it decide. You review the output. You handle the exceptions. You communicate with clients.
You don't need a data science team. You need a spreadsheet, a Python notebook, and a willingness to let the numbers drive the decision.
The Bigger Picture
I think this is going to become the norm. Not because AI is magic, but because the volume of decisions we make in marketing is outpacing our ability to make them well with raw intuition. We have more campaigns, more channels, more audiences, more data points than any human can hold in working memory. The model handles the volume. We handle the context.
My clients ask me, "What's your strategy?" and I say, "My strategy is to let the data make the mechanical decisions and spend my time on the strategic ones." The creative direction, the brand voice, the audience insight, the client relationship — those are still human work. But the "which of these five campaigns to cut" decision? That's a computation. And the machine is better at it.
I used to be proud of my gut. I still am. But I'm prouder of the 117 out of 120.
And if your gut says a campaign should be killed, check the data. And if the data says it should be kept, keep it. The machine doesn't care how you feel about the creative. And that's exactly what you need it to do.
