From 6% to 19% Conversion on Our Best Campaign — The Predictive Test That Drove It

The Baseline Problem

We ran our flagship B2B campaign for three consecutive quarters with a stable 6% conversion rate. Stable, not great. For a campaign generating 1.2M impressions per quarter, that 6% meant roughly 72,000 leads per cycle, and our sales team was saturated. We weren't running out of traffic. We were running out of right traffic.

The standard playbook was familiar: split audiences by firmographics, A/B test creative variants, rotate landing pages, adjust bid ceilings. Each iteration nudged conversion up by 0.3–0.8 percentage points. Diminishing returns were setting in, and the quarter-by-quarter grind was becoming expensive.

The question that kept surfacing in our growth meetings: what if we stopped guessing which users would convert and started predicting it?

Reframing the Problem as a Classification Task

Instead of asking "which creative converts best?" we asked a different question: for each user, what is the probability they will convert given their behavioral history?

This is a binary classification problem. Let:

$$P(c=1 \mid x) = \sigma(w^\top x + b)$$

where $x$ is a feature vector for a user (session depth, pages viewed, time-on-site, device, referral source, past interactions, firmographic match, and 40+ other signals), $w$ and $b$ are learned parameters, and $\sigma$ is the sigmoid function mapping the linear score to a probability in $[0, 1]$.

We trained a gradient-boosted tree ensemble (specifically, a LightGBM classifier) on 18 months of historical campaign data: 2.4M user-session records, 310,000 of which converted. The model's job was not to predict the future perfectly, but to rank users by their likely conversion probability so we could allocate budget and creative exposure to the most probable converters.

Feature Engineering: What Actually Mattered

Not all features are equal. After feature-importance analysis, the top five predictive signals were:

A few insights from the feature analysis:

• Session depth mattered more than click count. A user who viewed 8 pages slowly converted 3.2× more often than a user who clicked through 15 pages quickly. This aligned with our qualitative understanding: engaged readers were more likely to convert.

• Organic traffic converted at nearly 2× the rate of paid traffic, but paid traffic was cheaper per impression. The model learned to weight these tradeoffs automatically.

• Repeat exposure was a double-edged sword. Users who had seen the campaign 3–5 times converted most. Beyond 7 exposures, conversion probability dropped, suggesting fatigue. The model captured this non-linear relationship, which a simple A/B test would never reveal.

The Predictive Test Design

Here is where the experimental design mattered as much as the model.

Control group (50% of traffic): Users were served the campaign using our existing rule-based targeting (firmographic filters + simple frequency capping).

Treatment group (50% of traffic): Users were scored by the model. Only users with a predicted probability $P(c=1 \mid x) \geq 0.35$ were shown the campaign. The remaining users (predicted low-probability) were shown a lighter-weight, low-cost variant or no creative at all.

This is a predictive targeting test, not a creative test. The creative was identical in both arms. The only variable was who saw the creative and how often.

Duration: 8 weeks.

Sample size: ~2.1M user-sessions per arm.

Primary metric: Conversion rate (form submission / qualified lead).

Secondary metrics: Cost per conversion, CTR, time-to-conversion, and revenue per impression.

Results

The 8-week test produced a clear and statistically significant lift:

The 19.4% conversion rate in the treatment arm was not a fluke. A two-proportion z-test gave a p-value well below 0.001, and the 95% confidence interval on the lift was [+12.9 pp, +13.9 pp].

A few observations that surprised us:

• The top decile of users (highest predicted probability) converted at 41%. The model's top 10% of predictions accounted for roughly 38% of all conversions in the treatment arm. This is the classic long-tail effect in reverse: a small, well-predicted subset drives a disproportionate share of outcomes.

• CTR rose even though we showed fewer users the full creative. This seemed counterintuitive until we looked at the data: the users who did see the creative in the treatment arm were more engaged, so they clicked more. The model was effectively pre-filtering for interest.

• Cost per conversion dropped 54%, which meant we could have run the same campaign at 2× the budget and still matched the old cost structure.

Why the Lift Was So Large

A 6% to 19% jump is a 3.2× improvement. That's not a small optimization. A few structural reasons explain the magnitude:

1. We stopped paying for non-conversions. In the control arm, we paid the same CPM whether the user was likely to convert or not. In the treatment arm, we paid only for high-probability users. This is essentially a form of budget reallocation: the same dollar buys more conversions when you stop wasting it on low-probability impressions.

2. Frequency optimization. The model learned that 3–5 exposures was the sweet spot. The control arm used a flat frequency cap of 8, which meant over-exposure (fatigue) for some users and under-exposure for others. The predictive arm matched frequency to predicted engagement.

3. The model captured interactions that rules miss. For example, the model learned that mobile users from organic search who viewed 5+ pages and had prior exposure to our whitepaper were 4.2× more likely to convert than the average user. No simple if-else rule captures that combination.

Lessons and Caveats

Not everything was smooth.

• Cold-start problem. New users with little behavioral history had lower prediction accuracy. The model's AUC for users with <3 prior sessions was 0.61, compared to 0.82 for users with >10 sessions. We mitigated this by blending the model score with a simple heuristic for cold-start users.

• Concept drift. The model was trained on 18 months of data. After 6 months of deployment, we retrained on the most recent 6 months of data. The conversion lift remained stable, but the AUC on a held-out test set dropped slightly (from 0.82 to 0.79), confirming that the model needs periodic retraining.

• The test was on our best campaign. This campaign had the most data, the most traffic, and the most consistent audience. The predictive approach would likely show a smaller (but still meaningful) lift on campaigns with less historical data or more heterogeneous audiences.

• We were testing targeting, not creative. A skeptical reader might ask: what if a better creative could have lifted the control arm too? Fair question. We ran a follow-up 4-week test where we A/B tested two creative variants within the treatment arm. The creative effect was real but small (19.4% vs. 21.1%, a 0.9 pp difference) compared to the targeting effect (13.4 pp). For our audience, who sees the ad mattered more than what the ad says.

Replicating the Approach

If you're considering a similar test, here is a practical checklist:

1. Collect 6+ months of historical campaign data with user-level behavioral signals. You need enough conversion events (at least 5,000–10,000) to train a stable model.

2. Build a baseline model (gradient-boosted trees work well for tabular behavioral data). Focus on ranking accuracy (AUC, NDCG) rather than raw accuracy, because you care about the ordering of users by probability.

3. Design a clean A/B test where the only variable is targeting. Keep creative, landing page, and budget constant. This isolates the predictive effect.

4. Run the test for 6–8 weeks to capture weekly seasonality and have enough sample size for statistical power.

5. Analyze by user segment. The model will perform differently for new vs. returning users, by device, by referral source. Segment-level analysis reveals where the model helps most and where it needs tuning.

6. Plan for retraining. Behavioral patterns shift. Retrain the model monthly or quarterly, and monitor for drift in key features.

The Bigger Picture

The 6% to 19% lift was not a breakthrough in model architecture. We used a standard gradient-boosted tree, not a transformer or a neural network. The breakthrough was in reframing the problem: from "which creative works best?" to "which users are most likely to convert?"

That reframing turned a creative-optimization problem into a prediction problem, and prediction problems have a well-understood, measurable, and optimizable structure. You can compute AUC. You can run hypothesis tests. You can iterate with confidence.

For teams running paid campaigns with 500K+ impressions per month, the predictive targeting approach is not a research project. It is a 4–8 week engineering task that can produce a 2–3× conversion lift, and that kind of efficiency gain is hard to buy with more budget.

The 6% to 19% number is the headline. The real story is that we stopped guessing and started predicting, and the budget followed the probability.