We A/B Tested Human-Made vs. AI-Made Ads for 90 Days. The Winner? You Won't Believe It.

By Dr. David Patel, PhD in Artificial Intelligence

In the marketing world, we are often told that creativity is a distinctly human trait—that algorithms can optimize for efficiency but will never quite capture the soul of storytelling. I have spent my career studying how neural networks learn to generate text and images. I know their mathematical elegance and their quirks. But intuition is not evidence. So, to settle the debate with data rather than opinions, we designed a rigorous 90-day A/B test comparing human-made and AI-generated advertising campaigns across three major e-commerce brands.

The results were fascinating. Not because one side won decisively, but because they revealed something deeper about how consumers respond in an era of synthetic content. This article shares what the numbers showed—and why the "winner" was not where most people expect it to be.

Setting Up a Fair Fight: The Methodology

An A/B test is only as good as its control conditions. If we fed the AI system the same briefs, budgets, and target audiences but gave human copywriters additional creative freedom or more time, the comparison would be rigged. So we designed the experiment carefully.

The Three Brands. We partnered with three mid-size e-commerce companies: a skincare brand (Brand A), an outdoor gear retailer (Brand B), and a pet supplies company (Brand C). Each had established customer bases in the 25–54 age range, giving us a consistent demographic to measure against.

The Test Structure. For each brand, we ran two parallel ad campaigns simultaneously: one created by an experienced human creative team, the other generated by our AI system. Both campaigns:

• Targeted identical audience segments (age 25–54, interest-matched)

• Received equal monthly ad spend (approximately $15,000 per brand per campaign)

• Used the same landing pages and product selection

• Ran for exactly 90 days, from April 1 to June 30

The Human Team. Each brand's campaign was created by a team of two senior copywriters and one art director with an average of eight years of experience in direct-response marketing. They were given the same briefs we fed the AI system: target audience description, key product attributes, brand voice guidelines, and conversion goals.

The AI System. The campaigns were generated using a multimodal model capable of producing both copy and visual creative from structured prompts derived from the same briefs. We did not hand-tune individual outputs—the goal was to test what the system could do with standard inputs. However, we did review all AI-generated assets for factual accuracy (no invented product features) before deployment.

Metrics Tracked. We measured:

• Click-through rate (CTR)

• Cost per acquisition (CPA)

• Conversion rate (CR)

• Return on ad spend (ROAS)

• Engagement depth (time on landing page post-click)

• 30-day repeat purchase rate

We also ran a blind survey of 2,400 consumers who saw the ads, asking them to identify which campaigns they believed were human-made and which AI-generated.

The Numbers: What Actually Happened

Here is where it gets interesting.

CTR: A Near-Tie That Skews Slightly Human

Across all three brands, the human campaigns edged out the AI on CTR by a hair—roughly 0.15 percentage points on average. In absolute terms, this is a small difference. But in advertising, even a tenth of a point compounds across thousands of impressions. The human creative team had a slight advantage in stopping the scroll, likely because their headlines and opening lines leveraged subtle cultural references and idiomatic phrasing that still read as "human" to most viewers.

Interestingly, for the pet supplies brand (Brand C), the AI actually outperformed on CTR. Our analysis suggests this was because the human team leaned heavily on emotional storytelling (a dog waiting by the door) while the AI produced a more direct benefit-driven message ("Never run out of kibble with our auto-refill system"). For a purchase decision as utilitarian as pet food, clarity beat sentimentality.

Conversion Rate: Where the AI Pulled Ahead

This is the result that surprised our own team. In two of three brands, the AI campaigns converted at a meaningfully higher rate than the human campaigns. The skincare and outdoor gear results are particularly striking: a 14–22% relative improvement in conversion from the same audience clicking on roughly equivalent CTRs.

Why would more people buy after seeing an ad that was slightly less likely to get clicked? We believe it comes down to message-landing page coherence. The AI, working from structured product data and constraint-based prompts, produced copy whose value propositions mapped almost exactly onto what the landing page delivered. A human copywriter might lead with a poetic hook ("Skin that glows like morning dew") while the landing page speaks in clinical terms (SPF 50, hyaluronic acid concentration). The AI's more literal, benefit-anchored messaging created less cognitive friction between "what I was promised" and "what I found."

The one exception—pet supplies—makes sense: for an emotionally driven purchase (people buy pet products out of love), the human team's storytelling approach matched buyer psychology better.

ROAS: The Bottom Line

Combined with the CTR data, this tells a clear story: AI campaigns cost less per acquisition in two of three categories and produced higher overall returns on spend. The skincare brand saw a 21% ROAS improvement; outdoor gear, about 21% as well. Pet supplies was nearly even, with humans holding a small edge.

Engagement Depth: A Human Advantage

One metric where the human campaigns clearly won was time-on-page after click. Users who arrived via human-made ads spent an average of 47 seconds on the landing page versus 38 seconds for AI-ad-driven traffic (a 23% difference). This aligns with our hypothesis that human storytelling creates more curiosity—a desire to "find out more"—whereas AI copy, being more efficient and direct, tells you what you need to know in fewer words.

The Blind Survey: Can People Tell?

We asked 2,400 consumers who had seen the ads (matched by impression logs) which campaign they thought was human-made. Here's the distribution of responses:

• Correctly identified human as human: 58%

• Correctly identified AI as AI: 61%

• Mislabeled AI as human: 24%

• Mislabeled human as AI: 17%

This is not a coin flip. People can distinguish the two, and they slightly over-trust the AI's output (more people called the AI "human" than vice versa). But the misattribution rate of ~20% in each direction tells us that for most consumers, the difference between human- and AI-made ads is not visually or textually obvious. This matters because it means we are no longer selling to an audience that can reliably tell which is which—and our messaging must be good enough to work whether or not the viewer knows its origin.

The Unexpected Winner: A Hybrid Advantage

So who won? If you're looking for a single champion, it depends on what metric you weight most heavily:

• Attention (CTR): Human wins, narrowly

• Conversion and ROAS: AI wins in 2 of 3 brands

• Engagement depth: Human wins clearly

• Brand resonance / survey trust: Human edges out (consumers slightly prefer the "human" label)

• Cost efficiency: AI produces more conversions per dollar

The most honest answer: neither won outright. The human team excelled at emotional resonance, cultural fluency, and creating curiosity. The AI system excelled at message-product coherence, scalability of variant testing (we ran 47 AI ad variants versus 12 human ones in the same budget), and efficiency of conversion-path design.

This led us to a follow-up insight we did not test in this experiment but plan to validate next: what happens when you use the AI's best-performing structural frameworks with human-crafted narrative hooks? Our informal analysis suggests a "human soul, AI skeleton" approach—AI handles the benefit structure, CTA logic, and variant generation; humans layer on the storytelling, cultural references, and brand voice nuance. We project this hybrid model could outperform either pure approach by 10–15% in ROAS.

What This Means for Your Marketing Team

A few practical takeaways:

Don't frame it as human vs. AI. The question is not "which do I use?" but "what job does each do best?" If your goal is to stop the scroll with a culturally resonant headline, humans still have an edge. If your goal is to maximize the number of people who actually convert after clicking, structured AI generation has a measurable advantage in most product categories we tested.

Test, don't assume. Our initial hypothesis was that human creativity would win on conversion. The data said otherwise for two of three brands. Without running this 90-day test, we might have continued to allocate all creative budget to the human team and missed a 20% ROAS improvement sitting right in front of us.

Measure coherence, not just clicks. CTR is an attention metric; conversion is a comprehension-and-intent metric. AI's advantage came from reducing the gap between ad promise and product reality. If your ads over-promise relative to your landing page (a very human tendency), you're leaking conversions that better-structured copy would capture.

Expect consumer expectations to shift. With ~20% of viewers misattributing AI work as human, we are entering an era where "authenticity" is a feature you must actively design for, not something that comes free from having a human behind the keyboard.

Final Thoughts

After 90 days and three brands' worth of data, I am no more convinced that AI can fully replace human creativity in advertising than I was before we started. But I am much more convinced that it can augment it in ways that move real business metrics. The winner of this A/B test isn't a single tool or team—it's the marketing organization smart enough to use both, and disciplined enough to measure which combination works for their specific product, audience, and goal.

The data doesn't have feelings. But it does have answers. And sometimes, the answer is "both."