The 300% Subject Line Experiment

By Dr. Elara Voss

AI Inspired

I've spent six years building language models. I've read papers on attention mechanisms, written loss functions that would make a statistician weep, and debugged transformer layers at 2 AM. And yet, when I decided to test what a small LLM could do with something as mundane as an email subject line, the result genuinely surprised me.

The setup was almost embarrassingly simple. I took a batch of 500 outgoing marketing emails — product updates, event invitations, newsletter digests — and asked a mid-tier open-source model to generate three subject line candidates for each. I then A/B tested the AI-generated lines against my own hand-written ones over a 6-week window. The baseline open rate was 4.2%. The AI-generated cohort landed at 16.8%.

That's not a 30% lift. That's a 300% lift.

What the Model Was Actually Doing

Here's the thing that kept me up at night (not from worry — from curiosity). The model wasn't writing "cleverer" lines. It was writing more honest ones.

My hand-written subject lines followed the standard marketing template:

"🔥 Don't Miss Out! Exclusive Offer Inside!"
"Big News You Need to See Today"
"Your One-Stop Solution for Everything"

The model's output looked like this:

"Q3 pricing change takes effect Oct 15"
"Here's what changed in v2.4"
"We moved the onboarding call to Thursday"

No exclamation marks. No urgency theater. Just... information. And people opened them like they were reading a note from a colleague.

Let me formalize what I think is happening. If we model the recipient's open probability as a function of perceived relevance $r$ and perceived cost of opening $c$:

$$P( \text{open}) = \sigma\left(\frac{r}{c} - \beta\right)$$

Hand-written marketing lines inflate $r$ artificially (exclamation marks, superlatives) but also inflate $c$ (you have to parse the hype, you suspect a sales pitch, you're mentally preparing for a commitment). The AI lines kept $r$ modest but honest, and kept $c$ low because the subject line read like a factual preview. The ratio $r/c$ went up, and so did the open rate.

The Prompt That Mattered

I tried four prompt structures. The difference was dramatic.

Prompt A (generic):

Write a catchy email subject line for: [email body]

Result: 8.1% open rate. Better than my baseline, but still in "marketing" register.

Prompt B (persona-based):

You are a busy engineer who writes short, factual emails.
Write a subject line for: [email body]

Result: 14.3%. Noticeably better. The persona constraint pulled the model toward brevity.

Prompt C (constraint-heavy):

Write a subject line under 45 characters.
No exclamation marks. No all-caps.
Must include a specific number, date, or name
if the email body contains one.

Result: 18.7%. This is where it got interesting.

Prompt D (constraint + negative examples):

Write a subject line under 45 characters.
No exclamation marks, no all-caps, no "don't miss out."
If the email body contains a number, date, or name,
include exactly one of them in the subject line.
Avoid: "exclusive," "ultimate," "game-changer," "secret."

Result: 21.4%. The negative examples did more than the positive instructions. Telling the model what not to do was more effective than telling it what to do.

This tracks with what we see in the literature on constrained generation. You're not just steering the distribution; you're pruning the high-probability-but-generic branches of the output space.

Where the AI Still Loses

I want to be honest about the failures. The 300% number is a headline. The full picture has some rough edges.

Tone calibration. The model nailed factual, informational emails. But when I asked it to write a subject line for a team celebration announcement, it produced:

"Team celebration event scheduled for Friday"

Technically correct. Emotionally flat. A human would write:

"Everyone's invited to Friday's pizza night 🍕"

The model optimized for information density and lost the social signal.

Audience specificity. My emails went to a mixed audience: engineers, managers, a few non-technical stakeholders. The model treated them all the same. A human writer would tweak the register per audience. The AI needed explicit audience context in the prompt to differentiate, and even then, it was a coarse approximation.

Brand voice consistency. Over 500 emails, the AI's output had a subtle homogeneity. Slightly different phrasing, but the same underlying structure. A human writer drifts, experiments, occasionally breaks their own rules. The model is consistent in a way that can feel mechanical if you're reading all 500 in sequence.

The Math That Actually Matters

Let's look at the numbers more carefully.

The unsubscribe rate actually dropped with the AI lines. That's the metric that should worry any marketer, and it's the one that tells me the lift wasn't just novelty. People weren't opening because they were curious about the clever trick. They were opening because the subject line told them exactly what they'd get, and it matched their interest.

If we think of this in terms of expected value:

$$E [\text{benefit}] = P(\text{open}) \times P(\text{click}|\text{open}) \times V(\text{conversion}) - C(\text{attention})$$

The AI improved both $P(\text{open})$ and $P(\text{click}|\text{open})$ while reducing $C(\text{attention})$ because the subject line was a truer preview. The compounding effect is what turns a 4x open rate improvement into a roughly 5x overall engagement improvement.

What This Says About LLMs in General

I'm an AI researcher, so I'm a little wary of "look what the model can do!" narratives. But this experiment taught me something I hadn't internalized: the model's weakness in creative, emotional, social-registered writing is also its strength in precise, informational, low-hype writing.

We spend a lot of time trying to make LLMs more creative, more empathetic, more "human." And that matters for marketing copy, for creative writing, for customer support. But for the 70% of professional communication that is just information transfer, the model is already good enough. Maybe better than most humans, because humans bring bias, mood, and ego into even the most factual email.

The model doesn't. It just reads the body and writes the shortest honest preview. And that, it turns out, is what people want to read.

The Reproducibility Note

If you want to replicate this:

1. Take your last 100 sent emails.

2. For each, generate 3 subject line candidates with a prompt like Prompt D above.

3. A/B test over 4-6 weeks, rotating between your baseline and the AI lines.

4. Track open rate, click-through, and unsubscribe rate separately.

5. Watch for audience segments where the AI underperforms (usually the most relationship-driven segments).

You don't need a fancy model. A 7B parameter open-source model with a good prompt gets you most of the way. The prompt engineering matters more than the model size for this specific task.

The 300% number will look different for your audience, your niche, and your baseline. If your baseline is already 15% because you write well, the lift might be 30%. If your baseline is 2% because you write in corporate-speak, the lift might be 400%. The direction is consistent. The magnitude is yours.

But the direction is the point. And the direction is: write less, write truer, and let the information do the selling. The model just made that principle executable at scale.