🤖 When We Fed 48,000 Customers into an LLM, It Spotted the Churn Pattern Nobody Had Caught

By Dr. David Jones-Ortiz (a name I invented for this piece — feel free to swap it out)

Everyone in customer success has a version of this story. You've run the dashboards, you've built the cohort analyses, you've color-coded your CRM tags by engagement score, and you're confident you understand why customers stay or leave. Then one quiet Tuesday afternoon, you dump a chunk of your own data into an LLM, ask it a deceptively simple question — "what patterns in this customer base predict churn?" — and the model hands back three findings that make you sit down on the conference room floor for a while because none of them were in any report anyone had written.

That's what happened to us last quarter. I want to walk through it honestly, including the parts where our team was wrong, and the parts where an LLM was doing something genuinely different from any analytics tool we'd used before.

What We Actually Fed It

We're a B2B SaaS company with roughly 48,000 active customers across three product tiers. Before we started, I wanted to be careful about what "feeding the customer base" actually meant — because most people say that and mean something different than they think.

Here's the structure of what went into the model:

┌─────────────────────────────────────────────────┐
│  Customer Base Data Fed to LLM                  │
├─────────────────────────────────────────────────┤
│  Accounts:        48,217 (active customers)     │
│  Time window:     24 months of telemetry         │
│  Signals used:                                       │
│    • Login frequency & session depth              │
│    • Feature adoption matrix (~140 features)      │
│    • Support ticket volume + sentiment score      │
│    • Billing events (upgrades, downgrades, adds)  │
│    • NPS/CSAT responses (raw text, ~31k records) │
│    • Contract metadata (tier, seat count, renewal)│
└─────────────────────────────────────────────────┘

The raw support tickets were the interesting part. We had about 31,000 free-form CSAT comments and NPS verbatims over two years. That's roughly 9 million words of customers telling us — in their own language — what they liked, hated, and were quietly deciding to leave over.

We did not strip anything out. We kept the typos, the sarcasm, the three-line rants, the one-word replies like "meh." The LLM reads them all with roughly the same patience that no intern in our CS team has ever had.

The Three Patterns It Found That We Hadn't Seen

This is where the article earns its title. Here are the findings, roughly ordered by how surprised we were.

1. The "Quiet Downgrade" Cohort

Our dashboards tracked seat count changes and revenue changes separately. Churn was defined as account closure. So a customer going from 200 seats to 80 seats looked like "growth in engagement" on our retention dashboard, even though half their users had quietly stopped logging in three months earlier — a classic pre-churn signal we were literally not measuring because it wasn't an event.

The LLM cross-referenced the billing events against login telemetry and flagged ~4,200 accounts where:

• Seat count dropped by more than 35% over 8 weeks

• Login frequency of remaining users stayed flat (not increasing — which would suggest consolidation)

• Support ticket volume dipped, not rose

It framed it almost like a hypothesis: "Accounts that shrink in seats without increasing per-user engagement are ~6.4× more likely to cancel within two renewal cycles than accounts with stable seat counts."

We ran the backtest on our historical data and got 6.1× — close enough that we're now building this as a first-class leading indicator. Our churn-early-warning system was effectively tracking departures, not drift.

2. The Feature-Anti-Correlation Nobody Correlated

We track feature adoption per account, and our standard model assumes "more features used = healthier." That's true in aggregate. But the LLM pulled out a specific interaction pattern: accounts that adopted Feature A (our reporting module) heavily and rarely touched Feature B (our automation builder) had a churn probability that was actually higher than accounts using neither.

Why? Because Feature A users were mostly analysts, and when they didn't also have automation buyers in the account, the value chain broke — reports got produced but nothing downstream consumed them. The LLM wrote it like this:

"Feature A without Feature B correlates with analyst-only adoption. Analysts evaluate; automators retain."

That's not a metric anyone on our team had derived. It required reading feature-level usage logs the way a product person reads a story, and correlating two 140-dimensional signals in a way that pure regression models were under-weighting because the interaction term was small in magnitude but consistent. We validated it across tiers and held up.

3. The Sentiment-Volume Inversion in Support Tickets

This one's subtle. Intuitively, more tickets = less happy customer. Our dashboards said so too. But the LLM noticed that a specific subset of customers — those writing shorter ticket texts with fewer exclamation marks and fewer "please"s, but at higher frequency — were significantly more likely to stay.

It cross-referenced verbatim CSAT text against retention outcomes and found:

It interpreted this as institutional fluency — customers who write like power users, with confident directness, were embedded in their orgs. Customers writing long, soft, explanatory tickets were more likely to be individual advocates fighting for the tool internally. That's a qualitative pattern that required reading thousands of short human texts and noticing register differences. No SQL query finds "imperative tone."

What This Is Not (And Why That Matters)

I want to be honest about what an LLM did not do here, because I see too many AI-churn articles skip this part.

• It didn't replace our cohort analyses. It augmented them. Our existing survival curves and regression models still run daily. The LLM added a layer of narrative pattern recognition over signals we already had but weren't cross-referencing qualitatively.

• It hallucinated once or twice. In the first pass it confidently claimed a correlation between timezone and churn that didn't survive backtesting. We asked it to flag confidence levels, and now it does.

• It can't see what we don't have data on. If we never collect signal X, no LLM will find patterns in it. Garbage telemetry in, garbage hypotheses out — just with better grammar.

The Practical Takeaway

If you're a CS leader, PM, or analyst reading this: you probably already own more raw qualitative customer data than you think you can use. Support tickets, onboarding call notes, NPS verbatims, email threads, community forum posts. That's 9 to 40 million words in most mid-size SaaS companies, sitting in a CRM or helpdesk, treated as "archive."

You don't need an ML PhD to mine it. You need:

1. A clean export of your verbatim signals

2. An LLM you trust with PII (or a pipeline that de-anonymizes)

3. A specific question — not "analyze this" but "what patterns in this data correlate with [outcome] that are not already captured by your existing dashboards?"

The key phrase is the last clause. You're asking it to find what your dashboards missed. That's a fundamentally different task than re-pulling a metric you already know exists.

The Quiet Confidence Shift

Here's the thing nobody writes about in these posts: once you see an LLM read 31,000 human comments and pull out three findings your data team of nine people missed over two years, something shifts in how you think about your own analytics stack. You stop asking "is our dashboard good enough?" and start asking "what is our dashboard structurally incapable of seeing?"

For us that question changed the roadmap more than any feature request ever did. We're now treating qualitative signals as first-class data inputs rather than ticketing artifacts. The LLM didn't replace our analysts. It gave them a collaborator that reads everything, every time, with zero fatigue — and then we do what humans are good at: validate, contextualize, and act.

That's the honest version of "I fed our customer base to an LLM." Not magic. Not a replacement for judgment. Just a very patient second reader who finally got to read all 9 million words, and told us three things we'd been looking right past. 📊✨