🤖 The Two-Hour Churn Bot That Humiliated Our Data Team

By Dr. Elara Patel, Ph.D. in Artificial Intelligence

Everyone has a data team. Everyone thinks their data team is the bottleneck for every predictive problem they run into. Mine was. For three years at my company, asking "which customers are about to leave?" meant opening Jira, writing a ticket, waiting two weeks for a model, and then arguing with the analysts about feature engineering choices that were mostly vibes.

Last Tuesday I got tired of it. I opened a laptop, borrowed our staging database credentials, and by Thursday morning — roughly two hours of focused work across three sessions — I had a churn prediction bot in production that outperformed the data team's quarterly model on every metric we care about. Not by a little. By a lot. And no one was mad about it, which is honestly the more impressive part.

This article walks through exactly what I built, why it worked so well, and what it says about how teams should be thinking about "data work" in an era where a competent engineer plus a good LLM can do a surprising amount of analytics on their own.

📊 The Problem (And Why It's Easier Than You Think)

Churn prediction sounds intimidating but is actually one of the friendliest supervised learning problems in production ML. You have:

• Labels: did the customer cancel? (binary, clean, unambiguous)

• Features: everything we already log — usage frequency, ticket count, support interactions, payment events, last login, plan tier, seat count

• No cold-start problem for existing customers; you're predicting over a known population

The data team's model was a gradient-boosted tree (GBDT) trained on ~40 hand-engineered features. Decent architecture, but the feature set had drifted out of sync with how our product actually changed over two years. Nobody updated it because it "worked well enough." That's the real enemy in enterprise ML: not underfitting or overfitting — stale assumptions.

🧠 What I Actually Built

The bot is a small Python service (~400 lines total, plus config). Architecture looks like this:

┌─────────────┐     └───────────────┐
│  Postgres   │────▶│ Feature Layer │
│ (raw events)│      │ (SQL views +  │
└─────────────┘      │  pandas agg)  │
                     └───────┬───────┘
                             ▼
                     ┌───────────────────┐
                     │ LightGBM Model   │◀── retrained weekly via cron
                     │ (60 features,    │     from a single SQL query
                     │  ~2.1M rows)     │
                     └───────┬──────────┘
                             ▼
                     ┌───────────────────┐
                     │  FastAPI endpoint │──▶ POST /risk?customer_id=X
                     └───────┬───────────┘
                             ▼
                     ┌───────────────────┐
                     │ Slack bot +       │
                     │ Daily digest      │
                     │ (top 50 at-risk) │
                     └───────────────────┘

A few design choices that mattered more than the model itself:

1. Features were generated, not engineered. I wrote one SQL view per signal family (activity, support, billing, tenure) and let a LLM help me draft aggregations for 60 features — rolling windows at 7/30/90 days, ratios, z-scores within cohort. The data team's model used ~40 features; I had 60 but the real win was that mine tracked current product behavior because they were derived from raw events rather than frozen in a notebook from 2023.

2. Model choice: LightGBM, not a neural net. For tabular data with n ≈ 2M and d ≈ 60, gradient boosting is still the workhorse. I spent zero time tuning hyperparameters — default num_leaves=31, learning_rate=0.05, 5-fold CV was enough to beat our team's model. Sometimes the second-best algorithm with fresh features beats the best algorithm with stale ones.

3. The "bot" part is just an API + a digest. A FastAPI endpoint that takes a customer ID and returns {risk_score, top_3_drivers} using SHAP values computed at inference time (cheap for GBDT). A cron job runs daily, pulls the top 50 by risk, and posts to a Slack channel with one-line explanations:

📉 Acme Corp — churn risk 0.82

• API calls down 63% vs. 30-day avg

• 4 support tickets in last week (was 1)

• Seat count dropped 12 → 7

That last detail is what made sales engineers actually use it. A raw score means nothing to a human; drivers make the bot feel like a colleague, not a spreadsheet.

📈 The Results (With Numbers So You Can Dispute Them)

Evaluated on the same holdout window (Q3 last year), comparing my model vs. the data team's production model:

Precision at the top decile is the number that matters operationally — it tells you how many "at-risk" customers a sales rep actually has to call. Going from 31% to 58% precision means half as many wasted calls, and the team felt that within a week.

A simple bar chart of recall across risk tiers:

Recall by tier (holdout)
Tier 0-20%   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓       (mine, 81%)
             ▓▓▓▓▓▓▓▓▓▓                  (theirs, 64%)

Tier 20-40%  ▓▓▓▓▓▓▓▓▓▓▓▓                (mine, 71%)
             ▓▓▓▓▓▓▓▓                    (theirs, 58%)

Tier 40-60%  ▓▓▓▓▓▓                      (mine, 52%)
             ▓▓▓▓                        (theirs, 41%)

The gap widens at the top of the distribution — which is exactly where you want it to widen.

🧪 Why It Worked (And What I Got Wrong)

What worked:

• Freshness over cleverness. The biggest lever was deriving features from current raw events rather than reusing a 2-year-old feature set. If your product changed, your assumptions about what predicts churn changed too.

• SHAP for explainability as a UX tool. Engineers and sales reps don't read probability curves. They read "API calls down 63%." The bot's value is in the narrative, not just the number.

• Small, bounded scope. I didn't try to predict LTV or upsell readiness or NPS. One question, answered well, shipped in two hours.

What I got wrong (and fixed):

• First version used a single 90-day window for all features. Customers with seasonal usage patterns (B2B construction firms, for instance) looked like churners every January. Adding multi-window features (7/30/90 day) plus cohort-normalized z-scores fixed most of the false positives.

• I under-weighted billing signals in v1. Add payment_delay_days and invoice_disputes_30d and precision at top-5% jumped another 4 points.

What I did not do:

• No feature store, no model registry, no A/B harness. It's a bot in a cron job, not a MLOps platform. And that's the point — for many teams, the "right" amount of infrastructure is just enough to be useful, not "enterprise-grade."

🤔 What This Says About Data Teams (With Respect)

I want to be careful: this isn't a takedown of my data team. They run 140+ production models, own the warehouse, and handle the gnarly problems I wouldn't know where to start with — multi-entity churn attribution, survival analysis for cohorts, causal inference on retention programs. The bot does one narrow job well and fast.

But it does expose a real pattern: in many orgs, the data team becomes the bottleneck not because they're slow or wrong, but because the cost of asking them is higher than the cost of building a v1 yourself. Tickets take days to triage. Backlogs are backlogs. And by the time your ticket ships, product has moved on.

The interesting implication isn't "fire your data team." It's: reallocate their time toward problems that actually need their depth, and let engineers own the 80% of predictive needs that can be solved with a good LLM, a GBDT, and a weekend.

In my case: the data team took over the bot's training pipeline after I handed them the notebook, and they built a proper feature store on top of it in about three weeks. We're now 12 months past that Tuesday, and the model has been retrained weekly with zero downtime. The original two-hour prototype is still doing most of the inference load because... honestly, replacing it would be more work than leaving it running.

📐 A Small Formalization (Because I'm a Ph.D., Not an Apologist)

For those who like the notation: let customer $c$ have feature vector $\mathbf{x}_c \in \mathbb{R}^{60}$, and let $y_c \in {0,1}$ be the churn indicator over a 30-day forward window. We fit:

$$f^ * = \arg\min_f \sum_{i=1}^{N} L(f(\mathbf{x}_i), y_i) + \lambda |f|_2^2$$

where $L$ is the standard logistic loss and $\lambda$ regularizes against overfitting. The bot outputs:

$$\ hat{p}_c = f^*(\mathbf{x}_c) = \frac{1}{1+e^{-z_c}}$$

and a driver set $\mathcal{D}c = {(k, \phi_k): \text{top-3 by } |\text{SHAP}{c,k}|}$ where $\phi_k$ is the SHAP contribution of feature $k$. The daily digest ranks customers by $\hat{p}_c$ and renders the top-$\lceil N/100 \rceil$ with their driver narratives.

That's the whole bot, mathematically speaking. The engineering work was in keeping it alive: monitoring drift (PSI on feature distributions), retraining triggers (AUC drop > 3% vs. last stable run), and graceful degradation (cache last good scores if training fails).

🛠️ Reproducing This (If You're Tempted)

You don't need my exact stack. The minimum viable version is:

1. One SQL view that aggregates your event log into per-customer features. Start with 20–30 columns. Don't over-engineer.

2. A GBDT (LightGBM, XGBoost, or even sklearn's GradientBoostingClassifier). 5-fold CV. Default hyperparameters. You'll be surprised how far it gets.

3. An endpoint that takes a customer ID and returns score + top drivers. SHAP for tree models is nearly free.

4. A digest channel. Slack, email, Teams — whatever your team already checks daily. Format: customer name, risk score, 2–3 one-line drivers. No graphs in the chat message; keep it skimmable.

Total: ~300 lines of Python, a cron job, and an API key. Two hours if you're fast, four if you're not. You do not need a feature store, model registry, or CI/CD pipeline to start. Add those when the bot earns it — i.e., when someone other than you is depending on it.

🎯 The Bigger Lesson

The article title says I "outperformed our data team." A more honest version: I did in two hours what would have taken them four weeks, and then they made my two-hour thing better. That's the new shape of work in applied AI — fast prototypes by people close to the problem, polished and scaled by people with depth. Neither side is redundant. The bottleneck was never talent or tooling; it was coordination overhead — tickets, meetings, context-switching.

The bot didn't replace my data team. It replaced the waiting, and in a lot of companies, that's where most of the value actually lived all along.

If your product changes faster than your model retraining cycle, build a v1 yourself this weekend. You don't need permission. You just need a database read key, a notebook, and an afternoon. 🚀