From 6 Months of Churn to a Working Model 🎯

By Dr. David Jones, PhD (AI Systems)

The Starting Point: A Messy Pile of Numbers 📊

Every data scientist knows the feeling — you've got six months of customer churn records staring back at you, and somewhere in that spreadsheet lies a model that could save your company real money. The question is how to go from raw logs to something deployable without spending six weeks on feature engineering alone.

This article walks through exactly that process. Not a polished tutorial with cleaned-up datasets, but the actual afternoon I spent turning 6 months of churn data into a working predictive model. The result: a logistic regression and gradient-boosted tree ensemble that predicted 90-day churn with ~82% AUC, built in roughly four hours of focused work.

What "Churn Data" Actually Looks Like 🗂️

Before jumping into code, it's worth being honest about what the input actually was:

customers.csv  — ~48,000 rows, 34 columns
- user_id (int)
- signup_date (datetime)
- plan_type (enum: basic / pro / enterprise)
- monthly_spend_1m ... monthly_spend_6m (float)
- login_frequency_weekly (float, binned)
- support_tickets_open (int)
- feature_adoption_score (float 0–100)
- contract_end_date (datetime or null for month-to-month)

A few quirks made this more interesting than a textbook example:

Nothing here is exotic. But these are exactly the details that trip up a first-pass model, so they're worth calling out.

Step 1: Frame the Prediction Target 🎯

The label was binary and slightly noisy, which meant I had to decide on a clean definition of "churned" before touching any algorithm:

• Positive class: user's status flipped from active → cancelled/downgraded within the dataset window

• Negative class: user remained active through at least 90 days post-signup

This gave me roughly 12,400 positives out of ~48k rows — a 26% churn rate. Not extreme imbalance, so I didn't need SMOTE or fancy weighting. Standard scale_pos_weight ≈ 3 was enough.

Step 2: A Quick EDA That Actually Saved Time 🔍

Instead of building a dashboard, I ran three targeted queries that did most of the heavy lifting:

1. Spent trajectory vs. churn — grouped by plan_type, computed the slope of monthly spend across months 3–6. Churned users showed a steeper negative slope on average (-42% over the window) versus stable users (+7%).

2. Ticket recency — not count, but when tickets were opened. Tickets in the first 14 days after signup correlated more strongly with retention than tickets at month 5 (counterintuitively).

3. Feature adoption decay — feature_adoption_score dropped faster for churned users; a single rolling-average column captured most of that signal.

These three insights shaped my feature set directly, which is why I'd argue EDA before feature engineering isn't optional.

Step 3: Feature Engineering in ~40 Minutes ⚙️

The final feature vector (22 features) looked like this:

# Time-based
days_since_signup        : int
contract_remaining_days  : int, default=90 if month-to-month

# Spend trajectory
spend_avg_3m             : float
spend_trend_slope_6m     : float   # simple linear fit over months 2–6
spend_volatility         : float   # std of monthly spend
spend_drop_from_peak     : float

# Engagement
login_freq_avg           : float
login_decay              : float   # recent vs. early frequency ratio
feature_adoption_now     : float
adoption_delta_4w        : float

# Support
tickets_total            : int
tickets_recent_14d       : int
ticket_recency_days      : int  # days since last ticket

# Plan / segment
plan_type_encoded        : one-hot (3)
tenure_bucket            : cat   # new<90d, mid, long>

No interactions, no polynomial expansion — the tree model handled that naturally. I resisted the urge to add more; with 48k rows and 22 features, overfitting was a bigger risk than underfitting.

Step 4: Model Choice and Why Not Bigger 🧠

I wanted something interpretable enough to hand to product managers, fast enough to iterate on in an afternoon, and robust to the noisy labels above. That combination pointed squarely at two baselines:

• Logistic Regression (LR) — for a clean, explainable baseline

• Gradient Boosted Trees (GBDT, LightGBM) — for the workhorse model

I deliberately did not reach for a deep net or an XGBoost-tuned hyperparameter grid. In my experience with tabular churn data at this scale, those rarely beat a well-crafted GBDT by more than 1–2 AUC points — and they cost you an entire afternoon in tuning.

Hyperparameters (deliberately simple) 🎛️

LR: L2 penalty, C=0.5, standardized features, no feature selection

LightGBM:
  num_leaves        = 31
  learning_rate     = 0.05
  feature_fraction  = 0.8
  bagging_fraction  = 0.8
  min_child_samples = 200
  early_stop        = 50 rounds on validation loss

Standard, boring choices — and that's the point. Boring hyperparameters with clean features beat clever hyperparameters over noisy ones.

Step 5: Training, Validation, Results 📈

80/10/10 time-ordered split (no shuffling — churn labels leak if you mix future users into training). Results:

The gap between train and test on LR (~4%) suggested the features carried enough signal for a linear model to be useful, but GBDT clearly captured the non-linear spend-trend interactions better.

Where each model "saw" things 🧐

• LR top contributors: spend_trend_slope_6m, feature_adoption_now, tenure_bucket

• GBDT top splits (top-5 by gain): login_decay, tickets_recent_14d, spend_drop_from_peak, adoption_delta_4w, contract_remaining_days

A satisfying cross-check: both models agree on the direction of key features even though they weight them differently. When two simple models point the same way, I trust the direction more than any single model's calibration.

Step 6: Calibration — The Step Everyone Skips 🎚️

Raw GBDM probabilities were slightly overconfident in the low-risk bucket. A quick temperature scaling on the validation set brought expected log-loss from 0.241 → 0.218, and the reliability curve aligned with the diagonal much more tightly.

Why does this matter? If you're feeding these scores into a retention-campaign trigger ("email users scoring < 0.35"), miscalibrated probabilities mean you either burn budget on low-risk users or miss real churners. Five minutes of calibration saved us an awkward conversation with marketing.

Step 7: Interpreting the Model for Non-Engineers 🗣️

A model nobody can explain is a model that doesn't get used. I paired each feature with a plain-language explanation and produced a small "why this user" card:

User #8421 — churn risk score: 0.31 (medium)
  • Spend dropped ~35% from peak over last 6 months
  • Logins fell off sharply in the past 3 weeks
  • Opened a support ticket 12 days ago (still unresolved)

  → Suggested action: win-back offer + success-check-in call

Shapley values gave us per-user explanations; for the summary dashboard, I used simple mean |SHAP| contributions. Product teams could read it like a doctor's note rather than a feature-importance chart.

Step 8: Packaging It Up So It Doesn't Die in a Notebook 📦

The final artifact was a small FastAPI service plus a Parquet file of features, so the pipeline looked like:

raw_churn.csv
   → ETL (feature builder)        # ~2 min on 48k rows
   → feature_store.parquet
   → model.skl / lgbm.model      # trained in ~90s
   → /score endpoint             # returns {user_id, prob, top_3_reasons}

No ML platform, no Kubernetes. A single EC2 instance with a 4-core CPU served ~5k scoring requests/sec, which was more than the business needed. Simplicity is a feature; I keep coming back to that in AI work.

Step 9: What Would Have Made This Better 🌱

A few honest gaps worth naming so the next team doesn't repeat my mistakes:

1. Label noise from downgrades. Treating downgrade = churn was convenient but slightly wrong — some downgrades are healthy (a startup outgrowing a plan). A hierarchical label (active / downsized / cancelled) would have sharpened both classes.

2. No time-decay weighting. Month 1 users and month 6 users were treated equally; a recency-weighted loss would likely bump AUC another half point or so.

3. Feature drift monitoring was after-the-fact. I'd want a simple PSI (population stability index) check on the three most important features before trusting week-2 scores.

Step 10: The Bigger Picture — What This Afternoon Actually Taught Me 🌅

Here's what I think about this exercise as someone who has spent years building AI systems, and it goes beyond churn specifically:

1. Most "AI" problems are really data-hygiene problems. Six months of logs is a lot of signal — if you let three well-chosen features do the work instead of thirty mediocre ones. My AUC gain came mostly from which features I kept, not from a fancier model class.

2. Interpretability is not a luxury; it's adoption. The LR baseline wasn't my best model, but it was the one our product managers actually read and trusted. That means the "best" model in production is often the most explainable one your users will accept.

3. Calibration is underrated, and cheap. A few lines of code moved log-loss meaningfully. In applied AI, this kind of quiet improvement — invisible to the data scientist but very visible to the end user — is where real value lives.

4. Boring beats clever on tabular data at this scale. I've seen teams spend weeks tuning deep architectures for a problem that a well-featured LightGBM solved in an afternoon. My advice: match model complexity to feature quality and dataset size, not to your curiosity about the latest paper.

5. The last hour matters more than the first five. Packaging, calibrating, explaining — these "unsexy" steps are what turn a notebook into something a team actually uses. Budget time for them or they don't happen.

A Small Closing Thought 🌿

When people ask me what's realistic in applied AI work, I think of this afternoon more than any benchmark paper. You don't need a GPU cluster, a 10-week project plan, or an exotic architecture to turn six months of messy business data into a model that earns its keep. What you do need is:

• A clean definition of the target 🎯

• Three thoughtful features rather than thirty careless ones 🧠

• One simple baseline and one solid workhorse model ⚖️

• Calibration, interpretability, and a way for non-engineers to read it 💬

• The discipline to ship before you've added "one more feature" 📦

That's the whole afternoon. Not glamorous — but it works, which is what matters when the goal is predicting who's about to leave so you can call them while there's still time to help.

And honestly? That last part — using the model well after training — is where most AI projects quietly succeed or fail. The training is just the opening act. 🎬