I Gave an LLM Our Raw Clickstream and Got a Full Strategy in 20 Minutes
I Gave an LLM Our Raw Clickstream and Got a Full Strategy in 20 Minutes
By Dr. Elias Thornwood, PhD (AI)
We all know the ritual. Marketing leads request a "deep-dive" analysis of the last quarter's user behavior. The data engineers pull 500 million rows of clickstream logs. The analysts spend three weeks building dashboards. The PMs stare at the dashboards, feel "smart" for a day, and then the strategy meeting degenerates into a debate about which chart to trust.
I wanted to compress that pipeline. Not to replace humans, but to test a hypothesis: if I feed a large language model our raw clickstream — not a tidy CSV, not a summary — can it produce a strategy document a senior product lead would actually sign off on? The answer, after several failed runs, was a resounding yes. And it took 20 minutes.
The Experiment Setup
Before I describe the output, let me be precise about what "raw" means. We were talking about a 1.2 GB JSONL file of session-level events:
{"ts": 1717123456, "user_id": "u_88231", "page": "/pricing", "action": "click", "elem": "btn_start_trial", "dwell_ms": 4120}No feature engineering. No pre-computed funnels. No "user segments" defined up front. Just events, timestamps, and DOM paths. I loaded this into a context window using a sliding-window retrieval scheme — a simple BM25 index over serialized page paths, so the model could pull relevant slices of the stream on demand rather than trying to hold all 40M events in memory at once.
The prompt was deceptively short:
"Here is our raw clickstream for Q1. Analyze it. Identify the 3 highest-leverage opportunities. For each, give: (1) the behavioral evidence, (2) a specific hypothesis, (3) an experiment design with a primary metric and sample size, (4) the expected impact on quarterly revenue. Be skeptical of your own findings. Flag at least two places where the data could be misleading."
That last instruction — be skeptical — turned out to matter more than any of the analytical asks.
What the Model Actually Found
The first two opportunities were almost embarrassingly obvious. The model noticed that 62% of users who clicked "Start Trial" on /pricing never completed the email field, and that the drop-off correlated strongly with session length < 90 seconds. Classic above-the-fold attention problem. The experiment it proposed — a progressive-disclosure form that defers email capture until after the user selects a plan tier — was exactly what a good growth PM would have written.
The third finding is where it got interesting.
The model traced a cluster of sessions where users landed on the /docs/api-reference page from a competitor's comparison page (inferred from referrer patterns and subsequent navigation: docs → pricing → 404 on a pricing subpage → docs again). It then cross-referenced dwell times and noticed these users spent 3x longer on the pricing page than users arriving from our own blog. The model's hypothesis: these users were doing a feature-parity check, and our pricing page answered the "what" but not the "why us." It recommended a comparison module — not a generic "why choose us" banner, but a table that directly addressed the three features the competitor's comparison page highlighted.
Here's the key: the model didn't just describe the pattern. It built a causal chain. Landing source → navigation path → dwell time → hypothesis → experiment. That's the shape of a strategy memo, not a data summary.
The Skepticism Clause Paid Off
This is the part I want to emphasize, because it's where LLMs are either genuinely useful or dangerously so.
The model flagged two self-criticisms in its own output:
Survivorship bias in the trial cohort. "Users who complete the email field are not a random sample. They are, by selection, more motivated. The 38% completion rate may overstate true intent. To test this, we should add a passive-completion variant (email captured only on second interaction) to separate genuine interest from form-completion friction."
Referrer attribution is correlational. "The competitor-landing users may not be comparing us to a specific competitor. They could be researchers, students, or analysts building internal tools. The comparison-module experiment should include a control that doesn't reference any specific competitor, to isolate whether it's the act of comparison or the specific comparison driving conversion."
These are the kind of caveats that junior analysts omit because they don't want to muddy the narrative. Senior analysts include them because they know where the analysis can be challenged. The prompt instruction to "be skeptical of your own findings" pulled this behavior out of the model in a way that a generic "analyze this data" prompt did not.
The 20-Minute Breakdown
Phase | Duration | What Happened |
|---|---|---|
Data loading & indexing | 6 min | BM25 index over 40M events |
First full pass | 5 min | Initial pattern identification |
Skeptical re-pass | 4 min | Self-critique, bias flags |
Experiment design | 3 min | Metric, sample size, A/B spec |
Memo formatting | 2 min | Strategy doc, not data dump |
Twenty minutes. Not twenty hours. Not two weeks. Twenty minutes, on a single GPU node, with no custom training, no fine-tuning, no RLHF. Just a good prompt, a good retrieval scheme, and a model with enough working memory to hold a 200-token context of events at a time.
Where It Still Fails
I want to be honest about the failure modes, because if I only report the wins, this reads like a press release.
Memory is the bottleneck. The model can't hold 40M events. It can hold a 200-event window. The quality of the output depends entirely on the quality of the retrieval layer. If the BM25 index misses a relevant slice of the stream, the model never sees it. I ran the same prompt against three different retrieval strategies and got three different strategies out. The model is only as good as its context window.
It's a pattern matcher, not a physicist. The model finds correlations and dresses them up as causal chains. It will tell you "users who do X convert 2.3x more" without checking whether X and conversion share a hidden common cause. The skepticism clause helps, but it doesn't eliminate the problem. You still need a human to ask "what's the confounder?"
It over-indexes on recency. Raw clickstream is time-ordered. The model's attention mechanism weights recent events more heavily. If you feed it a file that starts in January and ends in March, the March patterns dominate. You have to chunk and rotate the file to get a uniform view.
It can't tell you what you didn't log. If you didn't capture scroll depth, the model can't analyze scroll depth. If you didn't capture device type, the model can't segment by device. The output is bounded by the input. Garbage in, strategy-shaped garbage out.
The Strategy Document (Abridged)
For the record, here's the structure of the output document:
Title: Q1 Clickstream Analysis: Three Leverage Points
Executive Summary: (4 sentences. No fluff.)
Finding 1: Trial Form Completion
Evidence: 62% drop-off at email field, correlated with session < 90s
Hypothesis: Above-the-fold form density exceeds user attention budget
Experiment: Progressive disclosure form; primary metric: email completion rate; n=12,000 per arm; 4-week duration
Expected impact: +$180K/quarter at current trial-to-paid conversion
Caveat: Selection bias in completed cohort
Finding 2: Competitor-Parity Users
Evidence: 3x dwell time on /pricing for competitor-referrer sessions
Hypothesis: Feature-parity checkers need a comparison artifact, not a pricing table
Experiment: Comparison module vs. control; primary metric: pricing-page → trial conversion; n=8,000 per arm; 6-week duration
Expected impact: +$95K/quarter
Caveat: Referrer attribution is correlational
Finding 3: Docs-First Users
Evidence: /docs/api-reference is the #1 landing page for 11% of sessions
Hypothesis: Developer-first users need API docs linked from pricing, not the other way around
Experiment: Cross-link module; primary metric: docs → pricing CTR; n=5,000 per arm; 3-week duration
Expected impact: +$40K/quarter
Caveat: May be a small, self-selected cohort
Risk Register: (3 items, each with a mitigation)
Next Steps: (4 actions, each with an owner and a deadline)
That's a strategy document. Not a data summary. Not a "here are 12 charts" deck. A document a VP can put in front of the board and say "here's what we'll do and why."
The Deeper Point
What this experiment actually demonstrates is not that LLMs can do your analysts' jobs. It's that LLMs can compress the loop between raw data and strategic action. The bottleneck in most product organizations is not analytical capacity — it's the latency between "we have data" and "we know what to do with it." That latency is where money goes to die.
Twenty minutes versus two weeks. That's not a 10x improvement. That's a 60x improvement in decision latency. And in a market where your competitor is running the same experiment with a different prompt, 60x is the difference between shipping the feature and explaining why you didn't.
You don't need to trust the LLM's output. You need to trust your own judgment to read it, challenge it, and decide. The LLM is not your analyst. It's your analyst's draft. And a good draft, in 20 minutes, is worth more than a perfect report in two weeks.
Dr. Elias Thornwood holds a PhD in Artificial Intelligence from MIT and has spent 12 years building LLM systems for enterprise analytics. He is the author of "Context Windows and Causal Chains: A Practitioner's Guide to LLM-Driven Strategy."
