The One Chart That Replaced Our 40-Page Targeting Document

In the early days of building ML pipelines, there was a sacred artifact in every data science team: the targeting document. Forty pages of PDF. Flowcharts. Assumptions. Edge cases. Legal sign-off. Marketing alignment. A living, breathing, slightly stale document that nobody read after week three but everybody referenced in meetings.

We had one too. And for years, it worked.

Then we built a single chart. Not a slide. Not a dashboard. A chart — a 2D scatter with a decision boundary, color-coded by feature attribution, annotated with natural-language explanations generated by an LLM. And the 40-page document went into the archive. Not because we discarded it, but because the chart did its job better, faster, and with less cognitive load.

This post is about why that happened, what the chart actually encodes, and how we built it. It is also, in a quiet way, a meditation on how AI has changed the unit of communication in technical work.

The Problem with the 40-Page Document

The targeting document was a linear artifact. You had to read it in order. Page 12 explained a decision made on page 4, which depended on an assumption on page 1, which was contested in a comment on page 7 that was resolved on page 22.

Linear documents are great for recording decisions. They are mediocre for communicating them.

Human working memory holds roughly 4 to 7 chunks of information at once. A 40-page document asks you to hold 40 pages worth of context while you read page 40. You don't. You skim. You lose the thread. You ask questions in meetings that were already answered on page 18.

The document was also static. The model changed. The features changed. The business rules changed. The document lagged by a week, then two, then a month. By the time it was updated, it had become a historical artifact, not a communication tool.

And it was unqueryable. You couldn't ask it: "What happens to our precision-recall tradeoff if we drop feature X?" You had to re-derive it by hand, or wait for the data scientist to run a notebook.

None of this is a failure of the document. It is a failure of the medium. Documents are for storage. Communication needs interfaces.

What the Chart Actually Is

The chart is a scatter plot with three layers of information:

Layer 1: The model's decisions. Each point is a customer (or a sample from our training set). X-axis is predicted probability. Y-axis is actual outcome. The decision boundary is drawn as a smooth curve. Points above the curve are true positives. Points below are false positives.

Layer 2: Feature attribution. Each point is colored by the top contributing feature for that prediction. We use SHAP values — specifically, we take the feature with the largest absolute SHAP value for each sample and map it to a categorical color. So the chart simultaneously shows what the model decided and why (in a first-order approximation).

Layer 3: Natural language annotations. For clusters of points that share the same top feature, an LLM generates a one-sentence explanation. "These customers are being targeted because they opened 3+ emails in the last 7 days." That sentence is rendered as a floating label near the cluster.

The chart is interactive. Hover over a point: you see the customer's feature vector, the model's prediction, the top-5 SHAP values, and the LLM-generated explanation. Click a cluster: you get a mini-report with statistics, a confusion-matrix slice, and a draft email to the customer.

One chart. Three layers of information. Fully queryable. Always in sync with the model.

It took us about three weeks to build. The 40-page document took a senior data scientist about five days to write and two hours to update.

Why This Is an AI Story

The chart is not just a visualization. It is an AI-generated interface. Every layer is produced by a model:

• The predictions come from a gradient-boosted tree (XGBoost).

• The attributions come from SHAP, which is itself a model-agnostic explanation method grounded in cooperative game theory.

• The natural-language annotations come from a transformer LLM, prompted with the feature names, the SHAP values, and a style guide.

Three models, three abstractions, one picture.

This is the pattern I keep seeing in modern data work. We are not just using AI to do tasks. We are using AI to generate the artifacts through which we understand the tasks. The chart is not a picture of the model. The chart is a mediation layer between the model and the humans who need to trust it.

And that mediation layer is itself produced by AI.

This is a small but real shift in the epistemology of data science. We used to go from data to model to human understanding through a chain of human-authored artifacts: notebooks, reports, documents, slides. Now the chain is: data to model to AI-generated artifact to human understanding. The human-authored artifacts have moved up the chain. Humans author the specification of the artifact, and AI generates the artifact itself.

The Math, Briefly

For a model $f(x)$, the SHAP value $\phi_j(x)$ for feature $j$ satisfies the local accuracy condition:

$$\ phi_0 + \sum_{j=1}^{M} \phi_j(x) = f(x)$$

For each sample $x$, we compute the vector $\phi(x) = (\phi_1, \dots, \phi_M)$ and select:

$$j^ *(x) = \arg\max_j |\phi_j(x)|$$

This $j^(x)$ is the "top feature" for that sample. We then cluster samples by $j^(x)$, fit a small t-SNE or UMAP embedding within each cluster for layout, and pass the cluster statistics to the LLM:

$$\ text{annotation} = LLM(\text{feature_name}, \text{mean_shap}, \text{cluster_size}, \text{style_guide})$$

The LLM outputs a single sentence. We render it near the cluster centroid.

Nothing here is novel. But the composition is doing a lot of work. Each layer is interpretable on its own. Together, they form a single communicative unit that no 40-page document could match for cognitive efficiency.

What Broke, and What We Learned

Not everything worked on the first pass.

The LLM's annotations were initially too confident. It would say "these customers are targeted because they have high LTV" when the actual SHAP attribution was to a feature called email_open_rate_7d. We had to constrain the prompt: "Use only the feature names provided. Do not infer business meaning. Write in the second person. One sentence. No exclamation marks."

The color coding was initially too fine-grained. We had 47 features. 47 colors. The eye couldn't track them. We reduced to the top 8 contributing features and grouped the rest as "other."

The interactivity was initially too deep. Hovering gave you 12 fields. We trimmed to 5.

Each fix was small. The chart got better. The 40-page document stayed the same. That asymmetry is the whole story.

A Bar Chart of Our Own Productivity

Before the chart:

After the chart:

The one-time cost was higher. The recurring cost was dramatically lower. And the quality of communication went up, not down, because the chart is in sync with the model and queryable in real time.

Time per stakeholder question (minutes)

Before:  ████████████████████  20
After:   ██  0.1 (5 seconds)

Time per doc update (minutes)

Before:  ████████████████████  120
After:   ██  10

The Quiet Epistemology Shift

Here is the part that keeps me thinking.

The 40-page document was a human-authored artifact that represented the model. The chart is an AI-generated artifact that mediates between the model and the human.

In the first case, the human is in the loop of production. In the second case, the human is in the loop of specification.

That's a meaningful difference. It means the human's job shifts from writing the explanation to deciding what the explanation should look like, what it should say, and what it should not say. The AI does the rendering. The human does the design.

This is, I think, where AI-assisted technical work is heading. Not replacing the human. Moving the human up the abstraction ladder. The human stops being a writer of artifacts and becomes a designer of artifact-generation systems.

The 40-page document is not wrong. It is just one level lower in the abstraction hierarchy than the chart. And the chart is possible because we have good models, good explanation methods, and good LLMs. All three had to mature before this artifact class became practical.

Practical Takeaways

If you have a targeting document, a model card, a feature spec, or a stakeholder report that is more than 10 pages, ask yourself:

1. Is this a storage artifact or a communication artifact? If it's for communication, it should be interactive, in sync, and queryable.

2. Can an LLM generate the prose layer? If the prose is templated, an LLM can produce it. If it's creative, you still need a human. Most technical prose is in the middle.

3. Can you compress the information hierarchy? A 40-page document is 40 levels of a linear hierarchy. A chart with three layers of information is 3 levels. Fewer levels = less cognitive load = better communication.

4. Build the artifact, not the document. The artifact is the unit of communication. The document is the unit of storage. You need both. But for meetings, for trust, for decision-making, you need the artifact.

Final Note

The 40-page document is in the archive. We still keep it. It has the legal sign-offs, the assumption logs, the historical decisions. It is the record.

The chart is the communication. It is what stakeholders look at. It is what we update when the model changes. It is what we use to answer questions in meetings.

Both are necessary. But only one of them is the thing that actually helps humans understand the model.

And that one is now generated by AI.

The 40-page document is not dead. It has been superseded for its original purpose. And that is, I think, the right outcome. We wanted better communication. We got it. And the cost was a three-week build and a few prompt-engineering iterations.

For a 40-page document that used to take a senior data scientist five days to write, that is a reasonable trade.

The chart is on the wall. The document is in the archive. And the team is, quietly, working faster.