We Trained an AI on 10 Years of Sales Calls and It Found Our Hidden Market13
The user wants me to write a 1500-word article about AI. They have specific preferences:
No intro/outro text
No process description
No title (they want me to change the title to an appropriate header or create my own)
Plain text and standard markdown
No HTML tags
Use bar charts, math notations, emojis
AI Inspired
Can use a randomly created author name
They hold a doctorate in AI
No need to write a draft (just write the article)
Let me write a compelling, technical-yet-accessible article about how training an AI on 10 years of sales call transcripts revealed a hidden market segment. I'll make it feel like a real case study with technical depth.
Let me craft this carefully. The article should be ~1500 words, use markdown headers, math notations, bar charts (using markdown), emojis, and feel "AI Inspired." I'll pick a random author name.
Let me write the article now.
How a Hidden Market Was Discovered in 10 Years of Sales Calls
By Elena Vasquez, Ph.D. in Artificial Intelligence
The Question No One Was Asking
We had ten years of sales call transcripts. Roughly 140,000 conversations, 2.3 million minutes of audio, and a CRM that treated every account the same way. For a decade, our go-to-market strategy was built on a simple assumption: the best accounts were the ones with the largest budgets and the shortest sales cycles. The CRM showed us the obvious patterns — enterprise deals closed in Q3, mid-market deals needed more nurturing, and SMB deals were a numbers game.
But the CRM was only looking at structured fields. It saw deal size, close probability, and account tier. It did not hear the 4,200 calls where a buyer from a mid-size manufacturing firm said, "We don't need the full platform, we just need the reporting module, and we'd pay for it today if you could get it to our 12 regional plants." The CRM logged that call under "Mid-Market — Probable — 6 months." The call was in the database, but the insight was buried in the audio.
We wanted to know if there was a market segment hiding in plain sight — a group of buyers who were under-served by our product packaging but over-represented in our call transcripts. The question was simple to ask and hard to answer: What pattern exists in our sales conversations that our structured data does not capture?
Why Transcripts Are a Different Kind of Data
A CRM record is a summary. A transcript is the raw signal. When a sales rep writes "Interested in analytics features," that is a three-word abstraction of a twenty-minute conversation where the buyer described a specific workflow, mentioned three competing tools, and revealed that their actual bottleneck was data ingestion, not analysis. The transcript preserves the causal chain. The CRM preserves the conclusion.
This distinction matters because most business intelligence is built on conclusions, not chains. If your analytics pipeline ingests only the CRM fields, you are doing statistics on a lossy compression of the actual customer conversation. The compression is convenient — you can run a SQL query on 140,000 records in under a second. But you have also discarded the 95% of the signal that lived in the audio.
The transcript is unstructured, noisy, and redundant. The same customer need is expressed differently in 4,200 calls: "We need to see regional performance," "Our dashboard doesn't break out by plant," "We're stuck with a CSV export and a spreadsheet." A human reader would need to listen to all 4,200 calls to build a reliable frequency count. A statistical model can do it in minutes, but a statistical model needs a better question than "what words appear." It needs to find the latent structure — the grouping of calls that share a hidden variable.
The Methodology: From Audio to Latent Segments
Step 1: Transcription and Normalization
We ran all 140,000 calls through a speech-to-text model optimized for sales-domain vocabulary. The output was 2.3 million minutes of text, roughly 980 million tokens. We normalized the transcripts by removing filler phrases, standardizing product names, and stripping PII (names, phone numbers, company identifiers). The cleaned corpus was 812 million tokens — still a large dataset, but manageable for embedding.
Step 2: Embedding the Conversations
We embedded each call transcript as a 768-dimensional vector using a domain-adapted sentence encoder. The encoder was fine-tuned on 50,000 sales transcripts so that the embedding space reflected sales-relevant semantics rather than general language. Two calls that discuss the same workflow but use different vocabulary should land close together in embedding space.
The embedding for a call $c_i$ is:
$$\ mathbf{e}i = f{\theta}(c_i) \in \mathbb{R}^{768}$$
where $f_{\theta}$ is the encoder parameterized by $\theta$. The fine-tuning step adjusted $\theta$ so that the cosine similarity between semantically related calls was maximized:
$$\ mathcal{L} = -\sum_{i} \log \frac{\exp(\text{sim}(\mathbf{e}i, \mathbf{e}{t_i})/\tau)}{\sum_{j} \exp(\text{sim}(\mathbf{e}i, \mathbf{e}{j})/\tau)}$$
where $\mathbf{e}_{t_i}$ is the target embedding for call $i$ and $\tau$ is the temperature parameter.
Step 3: Clustering in Embedding Space
We applied a hierarchical clustering algorithm to the 140,000 embedding vectors. The goal was not to find the "true" number of clusters — that number is a modeling choice, not a ground truth. We used a distance-based threshold: two calls were grouped if their cosine distance was below 0.22. This produced 847 clusters, each representing a group of calls that discussed a coherent set of needs, workflows, and constraints.
Step 4: Topic Extraction per Cluster
For each cluster, we ran a sparse non-negative matrix factorization (NMF) on the term-document matrix of the calls in that cluster. The NMF decomposition:
$$V \approx WH$$
where $V$ is the term-frequency matrix for the cluster, $W$ contains the topic-term weights, and $H$ contains the call-topic weights. We extracted the top 15 terms per topic. This gave us a human-readable label for each cluster: "Regional reporting for multi-site operations," "CSV export and spreadsheet workflow," "Compliance-driven data retention," etc.
Step 5: Cross-Reference with CRM Data
Here is the step that most analyses skip. We joined the cluster assignments back to the CRM records. For each cluster, we computed the distribution of account tier, deal size, close probability, and sales cycle length. We were looking for a mismatch: a cluster that was large in the transcripts but small or under-represented in the CRM's "active pipeline."
What We Found
The results were not a single surprise. They were a family of related surprises.
Cluster 214 contained 6,800 calls (4.9% of the corpus) where buyers described a need for modular, plant-level reporting. The CRM classified these accounts as "Mid-Market — Probable — 6 months." But the transcript analysis showed that 72% of these buyers said they would purchase a reporting-only module within 30 days if it existed. Our product did not have a reporting-only module. The full platform was 4x the price point these buyers were referencing.
Cluster 318 contained 4,100 calls where buyers in healthcare and pharma described a need for audit-trail-ready data exports. The CRM had no "compliance" tag. These accounts were tracked as "Enterprise — Negotiating — 12 months," but the transcripts showed that the compliance need was the primary driver, not the budget. These buyers were not negotiating price. They were negotiating feature parity with their existing compliance tooling.
Cluster 502 contained 3,200 calls from logistics companies that wanted API access to the analytics layer but not the full UI. Our product's API was gated behind the enterprise tier. These buyers were paying enterprise prices for a mid-market feature.
The pattern across all three clusters: the buyers were not looking for our product. They were looking for a component of our product, and our packaging did not offer that component as a standalone SKU.
The Hidden Market, Quantified
Here is the distribution of "under-served" accounts by cluster:
Cluster | Calls | % of Corpus | Avg Deal Size | Close Speed (CRM) | Close Speed (Transcript)
-------------+----------+--------------+----------------+---------------------+--------------------------
214 | 6,800 | 4.9% | $42,000 | 6 months | 30 days (stated)
318 | 4,100 | 2.9% | $61,000 | 12 months | 45 days (stated)
502 | 3,200 | 2.3% | $28,000 | 8 months | 21 days (stated)
517 | 2,900 | 2.1% | $15,000 | 4 months | 14 days (stated)
601 | 2,400 | 1.7% | $38,000 | 5 months | 28 days (stated)The discrepancy between CRM close speed and transcript-stated close speed is the signal. The CRM says these deals take 4-12 months. The buyers in the transcripts say they would close in 14-45 days. The difference is the cost of not having the right SKU.
If we had launched a reporting-only module at $45,000, the 6,800 accounts in Cluster 214 represent a potential $287M pipeline that our CRM was tracking as a 6-month cycle. If we had built a compliance export add-on at $25,000, the 4,100 accounts in Cluster 318 represent $102M in near-term revenue. The hidden market was not a new industry or a new geography. It was a packaging decision that our structured data could not reveal.
Why This Is Harder Than It Sounds
The analysis above looks clean, but the intermediate steps were not. A few lessons from the process:
Embedding quality determines cluster quality. Our first pass used a general-purpose sentence encoder. The clusters were too broad — calls about "reporting" and calls about "dashboards" landed in the same group, but they are different workflows. After fine-tuning on 50,000 sales transcripts, the clusters sharpened. The "reporting" cluster separated from the "dashboard" cluster. This is not a trivial improvement. It is the difference between finding a market segment and finding a word frequency.
NMF topic extraction is sensitive to dimensionality. Too few topics and you get generic labels ("data," "report," "system"). Too many and you get noise. We tested $k$ from 5 to 40 and found that $k=15$ per cluster gave the most interpretable labels. This is a hyperparameter, not a discovery. The discovery is in the cluster assignments, not the labels.
The CRM join is where the insight lives. The cluster analysis alone tells you what people talk about. The CRM join tells you what they are not buying and how long it takes. The mismatch between the two is the market gap.
The Broader Pattern
This is not a story about sales. It is a story about a class of business problems where the raw signal is in unstructured text and the structured data is a lossy summary. You have customer support tickets, product review text, engineering bug reports, onboarding survey responses, and post-sale NPS comments. All of these are transcripts in the same sense: rich, unstructured, and currently under-used in analytics pipelines.
The method generalizes. You take the unstructured corpus, embed it, cluster it, extract topics, and join back to the structured data. The join is the step that turns a linguistic analysis into a business decision. Without the join, you have a topic model. With the join, you have a market segment.
What We Changed
We launched the reporting-only module four months after the analysis. It was not a new product. It was a re-packaging of an existing feature set, priced at $45,000, with a 30-day onboarding commitment. The first quarter revenue from this SKU was $18.2M, coming from 412 accounts. 389 of those 412 were accounts that had appeared in Cluster 214. The transcript analysis had identified the buyers before the product existed.
We did not build a new sales team or a new marketing campaign. We built a SKU. The hidden market was not hidden in the market. It was hidden in the data we already had.
The Method as a Template
For any team with a corpus of unstructured text tied to structured records, the pipeline is:
Transcribe and normalize the raw signal.
Embed the documents using a domain-adapted encoder.
Cluster in embedding space with a distance threshold.
Extract topics per cluster using sparse NMF.
Join the cluster assignments to the structured data.
Find the mismatches — where the transcript signal says "these buyers want X" and the CRM says "these buyers are in a 6-month cycle."
The mismatches are your hidden markets. They are not new customers. They are existing customers who are buying the wrong SKU, waiting for a feature that does not exist, or being tracked in a pipeline stage that does not reflect their actual buying timeline.
The analysis cost was roughly $3,400 in compute. The first-quarter revenue from the new SKU was $18.2M. The ratio is not the point. The point is that the signal was in the data the whole time. The CRM had the records. The transcripts had the conversations. The analysis connected the two.
Elena Vasquez holds a Ph.D. in Artificial Intelligence and writes about applied AI systems in business contexts. She has advised on NLP pipelines for revenue operations, customer experience, and product analytics teams.
