How a Simple 'Hi' Can Be Worth $1,000s (You're Doing It Wrong)
The Hidden Economics of "Hello": Why Your Greeting Is More Expensive Than You Think
You type "hi" into your chat window and send it. Two characters. One keystroke after the other. A microsecond of screen time. And yet, somewhere in a server farm humming with liquid-cooled GPUs, those two letters cost you real money—sometimes hundreds of dollars' worth of compute per hour if you're not careful.
This isn't marketing fluff or a vendor trying to scare you into buying an enterprise plan. It's the quiet arithmetic of how modern AI systems actually work, and understanding it changes how you build, use, and pay for everything that talks to a large language model.
The Token Economy You Never See
Every time you interact with an LLM—chatbot, coding assistant, customer service agent, document summarizer—the system doesn't count words. It counts tokens. A token is roughly ¾ of a word in English: "tokenization" might be 2 tokens, while "hi" is 1 or 2 depending on the tokenizer's dictionary.
Here's the part most users miss: the model reads your entire conversation history every single time you send a message. Not just your last line. All of it. Every previous exchange, every system prompt, every tool definition, every retrieved document chunk—all of it is re-fed to the model as input tokens on every turn.
So if your chat has grown to 12,000 tokens of history and you send a simple "hi," the model still processes all 12,000+ tokens of context plus your 1–2 token message. You paid for roughly 12,000 input tokens to say hello.
Let's make this concrete with some math. For a mid-tier API model priced around $3 per million input tokens:
$$\ text{Cost} = \frac{12{,}000}{1{,}000{,}000} \times $3 = $0.036$$
Thirty-six cents for a "hi." For a high-end model at $30 per million input tokens? That same greeting costs you $0.36. And if your context window has bloated to 100,000 tokens (easy with RAG pipelines, long documents, or multi-turn agents), a single "hi" can approach $3–$4 in input cost alone before output is even counted.
Scale that across a customer support team running 50 concurrent chats, each with rich context, and you're looking at thousands of dollars per hour in pure context re-reading—money spent on the model re-processing information it has already seen.
Where the Money Actually Goes
Not all tokens are created equal in cost impact. The bill breaks down into several line items:
Component | Typical Share | Notes |
|---|---|---|
System prompt + tool schemas | 2,000–15,000 tokens per turn | Fixed overhead on every call |
Conversation history | 5,000–80,000 tokens | Grows linearly with turns |
RAG / retrieved chunks | 3,000–40,000 tokens | Scales with retrieval breadth |
User message (your "hi") | 1–200 tokens | Tiny, but you pay for all of the above too |
The counterintuitive insight: your message is the cheapest part of your bill. The expensive part is everything the model must re-read to understand where your two-character message fits. Context is a tax on every interaction.
This is why naive "just keep the chat going" architectures get surprisingly expensive, and why teams building production agents obsess over context management the way database engineers obsess over query plans.
The Architecture Levers That Matter
Knowing that context is the cost driver gives you concrete levers:
1. Context pruning and summarization. After N turns, compress earlier history into a summary. A 50-turn conversation at 80k tokens might become an 8k-token summary plus the last 5 full turns. You keep coherence while cutting input costs by 60–70%. The math is simple: if you reduce context from 80,000 to 30,000 tokens, your per-turn cost drops by more than half.
2. Prompt caching (where the API supports it). Services like Anthropic's cache or OpenAI's automatic caching let repeated prefixes (system prompts, tool definitions) be stored and billed at a fraction of normal rates—often 50–90% cheaper for cached tokens. If your system prompt is stable across thousands of calls, this is nearly free money back.
3. Right-size the model. A "hi" doesn't need a frontier reasoning model. Routing simple greetings, FAQs, and classification tasks to a smaller, cheaper model—and reserving the expensive one for complex generation—can reduce blended cost by an order of magnitude. This is sometimes called model cascading or semantic routing.
4. Tighten tool schemas. Every function definition in your system prompt costs tokens on every call. A 20-tool agent with verbose JSON schemas can burn 8,000–15,000 tokens of overhead per turn that a 5-tool agent doesn't. Audit them. Remove what you don't use. Shorten descriptions.
5. Batch and cache RAG. If your retrieval step pulls 20 chunks but the model only needs 3 to answer, you paid for 17 unused chunks in the context window. Rank-retrieval, MMR diversity control, or a small reranker can shrink that set dramatically.
A Worked Example: The Cost of a Support Bot
Suppose your support bot has:
System prompt + tools: 6,000 tokens (cached at 10% rate)
RAG context: 25,000 tokens (uncached)
History: 40,000 tokens (uncached)
User message: 100 tokens
At $3/M input and $15/M output, with ~200 output tokens per turn:
$$C _{\text{input}} = \frac{6{,}000}{10^6}\times$0.3 + \frac{65{,}000}{10^6}\times$3 \approx $0.002 + $0.195$$
$$C _{\text{output}} = \frac{200}{10^6}\times$15 \approx $0.003$$
Total ≈ $0.19 per turn. Multiply by 2,000 turns/day and you're at ~$380/day just in token costs—before infrastructure, retrieval, logging, or the human review layer. Now imagine 40% of those turns are simple acknowledgments ("hi," "ok," "thanks") that could've been handled by a $0.01 model or even a state machine. You're paying frontier-model prices for chit-chat.
The Deeper Lesson: Context Is a Design Variable, Not an Afterthought
The shift from "AI is magic" to "AI is a token economy with specific cost curves" is the maturation moment of applied LLM engineering. Your architecture choices—how much context you carry, which model handles which task, how aggressively you prune and cache—are as consequential for your P&L as database indexing or CDN choice was in web-scale systems a decade ago.
The simple "hi" isn't expensive by itself. It's expensive because of everything you decided to keep the model aware of while it said hello back. And that is where your engineering judgment, budget, and user experience all meet.
So next time you wonder why the AI bill looks high, don't just ask "how many tokens?" Ask: "what context did I choose to carry into this turn, and was every token of it earning its keep?" That question—asked consistently across your pipeline—is where thousands of dollars live or die. And a simple greeting is the clearest place to start asking it.
