How to Count Tokens in C#

Everything about working with LLMs is measured in tokens: cost is per token, context windows are sized in tokens, and requests fail when you exceed the limit. So it pays to count them before you send a request, not discover the problem in an error or an invoice. This guide shows how to count tokens in C#.

What a token is

A token is a chunk of text — roughly ¾ of a word on average in English, but it varies. “Hello” might be one token; “internationalization” several. The model sees tokens, not characters or words, so a rough character count is a poor estimate. For anything cost- or limit-sensitive, count properly.

Counting tokens with a tokenizer

Don’t guess — use a real tokenizer that matches your model’s encoding. In .NET, Microsoft.ML.Tokenizers provides tokenizers (including the tiktoken encodings OpenAI uses):

using Microsoft.ML.Tokenizers;

// Get the tokenizer for your model's encoding
Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4o-mini");

string text = "How many tokens is this sentence?";
int count = tokenizer.CountTokens(text);

Console.WriteLine($"Tokens: {count}");

Now you have an exact count, not an estimate.

Construct the tokenizer once. It loads and parses a vocabulary file, so building one per call is a real cost for no benefit. It is safe to share across threads for counting, which makes a static field or a DI singleton the right home:

public static class Tokens
{
    private static readonly Tokenizer Encoder =
        TiktokenTokenizer.CreateForModel("gpt-4o-mini");

    public static int Count(string text) => Encoder.CountTokens(text);
}

The trap: a chat request is more than its text

This is where most token estimates go wrong. Counting the strings you wrote counts the content, but the model is billed for the request — and a chat request carries structure that the content does not.

Each message contributes a few tokens of role and delimiter overhead. The conversation itself adds a small priming cost. And any tool or function definitions you attach are serialised and counted in full, every single turn, whether or not the model calls them.

The practical effect: count only the message text and you will typically undershoot by 5–15% on a short conversation, and considerably more on an agent with a dozen tools registered. That is fine when you are budgeting and fatal when you are checking a hard limit.

// Approximate the request, not just the prose. The per-message constant varies by
// model family — treat it as a safety margin rather than an exact figure.
const int PerMessageOverhead = 4;   // role + delimiters
const int ConversationPriming = 3;  // per request

int CountRequest(IEnumerable<ChatMessage> messages, string? toolsJson = null)
{
    var total = ConversationPriming;

    foreach (var m in messages)
        total += PerMessageOverhead + Tokens.Count(m.Text ?? string.Empty);

    // Tool schemas are sent on every turn — count them every turn.
    if (toolsJson is not null)
        total += Tokens.Count(toolsJson);

    return total;
}

If an exact number matters, the honest answer is to read Usage.InputTokenCount off the response and reconcile. Use the local count to decide whether to send; use the reported count for billing and for tuning your overhead constant.

That reconciliation is worth automating. Log both numbers on every call and alert when the ratio between them moves. A drift in that ratio is usually the first visible sign that someone added a tool, changed the system prompt, or switched the deployment to a model with a different chat format — all things that will otherwise show up as a surprise on an invoice weeks later.

One caveat on the reported number: if you stream responses, some providers omit usage from the streamed result unless you explicitly ask for it. Check whether your client surfaces usage on streamed calls before concluding that your streaming path is free.

Match the tokenizer to the model, and check it at startup

CreateForModel maps a model name to an encoding. That mapping is a lookup table shipped with the package, which has two consequences.

A model released after your version of Microsoft.ML.Tokenizers will not be in the table, and construction fails rather than quietly falling back to a default encoding. That is the right behaviour — a silent fallback would give you plausible numbers that are wrong — but it means a model upgrade can break your service at startup rather than in a test. If you know the encoding your model uses, CreateForEncoding sidesteps the model-name lookup entirely and is the more stable choice for anything long-lived.

The worse case is the one that does not throw: pointing a tokenizer for one model family at text destined for another. Nothing fails, the counts are simply wrong, and you find out when a request you calculated at 90% of the limit gets rejected. If you route between models — and cost control says you should — keep one tokenizer per model in a dictionary and resolve it from the same value you send as the model name, so the two cannot drift apart.

What going over the limit actually looks like

The failure is a 400 from the provider, not a truncated response. With OpenAI and Azure OpenAI the error code is context_length_exceeded, and the message tells you the model’s maximum and what your messages actually came to — which, incidentally, is the cheapest way to calibrate your overhead constants, since the API is telling you the real number for free.

It is a failure worth handling deliberately, because the naive retry makes it worse: the request is deterministic, so retrying sends the same oversized payload and fails identically while your retry policy adds latency. Catch it, trim, and resend once — or fail fast with a message the user can act on.

The subtler failure is the opposite end. Set max_tokens too low and the call succeeds, but the response stops mid-sentence with a finish reason of length. If you asked for JSON, you now have invalid JSON and a deserialisation exception three layers away from the cause. Always check the finish reason before parsing a structured response, and treat a length stop as an error rather than as data.

Trimming without breaking the request

Counting is only useful if you can act on it, and the action is almost always trimming. Two rules make the difference between trimming that works and trimming that produces new errors.

Never drop the system message. It is usually the smallest part of the payload and the only part that determines behaviour, so an eviction policy that treats messages uniformly will eventually delete the instructions and leave the history. Pin it and trim the rest.

Never split a tool call from its result. In a chat history, an assistant message that requests a tool and the message carrying that tool’s output are a matched pair keyed by an ID. Drop the request and keep the result — which is exactly what a naive “remove the oldest message” loop does — and the provider rejects the request because it contains a tool response with no corresponding call. Trim in pairs, or trim whole turns.

For trimming a single large document rather than a conversation, the tokenizer gives you a better tool than a character-count guess: GetIndexByTokenCount returns the index at which the text reaches a token budget, and GetIndexByTokenCountFromEnd does the same from the other end. Cutting at a token boundary avoids the mildly maddening case where a substring you sliced by characters counts as more tokens than the same text did in context, because the cut split a token that had merged with its neighbour.

Counting itself is not free. Tokenising a 100K-token prompt on every request costs real CPU, and on a busy service that shows up as latency before it shows up as anything else. Cache counts against content that does not change — your system prompt, tool schemas, retrieved chunks stored alongside their token count at index time — and only count the part that varies.

Estimating cost before you call

Once you can count tokens, you can price a request before making it. Cost is (input tokens × input rate) + (output tokens × output rate):

int inputTokens = tokenizer.CountTokens(prompt);
decimal estInputCost = inputTokens / 1000m * inputRatePer1K;
// You won't know output tokens exactly in advance — cap them with max_tokens and price the worst case.

This is invaluable for budgeting and for showing users an estimated cost before an expensive operation.

Staying under the context limit

Every model has a maximum context (input + output tokens). Exceed it and the call fails. Count before sending and trim if needed — this is the core of managing long conversations and large RAG contexts:

if (tokenizer.CountTokens(fullPrompt) > maxContextTokens - reservedForResponse)
{
    // Trim history or retrieved context before calling — see agent memory.
}

Reserve room for the response: if the context limit is 128K and you want up to 4K of output, keep the input under ~124K.

Leave a margin on top of that. Your overhead constants are approximate, tool schemas change as someone registers a new function, and the request that finally tips over the limit will be the one from your largest customer at the worst possible moment. Budgeting to 90% of the real limit costs almost nothing and removes an entire class of production incident.

Tokens are not equally priced across languages

One detail that catches teams serving a multilingual audience: token efficiency depends on how well a language is represented in the tokenizer’s vocabulary.

English prose runs at roughly four characters per token. Text in scripts that are under-represented — Hindi, Tamil, Arabic, Thai — can consume several times more tokens for the same meaning, because the tokenizer falls back to shorter and shorter pieces. Code and JSON also tokenize worse than prose, since punctuation and indentation rarely merge into larger tokens.

The consequences are practical rather than theoretical: the same conversation can cost noticeably more and hit the context limit sooner in one language than another. If you serve users in several languages, measure your worst case rather than pricing the English one and hoping.

// Worth running once against real samples from each language you support.
foreach (var (language, sample) in samples)
    Console.WriteLine($"{language,-10} {Tokens.Count(sample),5} tokens " +
                      $"for {sample.Length} chars " +
                      $"({(double)sample.Length / Tokens.Count(sample):F2} chars/token)");

Where this fits

Token counting underpins two things covered elsewhere:

• Cost control — you can’t optimize spend you don’t measure; token counts are the raw signal.
• Agent memory — windowing and summarizing conversation history is driven by staying under the token budget.

Note: tokenizer package names and model encodings change; verify against Microsoft.ML.Tokenizers and your provider’s current docs. The principle — count with the model’s real tokenizer, price and trim against the count — is stable.

Takeaway

Tokens are the unit of cost and the hard limit on context, so count them before you call — with a real tokenizer like Microsoft.ML.Tokenizers, not a character estimate. Counting lets you price a request in advance, keep prompts under the context window, and drive the trimming and budgeting that make production AI affordable. It’s a small utility that prevents a lot of surprise errors and invoices.