Token and context-budget estimator
Estimate how many tokens a text uses and how much of a model’s context window it takes, with a stated margin of error.
A clear answer. Here’s how.
Adds up weighted counts of what the text contains: Latin words and letters, accented letters, letters of other scripts, Chinese, Japanese and Korean characters, digits, punctuation, spaces, line breaks and symbols. The weights were fitted to the token counts of OpenAI’s public o200k_base tokenizer on 3,630 passages: Wikipedia text in 11 languages, source code, JSON, Markdown and CSS, with half held back for testing. On the held-out half, 9 in 10 estimates were within about 12 to 19% for Latin-script text, code and Markdown (wider for passages under a few hundred tokens), 10 to 17% for Chinese, Japanese and Korean, and 18 to 33% for other scripts; the range shown uses the figure for your text’s kind and length. A separate check on 20 fresh short passages (50 to 100 tokens each) found 18 inside their ranges, with Hindi and CSS the worst. This is an estimate only: real tokenizers differ between models and none is reproduced here, and older OpenAI tokenizers give up to 60% more tokens for non-Latin scripts. For billing or a hard limit, use the provider’s own token counter.
How this is checked
5 reference checks passed These 5 cases were run when this page was built, on 20 October 2026, and all reproduce within the stated tolerance. The same cases are part of the automated tests that run on every change to Toolmoz. Engine version 1.
See the checks and where each expected answer comes from
Case 1
- Inputs
- Text The committee met on Tuesday to review the proposed changes to the library opening hours. After a long discussion, members agreed that weekend hours should be extended, but only during the summer term, and that students would be consulted before any final decision was made. A short survey will be sent out next week.
- Expected, and what Toolmoz gave
- Estimated tokens expected 60, Toolmoz 62Characters expected 317, Toolmoz 317
- Where the expected answer comes from
- OpenAI o200k_base token count of this text, computed with the gpt-tokenizer package (version 4.0.3) (60); the character count is exact
- Allowed difference
- ±0.15 relative
Case 2
- Inputs
- Text export function parseRange(text) { const match = /^(\d+)-(\d+)$/.exec(text.trim()); if (!match) throw new Error("Expected a range such as 3-7"); const [from, to] = [Number(match[1]), Number(match[2])]; return Array.from({ length: to - from + 1 }, (_, i) => from + i); }
- Expected, and what Toolmoz gave
- Estimated tokens expected 87, Toolmoz 81
- Where the expected answer comes from
- OpenAI o200k_base token count of this text, computed with the gpt-tokenizer package (version 4.0.3)
- Allowed difference
- ±0.15 relative
Case 3
- Inputs
- Text {"id": 1042, "name": "Walnut desk", "tags": ["furniture", "office", "oak"], "dimensions": {"width": 120, "depth": 60, "height": 74}, "inStock": true, "price": 249.99, "supplier": {"name": "Northfield Joinery", "country": "GB"}}
- Expected, and what Toolmoz gave
- Estimated tokens expected 81, Toolmoz 73
- Where the expected answer comes from
- OpenAI o200k_base token count of this text, computed with the gpt-tokenizer package (version 4.0.3)
- Allowed difference
- ±0.15 relative
Case 4
- Inputs
- Text 委员会周二开会，审议了对图书馆开放时间的拟议修改。经过长时间的讨论，成员们同意延长周末开放时间，但仅限于夏季学期，并且在作出最终决定之前将征求学生的意见。下周将发送一份简短的问卷调查。
- Expected, and what Toolmoz gave
- Estimated tokens expected 72, Toolmoz 73Content type, which sets the range expected Chinese, Japanese or Korean, Toolmoz Chinese, Japanese or Korean
- Where the expected answer comes from
- OpenAI o200k_base token count of this text, computed with the gpt-tokenizer package (version 4.0.3)
- Allowed difference
- ±0.15 relative
Case 5
- Inputs
- Text Hello worldContext window of your model (tokens) 200Tokens to keep free for the answer 150
- Expected, and what Toolmoz gave
- Estimated tokens expected 2, Toolmoz 2Characters expected 11, Toolmoz 11Lines expected 1, Toolmoz 1Share of the context window used by this text expected 1, Toolmoz 1Window left after this text and the reserved answer expected 48, Toolmoz 48
- Where the expected answer comes from
- OpenAI o200k_base counts “Hello world” as 2 tokens; 2 ÷ 200 = 1% and 200 − 150 − 2 = 48 (hand calculation)
- Allowed difference
- exact
What this shows: the engine reproduces independently worked values for these cases. It does not prove every possible input is right. To test an answer you were given, see how to check a number; to see what is and is not checked across Toolmoz, see the accuracy ledger.
See a worked example
Worked example
A fixed demonstration, separate from the values you enter above.
- Text
- The committee met on Tuesday to review the proposed changes to the library opening hours. After a long discussion, members agreed that weekend hours should be extended, but only during the summer term, and that students would be consulted before any final decision was made. A short survey will be sent out next week.
- Context window of your model (tokens)
- 128000
- Tokens to keep free for the answer
- 4000
Example result
- Estimated tokens
- 62
- Likely range (about 9 times in 10)
- 51 to 74
- Characters
- 317
- Lines
- 1
- Characters per token (estimate)
- 5.11
- Content type, which sets the range
- Latin-script text, code or Markdown
- Share of the context window used by this text
- 0.0%
- Window left after this text and the reserved answer
- 123,938
An estimate, not a count. Tokenizers differ by model, and none is reproduced here: this adds up weighted counts of letters, words, digits, punctuation and symbols, fitted to OpenAI’s o200k_base tokenizer. On held-out test text about 9 in 10 estimates were within ±19% of that tokenizer for texts of this kind and length (the range shown). Other models (including Claude and Gemini) can differ by more, especially for non-Latin scripts and code; short texts vary by a few tokens. For billing or hard limits use the provider’s own counter.