🏆 #981 overall#14 of 178 in Tokenization & Preprocessing

dqbd /tiktokenizer

Online playground for OpenAPI tokenizers

$ git clone https://github.com/dqbd/tiktokenizer.git
GitHub social preview for dqbd/tiktokenizer
Stars
1.7K
1,721
This week
0 ★
+16 this month
Forks
188
188
Language
TypeScript
License
MIT
Created
Mar 2, 2023
3.7 years old
Last push
Apr 24, 2025

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 1.7K▲ +16★ 1.7K

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

botisan-ai/gpt3-tokenizer

Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.

TypeScriptMITupdated Jan 27, 2023
GitHub ↗★ 171⑂ 16
Tokenization & Preprocessing

zurawiki/tiktoken-rs

▲ +2 ★ this week

Ready-made tokenizer library for working with GPT and tiktoken

RustMITupdated Sep 24, 2026
GitHub ↗★ 412⑂ 76
Tokenization & Preprocessing

dmitry-brazhenko/SharpToken

▲ +1 ★ this week

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

C#MITupdated Mar 25, 2026
GitHub ↗★ 264⑂ 19
Tokenization & Preprocessing

microsoft/Tokenizer

▲ +1 ★ this week

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

C#MITupdated Oct 19, 2026
GitHub ↗★ 215⑂ 36
Tokenization & Preprocessing

johannschopplich/tokenx🔥 active

▲ +2 ★ this week

📐 Fast token estimation with 95%+ average accuracy in a 2kB bundle

TypeScriptMITupdated Nov 14, 2026
GitHub ↗★ 190⑂ 8
Tokenization & Preprocessing

dluc/openai-tools

A collection of tools for working with OpenAI

C#CC0-1.0updated May 4, 2023
GitHub ↗★ 102⑂ 14

Data from GitHub · snapshot Nov 16, 2026