Tokenization & Preprocessing
microsoft/Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
C#MITupdated Sep 10, 2026
🏆 #2,156 overall#73 of 148 in Tokenization & Preprocessing
dmitry-brazhenko /SharpToken
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
$ git clone https://github.com/dmitry-brazhenko/SharpToken.gitStars
262
262
Forks
19
19
Language
C#
License
MIT
Created
Mar 28, 2023
3.5 years old
Last push
Mar 25, 2026
Categories
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
botisan-ai/gpt3-tokenizer
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
TypeScriptMITupdated Jan 27, 2023
Tokenization & Preprocessing
Tokenization & Preprocessing
Tokenization & Preprocessing
niieani/gpt-tokenizer
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
TypeScriptMITupdated Aug 16, 2026
Tokenization & Preprocessing
zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
RustMITupdated Jul 1, 2026
Data from GitHub · snapshot Sep 24, 2026