Tokenization & Preprocessing
microsoft/Tokenizer
▲ +1 ★ this week
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
C#MITupdated Oct 19, 2026
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
$ git clone https://github.com/dmitry-brazhenko/SharpToken.git▲ +1 ★ this week
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
▲ +4 ★ this week
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
▲ +2 ★ this week
Ready-made tokenizer library for working with GPT and tiktoken
Data from GitHub · snapshot Nov 16, 2026