Tokenization & Preprocessing
dmitry-brazhenko/SharpToken
▲ +1 ★ this week
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
C#MITupdated Mar 25, 2026
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
$ git clone https://github.com/microsoft/Tokenizer.git▲ +1 ★ this week
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
▲ +499 ★ this week
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
▲ +2 ★ this week
Language model tokenization at GB/s
▲ +1 ★ this week
Easy token price estimates for 400+ LLMs. TokenOps.
Data from GitHub · snapshot Nov 16, 2026