Tokenization & Preprocessing
dmitry-brazhenko/SharpToken
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
C#MITupdated Mar 25, 2026
🏆 #2,357 overall#87 of 148 in Tokenization & Preprocessing
microsoft /Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
$ git clone https://github.com/microsoft/Tokenizer.gitStars
213
213
Forks
37
37
Language
C#
License
MIT
Created
Mar 28, 2023
3.5 years old
Last push
Sep 10, 2026
Categories
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
jshuadvd/LongRoPE
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
🤖 Language Models🧠 Natural Language Processing
Pythonno licenseupdated Jul 20, 2024
Tokenization & Preprocessing
rasbt/LLMs-from-scratch🔥 active
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
🤖 Language Models🧠 Natural Language Processing
Jupyter NotebookOtherupdated Sep 22, 2026
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
Tokenization & Preprocessing
AgentOps-AI/tokencost
Easy token price estimates for 400+ LLMs. TokenOps.
🤖 Language Models
PythonMITupdated Sep 5, 2025
Tokenization & Preprocessing
Data from GitHub · snapshot Sep 24, 2026