Tokenization & Preprocessing
🏆 #1,739 overall#53 of 148 in Tokenization & Preprocessing
zurawiki /tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
$ git clone https://github.com/zurawiki/tiktoken-rs.gitStars
411
411
Forks
75
75
Language
Rust
License
MIT
Created
Feb 2, 2023
3.6 years old
Last push
Jul 1, 2026
Categories
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
daac-tools/vibrato🔥 active
🎤 vibrato: Viterbi-based accelerated tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Sep 19, 2026
Tokenization & Preprocessing
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
🌍 Machine Translation🧠 Natural Language Processing
C++MITupdated Jan 10, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
dmitry-brazhenko/SharpToken
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
C#MITupdated Mar 25, 2026
Tokenization & Preprocessing
microsoft/Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
C#MITupdated Sep 10, 2026
Data from GitHub · snapshot Sep 24, 2026