Tokenization & Preprocessing
🏆 #2,806 overall#61 of 178 in Tokenization & Preprocessing
zurawiki /tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
$ git clone https://github.com/zurawiki/tiktoken-rs.gitStars
412
412
This week
+2 ★
+4 this month
Forks
76
76
Language
Rust
License
MIT
Created
Feb 2, 2023
3.8 years old
Last push
Sep 24, 2026
Stars, last 30 days
Oct 15, 2026 → Nov 16, 2026Categories
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
daac-tools/vibrato
▲ +2 ★ this week
🎤 vibrato: Viterbi-based accelerated tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Oct 25, 2026
Tokenization & Preprocessing
OpenNMT/Tokenizer
▲ +1 ★ this week
Fast and customizable text tokenization library with BPE and SentencePiece support
🌍 Machine Translation🧠 Natural Language Processing
C++MITupdated Jan 10, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
▲ +2 ★ this week
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Nov 3, 2026
Tokenization & Preprocessing
dmitry-brazhenko/SharpToken
▲ +1 ★ this week
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
C#MITupdated Mar 25, 2026
Tokenization & Preprocessing
microsoft/Tokenizer
▲ +1 ★ this week
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
C#MITupdated Oct 19, 2026
Data from GitHub · snapshot Nov 16, 2026