Tokenization & Preprocessing
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
🌍 Machine Translation🧠 Natural Language Processing
C++MITupdated Jan 10, 2026
🏆 #2,957 overall#131 of 148 in Tokenization & Preprocessing🔥 active this week
actual-computer /toks
exact tokenizers ids, as fast as the hardware allows
$ git clone https://github.com/actual-computer/toks.gitStars
118
118
Forks
10
10
Language
C
License
Apache-2.0
Created
Oct 5, 2026
0.1 years old
Last push
Nov 5, 2026
🔥 this week
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
lionsoul2014/friso
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
CApache-2.0updated Oct 29, 2023
Tokenization & Preprocessing
zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
RustMITupdated Sep 24, 2026
Tokenization & Preprocessing
Tokenization & Preprocessing
theseer/tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
PHPOtherupdated Feb 3, 2026
Tokenization & Preprocessing
marcelroed/gigatoken🔥 active
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
Data from GitHub · snapshot Nov 9, 2026