Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
🏆 #2,876 overall#124 of 148 in Tokenization & Preprocessing
clipperhouse /uax29
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
$ git clone https://github.com/clipperhouse/uax29.gitStars
123
123
Forks
7
7
Language
Go
License
MIT
Created
Apr 15, 2020
6.4 years old
Last push
Feb 16, 2026
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
cbaziotis/ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
🧠 Natural Language Processing
PythonMITupdated Jun 2, 2025
Tokenization & Preprocessing
daac-tools/vibrato🔥 active
🎤 vibrato: Viterbi-based accelerated tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Sep 19, 2026
Tokenization & Preprocessing
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
🌍 Machine Translation🧠 Natural Language Processing
C++MITupdated Jan 10, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
clipperhouse/jargon
Tokenizers and lemmatizers for Go
🧠 Natural Language Processing
GoMITupdated Sep 2, 2025
Data from GitHub · snapshot Sep 24, 2026