Tokenization & Preprocessing
clipperhouse/uax29
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
🧠 Natural Language Processing
GoMITupdated Feb 16, 2026
Fast and customizable text tokenization library with BPE and SentencePiece support
$ git clone https://github.com/OpenNMT/Tokenizer.gitA tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
▲ +4 ★ this week
State of the Art Natural Language Processing
▲ +1 ★ this week
Python port of Moses tokenizer, truecaser and normalizer
▲ +1 ★ this week
Fast bare-bones BPE for modern tokenizer training
exact tokenizers ids, as fast as the hardware allows
▲ +2 ★ this week
Language model tokenization at GB/s
Data from GitHub · snapshot Nov 16, 2026