OpenNMT/Tokenizer
▲ +1 ★ this week
Fast and customizable text tokenization library with BPE and SentencePiece support
Fast bare-bones BPE for modern tokenizer training
$ git clone https://github.com/gautierdag/bpeasy.git▲ +1 ★ this week
Fast and customizable text tokenization library with BPE and SentencePiece support
▲ +2 ★ this week
Language model tokenization at GB/s
▲ +1 ★ this week
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
▲ +3 ★ this week
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
▲ +2 ★ this week
🎤 vibrato: Viterbi-based accelerated tokenizer
▲ +2 ★ this week
Ready-made tokenizer library for working with GPT and tiktoken
Data from GitHub · snapshot Nov 16, 2026