Language model tokenization at GB/s
$ git clone https://github.com/marcelroed/gigatoken.gitEkphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
🎤 vibrato: Viterbi-based accelerated tokenizer
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
Token Cost Parity: Multilingual LLM Efficiency Analysis 2026
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Data from GitHub · snapshot Sep 24, 2026