Tokenization & Preprocessing
daac-tools/vibrato🔥 active
🎤 vibrato: Viterbi-based accelerated tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Sep 19, 2026
🏆 #2,332 overall#85 of 148 in Tokenization & Preprocessing🔥 active this week
adbar /simplemma
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
$ git clone https://github.com/adbar/simplemma.gitStars
219
219
Forks
17
17
Language
Python
License
MIT
Created
Jan 18, 2021
5.7 years old
Last push
Sep 21, 2026
🔥 this week
More in Tokenization & Preprocessing
Tokenization & Preprocessing
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
adbar/trafilatura🔥 active
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
🧠 Natural Language Processing💎 Text Mining
PythonApache-2.0updated Sep 21, 2026
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
Tokenization & Preprocessing
Tokenization & Preprocessing
cbaziotis/ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
🧠 Natural Language Processing
PythonMITupdated Jun 2, 2025
Data from GitHub · snapshot Sep 24, 2026