Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
🏆 #1,515 overall#40 of 148 in Tokenization & Preprocessing
yooper /php-text-analysis
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
$ git clone https://github.com/yooper/php-text-analysis.gitStars
534
534
Forks
91
91
Language
PHP
License
MIT
Created
May 21, 2012
14.3 years old
Last push
Dec 28, 2024
More in Tokenization & Preprocessing
Tokenization & Preprocessing
cbaziotis/ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
🧠 Natural Language Processing
PythonMITupdated Jun 2, 2025
Tokenization & Preprocessing
daac-tools/vibrato🔥 active
🎤 vibrato: Viterbi-based accelerated tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Sep 19, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
natasha/razdel
Rule-based token, sentence segmentation for Russian language
🧠 Natural Language Processing
PythonMITupdated Apr 13, 2026
Tokenization & Preprocessing
adbar/simplemma🔥 active
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
🧠 Natural Language Processing
PythonMITupdated Sep 21, 2026
Data from GitHub · snapshot Sep 24, 2026