Tokenization & Preprocessing
ropensci/tokenizers
Fast, Consistent Tokenization of Natural Language Text
🧠 Natural Language Processing💎 Text Mining
ROtherupdated Mar 27, 2024
🏆 #2,947 overall#132 of 148 in Tokenization & Preprocessing
ropensci /hunspell
High-Performance Stemmer, Tokenizer, and Spell Checker for R
$ git clone https://github.com/ropensci/hunspell.gitStars
116
116
Forks
47
47
Language
C++
License
Other
Created
Mar 9, 2016
10.5 years old
Last push
Feb 12, 2026
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
bnosac/udpipe
R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit
🧠 Natural Language Processing💎 Text Mining
C++MPL-2.0updated Mar 2, 2026
Tokenization & Preprocessing
theseer/tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
PHPOtherupdated Feb 3, 2026
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
Tokenization & Preprocessing
andialbrecht/sqlparse
A non-validating SQL parser module for Python
PythonBSD-3-Clauseupdated Aug 13, 2026
Tokenization & Preprocessing
Chevrotain/chevrotain🔥 active
Parser Building Toolkit for JavaScript
TypeScriptApache-2.0updated Sep 24, 2026
Data from GitHub · snapshot Sep 24, 2026