bnosac/udpipe
R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit
Fast, Consistent Tokenization of Natural Language Text
$ git clone https://github.com/ropensci/tokenizers.gitR package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit
High-Performance Stemmer, Tokenizer, and Spell Checker for R
Text preprocessing, representation and visualization from zero to hero.
▲ +1 ★ this week
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
▲ +2 ★ this week
Language model tokenization at GB/s
Data from GitHub · snapshot Nov 16, 2026