🏆 #4,721 overall#116 of 178 in Tokenization & Preprocessing

ropensci /tokenizers

Fast, Consistent Tokenization of Natural Language Text

$ git clone https://github.com/ropensci/tokenizers.git
GitHub social preview for ropensci/tokenizers
Stars
188
188
This week
0 ★
0 this month
Forks
23
23
Language
R
License
Other
Created
Mar 25, 2016
10.6 years old
Last push
Mar 27, 2024

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 188▲ +0★ 188

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

bnosac/udpipe

R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit

🧠 Natural Language Processing💎 Text Mining
C++MPL-2.0updated Mar 2, 2026
GitHub ↗★ 225⑂ 35
Tokenization & Preprocessing

ropensci/hunspell

High-Performance Stemmer, Tokenizer, and Spell Checker for R

C++Otherupdated Feb 12, 2026
GitHub ↗★ 116⑂ 47
Tokenization & Preprocessing

jbesomi/texthero

Text preprocessing, representation and visualization from zero to hero.

🧬 Embeddings🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Aug 29, 2023
GitHub ↗★ 2.9K⑂ 235
Tokenization & Preprocessing

cbaziotis/ekphrasis

▲ +1 ★ this week

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Jun 2, 2025
GitHub ↗★ 672⑂ 91
Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225

Data from GitHub · snapshot Nov 16, 2026