Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
π§ Natural Language Processing
RustMITupdated Sep 2, 2026
π #1,355 overall#31 of 148 in Tokenization & Preprocessing
cbaziotis /ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
$ git clone https://github.com/cbaziotis/ekphrasis.gitStars
673
673
Forks
92
92
Language
Python
License
MIT
Created
Feb 7, 2017
9.6 years old
Last push
Jun 2, 2025
More in Tokenization & Preprocessing
Tokenization & Preprocessing
daac-tools/vibratoπ₯ active
π€ vibrato: Viterbi-based accelerated tokenizer
π§ Natural Language Processing
RustApache-2.0updated Sep 19, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
π₯ Vaporetto: Very accelerated pointwise prediction based tokenizer
π§ Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
clipperhouse/uax29
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
π§ Natural Language Processing
GoMITupdated Feb 16, 2026
Tokenization & Preprocessing
natasha/natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
π·οΈ Named Entity Recognitionπ§ Natural Language Processing
PythonMITupdated Apr 13, 2026
Tokenization & Preprocessing
lovit/soynlp
νκ΅μ΄ μμ°μ΄μ²λ¦¬λ₯Ό μν νμ΄μ¬ λΌμ΄λΈλ¬λ¦¬μ λλ€. λ¨μ΄ μΆμΆ/ ν ν¬λμ΄μ / νμ¬νλ³/ μ μ²λ¦¬μ κΈ°λ₯μ μ 곡ν©λλ€.
π§ Natural Language Processing
PythonOtherupdated Mar 10, 2026
Data from GitHub Β· snapshot Sep 24, 2026