🏆 #1,969 overall#39 of 178 in Tokenization & Preprocessing

cbaziotis /ekphrasis

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

$ git clone https://github.com/cbaziotis/ekphrasis.git
GitHub social preview for cbaziotis/ekphrasis
Stars
672
672
This week
+1 ★
-1 this month
Forks
91
91
Language
Python
License
MIT
Created
Feb 7, 2017
9.8 years old
Last push
Jun 2, 2025

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 673▼ 1★ 672

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

daac-tools/vibrato

▲ +2 ★ this week

🎤 vibrato: Viterbi-based accelerated tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Oct 25, 2026
GitHub ↗★ 430⑂ 27
Tokenization & Preprocessing

daac-tools/vaporetto

▲ +2 ★ this week

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Nov 3, 2026
GitHub ↗★ 302⑂ 12
Tokenization & Preprocessing

clipperhouse/uax29

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.

🧠 Natural Language Processing
GoMITupdated Feb 16, 2026
GitHub ↗★ 125⑂ 8
Tokenization & Preprocessing

open-korean-text/open-korean-text

Open Korean Text Processor - An Open-source Korean Text Processor

🧠 Natural Language Processing💎 Text Mining
ScalaApache-2.0updated Mar 12, 2024
GitHub ↗★ 668⑂ 95
Tokenization & Preprocessing

yooper/php-text-analysis

▲ +1 ★ this week

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

🧠 Natural Language Processing💎 Text Mining
PHPMITupdated Dec 28, 2024
GitHub ↗★ 535⑂ 91

Data from GitHub · snapshot Nov 16, 2026