marcelroed/gigatoken
▲ +2 ★ this week
Language model tokenization at GB/s
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
$ git clone https://github.com/cbaziotis/ekphrasis.git▲ +2 ★ this week
Language model tokenization at GB/s
▲ +2 ★ this week
🎤 vibrato: Viterbi-based accelerated tokenizer
▲ +2 ★ this week
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
Open Korean Text Processor - An Open-source Korean Text Processor
▲ +1 ★ this week
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
Data from GitHub · snapshot Nov 16, 2026