🏆 #2,417 overall#51 of 178 in Tokenization & Preprocessing

lionsoul2014 /friso

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.

$ git clone https://github.com/lionsoul2014/friso.git
GitHub social preview for lionsoul2014/friso
Stars
515
515
This week
+1 ★
+1 this month
Forks
93
93
Language
C
License
Apache-2.0
Created
Mar 31, 2014
12.6 years old
Last push
Oct 29, 2023

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 514▲ +1★ 515

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

howl-anderson/MicroTokenizer

一个轻量且功能全面的中文分词器,帮助学生了解分词器的工作原理。MicroTokenizer: A lightweight Chinese tokenizer designed for educational and research purposes. Provides a practical, hands-on approach to understanding NLP concepts, featuring multiple tokenization algorithms and customizable models. Ideal for students, researchers, and NLP enthusiasts..

PythonMITupdated Oct 18, 2024
GitHub ↗★ 160⑂ 22
Tokenization & Preprocessing

open-korean-text/open-korean-text

Open Korean Text Processor - An Open-source Korean Text Processor

🧠 Natural Language Processing💎 Text Mining
ScalaApache-2.0updated Mar 12, 2024
GitHub ↗★ 668⑂ 95
Tokenization & Preprocessing

ongjin/garu🔥 active

▲ +1 ★ this week

초경량 한국어 형태소 분석기 — 1MB 모델로 브라우저에서 실행(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko

🧠 Natural Language Processing
TypeScriptApache-2.0updated Nov 16, 2026
GitHub ↗★ 183⑂ 9
Tokenization & Preprocessing

actual-computer/toks

exact tokenizers ids, as fast as the hardware allows

CApache-2.0updated Nov 5, 2026
GitHub ↗★ 118⑂ 10
Tokenization & Preprocessing

theseer/tokenizer

▲ +2 ★ this week

A small library for converting tokenized PHP source code into XML (and potentially other formats)

PHPOtherupdated Feb 3, 2026
GitHub ↗★ 5.2K⑂ 25
Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225

Data from GitHub · snapshot Nov 16, 2026