High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
$ git clone https://github.com/lionsoul2014/friso.gitไธไธช่ฝป้ไธๅ่ฝๅ จ้ข็ไธญๆๅ่ฏๅจ๏ผๅธฎๅฉๅญฆ็ไบ่งฃๅ่ฏๅจ็ๅทฅไฝๅ็ใMicroTokenizer: A lightweight Chinese tokenizer designed for educational and research purposes. Provides a practical, hands-on approach to understanding NLP concepts, featuring multiple tokenization algorithms and customizable models. Ideal for students, researchers, and NLP enthusiasts..
Open Korean Text Processor - An Open-source Korean Text Processor
์ด๊ฒฝ๋ ํ๊ตญ์ด ํํ์ ๋ถ์๊ธฐ โ 1MB ๋ชจ๋ธ๋ก ๋ธ๋ผ์ฐ์ ์์ ์คํ(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
A small library for converting tokenized PHP source code into XML (and potentially other formats)
Language model tokenization at GB/s
A non-validating SQL parser module for Python
Data from GitHub ยท snapshot Sep 24, 2026