一个轻量且功能全面的中文分词器，帮助学生了解分词器的工作原理。MicroTokenizer: A lightweight Chinese tokenizer designed for educational and research purposes. Provides a practical, hands-on approach to understanding NLP concepts, featuring multiple tokenization algorithms and customizable models. Ideal for students, researchers, and NLP enthusiasts..
$ git clone https://github.com/howl-anderson/MicroTokenizer.gitHigh performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
A small library for converting tokenized PHP source code into XML (and potentially other formats)
Language model tokenization at GB/s
A non-validating SQL parser module for Python
Parser Building Toolkit for JavaScript
Data from GitHub · snapshot Sep 24, 2026