🏆 #1,604 overall#30 of 178 in Tokenization & Preprocessing

wangfenjin /simple

支持中文和拼音的 SQLite fts5 全文搜索扩展 | A SQLite3 fts5 tokenizer which supports Chinese and PinYin

$ git clone https://github.com/wangfenjin/simple.git
GitHub social preview for wangfenjin/simple
Stars
875
875
This week
0 ★
+5 this month
Forks
113
113
Language
C++
License
Other
Created
Mar 3, 2020
6.7 years old
Last push
May 17, 2026

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 870▲ +5★ 875

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

risesoft-y9/Data-Labeling

▲ +1 ★ this week

数据标注是一款专门对文本数据进行处理和标注的工具,通过简化快捷的文本标注流程和动态的算法反馈,支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间。数据标注的过程先由人工标注构建基础,再由自动标注反哺人工标注,最后由人工标注进行纠偏,从而大幅度提高标注的精准度和高效性。数据标注需要依赖开源的数字底座进行人员岗位管控。

JavaGPL-3.0updated Jun 23, 2025
GitHub ↗★ 702⑂ 106
Tokenization & Preprocessing

theseer/tokenizer

▲ +2 ★ this week

A small library for converting tokenized PHP source code into XML (and potentially other formats)

PHPOtherupdated Feb 3, 2026
GitHub ↗★ 5.2K⑂ 25
Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

andialbrecht/sqlparse

A non-validating SQL parser module for Python

PythonBSD-3-Clauseupdated Nov 1, 2026
GitHub ↗★ 4K⑂ 781
Tokenization & Preprocessing

Chevrotain/chevrotain🔥 active

▲ +2 ★ this week

Parser Building Toolkit for JavaScript

TypeScriptApache-2.0updated Nov 14, 2026
GitHub ↗★ 2.8K⑂ 222

Data from GitHub · snapshot Nov 16, 2026