Tokenization & Preprocessing
daac-tools/vaporetto
β² +2 β this week
π₯ Vaporetto: Very accelerated pointwise prediction based tokenizer
π§ Natural Language Processing
RustApache-2.0updated Nov 3, 2026
π€ vibrato: Viterbi-based accelerated tokenizer
$ git clone https://github.com/daac-tools/vibrato.gitβ² +2 β this week
π₯ Vaporetto: Very accelerated pointwise prediction based tokenizer
β² +1 β this week
Self-contained Japanese Morphological Analyzer written in pure Go
Juman++ (a Morphological Analyzer Toolkit)
β² +1 β this week
μ΄κ²½λ νκ΅μ΄ ννμ λΆμκΈ° β 1MB λͺ¨λΈλ‘ λΈλΌμ°μ μμ μ€ν(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
β² +2 β this week
Sudachi in Rust π¦ and new generation of SudachiPy
β² +2 β this week
Language model tokenization at GB/s
Data from GitHub Β· snapshot Nov 16, 2026