Machine Translation
hplt-project/sacremoses
Python port of Moses tokenizer, truecaser and normalizer
✂️ Tokenization & Preprocessing🧠 Natural Language Processing
PythonMITupdated Sep 7, 2026
🏆 #2,913 overall#96 of 105 in Machine Translation
kakaobrain /kortok
The code and models for "An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks" (AACL-IJCNLP 2020)
$ git clone https://github.com/kakaobrain/kortok.gitStars
119
119
Forks
10
10
Language
Python
License
Apache-2.0
Created
Oct 5, 2020
6.0 years old
Last push
Oct 8, 2020
More in Machine Translation
Machine Translation
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
✂️ Tokenization & Preprocessing🧠 Natural Language Processing
C++MITupdated Jan 10, 2026
Machine Translation
uzaymacar/attention-mechanisms
Implementations for a family of attention mechanisms, suitable for all kinds of natural language processing tasks and compatible with TensorFlow 2.0 and Keras.
🤖 Language Models🧠 Natural Language Processing
PythonMITupdated Feb 6, 2024
Machine Translation
adobe/NLP-Cube
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
⛏️ Information Extraction✂️ Tokenization & Preprocessing
HTMLApache-2.0updated Nov 3, 2024
Machine Translation
sebastianruder/NLP-progress
Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks.
🏷️ Named Entity Recognition🧠 Natural Language Processing
PythonMITupdated Jul 28, 2024
Machine Translation
NVIDIA-NeMo/Speech🔥 active
A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
PythonApache-2.0updated Sep 23, 2026
Data from GitHub · snapshot Sep 24, 2026