Machine Translation
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
โ๏ธ Tokenization & Preprocessing๐ง Natural Language Processing
C++MITupdated Jan 10, 2026
๐ #1,579 overall#42 of 105 in Machine Translation
hplt-project /sacremoses
Python port of Moses tokenizer, truecaser and normalizer
$ git clone https://github.com/hplt-project/sacremoses.gitStars
497
497
Forks
59
59
Language
Python
License
MIT
Created
Apr 20, 2018
8.4 years old
Last push
Sep 7, 2026
Categories
GitHub topics
More in Machine Translation
Machine Translation
argosopentech/argos-translate
Open-source offline translation library written in Python
๐ง Natural Language Processing
PythonMITupdated Aug 8, 2026
Machine Translation
CLUEbenchmark/CLUEDatasetSearch
ๆ็ดขๆๆไธญๆNLPๆฐๆฎ้๏ผ้ๅธธ็จ่ฑๆNLPๆฐๆฎ้
๐ Sentiment Analysis๐ท๏ธ Named Entity Recognition๐ Text Summarization๐๏ธ Text Classification๐ง Natural Language Processing
Pythonno licenseupdated Nov 21, 2022
Machine Translation
JohnSnowLabs/spark-nlp๐ฅ active
State of the Art Natural Language Processing
๐ Sentiment Analysis๐ท๏ธ Named Entity Recognition๐๏ธ Text Classificationโ Question Answering๐ง Natural Language Processing
ScalaApache-2.0updated Sep 24, 2026
Machine Translation
tensorflow/lingvo
Lingvo
๐๏ธ Speech Recognition๐ค Language Models๐ง Natural Language Processing
PythonApache-2.0updated Jun 22, 2026
Machine Translation
allenai/RL4LMs
A modular RL library to fine-tune language models to human preferences
๐ Text Summarization๐ง Natural Language Processing
PythonApache-2.0updated Mar 1, 2024
Data from GitHub ยท snapshot Sep 24, 2026