Fast and customizable text tokenization library with BPE and SentencePiece support
$ git clone https://github.com/OpenNMT/Tokenizer.gitToolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Neural machine translation and sequence learning using TensorFlow
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
Cross platform C++ library focusing on optimized machine translation on the consumer-grade device.
Python port of Moses tokenizer, truecaser and normalizer
Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks.
Data from GitHub · snapshot Sep 24, 2026