Tokenization & Preprocessing
bitextor/bitextor
Bitextor generates translation memories from multilingual websites
🌍 Machine Translation
PythonGPL-3.0updated Nov 11, 2024
🏆 #1,264 overall#27 of 148 in Tokenization & Preprocessing
BLKSerene /Wordless
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
$ git clone https://github.com/BLKSerene/Wordless.gitStars
758
758
Forks
99
99
Language
Python
License
GPL-3.0
Created
Aug 23, 2018
8.1 years old
Last push
Oct 30, 2025
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
adbar/simplemma🔥 active
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
🧠 Natural Language Processing
PythonMITupdated Sep 21, 2026
Tokenization & Preprocessing
Dadmatech/DadmaTools
DadmaTools is a Persian NLP tools developed by Dadmatech Co.
🏷️ Named Entity Recognition🧠 Natural Language Processing
PythonApache-2.0updated Aug 14, 2025
Tokenization & Preprocessing
adbar/trafilatura🔥 active
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
🧠 Natural Language Processing💎 Text Mining
PythonApache-2.0updated Sep 21, 2026
Tokenization & Preprocessing
Tokenization & Preprocessing
adobe/NLP-Cube
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
⛏️ Information Extraction🌍 Machine Translation
HTMLApache-2.0updated Nov 3, 2024
Data from GitHub · snapshot Sep 24, 2026