Tokenization & Preprocessing
jshuadvd/LongRoPE
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
๐ค Language Models๐ง Natural Language Processing
Pythonno licenseupdated Jul 20, 2024
๐ #2,248 overall#82 of 148 in Tokenization & Preprocessing๐ฅ active this week
myylogic /cevahir-ai
Full-stack open-source AI engine for building language models โ tokenizer training, transformer architecture, cognitive reasoning and chat pipeline.
$ git clone https://github.com/myylogic/cevahir-ai.gitStars
240
240
Forks
9
9
Language
Python
License
Apache-2.0
Created
Mar 9, 2026
0.5 years old
Last push
Sep 21, 2026
๐ฅ this week
More in Tokenization & Preprocessing
Tokenization & Preprocessing
rasbt/LLMs-from-scratch๐ฅ active
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
๐ค Language Models๐ง Natural Language Processing
Jupyter NotebookOtherupdated Sep 22, 2026
Tokenization & Preprocessing
chizkidd/Karpathy-Neural-Networks-Zero-to-Hero
Detailed python notes & code for lectures and exercises of Andrej Karpathy's course "Neural Networks: Zero to Hero." The course is focused on building neural networks from scratch.
๐ค Language Models๐ง Natural Language Processing
Jupyter NotebookMITupdated Jun 3, 2026
Tokenization & Preprocessing
bminixhofer/zett
Code for Zero-Shot Tokenizer Transfer
๐ค Language Models
Pythonno licenseupdated Jan 14, 2025
Tokenization & Preprocessing
sefineh-ai/Amharic-Tokenizer๐ฅ active
Syllable-aware BPE tokenizer for the Amharic language (แ แแญแ) โ fast, accurate, trainable.
๐ง Natural Language Processing
PythonMITupdated Sep 24, 2026
Tokenization & Preprocessing
nlp-uoregon/trankit
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
๐ค Language Models๐ง Natural Language Processing
PythonApache-2.0updated Jul 22, 2025
Data from GitHub ยท snapshot Sep 24, 2026