Repository navigation

All pull requests

Labels (9)Milestones (0)New pull request

Open

(25)

Closed

(42)

fix: correct BPE merge annotations in test_wikipedia_example
#111coding-shalabh opened Mar 15, 20261 task done
Fix typos in exercise instructions
#107timlt3 opened Jan 19, 2026
add parallel tokenizer
#99NinjaKelly opened Sep 20, 2025
1
Extract best-pair selection from _encode_chunk into find_best_pair() for clarity
#97manoogim opened Aug 7, 2025
add lexicographic ordering for breaking ties to make the tokenizer deterministic
#90dapopov-st opened Sep 21, 2024
Add pyproject toml
#89gianlucagiudice opened Sep 17, 2024
updating stats across merge to reduce computation
#88imdaredevil opened Sep 1, 2024
Update README.md
#86wenming-ma opened Jul 1, 2024
Optimal algorithm for _encode_chunk(): 20% faster encoding, with 0.5% better COMPRESSION
#84Majdoddin opened Jun 17, 2024
Deduplication of text chunks with frequency count, training and encoding 5x speedup
#82Majdoddin opened Jun 8, 2024
4
calling len(ids) in merge() function only once to increase performance
#76crpatil1901 opened May 14, 2024
Link to Mojo port added
#75dorjeduck opened May 14, 2024
Count only nonoverlapping occurences of a pair
#72Majdoddin opened May 6, 2024
1
Update regex.py to correctly parse scripts with combining marks
#71ajaykg opened May 5, 2024
5
Updated decode() method in GPT4Tokenizer so that it handles special t…
#63Vakarva opened Apr 7, 2024
Handle error when running out of pairs to merge
#54vinhdq842 opened Mar 6, 2024
updated self.vocab initialization and reuse self._build_vocab()
#53muerghq opened Mar 2, 2024
Video2Post Generation Workflow
#49xihajun opened Feb 26, 20248 of 17 tasks
3
Update lecture.md based on video tutorial content from 08:15 through 28:23
#42astaff opened Feb 23, 2024
Automating testing using github actions
#41nobleaustine opened Feb 22, 2024Contributor
Create setup.py
#39Biswajit2902 opened Feb 22, 2024
1
Train BasicTokenizer on GPU with PyTorch, 100x speedup
#38kuprel opened Feb 22, 2024
5
Fix small typos
#34ggaughan opened Feb 21, 2024
1
Simplify generation of printable representation
#26SultanOrazbayev opened Feb 21, 2024
2
Batch encoding decoding
#22ezzaldeeen opened Feb 20, 2024
2

Pull requests