TokenizerChanger documentation
Contents:
About
Python script for manipulating the existing tokenizer.
The solution was tested on Llama3-8B tokenizer.
Installation
pip install TokenizerChanger
Requirements
Python 3.9+
tokenizers>=0.21.0
transformers>=4.47.0
tqdm>=4.66.4