TokenizerChanger documentation
==============================


.. toctree::
   :maxdepth: 2
   :caption: Contents:

   modules

About
==================

Python script for manipulating the existing tokenizer.

The solution was tested on Llama3-8B tokenizer.

Installation
==================

.. code-block:: bash

    pip install TokenizerChanger


Requirements
==================
- Python 3.9+
- tokenizers>=0.21.0
- transformers>=4.47.0
- tqdm>=4.66.4

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`