TokenizerChanger
- TokenizerChanger package
- Submodules
- TokenizerChanger.tokenizer_changer module
TokenizerChangerTokenizerChanger.add_merges()TokenizerChanger.add_token_suggestion()TokenizerChanger.add_tokens()TokenizerChanger.baseline_tokenizerTokenizerChanger.delete_inappropriate_merges()TokenizerChanger.delete_k_tail_tokens()TokenizerChanger.delete_merges()TokenizerChanger.delete_overlaps()TokenizerChanger.delete_tokens()TokenizerChanger.find_tail_tokens()TokenizerChanger.find_token_id_gap()TokenizerChanger.find_tokens()TokenizerChanger.format_merges()TokenizerChanger.get_overlapping_merges()TokenizerChanger.get_overlapping_tokens()TokenizerChanger.load_tokenizer()TokenizerChanger.none_keysTokenizerChanger.replace_tokens()TokenizerChanger.save_tokenizer()TokenizerChanger.set_space_marker()TokenizerChanger.set_space_sign()TokenizerChanger.space_markerTokenizerChanger.tokens_to_deleteTokenizerChanger.updated_tokenizer()
- Module contents
TokenizerChangerTokenizerChanger.add_merges()TokenizerChanger.add_token_suggestion()TokenizerChanger.add_tokens()TokenizerChanger.baseline_tokenizerTokenizerChanger.delete_inappropriate_merges()TokenizerChanger.delete_k_tail_tokens()TokenizerChanger.delete_merges()TokenizerChanger.delete_overlaps()TokenizerChanger.delete_tokens()TokenizerChanger.find_tail_tokens()TokenizerChanger.find_token_id_gap()TokenizerChanger.find_tokens()TokenizerChanger.format_merges()TokenizerChanger.get_overlapping_merges()TokenizerChanger.get_overlapping_tokens()TokenizerChanger.load_tokenizer()TokenizerChanger.none_keysTokenizerChanger.replace_tokens()TokenizerChanger.save_tokenizer()TokenizerChanger.set_space_marker()TokenizerChanger.set_space_sign()TokenizerChanger.space_markerTokenizerChanger.tokens_to_deleteTokenizerChanger.updated_tokenizer()