1.7b-MixtureVitae-web_instruct-100BT / tokenizer_config.json
Harsh1729's picture
Initial upload from /leonardo_work/AIFAC_L01_028/hraj0000/megatron_lm_reference/converted_checkpoints/1.7b_data-MixtureVitae-300BT-web_instruct-100BT/hf/iter_0024221
fe1f3e0 verified
raw
history blame contribute delete
156 Bytes
{"unk_token": "<|endoftext|>", "bos_token": "<|endoftext|>", "eos_token": "<|endoftext|>", "add_prefix_space": false, "tokenizer_class": "GPTNeoXTokenizer"}