Add llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123 LoRA model

.gitattributes

@@ -45,3 +45,5 @@ llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0002_data_size1000_max_
 llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0002_data_size1000_max_steps=500_seed=123/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr5e-05_data_size1000_max_steps=500_seed=123/checkpoint-234/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr5e-05_data_size1000_max_steps=500_seed=123/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/tokenizer.json filter=lfs diff=lfs merge=lfs -text

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/README.md

@@ -0,0 +1,202 @@
+---
+base_model: meta-llama/Llama-3.1-8B-Instruct
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.1

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/adapter_config.json

@@ -0,0 +1,39 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "down_proj",
+    "v_proj",
+    "o_proj",
+    "k_proj",
+    "up_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89a490fbab4bf72d16ed6c346c41d8eb1f74c40c04d7dd7cade4cc74fa6b2bea
+size 167832240

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/README.md

@@ -0,0 +1,202 @@
+---
+base_model: meta-llama/Llama-3.1-8B-Instruct
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.1

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/adapter_config.json

@@ -0,0 +1,39 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "down_proj",
+    "v_proj",
+    "o_proj",
+    "k_proj",
+    "up_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89a490fbab4bf72d16ed6c346c41d8eb1f74c40c04d7dd7cade4cc74fa6b2bea
+size 167832240

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17f6f4891734ba7b235960a86bd5ece117874fdca0daee7a07467509c963cb9d
+size 335922386

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8bee392b92208cf1f877ac974af60e08f54e728de0e26d7e8a126a0e4f5de132
+size 14244

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4babda782745ed577ce5433176de05b6f42769f421fa14e1b4f3eb312fc3a8da
+size 1064

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/special_tokens_map.json

@@ -0,0 +1,17 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|eot_id|>"
+}

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
+size 17209920

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/tokenizer_config.json

@@ -0,0 +1,2064 @@
+{
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|begin_of_text|>",
+  "chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n    {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n    {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n    {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n    {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n    {%- set system_message = messages[0]['content']|trim %}\n    {%- set messages = messages[1:] %}\n{%- else %}\n    {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n    {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n    {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n    {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n    {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n    {{- \"Do not use variables.\\n\\n\" }}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n    {#- Extract the first user message so we can plug it in here #}\n    {%- if messages | length != 0 %}\n        {%- set first_user_message = messages[0]['content']|trim %}\n        {%- set messages = messages[1:] %}\n    {%- else %}\n        {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n    {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n    {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n    {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n    {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n    {{- \"Do not use variables.\\n\\n\" }}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n    {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n    {%- elif 'tool_calls' in message %}\n        {%- if not message.tool_calls|length == 1 %}\n            {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n        {%- endif %}\n        {%- set tool_call = message.tool_calls[0].function %}\n        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n            {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n            {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n            {%- for arg_name, arg_val in tool_call.arguments | items %}\n                {{- arg_name + '=\"' + arg_val + '\"' }}\n                {%- if not loop.last %}\n                    {{- \", \" }}\n                {%- endif %}\n                {%- endfor %}\n            {{- \")\" }}\n        {%- else  %}\n            {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n            {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n            {{- '\"parameters\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- \"}\" }}\n        {%- endif %}\n        {%- if builtin_tools is defined %}\n            {#- This means we're in ipython mode #}\n            {{- \"<|eom_id|>\" }}\n        {%- else %}\n            {{- \"<|eot_id|>\" }}\n        {%- endif %}\n    {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n        {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n        {%- if message.content is mapping or message.content is iterable %}\n            {{- message.content | tojson }}\n        {%- else %}\n            {{- message.content }}\n        {%- endif %}\n        {{- \"<|eot_id|>\" }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "pad_token": "<|eot_id|>",
+  "tokenizer_class": "PreTrainedTokenizer"
+}

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/trainer_state.json

@@ -0,0 +1,2933 @@
+{
+  "best_global_step": 340,
+  "best_metric": 9.164205039269291e-06,
+  "best_model_checkpoint": "/leonardo_scratch/fast/IscrC_dynaLORA/moose_experts_loras/web_of_lies/llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340",
+  "epoch": 5.4,
+  "eval_steps": 2,
+  "global_step": 340,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.032,
+      "grad_norm": 1.5802136659622192,
+      "learning_rate": 2.0000000000000003e-06,
+      "loss": 0.3254,
+      "step": 2
+    },
+    {
+      "epoch": 0.032,
+      "eval_loss": 0.34003403782844543,
+      "eval_mean_token_accuracy": 0.79,
+      "eval_num_tokens": 3788.0,
+      "eval_runtime": 11.4868,
+      "eval_samples_per_second": 17.411,
+      "eval_steps_per_second": 17.411,
+      "step": 2
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 1.5083140134811401,
+      "learning_rate": 6e-06,
+      "loss": 0.3662,
+      "step": 4
+    },
+    {
+      "epoch": 0.064,
+      "eval_loss": 0.3350926339626312,
+      "eval_mean_token_accuracy": 0.795,
+      "eval_num_tokens": 7870.0,
+      "eval_runtime": 11.5982,
+      "eval_samples_per_second": 17.244,
+      "eval_steps_per_second": 17.244,
+      "step": 4
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 2.1517977714538574,
+      "learning_rate": 1e-05,
+      "loss": 0.3832,
+      "step": 6
+    },
+    {
+      "epoch": 0.096,
+      "eval_loss": 0.3295806050300598,
+      "eval_mean_token_accuracy": 0.7875,
+      "eval_num_tokens": 11870.0,
+      "eval_runtime": 11.4814,
+      "eval_samples_per_second": 17.42,
+      "eval_steps_per_second": 17.42,
+      "step": 6
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 4.416848182678223,
+      "learning_rate": 1.4000000000000001e-05,
+      "loss": 0.3482,
+      "step": 8
+    },
+    {
+      "epoch": 0.128,
+      "eval_loss": 0.32734423875808716,
+      "eval_mean_token_accuracy": 0.78,
+      "eval_num_tokens": 15870.0,
+      "eval_runtime": 11.583,
+      "eval_samples_per_second": 17.267,
+      "eval_steps_per_second": 17.267,
+      "step": 8
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 1.6268564462661743,
+      "learning_rate": 1.8e-05,
+      "loss": 0.31,
+      "step": 10
+    },
+    {
+      "epoch": 0.16,
+      "eval_loss": 0.3259817063808441,
+      "eval_mean_token_accuracy": 0.785,
+      "eval_num_tokens": 20038.0,
+      "eval_runtime": 11.6132,
+      "eval_samples_per_second": 17.222,
+      "eval_steps_per_second": 17.222,
+      "step": 10
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 2.735539674758911,
+      "learning_rate": 2.2000000000000003e-05,
+      "loss": 0.3275,
+      "step": 12
+    },
+    {
+      "epoch": 0.192,
+      "eval_loss": 0.32298287749290466,
+      "eval_mean_token_accuracy": 0.7775,
+      "eval_num_tokens": 24242.0,
+      "eval_runtime": 11.4723,
+      "eval_samples_per_second": 17.433,
+      "eval_steps_per_second": 17.433,
+      "step": 12
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 1.3300482034683228,
+      "learning_rate": 2.6000000000000002e-05,
+      "loss": 0.3719,
+      "step": 14
+    },
+    {
+      "epoch": 0.224,
+      "eval_loss": 0.3237625062465668,
+      "eval_mean_token_accuracy": 0.79,
+      "eval_num_tokens": 28448.0,
+      "eval_runtime": 11.5436,
+      "eval_samples_per_second": 17.326,
+      "eval_steps_per_second": 17.326,
+      "step": 14
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 1.247140884399414,
+      "learning_rate": 3e-05,
+      "loss": 0.3338,
+      "step": 16
+    },
+    {
+      "epoch": 0.256,
+      "eval_loss": 0.3177359104156494,
+      "eval_mean_token_accuracy": 0.79,
+      "eval_num_tokens": 32722.0,
+      "eval_runtime": 11.6243,
+      "eval_samples_per_second": 17.205,
+      "eval_steps_per_second": 17.205,
+      "step": 16
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 2.2500052452087402,
+      "learning_rate": 3.4000000000000007e-05,
+      "loss": 0.317,
+      "step": 18
+    },
+    {
+      "epoch": 0.288,
+      "eval_loss": 0.3118344247341156,
+      "eval_mean_token_accuracy": 0.81,
+      "eval_num_tokens": 36576.0,
+      "eval_runtime": 11.4432,
+      "eval_samples_per_second": 17.478,
+      "eval_steps_per_second": 17.478,
+      "step": 18
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 1.9043917655944824,
+      "learning_rate": 3.8e-05,
+      "loss": 0.3333,
+      "step": 20
+    },
+    {
+      "epoch": 0.32,
+      "eval_loss": 0.3012201189994812,
+      "eval_mean_token_accuracy": 0.8175,
+      "eval_num_tokens": 40666.0,
+      "eval_runtime": 11.4526,
+      "eval_samples_per_second": 17.463,
+      "eval_steps_per_second": 17.463,
+      "step": 20
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 1.2278327941894531,
+      "learning_rate": 4.2e-05,
+      "loss": 0.3112,
+      "step": 22
+    },
+    {
+      "epoch": 0.352,
+      "eval_loss": 0.30098074674606323,
+      "eval_mean_token_accuracy": 0.7775,
+      "eval_num_tokens": 44726.0,
+      "eval_runtime": 11.5577,
+      "eval_samples_per_second": 17.304,
+      "eval_steps_per_second": 17.304,
+      "step": 22
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 2.695115327835083,
+      "learning_rate": 4.600000000000001e-05,
+      "loss": 0.2945,
+      "step": 24
+    },
+    {
+      "epoch": 0.384,
+      "eval_loss": 0.2975901663303375,
+      "eval_mean_token_accuracy": 0.7775,
+      "eval_num_tokens": 48682.0,
+      "eval_runtime": 11.5218,
+      "eval_samples_per_second": 17.358,
+      "eval_steps_per_second": 17.358,
+      "step": 24
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 4.567770004272461,
+      "learning_rate": 5e-05,
+      "loss": 0.2739,
+      "step": 26
+    },
+    {
+      "epoch": 0.416,
+      "eval_loss": 0.29053255915641785,
+      "eval_mean_token_accuracy": 0.82,
+      "eval_num_tokens": 52620.0,
+      "eval_runtime": 11.4284,
+      "eval_samples_per_second": 17.5,
+      "eval_steps_per_second": 17.5,
+      "step": 26
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 2.587909698486328,
+      "learning_rate": 5.4000000000000005e-05,
+      "loss": 0.2695,
+      "step": 28
+    },
+    {
+      "epoch": 0.448,
+      "eval_loss": 0.2993077039718628,
+      "eval_mean_token_accuracy": 0.83,
+      "eval_num_tokens": 56574.0,
+      "eval_runtime": 11.485,
+      "eval_samples_per_second": 17.414,
+      "eval_steps_per_second": 17.414,
+      "step": 28
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 7.742918014526367,
+      "learning_rate": 5.8e-05,
+      "loss": 0.3712,
+      "step": 30
+    },
+    {
+      "epoch": 0.48,
+      "eval_loss": 0.28641292452812195,
+      "eval_mean_token_accuracy": 0.815,
+      "eval_num_tokens": 60662.0,
+      "eval_runtime": 11.5022,
+      "eval_samples_per_second": 17.388,
+      "eval_steps_per_second": 17.388,
+      "step": 30
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 2.3204309940338135,
+      "learning_rate": 6.2e-05,
+      "loss": 0.2859,
+      "step": 32
+    },
+    {
+      "epoch": 0.512,
+      "eval_loss": 0.29522469639778137,
+      "eval_mean_token_accuracy": 0.81,
+      "eval_num_tokens": 64892.0,
+      "eval_runtime": 11.3681,
+      "eval_samples_per_second": 17.593,
+      "eval_steps_per_second": 17.593,
+      "step": 32
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 1.2671048641204834,
+      "learning_rate": 6.6e-05,
+      "loss": 0.3264,
+      "step": 34
+    },
+    {
+      "epoch": 0.544,
+      "eval_loss": 0.29506978392601013,
+      "eval_mean_token_accuracy": 0.8025,
+      "eval_num_tokens": 69050.0,
+      "eval_runtime": 11.4358,
+      "eval_samples_per_second": 17.489,
+      "eval_steps_per_second": 17.489,
+      "step": 34
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 2.1717748641967773,
+      "learning_rate": 7e-05,
+      "loss": 0.309,
+      "step": 36
+    },
+    {
+      "epoch": 0.576,
+      "eval_loss": 0.28752461075782776,
+      "eval_mean_token_accuracy": 0.8075,
+      "eval_num_tokens": 73246.0,
+      "eval_runtime": 11.402,
+      "eval_samples_per_second": 17.541,
+      "eval_steps_per_second": 17.541,
+      "step": 36
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 1.0603128671646118,
+      "learning_rate": 7.4e-05,
+      "loss": 0.302,
+      "step": 38
+    },
+    {
+      "epoch": 0.608,
+      "eval_loss": 0.28400492668151855,
+      "eval_mean_token_accuracy": 0.7975,
+      "eval_num_tokens": 77564.0,
+      "eval_runtime": 11.4338,
+      "eval_samples_per_second": 17.492,
+      "eval_steps_per_second": 17.492,
+      "step": 38
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 2.5529773235321045,
+      "learning_rate": 7.800000000000001e-05,
+      "loss": 0.3403,
+      "step": 40
+    },
+    {
+      "epoch": 0.64,
+      "eval_loss": 0.2851684093475342,
+      "eval_mean_token_accuracy": 0.8225,
+      "eval_num_tokens": 81926.0,
+      "eval_runtime": 11.4115,
+      "eval_samples_per_second": 17.526,
+      "eval_steps_per_second": 17.526,
+      "step": 40
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 1.311371922492981,
+      "learning_rate": 8.2e-05,
+      "loss": 0.2668,
+      "step": 42
+    },
+    {
+      "epoch": 0.672,
+      "eval_loss": 0.29013916850090027,
+      "eval_mean_token_accuracy": 0.8175,
+      "eval_num_tokens": 86010.0,
+      "eval_runtime": 11.4191,
+      "eval_samples_per_second": 17.515,
+      "eval_steps_per_second": 17.515,
+      "step": 42
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 1.3436343669891357,
+      "learning_rate": 8.6e-05,
+      "loss": 0.3137,
+      "step": 44
+    },
+    {
+      "epoch": 0.704,
+      "eval_loss": 0.28718769550323486,
+      "eval_mean_token_accuracy": 0.78,
+      "eval_num_tokens": 90162.0,
+      "eval_runtime": 11.472,
+      "eval_samples_per_second": 17.434,
+      "eval_steps_per_second": 17.434,
+      "step": 44
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 1.3073921203613281,
+      "learning_rate": 9e-05,
+      "loss": 0.262,
+      "step": 46
+    },
+    {
+      "epoch": 0.736,
+      "eval_loss": 0.330881804227829,
+      "eval_mean_token_accuracy": 0.7925,
+      "eval_num_tokens": 94218.0,
+      "eval_runtime": 11.4956,
+      "eval_samples_per_second": 17.398,
+      "eval_steps_per_second": 17.398,
+      "step": 46
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 1.8404035568237305,
+      "learning_rate": 9.4e-05,
+      "loss": 0.2322,
+      "step": 48
+    },
+    {
+      "epoch": 0.768,
+      "eval_loss": 0.3728793263435364,
+      "eval_mean_token_accuracy": 0.79,
+      "eval_num_tokens": 98112.0,
+      "eval_runtime": 11.4707,
+      "eval_samples_per_second": 17.436,
+      "eval_steps_per_second": 17.436,
+      "step": 48
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 1.3995583057403564,
+      "learning_rate": 9.8e-05,
+      "loss": 0.3132,
+      "step": 50
+    },
+    {
+      "epoch": 0.8,
+      "eval_loss": 0.2860886752605438,
+      "eval_mean_token_accuracy": 0.7775,
+      "eval_num_tokens": 102216.0,
+      "eval_runtime": 11.4864,
+      "eval_samples_per_second": 17.412,
+      "eval_steps_per_second": 17.412,
+      "step": 50
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 2.288776159286499,
+      "learning_rate": 9.999878153526974e-05,
+      "loss": 0.2926,
+      "step": 52
+    },
+    {
+      "epoch": 0.832,
+      "eval_loss": 0.3439726233482361,
+      "eval_mean_token_accuracy": 0.815,
+      "eval_num_tokens": 106396.0,
+      "eval_runtime": 11.4605,
+      "eval_samples_per_second": 17.451,
+      "eval_steps_per_second": 17.451,
+      "step": 52
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 1.2774162292480469,
+      "learning_rate": 9.998903417374228e-05,
+      "loss": 0.3251,
+      "step": 54
+    },
+    {
+      "epoch": 0.864,
+      "eval_loss": 0.27272847294807434,
+      "eval_mean_token_accuracy": 0.8225,
+      "eval_num_tokens": 110438.0,
+      "eval_runtime": 11.4308,
+      "eval_samples_per_second": 17.497,
+      "eval_steps_per_second": 17.497,
+      "step": 54
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 1.7763952016830444,
+      "learning_rate": 9.99695413509548e-05,
+      "loss": 0.3134,
+      "step": 56
+    },
+    {
+      "epoch": 0.896,
+      "eval_loss": 0.26282021403312683,
+      "eval_mean_token_accuracy": 0.8125,
+      "eval_num_tokens": 114626.0,
+      "eval_runtime": 11.4799,
+      "eval_samples_per_second": 17.422,
+      "eval_steps_per_second": 17.422,
+      "step": 56
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 0.6160733699798584,
+      "learning_rate": 9.99403068670717e-05,
+      "loss": 0.2437,
+      "step": 58
+    },
+    {
+      "epoch": 0.928,
+      "eval_loss": 0.2630641758441925,
+      "eval_mean_token_accuracy": 0.8225,
+      "eval_num_tokens": 118598.0,
+      "eval_runtime": 11.4821,
+      "eval_samples_per_second": 17.418,
+      "eval_steps_per_second": 17.418,
+      "step": 58
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 2.4143295288085938,
+      "learning_rate": 9.990133642141359e-05,
+      "loss": 0.3038,
+      "step": 60
+    },
+    {
+      "epoch": 0.96,
+      "eval_loss": 0.2610669732093811,
+      "eval_mean_token_accuracy": 0.84,
+      "eval_num_tokens": 122706.0,
+      "eval_runtime": 11.4326,
+      "eval_samples_per_second": 17.494,
+      "eval_steps_per_second": 17.494,
+      "step": 60
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 0.7448087334632874,
+      "learning_rate": 9.985263761134602e-05,
+      "loss": 0.288,
+      "step": 62
+    },
+    {
+      "epoch": 0.992,
+      "eval_loss": 0.3012274503707886,
+      "eval_mean_token_accuracy": 0.7925,
+      "eval_num_tokens": 126668.0,
+      "eval_runtime": 11.4687,
+      "eval_samples_per_second": 17.439,
+      "eval_steps_per_second": 17.439,
+      "step": 62
+    },
+    {
+      "epoch": 1.016,
+      "grad_norm": 1.1838434934616089,
+      "learning_rate": 9.979421993079852e-05,
+      "loss": 0.2814,
+      "step": 64
+    },
+    {
+      "epoch": 1.016,
+      "eval_loss": 0.2992355227470398,
+      "eval_mean_token_accuracy": 0.795,
+      "eval_num_tokens": 129714.0,
+      "eval_runtime": 11.436,
+      "eval_samples_per_second": 17.489,
+      "eval_steps_per_second": 17.489,
+      "step": 64
+    },
+    {
+      "epoch": 1.048,
+      "grad_norm": 0.8440607190132141,
+      "learning_rate": 9.972609476841367e-05,
+      "loss": 0.3005,
+      "step": 66
+    },
+    {
+      "epoch": 1.048,
+      "eval_loss": 0.25750571489334106,
+      "eval_mean_token_accuracy": 0.82,
+      "eval_num_tokens": 133980.0,
+      "eval_runtime": 11.41,
+      "eval_samples_per_second": 17.528,
+      "eval_steps_per_second": 17.528,
+      "step": 66
+    },
+    {
+      "epoch": 1.08,
+      "grad_norm": 1.609228491783142,
+      "learning_rate": 9.964827540532685e-05,
+      "loss": 0.193,
+      "step": 68
+    },
+    {
+      "epoch": 1.08,
+      "eval_loss": 0.2647652328014374,
+      "eval_mean_token_accuracy": 0.82,
+      "eval_num_tokens": 137840.0,
+      "eval_runtime": 11.4501,
+      "eval_samples_per_second": 17.467,
+      "eval_steps_per_second": 17.467,
+      "step": 68
+    },
+    {
+      "epoch": 1.112,
+      "grad_norm": 1.7681248188018799,
+      "learning_rate": 9.956077701257709e-05,
+      "loss": 0.2691,
+      "step": 70
+    },
+    {
+      "epoch": 1.112,
+      "eval_loss": 0.28832611441612244,
+      "eval_mean_token_accuracy": 0.815,
+      "eval_num_tokens": 142076.0,
+      "eval_runtime": 11.394,
+      "eval_samples_per_second": 17.553,
+      "eval_steps_per_second": 17.553,
+      "step": 70
+    },
+    {
+      "epoch": 1.144,
+      "grad_norm": 6.933201789855957,
+      "learning_rate": 9.946361664814943e-05,
+      "loss": 0.2987,
+      "step": 72
+    },
+    {
+      "epoch": 1.144,
+      "eval_loss": 0.23903293907642365,
+      "eval_mean_token_accuracy": 0.85,
+      "eval_num_tokens": 146130.0,
+      "eval_runtime": 11.3536,
+      "eval_samples_per_second": 17.616,
+      "eval_steps_per_second": 17.616,
+      "step": 72
+    },
+    {
+      "epoch": 1.176,
+      "grad_norm": 2.0388996601104736,
+      "learning_rate": 9.93568132536494e-05,
+      "loss": 0.1763,
+      "step": 74
+    },
+    {
+      "epoch": 1.176,
+      "eval_loss": 0.24082627892494202,
+      "eval_mean_token_accuracy": 0.8475,
+      "eval_num_tokens": 150152.0,
+      "eval_runtime": 11.3628,
+      "eval_samples_per_second": 17.601,
+      "eval_steps_per_second": 17.601,
+      "step": 74
+    },
+    {
+      "epoch": 1.208,
+      "grad_norm": 2.7526791095733643,
+      "learning_rate": 9.924038765061042e-05,
+      "loss": 0.2776,
+      "step": 76
+    },
+    {
+      "epoch": 1.208,
+      "eval_loss": 0.2463812232017517,
+      "eval_mean_token_accuracy": 0.8375,
+      "eval_num_tokens": 154218.0,
+      "eval_runtime": 11.3974,
+      "eval_samples_per_second": 17.548,
+      "eval_steps_per_second": 17.548,
+      "step": 76
+    },
+    {
+      "epoch": 1.24,
+      "grad_norm": 2.1134414672851562,
+      "learning_rate": 9.911436253643445e-05,
+      "loss": 0.1951,
+      "step": 78
+    },
+    {
+      "epoch": 1.24,
+      "eval_loss": 0.23271098732948303,
+      "eval_mean_token_accuracy": 0.845,
+      "eval_num_tokens": 157958.0,
+      "eval_runtime": 11.427,
+      "eval_samples_per_second": 17.502,
+      "eval_steps_per_second": 17.502,
+      "step": 78
+    },
+    {
+      "epoch": 1.272,
+      "grad_norm": 1.4288508892059326,
+      "learning_rate": 9.89787624799672e-05,
+      "loss": 0.2175,
+      "step": 80
+    },
+    {
+      "epoch": 1.272,
+      "eval_loss": 0.2633565664291382,
+      "eval_mean_token_accuracy": 0.85,
+      "eval_num_tokens": 162208.0,
+      "eval_runtime": 11.4183,
+      "eval_samples_per_second": 17.516,
+      "eval_steps_per_second": 17.516,
+      "step": 80
+    },
+    {
+      "epoch": 1.304,
+      "grad_norm": 4.318436145782471,
+      "learning_rate": 9.88336139167084e-05,
+      "loss": 0.3455,
+      "step": 82
+    },
+    {
+      "epoch": 1.304,
+      "eval_loss": 0.23306137323379517,
+      "eval_mean_token_accuracy": 0.8475,
+      "eval_num_tokens": 166336.0,
+      "eval_runtime": 11.4655,
+      "eval_samples_per_second": 17.444,
+      "eval_steps_per_second": 17.444,
+      "step": 82
+    },
+    {
+      "epoch": 1.336,
+      "grad_norm": 1.1009011268615723,
+      "learning_rate": 9.867894514365802e-05,
+      "loss": 0.1921,
+      "step": 84
+    },
+    {
+      "epoch": 1.336,
+      "eval_loss": 0.23506759107112885,
+      "eval_mean_token_accuracy": 0.85,
+      "eval_num_tokens": 170294.0,
+      "eval_runtime": 11.45,
+      "eval_samples_per_second": 17.467,
+      "eval_steps_per_second": 17.467,
+      "step": 84
+    },
+    {
+      "epoch": 1.3679999999999999,
+      "grad_norm": 0.9112859964370728,
+      "learning_rate": 9.851478631379982e-05,
+      "loss": 0.1739,
+      "step": 86
+    },
+    {
+      "epoch": 1.3679999999999999,
+      "eval_loss": 0.24006782472133636,
+      "eval_mean_token_accuracy": 0.815,
+      "eval_num_tokens": 174026.0,
+      "eval_runtime": 11.4216,
+      "eval_samples_per_second": 17.511,
+      "eval_steps_per_second": 17.511,
+      "step": 86
+    },
+    {
+      "epoch": 1.4,
+      "grad_norm": 0.8116211295127869,
+      "learning_rate": 9.834116943022298e-05,
+      "loss": 0.2797,
+      "step": 88
+    },
+    {
+      "epoch": 1.4,
+      "eval_loss": 0.2299848347902298,
+      "eval_mean_token_accuracy": 0.835,
+      "eval_num_tokens": 178228.0,
+      "eval_runtime": 11.5224,
+      "eval_samples_per_second": 17.357,
+      "eval_steps_per_second": 17.357,
+      "step": 88
+    },
+    {
+      "epoch": 1.432,
+      "grad_norm": 1.4852060079574585,
+      "learning_rate": 9.815812833988291e-05,
+      "loss": 0.2607,
+      "step": 90
+    },
+    {
+      "epoch": 1.432,
+      "eval_loss": 0.24036872386932373,
+      "eval_mean_token_accuracy": 0.845,
+      "eval_num_tokens": 182190.0,
+      "eval_runtime": 11.4941,
+      "eval_samples_per_second": 17.4,
+      "eval_steps_per_second": 17.4,
+      "step": 90
+    },
+    {
+      "epoch": 1.464,
+      "grad_norm": 0.6389371752738953,
+      "learning_rate": 9.796569872700288e-05,
+      "loss": 0.2719,
+      "step": 92
+    },
+    {
+      "epoch": 1.464,
+      "eval_loss": 0.22846919298171997,
+      "eval_mean_token_accuracy": 0.8525,
+      "eval_num_tokens": 186412.0,
+      "eval_runtime": 11.4973,
+      "eval_samples_per_second": 17.395,
+      "eval_steps_per_second": 17.395,
+      "step": 92
+    },
+    {
+      "epoch": 1.496,
+      "grad_norm": 1.8614541292190552,
+      "learning_rate": 9.776391810611718e-05,
+      "loss": 0.2931,
+      "step": 94
+    },
+    {
+      "epoch": 1.496,
+      "eval_loss": 0.21627165377140045,
+      "eval_mean_token_accuracy": 0.835,
+      "eval_num_tokens": 190736.0,
+      "eval_runtime": 11.4916,
+      "eval_samples_per_second": 17.404,
+      "eval_steps_per_second": 17.404,
+      "step": 94
+    },
+    {
+      "epoch": 1.528,
+      "grad_norm": 2.3089845180511475,
+      "learning_rate": 9.755282581475769e-05,
+      "loss": 0.2017,
+      "step": 96
+    },
+    {
+      "epoch": 1.528,
+      "eval_loss": 0.24827617406845093,
+      "eval_mean_token_accuracy": 0.8425,
+      "eval_num_tokens": 194820.0,
+      "eval_runtime": 11.4844,
+      "eval_samples_per_second": 17.415,
+      "eval_steps_per_second": 17.415,
+      "step": 96
+    },
+    {
+      "epoch": 1.56,
+      "grad_norm": 1.1124815940856934,
+      "learning_rate": 9.733246300578483e-05,
+      "loss": 0.2282,
+      "step": 98
+    },
+    {
+      "epoch": 1.56,
+      "eval_loss": 0.2294478565454483,
+      "eval_mean_token_accuracy": 0.86,
+      "eval_num_tokens": 199110.0,
+      "eval_runtime": 11.5211,
+      "eval_samples_per_second": 17.359,
+      "eval_steps_per_second": 17.359,
+      "step": 98
+    },
+    {
+      "epoch": 1.592,
+      "grad_norm": 2.3057920932769775,
+      "learning_rate": 9.710287263936484e-05,
+      "loss": 0.2932,
+      "step": 100
+    },
+    {
+      "epoch": 1.592,
+      "eval_loss": 0.20428486168384552,
+      "eval_mean_token_accuracy": 0.8525,
+      "eval_num_tokens": 203366.0,
+      "eval_runtime": 11.4704,
+      "eval_samples_per_second": 17.436,
+      "eval_steps_per_second": 17.436,
+      "step": 100
+    },
+    {
+      "epoch": 1.624,
+      "grad_norm": 1.4235131740570068,
+      "learning_rate": 9.686409947459458e-05,
+      "loss": 0.2334,
+      "step": 102
+    },
+    {
+      "epoch": 1.624,
+      "eval_loss": 0.20842128992080688,
+      "eval_mean_token_accuracy": 0.8725,
+      "eval_num_tokens": 207652.0,
+      "eval_runtime": 11.5053,
+      "eval_samples_per_second": 17.383,
+      "eval_steps_per_second": 17.383,
+      "step": 102
+    },
+    {
+      "epoch": 1.6560000000000001,
+      "grad_norm": 0.8341189026832581,
+      "learning_rate": 9.661619006077561e-05,
+      "loss": 0.2042,
+      "step": 104
+    },
+    {
+      "epoch": 1.6560000000000001,
+      "eval_loss": 0.18993881344795227,
+      "eval_mean_token_accuracy": 0.87,
+      "eval_num_tokens": 211784.0,
+      "eval_runtime": 11.4668,
+      "eval_samples_per_second": 17.442,
+      "eval_steps_per_second": 17.442,
+      "step": 104
+    },
+    {
+      "epoch": 1.688,
+      "grad_norm": 1.6879816055297852,
+      "learning_rate": 9.635919272833938e-05,
+      "loss": 0.1601,
+      "step": 106
+    },
+    {
+      "epoch": 1.688,
+      "eval_loss": 0.19051876664161682,
+      "eval_mean_token_accuracy": 0.87,
+      "eval_num_tokens": 215766.0,
+      "eval_runtime": 11.4304,
+      "eval_samples_per_second": 17.497,
+      "eval_steps_per_second": 17.497,
+      "step": 106
+    },
+    {
+      "epoch": 1.72,
+      "grad_norm": 2.563009262084961,
+      "learning_rate": 9.609315757942503e-05,
+      "loss": 0.12,
+      "step": 108
+    },
+    {
+      "epoch": 1.72,
+      "eval_loss": 0.23968438804149628,
+      "eval_mean_token_accuracy": 0.8875,
+      "eval_num_tokens": 219608.0,
+      "eval_runtime": 11.4338,
+      "eval_samples_per_second": 17.492,
+      "eval_steps_per_second": 17.492,
+      "step": 108
+    },
+    {
+      "epoch": 1.752,
+      "grad_norm": 4.671048164367676,
+      "learning_rate": 9.581813647811198e-05,
+      "loss": 0.2151,
+      "step": 110
+    },
+    {
+      "epoch": 1.752,
+      "eval_loss": 0.2705950438976288,
+      "eval_mean_token_accuracy": 0.8975,
+      "eval_num_tokens": 223740.0,
+      "eval_runtime": 11.3787,
+      "eval_samples_per_second": 17.577,
+      "eval_steps_per_second": 17.577,
+      "step": 110
+    },
+    {
+      "epoch": 1.784,
+      "grad_norm": 3.9894750118255615,
+      "learning_rate": 9.553418304030886e-05,
+      "loss": 0.1924,
+      "step": 112
+    },
+    {
+      "epoch": 1.784,
+      "eval_loss": 0.24154651165008545,
+      "eval_mean_token_accuracy": 0.885,
+      "eval_num_tokens": 227726.0,
+      "eval_runtime": 11.4077,
+      "eval_samples_per_second": 17.532,
+      "eval_steps_per_second": 17.532,
+      "step": 112
+    },
+    {
+      "epoch": 1.8159999999999998,
+      "grad_norm": 8.894902229309082,
+      "learning_rate": 9.524135262330098e-05,
+      "loss": 0.2466,
+      "step": 114
+    },
+    {
+      "epoch": 1.8159999999999998,
+      "eval_loss": 0.2230948954820633,
+      "eval_mean_token_accuracy": 0.8575,
+      "eval_num_tokens": 231626.0,
+      "eval_runtime": 11.4032,
+      "eval_samples_per_second": 17.539,
+      "eval_steps_per_second": 17.539,
+      "step": 114
+    },
+    {
+      "epoch": 1.8479999999999999,
+      "grad_norm": 8.400607109069824,
+      "learning_rate": 9.493970231495835e-05,
+      "loss": 0.3571,
+      "step": 116
+    },
+    {
+      "epoch": 1.8479999999999999,
+      "eval_loss": 0.20152923464775085,
+      "eval_mean_token_accuracy": 0.855,
+      "eval_num_tokens": 235706.0,
+      "eval_runtime": 11.4361,
+      "eval_samples_per_second": 17.488,
+      "eval_steps_per_second": 17.488,
+      "step": 116
+    },
+    {
+      "epoch": 1.88,
+      "grad_norm": 0.9127203226089478,
+      "learning_rate": 9.462929092260628e-05,
+      "loss": 0.1996,
+      "step": 118
+    },
+    {
+      "epoch": 1.88,
+      "eval_loss": 0.19683368504047394,
+      "eval_mean_token_accuracy": 0.865,
+      "eval_num_tokens": 239652.0,
+      "eval_runtime": 11.429,
+      "eval_samples_per_second": 17.499,
+      "eval_steps_per_second": 17.499,
+      "step": 118
+    },
+    {
+      "epoch": 1.912,
+      "grad_norm": 0.8041687607765198,
+      "learning_rate": 9.431017896156074e-05,
+      "loss": 0.2366,
+      "step": 120
+    },
+    {
+      "epoch": 1.912,
+      "eval_loss": 0.2048008292913437,
+      "eval_mean_token_accuracy": 0.8625,
+      "eval_num_tokens": 243802.0,
+      "eval_runtime": 11.3113,
+      "eval_samples_per_second": 17.681,
+      "eval_steps_per_second": 17.681,
+      "step": 120
+    },
+    {
+      "epoch": 1.944,
+      "grad_norm": 0.8501959443092346,
+      "learning_rate": 9.398242864333083e-05,
+      "loss": 0.2125,
+      "step": 122
+    },
+    {
+      "epoch": 1.944,
+      "eval_loss": 0.20324259996414185,
+      "eval_mean_token_accuracy": 0.8775,
+      "eval_num_tokens": 247894.0,
+      "eval_runtime": 11.4536,
+      "eval_samples_per_second": 17.462,
+      "eval_steps_per_second": 17.462,
+      "step": 122
+    },
+    {
+      "epoch": 1.976,
+      "grad_norm": 0.7903106212615967,
+      "learning_rate": 9.364610386349049e-05,
+      "loss": 0.1722,
+      "step": 124
+    },
+    {
+      "epoch": 1.976,
+      "eval_loss": 0.20135432481765747,
+      "eval_mean_token_accuracy": 0.8725,
+      "eval_num_tokens": 251900.0,
+      "eval_runtime": 11.4609,
+      "eval_samples_per_second": 17.451,
+      "eval_steps_per_second": 17.451,
+      "step": 124
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 1.7705943584442139,
+      "learning_rate": 9.330127018922194e-05,
+      "loss": 0.182,
+      "step": 126
+    },
+    {
+      "epoch": 2.0,
+      "eval_loss": 0.18991686403751373,
+      "eval_mean_token_accuracy": 0.885,
+      "eval_num_tokens": 255064.0,
+      "eval_runtime": 11.5169,
+      "eval_samples_per_second": 17.366,
+      "eval_steps_per_second": 17.366,
+      "step": 126
+    },
+    {
+      "epoch": 2.032,
+      "grad_norm": 1.3064204454421997,
+      "learning_rate": 9.294799484653323e-05,
+      "loss": 0.188,
+      "step": 128
+    },
+    {
+      "epoch": 2.032,
+      "eval_loss": 0.19707800447940826,
+      "eval_mean_token_accuracy": 0.8775,
+      "eval_num_tokens": 258998.0,
+      "eval_runtime": 11.5071,
+      "eval_samples_per_second": 17.381,
+      "eval_steps_per_second": 17.381,
+      "step": 128
+    },
+    {
+      "epoch": 2.064,
+      "grad_norm": 4.833868980407715,
+      "learning_rate": 9.258634670715238e-05,
+      "loss": 0.188,
+      "step": 130
+    },
+    {
+      "epoch": 2.064,
+      "eval_loss": 0.20698383450508118,
+      "eval_mean_token_accuracy": 0.8775,
+      "eval_num_tokens": 263028.0,
+      "eval_runtime": 11.4934,
+      "eval_samples_per_second": 17.401,
+      "eval_steps_per_second": 17.401,
+      "step": 130
+    },
+    {
+      "epoch": 2.096,
+      "grad_norm": 1.8930493593215942,
+      "learning_rate": 9.221639627510076e-05,
+      "loss": 0.1556,
+      "step": 132
+    },
+    {
+      "epoch": 2.096,
+      "eval_loss": 0.17914585769176483,
+      "eval_mean_token_accuracy": 0.9,
+      "eval_num_tokens": 267098.0,
+      "eval_runtime": 11.4799,
+      "eval_samples_per_second": 17.422,
+      "eval_steps_per_second": 17.422,
+      "step": 132
+    },
+    {
+      "epoch": 2.128,
+      "grad_norm": 1.406873106956482,
+      "learning_rate": 9.183821567294809e-05,
+      "loss": 0.1927,
+      "step": 134
+    },
+    {
+      "epoch": 2.128,
+      "eval_loss": 0.1733633428812027,
+      "eval_mean_token_accuracy": 0.9,
+      "eval_num_tokens": 271360.0,
+      "eval_runtime": 11.4998,
+      "eval_samples_per_second": 17.392,
+      "eval_steps_per_second": 17.392,
+      "step": 134
+    },
+    {
+      "epoch": 2.16,
+      "grad_norm": 1.1455000638961792,
+      "learning_rate": 9.145187862775209e-05,
+      "loss": 0.2225,
+      "step": 136
+    },
+    {
+      "epoch": 2.16,
+      "eval_loss": 0.19100509583950043,
+      "eval_mean_token_accuracy": 0.8725,
+      "eval_num_tokens": 275678.0,
+      "eval_runtime": 11.5068,
+      "eval_samples_per_second": 17.381,
+      "eval_steps_per_second": 17.381,
+      "step": 136
+    },
+    {
+      "epoch": 2.192,
+      "grad_norm": 1.0726306438446045,
+      "learning_rate": 9.105746045668521e-05,
+      "loss": 0.1864,
+      "step": 138
+    },
+    {
+      "epoch": 2.192,
+      "eval_loss": 0.248301699757576,
+      "eval_mean_token_accuracy": 0.8625,
+      "eval_num_tokens": 279702.0,
+      "eval_runtime": 11.5267,
+      "eval_samples_per_second": 17.351,
+      "eval_steps_per_second": 17.351,
+      "step": 138
+    },
+    {
+      "epoch": 2.224,
+      "grad_norm": 0.8405861258506775,
+      "learning_rate": 9.065503805235138e-05,
+      "loss": 0.1547,
+      "step": 140
+    },
+    {
+      "epoch": 2.224,
+      "eval_loss": 0.1965908408164978,
+      "eval_mean_token_accuracy": 0.8775,
+      "eval_num_tokens": 283602.0,
+      "eval_runtime": 11.4832,
+      "eval_samples_per_second": 17.417,
+      "eval_steps_per_second": 17.417,
+      "step": 140
+    },
+    {
+      "epoch": 2.2560000000000002,
+      "grad_norm": 0.8089569211006165,
+      "learning_rate": 9.02446898677957e-05,
+      "loss": 0.2039,
+      "step": 142
+    },
+    {
+      "epoch": 2.2560000000000002,
+      "eval_loss": 0.18490749597549438,
+      "eval_mean_token_accuracy": 0.8875,
+      "eval_num_tokens": 287602.0,
+      "eval_runtime": 11.4503,
+      "eval_samples_per_second": 17.467,
+      "eval_steps_per_second": 17.467,
+      "step": 142
+    },
+    {
+      "epoch": 2.288,
+      "grad_norm": 1.3834277391433716,
+      "learning_rate": 8.982649590120982e-05,
+      "loss": 0.1158,
+      "step": 144
+    },
+    {
+      "epoch": 2.288,
+      "eval_loss": 0.199040949344635,
+      "eval_mean_token_accuracy": 0.8725,
+      "eval_num_tokens": 291536.0,
+      "eval_runtime": 11.4543,
+      "eval_samples_per_second": 17.461,
+      "eval_steps_per_second": 17.461,
+      "step": 144
+    },
+    {
+      "epoch": 2.32,
+      "grad_norm": 4.859565258026123,
+      "learning_rate": 8.940053768033609e-05,
+      "loss": 0.2542,
+      "step": 146
+    },
+    {
+      "epoch": 2.32,
+      "eval_loss": 0.16568240523338318,
+      "eval_mean_token_accuracy": 0.9075,
+      "eval_num_tokens": 295878.0,
+      "eval_runtime": 11.4719,
+      "eval_samples_per_second": 17.434,
+      "eval_steps_per_second": 17.434,
+      "step": 146
+    },
+    {
+      "epoch": 2.352,
+      "grad_norm": 1.3032429218292236,
+      "learning_rate": 8.896689824657372e-05,
+      "loss": 0.1435,
+      "step": 148
+    },
+    {
+      "epoch": 2.352,
+      "eval_loss": 0.1407879889011383,
+      "eval_mean_token_accuracy": 0.9225,
+      "eval_num_tokens": 300074.0,
+      "eval_runtime": 11.4369,
+      "eval_samples_per_second": 17.487,
+      "eval_steps_per_second": 17.487,
+      "step": 148
+    },
+    {
+      "epoch": 2.384,
+      "grad_norm": 1.1557220220565796,
+      "learning_rate": 8.852566213878947e-05,
+      "loss": 0.122,
+      "step": 150
+    },
+    {
+      "epoch": 2.384,
+      "eval_loss": 0.13803181052207947,
+      "eval_mean_token_accuracy": 0.9275,
+      "eval_num_tokens": 304160.0,
+      "eval_runtime": 11.4137,
+      "eval_samples_per_second": 17.523,
+      "eval_steps_per_second": 17.523,
+      "step": 150
+    },
+    {
+      "epoch": 2.416,
+      "grad_norm": 1.196094274520874,
+      "learning_rate": 8.807691537683685e-05,
+      "loss": 0.0878,
+      "step": 152
+    },
+    {
+      "epoch": 2.416,
+      "eval_loss": 0.16675826907157898,
+      "eval_mean_token_accuracy": 0.9225,
+      "eval_num_tokens": 308168.0,
+      "eval_runtime": 11.4028,
+      "eval_samples_per_second": 17.54,
+      "eval_steps_per_second": 17.54,
+      "step": 152
+    },
+    {
+      "epoch": 2.448,
+      "grad_norm": 1.1323683261871338,
+      "learning_rate": 8.762074544478623e-05,
+      "loss": 0.218,
+      "step": 154
+    },
+    {
+      "epoch": 2.448,
+      "eval_loss": 0.11720375716686249,
+      "eval_mean_token_accuracy": 0.94,
+      "eval_num_tokens": 312074.0,
+      "eval_runtime": 11.4178,
+      "eval_samples_per_second": 17.517,
+      "eval_steps_per_second": 17.517,
+      "step": 154
+    },
+    {
+      "epoch": 2.48,
+      "grad_norm": 5.290878772735596,
+      "learning_rate": 8.715724127386972e-05,
+      "loss": 0.134,
+      "step": 156
+    },
+    {
+      "epoch": 2.48,
+      "eval_loss": 0.17898881435394287,
+      "eval_mean_token_accuracy": 0.91,
+      "eval_num_tokens": 316202.0,
+      "eval_runtime": 11.4718,
+      "eval_samples_per_second": 17.434,
+      "eval_steps_per_second": 17.434,
+      "step": 156
+    },
+    {
+      "epoch": 2.512,
+      "grad_norm": 2.439903974533081,
+      "learning_rate": 8.668649322514382e-05,
+      "loss": 0.1365,
+      "step": 158
+    },
+    {
+      "epoch": 2.512,
+      "eval_loss": 0.12945598363876343,
+      "eval_mean_token_accuracy": 0.92,
+      "eval_num_tokens": 320012.0,
+      "eval_runtime": 11.4286,
+      "eval_samples_per_second": 17.5,
+      "eval_steps_per_second": 17.5,
+      "step": 158
+    },
+    {
+      "epoch": 2.544,
+      "grad_norm": 3.3977372646331787,
+      "learning_rate": 8.620859307187339e-05,
+      "loss": 0.1681,
+      "step": 160
+    },
+    {
+      "epoch": 2.544,
+      "eval_loss": 0.1133602112531662,
+      "eval_mean_token_accuracy": 0.9375,
+      "eval_num_tokens": 324418.0,
+      "eval_runtime": 11.4514,
+      "eval_samples_per_second": 17.465,
+      "eval_steps_per_second": 17.465,
+      "step": 160
+    },
+    {
+      "epoch": 2.576,
+      "grad_norm": 14.800116539001465,
+      "learning_rate": 8.572363398164017e-05,
+      "loss": 0.1669,
+      "step": 162
+    },
+    {
+      "epoch": 2.576,
+      "eval_loss": 0.1517494171857834,
+      "eval_mean_token_accuracy": 0.9325,
+      "eval_num_tokens": 328562.0,
+      "eval_runtime": 11.4023,
+      "eval_samples_per_second": 17.54,
+      "eval_steps_per_second": 17.54,
+      "step": 162
+    },
+    {
+      "epoch": 2.608,
+      "grad_norm": 0.46428850293159485,
+      "learning_rate": 8.523171049817974e-05,
+      "loss": 0.1727,
+      "step": 164
+    },
+    {
+      "epoch": 2.608,
+      "eval_loss": 0.1435018926858902,
+      "eval_mean_token_accuracy": 0.9425,
+      "eval_num_tokens": 332708.0,
+      "eval_runtime": 11.487,
+      "eval_samples_per_second": 17.411,
+      "eval_steps_per_second": 17.411,
+      "step": 164
+    },
+    {
+      "epoch": 2.64,
+      "grad_norm": 4.931906223297119,
+      "learning_rate": 8.473291852294987e-05,
+      "loss": 0.0871,
+      "step": 166
+    },
+    {
+      "epoch": 2.64,
+      "eval_loss": 0.1461663395166397,
+      "eval_mean_token_accuracy": 0.955,
+      "eval_num_tokens": 336896.0,
+      "eval_runtime": 11.5628,
+      "eval_samples_per_second": 17.297,
+      "eval_steps_per_second": 17.297,
+      "step": 166
+    },
+    {
+      "epoch": 2.672,
+      "grad_norm": 6.912172794342041,
+      "learning_rate": 8.422735529643444e-05,
+      "loss": 0.2471,
+      "step": 168
+    },
+    {
+      "epoch": 2.672,
+      "eval_loss": 0.07861819863319397,
+      "eval_mean_token_accuracy": 0.9625,
+      "eval_num_tokens": 340918.0,
+      "eval_runtime": 11.5061,
+      "eval_samples_per_second": 17.382,
+      "eval_steps_per_second": 17.382,
+      "step": 168
+    },
+    {
+      "epoch": 2.7039999999999997,
+      "grad_norm": 1.6310043334960938,
+      "learning_rate": 8.371511937918616e-05,
+      "loss": 0.1027,
+      "step": 170
+    },
+    {
+      "epoch": 2.7039999999999997,
+      "eval_loss": 0.05627990886569023,
+      "eval_mean_token_accuracy": 0.97,
+      "eval_num_tokens": 345050.0,
+      "eval_runtime": 11.5107,
+      "eval_samples_per_second": 17.375,
+      "eval_steps_per_second": 17.375,
+      "step": 170
+    },
+    {
+      "epoch": 2.7359999999999998,
+      "grad_norm": 0.32810577750205994,
+      "learning_rate": 8.319631063261209e-05,
+      "loss": 0.0623,
+      "step": 172
+    },
+    {
+      "epoch": 2.7359999999999998,
+      "eval_loss": 0.06946524977684021,
+      "eval_mean_token_accuracy": 0.9725,
+      "eval_num_tokens": 349110.0,
+      "eval_runtime": 11.4864,
+      "eval_samples_per_second": 17.412,
+      "eval_steps_per_second": 17.412,
+      "step": 172
+    },
+    {
+      "epoch": 2.768,
+      "grad_norm": 5.873373031616211,
+      "learning_rate": 8.267103019950529e-05,
+      "loss": 0.2089,
+      "step": 174
+    },
+    {
+      "epoch": 2.768,
+      "eval_loss": 0.048328179866075516,
+      "eval_mean_token_accuracy": 0.975,
+      "eval_num_tokens": 353242.0,
+      "eval_runtime": 11.5031,
+      "eval_samples_per_second": 17.387,
+      "eval_steps_per_second": 17.387,
+      "step": 174
+    },
+    {
+      "epoch": 2.8,
+      "grad_norm": 1.1473877429962158,
+      "learning_rate": 8.213938048432697e-05,
+      "loss": 0.0432,
+      "step": 176
+    },
+    {
+      "epoch": 2.8,
+      "eval_loss": 0.06747272610664368,
+      "eval_mean_token_accuracy": 0.98,
+      "eval_num_tokens": 357332.0,
+      "eval_runtime": 11.5304,
+      "eval_samples_per_second": 17.345,
+      "eval_steps_per_second": 17.345,
+      "step": 176
+    },
+    {
+      "epoch": 2.832,
+      "grad_norm": 0.7021322846412659,
+      "learning_rate": 8.160146513324254e-05,
+      "loss": 0.0933,
+      "step": 178
+    },
+    {
+      "epoch": 2.832,
+      "eval_loss": 0.0363197959959507,
+      "eval_mean_token_accuracy": 0.9775,
+      "eval_num_tokens": 361614.0,
+      "eval_runtime": 11.4788,
+      "eval_samples_per_second": 17.423,
+      "eval_steps_per_second": 17.423,
+      "step": 178
+    },
+    {
+      "epoch": 2.864,
+      "grad_norm": 1.706747055053711,
+      "learning_rate": 8.105738901391552e-05,
+      "loss": 0.0608,
+      "step": 180
+    },
+    {
+      "epoch": 2.864,
+      "eval_loss": 0.06278806924819946,
+      "eval_mean_token_accuracy": 0.97,
+      "eval_num_tokens": 365528.0,
+      "eval_runtime": 11.4832,
+      "eval_samples_per_second": 17.417,
+      "eval_steps_per_second": 17.417,
+      "step": 180
+    },
+    {
+      "epoch": 2.896,
+      "grad_norm": 1.9492478370666504,
+      "learning_rate": 8.05072581950634e-05,
+      "loss": 0.0335,
+      "step": 182
+    },
+    {
+      "epoch": 2.896,
+      "eval_loss": 0.09729886054992676,
+      "eval_mean_token_accuracy": 0.9725,
+      "eval_num_tokens": 369352.0,
+      "eval_runtime": 11.4331,
+      "eval_samples_per_second": 17.493,
+      "eval_steps_per_second": 17.493,
+      "step": 182
+    },
+    {
+      "epoch": 2.928,
+      "grad_norm": 21.485374450683594,
+      "learning_rate": 7.99511799257793e-05,
+      "loss": 0.2721,
+      "step": 184
+    },
+    {
+      "epoch": 2.928,
+      "eval_loss": 0.04741651937365532,
+      "eval_mean_token_accuracy": 0.98,
+      "eval_num_tokens": 373516.0,
+      "eval_runtime": 11.6699,
+      "eval_samples_per_second": 17.138,
+      "eval_steps_per_second": 17.138,
+      "step": 184
+    },
+    {
+      "epoch": 2.96,
+      "grad_norm": 0.45478004217147827,
+      "learning_rate": 7.938926261462366e-05,
+      "loss": 0.044,
+      "step": 186
+    },
+    {
+      "epoch": 2.96,
+      "eval_loss": 0.039359673857688904,
+      "eval_mean_token_accuracy": 0.9875,
+      "eval_num_tokens": 377460.0,
+      "eval_runtime": 11.3414,
+      "eval_samples_per_second": 17.635,
+      "eval_steps_per_second": 17.635,
+      "step": 186
+    },
+    {
+      "epoch": 2.992,
+      "grad_norm": 2.0323572158813477,
+      "learning_rate": 7.882161580848967e-05,
+      "loss": 0.0702,
+      "step": 188
+    },
+    {
+      "epoch": 2.992,
+      "eval_loss": 0.06572172045707703,
+      "eval_mean_token_accuracy": 0.9825,
+      "eval_num_tokens": 381576.0,
+      "eval_runtime": 11.3844,
+      "eval_samples_per_second": 17.568,
+      "eval_steps_per_second": 17.568,
+      "step": 188
+    },
+    {
+      "epoch": 3.016,
+      "grad_norm": 0.6514262557029724,
+      "learning_rate": 7.82483501712469e-05,
+      "loss": 0.0189,
+      "step": 190
+    },
+    {
+      "epoch": 3.016,
+      "eval_loss": 0.054220449179410934,
+      "eval_mean_token_accuracy": 0.985,
+      "eval_num_tokens": 384716.0,
+      "eval_runtime": 11.5834,
+      "eval_samples_per_second": 17.266,
+      "eval_steps_per_second": 17.266,
+      "step": 190
+    },
+    {
+      "epoch": 3.048,
+      "grad_norm": 0.48469850420951843,
+      "learning_rate": 7.766957746216721e-05,
+      "loss": 0.0183,
+      "step": 192
+    },
+    {
+      "epoch": 3.048,
+      "eval_loss": 0.04004016891121864,
+      "eval_mean_token_accuracy": 0.975,
+      "eval_num_tokens": 388956.0,
+      "eval_runtime": 11.6179,
+      "eval_samples_per_second": 17.215,
+      "eval_steps_per_second": 17.215,
+      "step": 192
+    },
+    {
+      "epoch": 3.08,
+      "grad_norm": 11.357516288757324,
+      "learning_rate": 7.7085410514137e-05,
+      "loss": 0.0829,
+      "step": 194
+    },
+    {
+      "epoch": 3.08,
+      "eval_loss": 0.02486039325594902,
+      "eval_mean_token_accuracy": 0.985,
+      "eval_num_tokens": 393244.0,
+      "eval_runtime": 11.4446,
+      "eval_samples_per_second": 17.476,
+      "eval_steps_per_second": 17.476,
+      "step": 194
+    },
+    {
+      "epoch": 3.112,
+      "grad_norm": 0.47729116678237915,
+      "learning_rate": 7.649596321166024e-05,
+      "loss": 0.0042,
+      "step": 196
+    },
+    {
+      "epoch": 3.112,
+      "eval_loss": 0.020039737224578857,
+      "eval_mean_token_accuracy": 0.99,
+      "eval_num_tokens": 397206.0,
+      "eval_runtime": 11.4496,
+      "eval_samples_per_second": 17.468,
+      "eval_steps_per_second": 17.468,
+      "step": 196
+    },
+    {
+      "epoch": 3.144,
+      "grad_norm": 3.093353509902954,
+      "learning_rate": 7.590135046865651e-05,
+      "loss": 0.0591,
+      "step": 198
+    },
+    {
+      "epoch": 3.144,
+      "eval_loss": 0.0175277441740036,
+      "eval_mean_token_accuracy": 0.99,
+      "eval_num_tokens": 401516.0,
+      "eval_runtime": 11.4765,
+      "eval_samples_per_second": 17.427,
+      "eval_steps_per_second": 17.427,
+      "step": 198
+    },
+    {
+      "epoch": 3.176,
+      "grad_norm": 0.6326797008514404,
+      "learning_rate": 7.530168820605818e-05,
+      "loss": 0.1258,
+      "step": 200
+    },
+    {
+      "epoch": 3.176,
+      "eval_loss": 0.016891375184059143,
+      "eval_mean_token_accuracy": 0.9925,
+      "eval_num_tokens": 405566.0,
+      "eval_runtime": 11.4315,
+      "eval_samples_per_second": 17.495,
+      "eval_steps_per_second": 17.495,
+      "step": 200
+    },
+    {
+      "epoch": 3.208,
+      "grad_norm": 2.543379545211792,
+      "learning_rate": 7.469709332921155e-05,
+      "loss": 0.0205,
+      "step": 202
+    },
+    {
+      "epoch": 3.208,
+      "eval_loss": 0.016701769083738327,
+      "eval_mean_token_accuracy": 0.9925,
+      "eval_num_tokens": 409548.0,
+      "eval_runtime": 11.3574,
+      "eval_samples_per_second": 17.61,
+      "eval_steps_per_second": 17.61,
+      "step": 202
+    },
+    {
+      "epoch": 3.24,
+      "grad_norm": 0.17252640426158905,
+      "learning_rate": 7.408768370508576e-05,
+      "loss": 0.0057,
+      "step": 204
+    },
+    {
+      "epoch": 3.24,
+      "eval_loss": 0.017669010907411575,
+      "eval_mean_token_accuracy": 0.9925,
+      "eval_num_tokens": 413698.0,
+      "eval_runtime": 11.377,
+      "eval_samples_per_second": 17.579,
+      "eval_steps_per_second": 17.579,
+      "step": 204
+    },
+    {
+      "epoch": 3.2720000000000002,
+      "grad_norm": 1.0142366886138916,
+      "learning_rate": 7.347357813929454e-05,
+      "loss": 0.0359,
+      "step": 206
+    },
+    {
+      "epoch": 3.2720000000000002,
+      "eval_loss": 0.01496225968003273,
+      "eval_mean_token_accuracy": 0.9925,
+      "eval_num_tokens": 417858.0,
+      "eval_runtime": 11.4371,
+      "eval_samples_per_second": 17.487,
+      "eval_steps_per_second": 17.487,
+      "step": 206
+    },
+    {
+      "epoch": 3.304,
+      "grad_norm": 0.7629784941673279,
+      "learning_rate": 7.285489635293472e-05,
+      "loss": 0.0202,
+      "step": 208
+    },
+    {
+      "epoch": 3.304,
+      "eval_loss": 0.011429696343839169,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 421754.0,
+      "eval_runtime": 11.3744,
+      "eval_samples_per_second": 17.583,
+      "eval_steps_per_second": 17.583,
+      "step": 208
+    },
+    {
+      "epoch": 3.336,
+      "grad_norm": 0.8358567357063293,
+      "learning_rate": 7.223175895924638e-05,
+      "loss": 0.0301,
+      "step": 210
+    },
+    {
+      "epoch": 3.336,
+      "eval_loss": 0.010167855769395828,
+      "eval_mean_token_accuracy": 0.9975,
+      "eval_num_tokens": 425706.0,
+      "eval_runtime": 11.4121,
+      "eval_samples_per_second": 17.525,
+      "eval_steps_per_second": 17.525,
+      "step": 210
+    },
+    {
+      "epoch": 3.368,
+      "grad_norm": 0.0006723266560584307,
+      "learning_rate": 7.160428744009912e-05,
+      "loss": 0.0032,
+      "step": 212
+    },
+    {
+      "epoch": 3.368,
+      "eval_loss": 0.010096004232764244,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 429642.0,
+      "eval_runtime": 11.4588,
+      "eval_samples_per_second": 17.454,
+      "eval_steps_per_second": 17.454,
+      "step": 212
+    },
+    {
+      "epoch": 3.4,
+      "grad_norm": 0.0526227205991745,
+      "learning_rate": 7.097260412230886e-05,
+      "loss": 0.0344,
+      "step": 214
+    },
+    {
+      "epoch": 3.4,
+      "eval_loss": 0.012297773733735085,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 433732.0,
+      "eval_runtime": 11.4417,
+      "eval_samples_per_second": 17.48,
+      "eval_steps_per_second": 17.48,
+      "step": 214
+    },
+    {
+      "epoch": 3.432,
+      "grad_norm": 0.4703904390335083,
+      "learning_rate": 7.033683215379002e-05,
+      "loss": 0.0032,
+      "step": 216
+    },
+    {
+      "epoch": 3.432,
+      "eval_loss": 0.018655912950634956,
+      "eval_mean_token_accuracy": 0.9925,
+      "eval_num_tokens": 437680.0,
+      "eval_runtime": 11.4685,
+      "eval_samples_per_second": 17.439,
+      "eval_steps_per_second": 17.439,
+      "step": 216
+    },
+    {
+      "epoch": 3.464,
+      "grad_norm": 10.907818794250488,
+      "learning_rate": 6.969709547954756e-05,
+      "loss": 0.1169,
+      "step": 218
+    },
+    {
+      "epoch": 3.464,
+      "eval_loss": 0.019289858639240265,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 442164.0,
+      "eval_runtime": 11.4984,
+      "eval_samples_per_second": 17.394,
+      "eval_steps_per_second": 17.394,
+      "step": 218
+    },
+    {
+      "epoch": 3.496,
+      "grad_norm": 0.0031443352345377207,
+      "learning_rate": 6.905351881751372e-05,
+      "loss": 0.001,
+      "step": 220
+    },
+    {
+      "epoch": 3.496,
+      "eval_loss": 0.021614665165543556,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 446202.0,
+      "eval_runtime": 11.453,
+      "eval_samples_per_second": 17.463,
+      "eval_steps_per_second": 17.463,
+      "step": 220
+    },
+    {
+      "epoch": 3.528,
+      "grad_norm": 0.049350984394550323,
+      "learning_rate": 6.840622763423391e-05,
+      "loss": 0.0631,
+      "step": 222
+    },
+    {
+      "epoch": 3.528,
+      "eval_loss": 0.014943327754735947,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 450296.0,
+      "eval_runtime": 11.4503,
+      "eval_samples_per_second": 17.467,
+      "eval_steps_per_second": 17.467,
+      "step": 222
+    },
+    {
+      "epoch": 3.56,
+      "grad_norm": 3.5975873470306396,
+      "learning_rate": 6.775534812040685e-05,
+      "loss": 0.051,
+      "step": 224
+    },
+    {
+      "epoch": 3.56,
+      "eval_loss": 0.01427337434142828,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 454566.0,
+      "eval_runtime": 11.4931,
+      "eval_samples_per_second": 17.402,
+      "eval_steps_per_second": 17.402,
+      "step": 224
+    },
+    {
+      "epoch": 3.592,
+      "grad_norm": 0.4167959988117218,
+      "learning_rate": 6.710100716628344e-05,
+      "loss": 0.0375,
+      "step": 226
+    },
+    {
+      "epoch": 3.592,
+      "eval_loss": 0.01604151912033558,
+      "eval_mean_token_accuracy": 0.9925,
+      "eval_num_tokens": 458816.0,
+      "eval_runtime": 11.5104,
+      "eval_samples_per_second": 17.376,
+      "eval_steps_per_second": 17.376,
+      "step": 226
+    },
+    {
+      "epoch": 3.624,
+      "grad_norm": 2.67622447013855,
+      "learning_rate": 6.644333233692916e-05,
+      "loss": 0.0291,
+      "step": 228
+    },
+    {
+      "epoch": 3.624,
+      "eval_loss": 0.023203900083899498,
+      "eval_mean_token_accuracy": 0.9925,
+      "eval_num_tokens": 462872.0,
+      "eval_runtime": 11.478,
+      "eval_samples_per_second": 17.425,
+      "eval_steps_per_second": 17.425,
+      "step": 228
+    },
+    {
+      "epoch": 3.656,
+      "grad_norm": 0.987598717212677,
+      "learning_rate": 6.578245184735513e-05,
+      "loss": 0.0336,
+      "step": 230
+    },
+    {
+      "epoch": 3.656,
+      "eval_loss": 0.030808784067630768,
+      "eval_mean_token_accuracy": 0.99,
+      "eval_num_tokens": 466968.0,
+      "eval_runtime": 11.5082,
+      "eval_samples_per_second": 17.379,
+      "eval_steps_per_second": 17.379,
+      "step": 230
+    },
+    {
+      "epoch": 3.6879999999999997,
+      "grad_norm": 0.026162542402744293,
+      "learning_rate": 6.511849453752223e-05,
+      "loss": 0.0028,
+      "step": 232
+    },
+    {
+      "epoch": 3.6879999999999997,
+      "eval_loss": 0.025599656626582146,
+      "eval_mean_token_accuracy": 0.99,
+      "eval_num_tokens": 470960.0,
+      "eval_runtime": 11.4574,
+      "eval_samples_per_second": 17.456,
+      "eval_steps_per_second": 17.456,
+      "step": 232
+    },
+    {
+      "epoch": 3.7199999999999998,
+      "grad_norm": 0.20970286428928375,
+      "learning_rate": 6.445158984722358e-05,
+      "loss": 0.0032,
+      "step": 234
+    },
+    {
+      "epoch": 3.7199999999999998,
+      "eval_loss": 0.02075776644051075,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 475096.0,
+      "eval_runtime": 11.4514,
+      "eval_samples_per_second": 17.465,
+      "eval_steps_per_second": 17.465,
+      "step": 234
+    },
+    {
+      "epoch": 3.752,
+      "grad_norm": 0.004214724991470575,
+      "learning_rate": 6.378186779084995e-05,
+      "loss": 0.0,
+      "step": 236
+    },
+    {
+      "epoch": 3.752,
+      "eval_loss": 0.018998321145772934,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 479048.0,
+      "eval_runtime": 11.4221,
+      "eval_samples_per_second": 17.51,
+      "eval_steps_per_second": 17.51,
+      "step": 236
+    },
+    {
+      "epoch": 3.784,
+      "grad_norm": 0.000873790995683521,
+      "learning_rate": 6.310945893204324e-05,
+      "loss": 0.044,
+      "step": 238
+    },
+    {
+      "epoch": 3.784,
+      "eval_loss": 0.015460082329809666,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 483158.0,
+      "eval_runtime": 11.4054,
+      "eval_samples_per_second": 17.536,
+      "eval_steps_per_second": 17.536,
+      "step": 238
+    },
+    {
+      "epoch": 3.816,
+      "grad_norm": 34.56232452392578,
+      "learning_rate": 6.243449435824276e-05,
+      "loss": 0.1101,
+      "step": 240
+    },
+    {
+      "epoch": 3.816,
+      "eval_loss": 0.013121403753757477,
+      "eval_mean_token_accuracy": 0.9975,
+      "eval_num_tokens": 487444.0,
+      "eval_runtime": 11.3931,
+      "eval_samples_per_second": 17.554,
+      "eval_steps_per_second": 17.554,
+      "step": 240
+    },
+    {
+      "epoch": 3.848,
+      "grad_norm": 0.00021152305998839438,
+      "learning_rate": 6.17571056551295e-05,
+      "loss": 0.0,
+      "step": 242
+    },
+    {
+      "epoch": 3.848,
+      "eval_loss": 0.011188148520886898,
+      "eval_mean_token_accuracy": 0.9975,
+      "eval_num_tokens": 491270.0,
+      "eval_runtime": 11.3925,
+      "eval_samples_per_second": 17.555,
+      "eval_steps_per_second": 17.555,
+      "step": 242
+    },
+    {
+      "epoch": 3.88,
+      "grad_norm": 0.0012458531418815255,
+      "learning_rate": 6.107742488097338e-05,
+      "loss": 0.0666,
+      "step": 244
+    },
+    {
+      "epoch": 3.88,
+      "eval_loss": 0.002846868708729744,
+      "eval_mean_token_accuracy": 0.9975,
+      "eval_num_tokens": 495430.0,
+      "eval_runtime": 11.4171,
+      "eval_samples_per_second": 17.518,
+      "eval_steps_per_second": 17.518,
+      "step": 244
+    },
+    {
+      "epoch": 3.912,
+      "grad_norm": 0.0006587646785192192,
+      "learning_rate": 6.0395584540887963e-05,
+      "loss": 0.0,
+      "step": 246
+    },
+    {
+      "epoch": 3.912,
+      "eval_loss": 0.0023970839101821184,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 499282.0,
+      "eval_runtime": 11.4033,
+      "eval_samples_per_second": 17.539,
+      "eval_steps_per_second": 17.539,
+      "step": 246
+    },
+    {
+      "epoch": 3.944,
+      "grad_norm": 0.0006554482970386744,
+      "learning_rate": 5.97117175609986e-05,
+      "loss": 0.0,
+      "step": 248
+    },
+    {
+      "epoch": 3.944,
+      "eval_loss": 0.004124053753912449,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 503072.0,
+      "eval_runtime": 11.4426,
+      "eval_samples_per_second": 17.479,
+      "eval_steps_per_second": 17.479,
+      "step": 248
+    },
+    {
+      "epoch": 3.976,
+      "grad_norm": 0.022889109328389168,
+      "learning_rate": 5.902595726252801e-05,
+      "loss": 0.0002,
+      "step": 250
+    },
+    {
+      "epoch": 3.976,
+      "eval_loss": 0.006614612881094217,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 507388.0,
+      "eval_runtime": 11.4703,
+      "eval_samples_per_second": 17.436,
+      "eval_steps_per_second": 17.436,
+      "step": 250
+    },
+    {
+      "epoch": 4.0,
+      "grad_norm": 0.00430309260264039,
+      "learning_rate": 5.833843733580512e-05,
+      "loss": 0.0001,
+      "step": 252
+    },
+    {
+      "epoch": 4.0,
+      "eval_loss": 0.007905487902462482,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 510128.0,
+      "eval_runtime": 11.4818,
+      "eval_samples_per_second": 17.419,
+      "eval_steps_per_second": 17.419,
+      "step": 252
+    },
+    {
+      "epoch": 4.032,
+      "grad_norm": 0.26011374592781067,
+      "learning_rate": 5.764929181420191e-05,
+      "loss": 0.0025,
+      "step": 254
+    },
+    {
+      "epoch": 4.032,
+      "eval_loss": 0.008602648973464966,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 514206.0,
+      "eval_runtime": 11.5431,
+      "eval_samples_per_second": 17.326,
+      "eval_steps_per_second": 17.326,
+      "step": 254
+    },
+    {
+      "epoch": 4.064,
+      "grad_norm": 0.18739522993564606,
+      "learning_rate": 5.695865504800327e-05,
+      "loss": 0.0008,
+      "step": 256
+    },
+    {
+      "epoch": 4.064,
+      "eval_loss": 0.008637922815978527,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 518310.0,
+      "eval_runtime": 11.5927,
+      "eval_samples_per_second": 17.252,
+      "eval_steps_per_second": 17.252,
+      "step": 256
+    },
+    {
+      "epoch": 4.096,
+      "grad_norm": 0.010165493004024029,
+      "learning_rate": 5.6266661678215216e-05,
+      "loss": 0.0001,
+      "step": 258
+    },
+    {
+      "epoch": 4.096,
+      "eval_loss": 0.008439877070486546,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 522220.0,
+      "eval_runtime": 11.4057,
+      "eval_samples_per_second": 17.535,
+      "eval_steps_per_second": 17.535,
+      "step": 258
+    },
+    {
+      "epoch": 4.128,
+      "grad_norm": 0.022179462015628815,
+      "learning_rate": 5.557344661031627e-05,
+      "loss": 0.0001,
+      "step": 260
+    },
+    {
+      "epoch": 4.128,
+      "eval_loss": 0.00898431334644556,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 525924.0,
+      "eval_runtime": 11.2794,
+      "eval_samples_per_second": 17.731,
+      "eval_steps_per_second": 17.731,
+      "step": 260
+    },
+    {
+      "epoch": 4.16,
+      "grad_norm": 0.08368759602308273,
+      "learning_rate": 5.487914498795747e-05,
+      "loss": 0.0229,
+      "step": 262
+    },
+    {
+      "epoch": 4.16,
+      "eval_loss": 0.006476944778114557,
+      "eval_mean_token_accuracy": 0.995,
+      "eval_num_tokens": 530044.0,
+      "eval_runtime": 11.3326,
+      "eval_samples_per_second": 17.648,
+      "eval_steps_per_second": 17.648,
+      "step": 262
+    },
+    {
+      "epoch": 4.192,
+      "grad_norm": 0.0875498577952385,
+      "learning_rate": 5.418389216661579e-05,
+      "loss": 0.0007,
+      "step": 264
+    },
+    {
+      "epoch": 4.192,
+      "eval_loss": 0.004615248646587133,
+      "eval_mean_token_accuracy": 0.9975,
+      "eval_num_tokens": 534138.0,
+      "eval_runtime": 11.401,
+      "eval_samples_per_second": 17.542,
+      "eval_steps_per_second": 17.542,
+      "step": 264
+    },
+    {
+      "epoch": 4.224,
+      "grad_norm": 0.056955043226480484,
+      "learning_rate": 5.348782368720626e-05,
+      "loss": 0.0038,
+      "step": 266
+    },
+    {
+      "epoch": 4.224,
+      "eval_loss": 0.0034542647190392017,
+      "eval_mean_token_accuracy": 0.9975,
+      "eval_num_tokens": 538160.0,
+      "eval_runtime": 11.385,
+      "eval_samples_per_second": 17.567,
+      "eval_steps_per_second": 17.567,
+      "step": 266
+    },
+    {
+      "epoch": 4.256,
+      "grad_norm": 0.0016861628973856568,
+      "learning_rate": 5.279107524965819e-05,
+      "loss": 0.0007,
+      "step": 268
+    },
+    {
+      "epoch": 4.256,
+      "eval_loss": 0.0020931244362145662,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 542330.0,
+      "eval_runtime": 11.3941,
+      "eval_samples_per_second": 17.553,
+      "eval_steps_per_second": 17.553,
+      "step": 268
+    },
+    {
+      "epoch": 4.288,
+      "grad_norm": 0.08018090575933456,
+      "learning_rate": 5.209378268645998e-05,
+      "loss": 0.0372,
+      "step": 270
+    },
+    {
+      "epoch": 4.288,
+      "eval_loss": 0.0025086223613470793,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 546686.0,
+      "eval_runtime": 11.4057,
+      "eval_samples_per_second": 17.535,
+      "eval_steps_per_second": 17.535,
+      "step": 270
+    },
+    {
+      "epoch": 4.32,
+      "grad_norm": 0.03331761434674263,
+      "learning_rate": 5.139608193617845e-05,
+      "loss": 0.0003,
+      "step": 272
+    },
+    {
+      "epoch": 4.32,
+      "eval_loss": 0.0026831370778381824,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 550746.0,
+      "eval_runtime": 11.3596,
+      "eval_samples_per_second": 17.606,
+      "eval_steps_per_second": 17.606,
+      "step": 272
+    },
+    {
+      "epoch": 4.352,
+      "grad_norm": 0.007008640095591545,
+      "learning_rate": 5.0698109016957274e-05,
+      "loss": 0.0142,
+      "step": 274
+    },
+    {
+      "epoch": 4.352,
+      "eval_loss": 0.002527955686673522,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 554538.0,
+      "eval_runtime": 11.4109,
+      "eval_samples_per_second": 17.527,
+      "eval_steps_per_second": 17.527,
+      "step": 274
+    },
+    {
+      "epoch": 4.384,
+      "grad_norm": 0.009444222785532475,
+      "learning_rate": 5e-05,
+      "loss": 0.0002,
+      "step": 276
+    },
+    {
+      "epoch": 4.384,
+      "eval_loss": 0.0023918580263853073,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 558614.0,
+      "eval_runtime": 11.367,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 17.595,
+      "step": 276
+    },
+    {
+      "epoch": 4.416,
+      "grad_norm": 0.012622497975826263,
+      "learning_rate": 4.9301890983042744e-05,
+      "loss": 0.0001,
+      "step": 278
+    },
+    {
+      "epoch": 4.416,
+      "eval_loss": 0.002456513699144125,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 562700.0,
+      "eval_runtime": 11.3939,
+      "eval_samples_per_second": 17.553,
+      "eval_steps_per_second": 17.553,
+      "step": 278
+    },
+    {
+      "epoch": 4.448,
+      "grad_norm": 0.5172138214111328,
+      "learning_rate": 4.860391806382157e-05,
+      "loss": 0.0036,
+      "step": 280
+    },
+    {
+      "epoch": 4.448,
+      "eval_loss": 0.0019083315273746848,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 566758.0,
+      "eval_runtime": 11.3929,
+      "eval_samples_per_second": 17.555,
+      "eval_steps_per_second": 17.555,
+      "step": 280
+    },
+    {
+      "epoch": 4.48,
+      "grad_norm": 1.0472556352615356,
+      "learning_rate": 4.790621731354003e-05,
+      "loss": 0.0065,
+      "step": 282
+    },
+    {
+      "epoch": 4.48,
+      "eval_loss": 0.0011325461091473699,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 570822.0,
+      "eval_runtime": 11.3913,
+      "eval_samples_per_second": 17.557,
+      "eval_steps_per_second": 17.557,
+      "step": 282
+    },
+    {
+      "epoch": 4.5120000000000005,
+      "grad_norm": 0.22454571723937988,
+      "learning_rate": 4.720892475034181e-05,
+      "loss": 0.0016,
+      "step": 284
+    },
+    {
+      "epoch": 4.5120000000000005,
+      "eval_loss": 0.0005773272132501006,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 574862.0,
+      "eval_runtime": 11.3908,
+      "eval_samples_per_second": 17.558,
+      "eval_steps_per_second": 17.558,
+      "step": 284
+    },
+    {
+      "epoch": 4.5440000000000005,
+      "grad_norm": 0.06415335834026337,
+      "learning_rate": 4.6512176312793736e-05,
+      "loss": 0.0005,
+      "step": 286
+    },
+    {
+      "epoch": 4.5440000000000005,
+      "eval_loss": 0.0002438312367303297,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 578862.0,
+      "eval_runtime": 11.3788,
+      "eval_samples_per_second": 17.577,
+      "eval_steps_per_second": 17.577,
+      "step": 286
+    },
+    {
+      "epoch": 4.576,
+      "grad_norm": 0.05790300667285919,
+      "learning_rate": 4.5816107833384234e-05,
+      "loss": 0.0003,
+      "step": 288
+    },
+    {
+      "epoch": 4.576,
+      "eval_loss": 0.00011587580956984311,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 582968.0,
+      "eval_runtime": 11.2993,
+      "eval_samples_per_second": 17.7,
+      "eval_steps_per_second": 17.7,
+      "step": 288
+    },
+    {
+      "epoch": 4.608,
+      "grad_norm": 0.0013133377069607377,
+      "learning_rate": 4.512085501204253e-05,
+      "loss": 0.0001,
+      "step": 290
+    },
+    {
+      "epoch": 4.608,
+      "eval_loss": 6.511201354442164e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 587124.0,
+      "eval_runtime": 11.4294,
+      "eval_samples_per_second": 17.499,
+      "eval_steps_per_second": 17.499,
+      "step": 290
+    },
+    {
+      "epoch": 4.64,
+      "grad_norm": 0.00043543236097320914,
+      "learning_rate": 4.442655338968373e-05,
+      "loss": 0.0,
+      "step": 292
+    },
+    {
+      "epoch": 4.64,
+      "eval_loss": 4.0417387936031446e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 591394.0,
+      "eval_runtime": 11.4061,
+      "eval_samples_per_second": 17.534,
+      "eval_steps_per_second": 17.534,
+      "step": 292
+    },
+    {
+      "epoch": 4.672,
+      "grad_norm": 0.0006148144602775574,
+      "learning_rate": 4.373333832178478e-05,
+      "loss": 0.0,
+      "step": 294
+    },
+    {
+      "epoch": 4.672,
+      "eval_loss": 2.7705291358870454e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 595376.0,
+      "eval_runtime": 11.4153,
+      "eval_samples_per_second": 17.52,
+      "eval_steps_per_second": 17.52,
+      "step": 294
+    },
+    {
+      "epoch": 4.704,
+      "grad_norm": 0.00023849304125178605,
+      "learning_rate": 4.3041344951996746e-05,
+      "loss": 0.0,
+      "step": 296
+    },
+    {
+      "epoch": 4.704,
+      "eval_loss": 2.0575953385559842e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 599692.0,
+      "eval_runtime": 11.4111,
+      "eval_samples_per_second": 17.527,
+      "eval_steps_per_second": 17.527,
+      "step": 296
+    },
+    {
+      "epoch": 4.736,
+      "grad_norm": 0.0001566153223393485,
+      "learning_rate": 4.23507081857981e-05,
+      "loss": 0.0,
+      "step": 298
+    },
+    {
+      "epoch": 4.736,
+      "eval_loss": 1.6603422409389168e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 603796.0,
+      "eval_runtime": 11.4467,
+      "eval_samples_per_second": 17.472,
+      "eval_steps_per_second": 17.472,
+      "step": 298
+    },
+    {
+      "epoch": 4.768,
+      "grad_norm": 0.0009461111621931195,
+      "learning_rate": 4.166156266419489e-05,
+      "loss": 0.0,
+      "step": 300
+    },
+    {
+      "epoch": 4.768,
+      "eval_loss": 1.767077628755942e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 608042.0,
+      "eval_runtime": 11.4214,
+      "eval_samples_per_second": 17.511,
+      "eval_steps_per_second": 17.511,
+      "step": 300
+    },
+    {
+      "epoch": 4.8,
+      "grad_norm": 0.0001271154178539291,
+      "learning_rate": 4.0974042737472006e-05,
+      "loss": 0.0,
+      "step": 302
+    },
+    {
+      "epoch": 4.8,
+      "eval_loss": 1.4388447198143695e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 612088.0,
+      "eval_runtime": 11.3887,
+      "eval_samples_per_second": 17.561,
+      "eval_steps_per_second": 17.561,
+      "step": 302
+    },
+    {
+      "epoch": 4.832,
+      "grad_norm": 0.003310651518404484,
+      "learning_rate": 4.028828243900141e-05,
+      "loss": 0.0,
+      "step": 304
+    },
+    {
+      "epoch": 4.832,
+      "eval_loss": 1.3456246961141005e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 616228.0,
+      "eval_runtime": 11.3578,
+      "eval_samples_per_second": 17.609,
+      "eval_steps_per_second": 17.609,
+      "step": 304
+    },
+    {
+      "epoch": 4.864,
+      "grad_norm": 0.00014842761447653174,
+      "learning_rate": 3.960441545911204e-05,
+      "loss": 0.0,
+      "step": 306
+    },
+    {
+      "epoch": 4.864,
+      "eval_loss": 1.2526979844551533e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 620242.0,
+      "eval_runtime": 11.3498,
+      "eval_samples_per_second": 17.622,
+      "eval_steps_per_second": 17.622,
+      "step": 306
+    },
+    {
+      "epoch": 4.896,
+      "grad_norm": 0.00010382337495684624,
+      "learning_rate": 3.892257511902664e-05,
+      "loss": 0.0,
+      "step": 308
+    },
+    {
+      "epoch": 4.896,
+      "eval_loss": 1.1547385838639457e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 624174.0,
+      "eval_runtime": 11.4115,
+      "eval_samples_per_second": 17.526,
+      "eval_steps_per_second": 17.526,
+      "step": 308
+    },
+    {
+      "epoch": 4.928,
+      "grad_norm": 0.03389514237642288,
+      "learning_rate": 3.82428943448705e-05,
+      "loss": 0.0,
+      "step": 310
+    },
+    {
+      "epoch": 4.928,
+      "eval_loss": 1.1918379641429055e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 628502.0,
+      "eval_runtime": 11.3808,
+      "eval_samples_per_second": 17.574,
+      "eval_steps_per_second": 17.574,
+      "step": 310
+    },
+    {
+      "epoch": 4.96,
+      "grad_norm": 0.00018427451141178608,
+      "learning_rate": 3.756550564175727e-05,
+      "loss": 0.0,
+      "step": 312
+    },
+    {
+      "epoch": 4.96,
+      "eval_loss": 1.13857649921556e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 632674.0,
+      "eval_runtime": 11.3791,
+      "eval_samples_per_second": 17.576,
+      "eval_steps_per_second": 17.576,
+      "step": 312
+    },
+    {
+      "epoch": 4.992,
+      "grad_norm": 0.00014324556104838848,
+      "learning_rate": 3.6890541067956776e-05,
+      "loss": 0.0,
+      "step": 314
+    },
+    {
+      "epoch": 4.992,
+      "eval_loss": 1.0801677490235306e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 636674.0,
+      "eval_runtime": 11.2879,
+      "eval_samples_per_second": 17.718,
+      "eval_steps_per_second": 17.718,
+      "step": 314
+    },
+    {
+      "epoch": 5.016,
+      "grad_norm": 0.00010275087697664276,
+      "learning_rate": 3.6218132209150045e-05,
+      "loss": 0.0,
+      "step": 316
+    },
+    {
+      "epoch": 5.016,
+      "eval_loss": 1.0888848919421434e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 639496.0,
+      "eval_runtime": 11.3726,
+      "eval_samples_per_second": 17.586,
+      "eval_steps_per_second": 17.586,
+      "step": 316
+    },
+    {
+      "epoch": 5.048,
+      "grad_norm": 0.00010966294212266803,
+      "learning_rate": 3.554841015277641e-05,
+      "loss": 0.0,
+      "step": 318
+    },
+    {
+      "epoch": 5.048,
+      "eval_loss": 1.0286075848853216e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 643448.0,
+      "eval_runtime": 11.4133,
+      "eval_samples_per_second": 17.523,
+      "eval_steps_per_second": 17.523,
+      "step": 318
+    },
+    {
+      "epoch": 5.08,
+      "grad_norm": 0.00013332242087926716,
+      "learning_rate": 3.488150546247778e-05,
+      "loss": 0.0,
+      "step": 320
+    },
+    {
+      "epoch": 5.08,
+      "eval_loss": 1.022618107526796e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 647534.0,
+      "eval_runtime": 11.3974,
+      "eval_samples_per_second": 17.548,
+      "eval_steps_per_second": 17.548,
+      "step": 320
+    },
+    {
+      "epoch": 5.112,
+      "grad_norm": 0.0003446874034125358,
+      "learning_rate": 3.4217548152644885e-05,
+      "loss": 0.0,
+      "step": 322
+    },
+    {
+      "epoch": 5.112,
+      "eval_loss": 9.997980669140816e-06,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 651780.0,
+      "eval_runtime": 11.3821,
+      "eval_samples_per_second": 17.571,
+      "eval_steps_per_second": 17.571,
+      "step": 322
+    },
+    {
+      "epoch": 5.144,
+      "grad_norm": 0.00013512311852537096,
+      "learning_rate": 3.355666766307084e-05,
+      "loss": 0.0,
+      "step": 324
+    },
+    {
+      "epoch": 5.144,
+      "eval_loss": 9.994946594815701e-06,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 655544.0,
+      "eval_runtime": 11.4135,
+      "eval_samples_per_second": 17.523,
+      "eval_steps_per_second": 17.523,
+      "step": 324
+    },
+    {
+      "epoch": 5.176,
+      "grad_norm": 8.219212759286165e-05,
+      "learning_rate": 3.289899283371657e-05,
+      "loss": 0.0,
+      "step": 326
+    },
+    {
+      "epoch": 5.176,
+      "eval_loss": 1.0030768862634432e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 659442.0,
+      "eval_runtime": 11.3796,
+      "eval_samples_per_second": 17.575,
+      "eval_steps_per_second": 17.575,
+      "step": 326
+    },
+    {
+      "epoch": 5.208,
+      "grad_norm": 0.00018281866505276412,
+      "learning_rate": 3.224465187959316e-05,
+      "loss": 0.0,
+      "step": 328
+    },
+    {
+      "epoch": 5.208,
+      "eval_loss": 1.0034288607130293e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 663530.0,
+      "eval_runtime": 11.4271,
+      "eval_samples_per_second": 17.502,
+      "eval_steps_per_second": 17.502,
+      "step": 328
+    },
+    {
+      "epoch": 5.24,
+      "grad_norm": 0.00015049244393594563,
+      "learning_rate": 3.1593772365766105e-05,
+      "loss": 0.0,
+      "step": 330
+    },
+    {
+      "epoch": 5.24,
+      "eval_loss": 9.735815183375962e-06,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 667656.0,
+      "eval_runtime": 11.3965,
+      "eval_samples_per_second": 17.549,
+      "eval_steps_per_second": 17.549,
+      "step": 330
+    },
+    {
+      "epoch": 5.272,
+      "grad_norm": 0.00043579115299507976,
+      "learning_rate": 3.09464811824863e-05,
+      "loss": 0.0,
+      "step": 332
+    },
+    {
+      "epoch": 5.272,
+      "eval_loss": 1.0137660865439102e-05,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 671910.0,
+      "eval_runtime": 11.403,
+      "eval_samples_per_second": 17.539,
+      "eval_steps_per_second": 17.539,
+      "step": 332
+    },
+    {
+      "epoch": 5.304,
+      "grad_norm": 0.00011134936357848346,
+      "learning_rate": 3.0302904520452447e-05,
+      "loss": 0.0,
+      "step": 334
+    },
+    {
+      "epoch": 5.304,
+      "eval_loss": 9.440830581297632e-06,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 675762.0,
+      "eval_runtime": 11.4044,
+      "eval_samples_per_second": 17.537,
+      "eval_steps_per_second": 17.537,
+      "step": 334
+    },
+    {
+      "epoch": 5.336,
+      "grad_norm": 0.00011384551180526614,
+      "learning_rate": 2.9663167846209998e-05,
+      "loss": 0.0,
+      "step": 336
+    },
+    {
+      "epoch": 5.336,
+      "eval_loss": 9.757546649780124e-06,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 679834.0,
+      "eval_runtime": 11.3063,
+      "eval_samples_per_second": 17.689,
+      "eval_steps_per_second": 17.689,
+      "step": 336
+    },
+    {
+      "epoch": 5.368,
+      "grad_norm": 0.0001350200327578932,
+      "learning_rate": 2.9027395877691144e-05,
+      "loss": 0.0,
+      "step": 338
+    },
+    {
+      "epoch": 5.368,
+      "eval_loss": 9.599931217962876e-06,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 683864.0,
+      "eval_runtime": 11.4352,
+      "eval_samples_per_second": 17.49,
+      "eval_steps_per_second": 17.49,
+      "step": 338
+    },
+    {
+      "epoch": 5.4,
+      "grad_norm": 0.00011894023191416636,
+      "learning_rate": 2.8395712559900877e-05,
+      "loss": 0.0,
+      "step": 340
+    },
+    {
+      "epoch": 5.4,
+      "eval_loss": 9.164205039269291e-06,
+      "eval_mean_token_accuracy": 1.0,
+      "eval_num_tokens": 688042.0,
+      "eval_runtime": 11.3082,
+      "eval_samples_per_second": 17.686,
+      "eval_steps_per_second": 17.686,
+      "step": 340
+    }
+  ],
+  "logging_steps": 2,
+  "max_steps": 500,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 9,
+  "save_steps": 2,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 20,
+        "early_stopping_threshold": 0.0
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.1155371725799424e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/checkpoint-340/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c8be94a11838b4d50c6aec22352c893c29f922bf66e333e2e30a75634facb8ed
+size 6008

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/special_tokens_map.json

@@ -0,0 +1,17 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|eot_id|>"
+}

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
+size 17209920

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/tokenizer_config.json

@@ -0,0 +1,2064 @@
+{
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|begin_of_text|>",
+  "chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n    {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n    {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n    {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n    {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n    {%- set system_message = messages[0]['content']|trim %}\n    {%- set messages = messages[1:] %}\n{%- else %}\n    {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n    {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n    {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n    {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n    {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n    {{- \"Do not use variables.\\n\\n\" }}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n    {#- Extract the first user message so we can plug it in here #}\n    {%- if messages | length != 0 %}\n        {%- set first_user_message = messages[0]['content']|trim %}\n        {%- set messages = messages[1:] %}\n    {%- else %}\n        {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n    {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n    {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n    {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n    {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n    {{- \"Do not use variables.\\n\\n\" }}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n    {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n    {%- elif 'tool_calls' in message %}\n        {%- if not message.tool_calls|length == 1 %}\n            {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n        {%- endif %}\n        {%- set tool_call = message.tool_calls[0].function %}\n        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n            {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n            {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n            {%- for arg_name, arg_val in tool_call.arguments | items %}\n                {{- arg_name + '=\"' + arg_val + '\"' }}\n                {%- if not loop.last %}\n                    {{- \", \" }}\n                {%- endif %}\n                {%- endfor %}\n            {{- \")\" }}\n        {%- else  %}\n            {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n            {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n            {{- '\"parameters\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- \"}\" }}\n        {%- endif %}\n        {%- if builtin_tools is defined %}\n            {#- This means we're in ipython mode #}\n            {{- \"<|eom_id|>\" }}\n        {%- else %}\n            {{- \"<|eot_id|>\" }}\n        {%- endif %}\n    {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n        {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n        {%- if message.content is mapping or message.content is iterable %}\n            {{- message.content | tojson }}\n        {%- else %}\n            {{- message.content }}\n        {%- endif %}\n        {{- \"<|eot_id|>\" }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "pad_token": "<|eot_id|>",
+  "tokenizer_class": "PreTrainedTokenizer"
+}

llama_finetune_web_of_lies_r16_alpha=32_dropout=0.05_lr0.0001_data_size1000_max_steps=500_seed=123/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c8be94a11838b4d50c6aec22352c893c29f922bf66e333e2e30a75634facb8ed
+size 6008