Add better infilling documentation

README.md

@@ -40,7 +40,7 @@ CarperAI will be releasing larger LMs better tuned for code in the near future,
 | \\(n_{heads}\\)      | 16                                                                                                                                     |
 | \\(d_{head}\\)       | 128                                                                                                                                    |
 | \\(n_{ctx}\\)        | 2048                                                                                                                                   |
-| \\(n_{vocab}\\)      | 50254                                                                                                                     |
+| \\(n_{vocab}\\)      | 50280                                                                                                                     |
 | Positional Encoding  | [Rotary Position Embedding (RoPE)](https://arxiv.org/abs/2104.09864)                                                                   
 
 
@@ -105,27 +105,59 @@ language model output is generated after \<MID\> token!
 
 As a concrete example, here is a code snippet that should allow a model to perform infilling:
 
-```python
+There was an issue where the sentinel `<|SUF|>`, `<|PRE|>`, and `<|MID|>` tokens were not the correct ids in the uploaded tokenizer and model card! Please try clearing the Huggingface cache and redownloading the model :))
 
+Here is a minimal example of performing open-ended generation with this model, on a simple function `score(x, y)`:
+```
+def score(x,y) -> int:
+    """
+    
+```
 
-from transformers import AutoTokenizer, AutoModelForCausalLM
+and also infilling with the function and end of docstring already placed:
 
+```
+def score(x,y) -> int:
+    """
+    <|MID|> (infill here)
+    """
 
-tokenizer = AutoTokenizer.from_pretrained("CarperAI/FIM-NeoX-1.3B")
-model = AutoModelForCausalLM.from_pretrained("CarperAI/FIM-NeoX-1.3B")
+    score = x + y
+    return score
+```
 
-prelude = "this is some text preceding the cursor,"
-suffix = "and this is some text after it."
+```
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model = AutoModelForCausalLM.from_pretrained("CarperAI/FIM-NeoX-1.3B")
+tok = AutoTokenizer.from_pretrained("CarperAI/
 
+# infilling demo
+prefix = 'def score(x, y) -> int:\n"""\n'
+suffix = '"""\n\n    score = x + y\n    return score'
 
-model_tokenized_input = [50253, *tokenizer(suffix), 50254, *tokenizer(prelude), 50255]
+ model_input = [50277, *tok(suffix)["input_ids"], 50278, *tok(prefix)["input_ids"], 50279]
+ output = tok.decode(model.generate(torch.IntTensor(model_input).unsqueeze(0), max_length=40)[0])
 
-infilled = model.generate(model_tokenized_input)
+print(output)
+```
+outputs: `'<|SUF|>"""\n\n    score = x + y\n    return score<|PRE|>def score(x, y) -> int:\n"""\n<|MID|>    score(x, y) -> int\n<|endoftext|>'`
 
+```
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
 
+# non-infilling demo
+prefix = 'def score(x, y) -> int:\n"""\n'
+model_input = [*tok(prefix)["input_ids"]]
+output = tok.decode(model.generate(torch.IntTensor(model_input).unsqueeze(0), max_length=100)[0])
+print(output)
 ```
+outputs: `'def score(x, y) -> int:\n"""\n    Return the score of the given point.\n    """\n    return sum(x * y for x, y in zip(x_list, y_list))\n\ndef get_point_score(x, y) -> int:\n    """\n    Return the score of the given point.\n    """\n    return sum(x * y for x, y in zip(x_list, y'`
+
 
-We are working on making a better interface for this in future model releases or updates to the tokenizer.
+The sentinel tokens are now accessible via `tokenizer.decode(50277) = "<|SUF|>"`, `tokenizer.decode(50278) = "<|PRE|>"`, `tokenizer.decode(50279) = "<|MID|>"`. 
 
 
 ## Intended Uses and Limitations