Retrain v0.3.1 with 13 modern <|...|> special tokens (chat / code-block / tool-output boundaries; benchmark unchanged)

Promote v0.3.1 to main (4-domain SOTA at 100k vocab)

Honest OOD FineWeb benchmark: lead AR +13.5% vs GPT-4o, lose EN by 6.2%

Honest OOD FineWeb benchmark: lead AR +13.5% vs GPT-4o, lose EN by 6.2%

Honest OOD FineWeb benchmark: lead AR +13.5% vs GPT-4o, lose EN by 6.2%

Update README.md

Link to public SARFTokenizer-benchmark-eval dataset for reproducibility

README: add Gemma-4, Qwen3.6, Kimi-K2.6 benchmarks + head-to-head vs flagships

README: add Gemma-4, Qwen3.6, Kimi-K2.6 benchmarks + head-to-head vs flagships

Update README.md

Update README.md

Update README: Colab-ready code, benchmark, troubleshooting

v0.2: Unigram LM at 65k vocab — beats GPT-4o on AR and EN at 1/3 vocab

v0.2: Unigram LM at 65k vocab — beats GPT-4o on AR and EN at 1/3 vocab

v0.2: Unigram LM at 65k vocab — beats GPT-4o on AR and EN at 1/3 vocab

v0.2: Unigram LM at 65k vocab — beats GPT-4o on AR and EN at 1/3 vocab

v0.2: Unigram LM at 65k vocab — beats GPT-4o on AR and EN at 1/3 vocab

v0.2: Unigram LM at 65k vocab — beats GPT-4o on AR and EN at 1/3 vocab

Drop Fanar-2-Diwan from benchmark (unfair Arabic-maximalist peer)

Drop Fanar-2-Diwan from benchmark (unfair Arabic-maximalist peer)

Add benchmark_results.json: v0.1 benchmark vs 11 tokenizers

Add BENCHMARK.md: v0.1 benchmark vs 11 tokenizers

Clean up stale files from prior repo state

v0.1: initial SARFTokenizer release

Delete morfessor_models

Delete lexicons

Upload lexicons/lexicon_en.txt with huggingface_hub

Upload lexicons/lexicon_ar.txt with huggingface_hub

Upload morfessor_models/morf_map_reverse.json with huggingface_hub

Upload morfessor_models/morf_map.json with huggingface_hub

Upload morfessor_models/morfessor_en.bin with huggingface_hub

Upload morfessor_models/morfessor_ar.bin with huggingface_hub

Update README.md

Upload benchmark_pypi_full.py with huggingface_hub

Delete benchmark_pypi.py with huggingface_hub

Upload tokenizer_config.json with huggingface_hub

Upload special_tokens_map.json with huggingface_hub

Upload tokenizer.json with huggingface_hub

Update README.md

Update README.md

Update README.md

Update benchmark results with new tokenizers (Falcon-H1, ALLaM, Hala, Mistral)

Upload benchmark_parallel_results.json with huggingface_hub

Upload test_comprehensive_results.json with huggingface_hub

Upload BENCHMARK_REPORT.md with huggingface_hub

Upload test_comprehensive_million.py with huggingface_hub

Upload benchmark_tiktoken_style.py with huggingface_hub

Upload README.md with huggingface_hub

Upload README.md with huggingface_hub

Upload README.md with huggingface_hub