The BPE Model is wrong
#3
by
lianrzh
- opened
the vocab size of the bpe model is 500.
Can you reupload the right bpe model which vocab size is 6254 ?
vocab_set = english_bpe_set + chinese_char_set, please check the tokens.txt file for the chinese-english-mixed vocabulary
pfluo
changed discussion status to
closed
yet can I generate the chinese-english-mixed bpe-model with the tokens.txt?