| | --- |
| | license: apache-2.0 |
| | language: |
| | - zh |
| | - en |
| | pipeline_tag: text-generation |
| | --- |
| | <div align="left"> |
| | <h1> |
| | Mengzi3-13B-Base |
| | </h1> |
| | </div> |
| |
|
| | <p align="center"> |
| | <img src="https://raw.githubusercontent.com/Langboat/Mengzi3/main/assets/mengzi_logo.png" width="200"/> |
| | <p> |
| | |
| | <p align="center"> |
| | 🤗 <a href="https://huggingface.co/Langboat">Hugging Face</a> | 🤖 <a href="https://modelscope.cn/organization/Langboat">ModelScope</a> | <a href="https://gitee.com/mindspore/mindformers/blob/r1.0/research/mengzi3/mengzi3.md"><img src="https://www.mindspore.cn/_static/logo-zh-light.99fc9222.svg" width="50" style="white-space: nowrap;display: inline-block;overflow: hidden;max-width: 100%;"/></a> | <a href="https://wisemodel.cn/organization/Langboat">Wisemodel</a> ｜ 💬 <a href="https://github.com/Langboat/Mengzi3/blob/main/assets/wechat.png">WeChat</a> | <a href="https://www.langboat.com/document/mengzi/mengzi-gpt/call">API</a> | <a href="https://www.langboat.com/portal/mengzi-gpt"><img src="https://raw.githubusercontent.com/Langboat/Mengzi3/main/assets/mengzi_logo.png" width="16" style="white-space: nowrap;display: inline-block;overflow: hidden;max-width: 100%;"/> 孟子GPT</a> |
| | </p> |
| | |
| | # 模型介绍/Introduction |
| |
|
| | 本次开源Mengzi3 13B系列模型，模型的地址如下: |
| |
|
| | | | Mengzi3-13B-Base | Mengzi3-13B-Chat | |
| | | :-: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------: | |
| | | 13B | [🤗](https://huggingface.co/Langboat/Mengzi3-13B-Base) / [🤖](https://modelscope.cn/Langboat/Mengzi3-13B-Base) / [MindSpore](https://gitee.com/mindspore/mindformers/blob/r1.0/research/mengzi3/mengzi3.md) / [Wisemodel](https://wisemodel.cn/models/Langboat/Mengzi3-13B-Base) | 敬请期待 | |
| |
|
| | Mengzi3-13B模型基于Llama架构，语料精选自网页、百科、社交、媒体、新闻，以及高质量的开源数据集。通过在万亿tokens上进行多语言语料的继续训练，模型的中文能力突出并且兼顾多语言能力。 |
| |
|
| | Mengzi3-13B is based on the Llama architecture, and the corpus is selected from web pages, encyclopedias, social networking, media, news, and high-quality open source data sets. By continuing to train multilingual corpus on trillions of tokens, the model has outstanding Chinese capabilities and takes into account multilingual capabilities. |
| |
|
| | # 快速开始/Quickstart |
| |
|
| | ```python |
| | import torch |
| | from transformers import AutoModelForCausalLM, AutoTokenizer |
| | |
| | tokenizer = AutoTokenizer.from_pretrained("Langboat/Mengzi3-13B-Base", use_fast=False, trust_remote_code=True) |
| | model = AutoModelForCausalLM.from_pretrained("Langboat/Mengzi3-13B-Base", device_map="auto", trust_remote_code=True) |
| | inputs = tokenizer('指令：回答以下问题。输入：介绍一下孟子。输出：', return_tensors='pt') |
| | if torch.cuda.is_available(): |
| | inputs = inputs.to('cuda') |
| | pred = model.generate(**inputs, max_new_tokens=512, repetition_penalty=1.01, eos_token_id=tokenizer.eos_token_id) |
| | print(tokenizer.decode(pred[0], skip_special_tokens=True)) |
| | ``` |
| |
|
| | 详细的模型推理和微调代码见[Github](https://github.com/Langboat/Mengzi3) |
| |
|
| | Detailed code of model reasoning and finetune see [Github](https://github.com/Langboat) |
| |
|
| | # 性能评测/Evaluation |
| |
|
| | Mengzi3-13B-Base在各项基准测试中与同等参数量大语言模型相比，语言能力成绩领先，数学和编程能力位于前列。 |
| |
|
| | Mengzi3-13B-Base leads in language proficiency and is at the forefront in math and programming proficiency compared to the equivalent large language model in various benchmark tests. |
| |
|
| | | | MMLU | CMMLU | OCNLI | GSM8K | HumanEval | |
| | | :------------------------: | :---------------------: | :---------------------: | :---------------------: | :---: | :-------: | |
| | | Baichuan2-13B-Base | 0.530 | 0.489 | 0.433 | 0.528 | 0.171 | |
| | | Qwen-14B | 0.589 | 0.539 | 0.550 | 0.613 | 0.323 | |
| | | ChatGLM3-6B-base | 0.551 | 0.495 | 0.754 | 0.723 | - | |
| | | InternLM2-20B | 0.610 | 0.538 | 0.650 | 0.761 | 0.488 | |
| | | Skywork-13B-base | 0.557 | 0.524 | 0.426 | 0.558 | - | |
| | | LingoWhale-8B | 0.541 | 0.495 | 0.352 | 0.550 | 0.329 | |
| | | DeepSeek-7B | 0.436 | 0.424 | 0.356 | 0.174 | 0.262 | |
| | | DeepSeek-MoE-16B-base | 0.423 | 0.388 | 0.342 | 0.188 | 0.268 | |
| | | MindSource-7B | 0.498 | 0.425 | 0.528 | - | - | |
| | | **Mengzi3-13B-Base** | **0.651 (+6.7%)** | **0.588 (+9.1%)** | **0.776 (+2.9%)** | 0.631 | 0.287 | |
| |
|
| | > 以上结果基于5-shot，MMLU/CMMLU/OCNLI结果来自[FlagEval](https://flageval.baai.ac.cn/) |
| | > |
| | > The above results are based on 5-shot，MMLU/CMMLU/OCNLI results from [FlagEval](https://flageval.baai.ac.cn/) |
| |
|
| | # 协议/License Agreement |
| |
|
| | Mengzi3-13B-Base依照Apache 2.0协议开源，对学术研究完全开放，同时支持免费商用。如需申请商业许可证，请[联系我们](https://www.langboat.com/form?p=3)，其他商务合作请联系[bd@langboat.com](mailto:bd@langboat.com)。 |
| |
|
| | Mengzi3-13B-Base is open source under the Apache 2.0 protocol, fully open for academic research, and free for commercial use. If you need to apply for business license, please [contact us](https://www.langboat.com/en/form?p=3), other business cooperation, please contact [bd@langboat.com](mailto:bd@langboat.com). |
| |
|
