49 lines
3.5 KiB
Markdown
49 lines
3.5 KiB
Markdown
---
|
|
license: other
|
|
---
|
|
### Introduction
|
|
Basiclly an update to the old attempt of [vicuna-chinese-replication-beta](https://huggingface.co/keyfan/vicuna-chinese-replication-beta)
|
|
* We adopted an curriculum-learning like approch, starting from simple QAs to reasoning-intensive coding & mathamatical problems. Coincidentally, [Ziya](https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-13B-v1) adopted the same idea during SFT period.
|
|
* The base model was changed from [chinese-llama](https://huggingface.co/ziqingyang/chinese-llama-lora-13b) to [chinese-llama-plus](https://huggingface.co/ziqingyang/chinese-llama-plus-lora-13b). However, as observed by [BiLLa](https://github.com/Neutralzz/BiLLa), continue training on Chinese-only corpus significantly increases its perplexity on English corpus, which in turns undermines its abilities in fields like mathematical calculation in our preliminary experiment. The subject of continuing-training is under-studied, while using bilingual corpus may be a better alternative as shown so far.
|
|
* We changed to the Vicuna v1.1 conversative template and included more CoT training data.
|
|
|
|
Again, this is for research purpose only. There's no guarantee for its performance. All credits to the original authors of LLaMA and Chinese-LLaMA.
|
|
|
|
Compared with previous release, new model improves on coding and reasoning problem. However it still suffers from hallucinations and perform poorly on Chinese domain-specific problem, e.g. chinese literature and idioms.
|
|
|
|
### Usage
|
|
|
|
We use exactly the Vicuna template for training and inference. Sample code as below.
|
|
|
|
```
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
checkpoint = "keyfan/vicuna-chinese-replication-v1.1"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(checkpoint, use_fast=False)
|
|
model = AutoModelForCausalLM.from_pretrained(checkpoint).cuda()
|
|
|
|
template = ("A chat between a curious human and an artificial intelligence assistant. "
|
|
"The assistant gives helpful, detailed, and polite answers to the human's questions. "
|
|
"USER: {}\nASSISTANT:")
|
|
question = template.format("Who was the president of the United States in 1955?")
|
|
inputs = tokenizer.encode(question, return_tensors="pt").cuda()
|
|
outputs = model.generate(inputs, do_sample=True, temperature=0.2, max_new_tokens=512)
|
|
print(tokenizer.decode(outputs[0]))
|
|
```
|
|
|
|
### Evaluation
|
|
|
|
* Result on the [Chinese-LLaMA-Alpaca devset](https://github.com/ymcui/Chinese-LLaMA-Alpaca/tree/main/examples) compared with the result of Alpaca-Plus-13B. For simplity, we only sample one answer for each question without any cherry-picking. We used the template as provided in their repo. GPT-4 have strong bias for more detailed answers, so the score may not be consistent with human evaluation.
|
|
|
|
| Model | Macro-Average | QA | OQA | REASONING | LITERATURE | ENTERTAINMENT | GENERATION | TRANSLATION | CODE | ETHICS |
|
|
| - | - | - | - | - | - | - | - | - | - | - |
|
|
| Alpaca-Plus-13B | 77.3 | 70 | 74 | 70 | **80** | **77** | 82 | **89** | 64 | **90** |
|
|
| ours | **82.4** | **81** | **87** | **88** | 73 | **78** | **85** | 83 | **83** | 84 |
|
|
|
|
* Result on the newly released [C-Eval test set](https://cevalbenchmark.com/index.html#home) with 5-shot. We slightly modified [MOSS's code](https://github.com/SJTU-LIT/ceval/blob/main/code/evaluator_series/evaluators/moss.py) from ceval codebase by moving the '答案：' (Answer: ) suffix from the end of question to the beginning of the chatbot response.
|
|
|
|
| Average | Avg(Hard) | STEM | Social Science | Humanities | Others |
|
|
| - | - | - | - | - | - |
|
|
| 37.0 | 29.5 | 34.6 | 44.5 | 35.7 | 35.9 |
|