初始化项目，由ModelHub XC社区提供模型
Model: aari1995/germeo-7b-laser Source: Original Platform
This commit is contained in:
ModelHub XC
249
README.md
Normal file
249
README.md
Normal file
@@ -0,0 +1,249 @@
|
||||
---
|
||||
language:
|
||||
- de
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- hermeo
|
||||
- laser
|
||||
datasets:
|
||||
- LeoLM/OpenSchnabeltier
|
||||
pipeline_tag: conversational
|
||||
model-index:
|
||||
- name: germeo-7b-laser
|
||||
results:
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: AI2 Reasoning Challenge (25-Shot)
|
||||
type: ai2_arc
|
||||
config: ARC-Challenge
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 25
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 60.75
|
||||
name: normalized accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aari1995/germeo-7b-laser
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: HellaSwag (10-Shot)
|
||||
type: hellaswag
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 10
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 82.81
|
||||
name: normalized accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aari1995/germeo-7b-laser
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: MMLU (5-Shot)
|
||||
type: cais/mmlu
|
||||
config: all
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 60.57
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aari1995/germeo-7b-laser
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: TruthfulQA (0-shot)
|
||||
type: truthful_qa
|
||||
config: multiple_choice
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 0
|
||||
metrics:
|
||||
- type: mc2
|
||||
value: 53.83
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aari1995/germeo-7b-laser
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: Winogrande (5-shot)
|
||||
type: winogrande
|
||||
config: winogrande_xl
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 75.61
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aari1995/germeo-7b-laser
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: GSM8k (5-shot)
|
||||
type: gsm8k
|
||||
config: main
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 43.37
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=aari1995/germeo-7b-laser
|
||||
name: Open LLM Leaderboard
|
||||
---
|
||||
|
||||
(Evaluation WIP)
|
||||
|
||||
## Hermes + Leo + German Laser = Germeo
|
||||
|
||||
## Germeo-7B-Laser
|
||||
A German-English understanding, but German-only speaking model merged from Hermeo-7B.
|
||||
|
||||
### Model details
|
||||
|
||||
**Merged from**: leo-mistral-hessianai-7b-chat and DPOpenHermes-7B-v2
|
||||
|
||||
**Model type**: Causal decoder-only transformer language model
|
||||
|
||||
**Languages**: German replies with English Understanding Capabilities
|
||||
|
||||
**Laser-Data**: LeoLM/OpenSchnabeltier
|
||||
|
||||
|
||||
This is an early experiment on laser and its influence on language understanding. It generally improves the language understanding capabilities.
|
||||
The hypothesis is that it degrades the probability of English replies and increasing those of German replies. The models internal German capabilities are boosted.
|
||||
|
||||
Will keep you updated..
|
||||
|
||||
### Acknowledgements:
|
||||
|
||||
I would like to thank everyone that participated in making this model and its training possible:
|
||||
To [@malteos](https://huggingface.co/malteos) for hermeo
|
||||
To [@cognitivecomputations](https://huggingface.co/cognitivecomputations) and Fernando Fernandes Neto for their implementation of LASER
|
||||
To [@LeoLM](https://huggingface.co/LeoLM) and Björn for the OpenSchnabeltier dataset.
|
||||
|
||||
|
||||
### Prompt format:
|
||||
|
||||
```python
|
||||
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
|
||||
# Convert prompt to tokens
|
||||
prompt_template = """<|im_start|>system
|
||||
Du bist ein hilfreicher Assistent.<|im_end|>
|
||||
<|im_start|>user
|
||||
{prompt}<|im_end|>
|
||||
<|im_start|>assistant"""
|
||||
|
||||
prompt = "Schreibe eine Stellenanzeige für Data Scientist bei AXA!"
|
||||
|
||||
final_prompt = prompt_template.format(prompt=prompt)
|
||||
```
|
||||
|
||||
#### Limit the model to output reply-only:
|
||||
To solve this, you need to implement a custom stopping criteria:
|
||||
|
||||
```python
|
||||
from transformers import StoppingCriteria
|
||||
class GermeoStoppingCriteria(StoppingCriteria):
|
||||
def __init__(self, target_sequence, prompt):
|
||||
self.target_sequence = target_sequence
|
||||
self.prompt=prompt
|
||||
|
||||
def __call__(self, input_ids, scores, **kwargs):
|
||||
# Get the generated text as a string
|
||||
generated_text = tokenizer.decode(input_ids[0])
|
||||
generated_text = generated_text.replace(self.prompt,'')
|
||||
# Check if the target sequence appears in the generated text
|
||||
if self.target_sequence in generated_text:
|
||||
return True # Stop generation
|
||||
|
||||
return False # Continue generation
|
||||
|
||||
def __len__(self):
|
||||
return 1
|
||||
|
||||
def __iter__(self):
|
||||
yield self
|
||||
```
|
||||
This then expects your input prompt (formatted as given into the model), and a stopping criteria, in this case the im_end token. Simply add it to the generation:
|
||||
|
||||
```python
|
||||
generation_output = model.generate(
|
||||
tokens,
|
||||
streamer=streamer,
|
||||
max_new_tokens=1012,
|
||||
stopping_criteria=GermeoStoppingCriteria("<|im_end|>", prompt_template.format(prompt=prompt))
|
||||
)
|
||||
```
|
||||
|
||||
### German benchmarks
|
||||
|
||||
| **German tasks:** | **MMLU-DE** | **Hellaswag-DE** | **ARC-DE** |**Average** |
|
||||
|-------------------------------|-------------|---------------|--------------|--------------|
|
||||
| **Models / Few-shots:** | _(5 shots)_ | _(10 shots)_ | _(24 shots)_ | |
|
||||
| _7B parameters_ | | | | |
|
||||
| llama-2-7b | 0.400 | 0.513 | 0.381 | 0.431 |
|
||||
| leo-hessianai-7b | 0.400 | 0.609 | 0.429 | 0.479 |
|
||||
| bloom-6b4-clp-german | 0.274 | 0.550 | 0.351 | 0.392 |
|
||||
| mistral-7b | **0.524** | 0.588 | 0.473 | 0.528 |
|
||||
| leo-mistral-hessianai-7b | 0.481 | 0.663 | 0.485 | 0.543 |
|
||||
| leo-mistral-hessianai-7b-chat | 0.458 | 0.617 | 0.465 | 0.513 |
|
||||
| DPOpenHermes-7B-v2 | 0.517 | 0.603 | 0.515 | 0.545 |
|
||||
| hermeo-7b | 0.511 | **0.668** | **0.528** | **0.569** |
|
||||
| **germeo-7b-laser (this model)**| ? | ? | ? | ? |
|
||||
| _13B parameters_ | | | | |
|
||||
| llama-2-13b | 0.469 | 0.581 | 0.468 | 0.506 |
|
||||
| leo-hessianai-13b | **0.486** | **0.658** | **0.509** | **0.551** |
|
||||
| _70B parameters_ | | | | |
|
||||
| llama-2-70b | 0.597 | 0.674 | 0.561 | 0.611 |
|
||||
| leo-hessianai-70b | **0.653** | **0.721** | **0.600** | **0.658** |
|
||||
|
||||
|
||||
Even though the model does not generate English text without being explicitly asked, performance on English Benchmarks is still up:
|
||||
|
||||
### English benchmarks
|
||||
|
||||
| **English tasks:** | **MMLU** | **Hellaswag** | **ARC** | **Average** |
|
||||
|------------------------------------|-------------|---------------|--------------|-------------|
|
||||
| **Models / Few-shots:** | _(5 shots)_ | _(10 shots)_ | _(24 shots)_ | |
|
||||
| llama-2-7b | 0.466 | 0.786 | 0.530 | 0.594 |
|
||||
| leolm-hessianai-7b | 0.423 | 0.759 | 0.522 | 0.568 |
|
||||
| bloom-6b4-clp-german | 0.264 | 0.525 | 0.328 | 0.372 |
|
||||
| mistral-7b | **0.635** | **0.832** | 0.607 | **0.691** |
|
||||
| leolm-mistral-hessianai-7b | 0.550 | 0.777 | 0.518 | 0.615 |
|
||||
| hermeo-7b | 0.601 | 0.821 | **0.620** | 0.681 |
|
||||
| germeo-7b-laser (this model) | 0.601 | 0.828 | 0.608 | 0.679 |
|
||||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_aari1995__germeo-7b-laser)
|
||||
|
||||
| Metric |Value|
|
||||
|---------------------------------|----:|
|
||||
|Avg. |62.82|
|
||||
|AI2 Reasoning Challenge (25-Shot)|60.75|
|
||||
|HellaSwag (10-Shot) |82.81|
|
||||
|MMLU (5-Shot) |60.57|
|
||||
|TruthfulQA (0-shot) |53.83|
|
||||
|Winogrande (5-shot) |75.61|
|
||||
|GSM8k (5-shot) |43.37|
|
||||
|
||||
Reference in New Issue
Block a user