初始化项目，由ModelHub XC社区提供模型
Model: beomi/OPEN-SOLAR-KO-10.7B Source: Original Platform
This commit is contained in:
ModelHub XC
125
README.md
Normal file
125
README.md
Normal file
@@ -0,0 +1,125 @@
|
||||
---
|
||||
language:
|
||||
- ko
|
||||
- en
|
||||
pipeline_tag: text-generation
|
||||
inference: false
|
||||
tags:
|
||||
- solar
|
||||
- mistral
|
||||
- pytorch
|
||||
- solar-ko
|
||||
library_name: transformers
|
||||
license: apache-2.0
|
||||
---
|
||||
|
||||
**Update Log**
|
||||
|
||||
- 2024.01.08: Initial Test version Release of Solar-Ko
|
||||
|
||||
# **Open-Solar-Ko** ⭐🇰🇷
|
||||
|
||||
Solar-Ko represents an advanced iteration of the upstage/SOLAR-10.7B-v1.0 model, featuring an expanded vocabulary and the inclusion of a Korean corpus for enhanced pretraining.
|
||||
|
||||
Open-Solar-Ko exclusively utilizes publicly accessible Korean corpora, including sources such as [AI Hub](https://www.aihub.or.kr), [Modu Corpus, 모두의 말뭉치](https://corpus.korean.go.kr/), and [Korean Wikipedia](https://dumps.wikimedia.org/kowiki/).
|
||||
|
||||
As training was conducted solely with publicly available corpora, this model is open for unrestricted use by everyone, adhering to the Apache2.0 open source License.
|
||||
|
||||
## Model Details
|
||||
|
||||
**Model Developers:** Junbum Lee (Beomi)
|
||||
|
||||
**Variations:** Solar-Ko is available with one parameter sizes — 10B with Continual Pretrained version.
|
||||
|
||||
**Input:** The model accepts only text input.
|
||||
|
||||
**Output:** The model produces text output exclusively.
|
||||
|
||||
**Model Architecture:**
|
||||
|
||||
SOLAR-KO-10.7B is an auto-regressive language model that leverages an optimized transformer architecture derived from Llama-2.
|
||||
|
||||
| |Training Data|Parameters|Content Length|GQA|Tokens|Learning Rate|
|
||||
|---|---|---|---|---|---|---|
|
||||
|SOLAR-KO-10.7B|*A curated mix of Publicly Accessible Korean Corpora*|10.7B|4k|O|>15B*|5e<sup>-5</sup>|
|
||||
|
||||
**Training Corpus**
|
||||
|
||||
The model was trained using selected datasets from AIHub and Modu Corpus. Detailed information about the training datasets is available below:
|
||||
|
||||
- AI Hub: [corpus/AI_HUB](./corpus/AI_HUB)
|
||||
- Only the `Training` segment of the data was used.
|
||||
- The `Validation` and `Test` segments were deliberately excluded.
|
||||
- Modu Corpus: [corpus/MODU_CORPUS](./corpus/MODU_CORPUS)
|
||||
|
||||
The final JSONL dataset used to train this model is approximately 61GB in size.
|
||||
|
||||
Total token count: Approximately 15 billion tokens (*using the expanded tokenizer. With the original SOLAR tokenizer, >60 billion tokens.)
|
||||
|
||||
**Vocab Expansion**
|
||||
|
||||
| Model Name | Vocabulary Size | Description |
|
||||
| --- | --- | --- |
|
||||
| Original Solar | 32000 | Sentencepiece BPE |
|
||||
| **Expanded SOLAR-KO-10.7B** | 46592 | Sentencepiece BPE. Added Korean vocab and merges |
|
||||
|
||||
**Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."**
|
||||
|
||||
- SOLAR-10.7B: 26 tokens
|
||||
- SOLAR-KO-10.7b: 8 tokens
|
||||
|
||||
| Model | Tokens |
|
||||
| --- | --- |
|
||||
| SOLAR-10.7B | `['▁', '안', '<0xEB>', '<0x85>', '<0x95>', '하', '세', '요', ',', '▁', '오', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '날', '<0xEC>', '<0x94>', '<0xA8>', '가', '▁', '좋', '네', '요', '.']` |
|
||||
| SOLAR-KO-10.7B | `['▁안녕', '하세요', ',', '▁오늘은', '▁날', '씨가', '▁좋네요', '.']` |
|
||||
|
||||
**Tokenizing "Meet 10.7B Solar: Elevating Performance with Upstage Depth UP Scaling!"**
|
||||
|
||||
- SOLAR-10.7B: 22 tokens
|
||||
- SOLAR-KO-10.7b: 22 tokens
|
||||
|
||||
| Model | Tokens |
|
||||
| --- | --- |
|
||||
| SOLAR-10.7B | `['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!']` |
|
||||
| SOLAR-KO-10.7B | `['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!']` |
|
||||
|
||||
# LICENSE
|
||||
|
||||
Apache 2.0
|
||||
|
||||
# **Model Benchmark**
|
||||
|
||||
## LM Eval Harness - Korean (polyglot branch)
|
||||
|
||||
- Used EleutherAI's lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot
|
||||
|
||||
| | 0 | 5 | 10 | 50 |
|
||||
|:---------------------------------|---------:|---------:|---------:|---------:|
|
||||
| kobest_boolq (macro_f1) | 0.853949 | 0.88098 | 0.898139 | 0.902354 |
|
||||
| kobest_copa (macro_f1) | 0.804531 | 0.826736 | 0.837656 | 0.860899 |
|
||||
| kobest_hellaswag (macro_f1) | 0.507174 | 0.500983 | 0.487287 | 0.512182 |
|
||||
| kobest_sentineg (macro_f1) | 0.3517 | 0.972291 | 0.977321 | 0.984884 |
|
||||
| kohatespeech (macro_f1) | 0.258111 | 0.403957 | 0.386808 | 0.462393 |
|
||||
| kohatespeech_apeach (macro_f1) | 0.337667 | 0.651697 | 0.705337 | 0.827757 |
|
||||
| kohatespeech_gen_bias (macro_f1) | 0.124535 | 0.503464 | 0.498501 | 0.443218 |
|
||||
| korunsmile (f1) | 0.3814 | 0.356939 | 0.369989 | 0.296193 |
|
||||
| nsmc (acc) | 0.5356 | 0.87162 | 0.88654 | 0.89632 |
|
||||
| pawsx_ko (acc) | 0.5435 | 0.5245 | 0.5315 | 0.5385 |
|
||||
|
||||
## Citation
|
||||
|
||||
```
|
||||
@misc {solar_ko_junbum_2023,
|
||||
author = { {L. Junbum} },
|
||||
title = { Solar-Ko-10.7b },
|
||||
year = 2024,
|
||||
url = { https://huggingface.co/beomi/SOLAR-KO-10.7B },
|
||||
publisher = { Hugging Face }
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
- Training support was provided by the [TPU Research Cloud](https://sites.research.google/trc/) program.
|
||||
- The training corpus includes data from [AI Hub](https://www.aihub.or.kr/), [Modu Corpus](https://corpus.korean.go.kr/), and [Korean Wikipedia](https://dumps.wikimedia.org/kowiki/).
|
||||
Reference in New Issue
Block a user