初始化项目，由ModelHub XC社区提供模型
Model: davidkim205/Hunminai-1.0-12b Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
108
README.md
Normal file
108
README.md
Normal file
@@ -0,0 +1,108 @@
|
||||
---
|
||||
library_name: transformers
|
||||
language:
|
||||
- ko
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# Hunminai-1.0-12b
|
||||
|
||||
Hunminai-1.0 is a Korean-aligned language model based on [Google's Gemma-3](https://huggingface.co/collections/google/gemma-3-release-67c6c6f89c4f76621268bb6d) architecture. To improve performance on Korean natural language tasks, the model was fine-tuned on a corpus of 100k instruction examples using Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO). This approach enables the model to better align with user intents in Korean and enhances its applicability to downstream tasks such as dialogue generation, question answering, and long-form text generation.
|
||||
|
||||
## Model Details
|
||||
- **Base Model**: [google/gemma-3-12b-it](https://huggingface.co/google/gemma-3-12b-it)
|
||||
- **Base Model Release Date**: March 12, 2025
|
||||
- **Context Length**: 128k
|
||||
- **License**: [gemma](https://ai.google.dev/gemma/terms)
|
||||
- **Model Type**: Text Generation
|
||||
- **Fine-Tuning Techniques**: Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO)
|
||||
|
||||
## Usage
|
||||
|
||||
Gemma 3 is supported starting from version 4.50.0 of the Transformers library.
|
||||
|
||||
To update to the latest version, run the following command:
|
||||
```
|
||||
$ pip install -U transformers
|
||||
```
|
||||
|
||||
Install the required package and run the example code below to load the Hunminai-3-12B model and perform a simple Korean-language chat completion.
|
||||
|
||||
```python
|
||||
# pip install accelerate
|
||||
|
||||
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
|
||||
import torch
|
||||
|
||||
model_id = "davidkim205/Hunminai-1.0-12b"
|
||||
|
||||
model = Gemma3ForConditionalGeneration.from_pretrained(
|
||||
model_id, device_map="auto"
|
||||
).eval()
|
||||
|
||||
processor = AutoProcessor.from_pretrained(model_id)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": [{"type": "text", "text": "당신은 유용한 AI 비서입니다."}]
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "대한민국의 수도는 어디인가요?"}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
inputs = processor.apply_chat_template(
|
||||
messages, add_generation_prompt=True, tokenize=True,
|
||||
return_dict=True, return_tensors="pt"
|
||||
).to(model.device, dtype=torch.bfloat16)
|
||||
|
||||
input_len = inputs["input_ids"].shape[-1]
|
||||
|
||||
with torch.inference_mode():
|
||||
generation = model.generate(**inputs, max_new_tokens=128, do_sample=False)
|
||||
generation = generation[0][input_len:]
|
||||
|
||||
decoded = processor.decode(generation, skip_special_tokens=True)
|
||||
print(decoded)
|
||||
```
|
||||
|
||||
## Training Dataset
|
||||
|
||||
The model was trained on high-quality Korean instructional data. The dataset was curated to cover a broad range of Korean language contexts and task types, with a focus on aligning model outputs with user intent and natural language generation. It is currently not publicly available.
|
||||
|
||||
|
||||
## Evaluation
|
||||
|
||||
### Benchmarks Datasets
|
||||
|
||||
The table below contains a description of the Korean LLM evaluation benchmark dataset used for the model evaluation. More information on the benchmarks is available at [Blog](https://davidkim205.github.io/).
|
||||
|
||||
| Benchmark | Description | Abbreviation |
|
||||
|------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
|
||||
| [ko-bench](https://huggingface.co/datasets/davidkim205/ko-bench) | Korean-translated dataset of [MT-Bench](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/data/mt_bench/question.jsonl) questions | bench |
|
||||
| [ko-ged](https://huggingface.co/datasets/davidkim205/ko-ged) | Korean GED (elementary, middle, high school) open-ended question dataset<br/>Subjects: Korean, English, Mathematics, Science, Social Studies | ged |
|
||||
| [ko-ifeval](https://huggingface.co/datasets/davidkim205/ko-ifeval) | Instruction-following evaluation dataset translated from [IFEval](https://github.com/google-research/google-research/tree/master/instruction_following_eval), adapted for Korean language and culture | ifeval |
|
||||
| [ko-ged-mc-elementary](https://huggingface.co/datasets/davidkim205/ko-ged-mc-elementary) | Korean elementary school GED multiple-choice question dataset | ged\:E |
|
||||
| [ko-ged-mc-middle](https://huggingface.co/datasets/davidkim205/ko-ged-mc-middle) | Korean middle school GED multiple-choice question dataset | ged\:M |
|
||||
| [ko-ged-mc-high](https://huggingface.co/datasets/davidkim205/ko-ged-mc-high) | Korean high school GED multiple-choice question dataset | ged\:H |
|
||||
| [ko-gpqa](https://huggingface.co/datasets/davidkim205/ko-gpqa) | Korean version of GPQA containing challenging physics questions designed to test deep understanding and logical reasoning | gpqa |
|
||||
| [ko-math-500](https://huggingface.co/datasets/davidkim205/ko-math-500) | Korean-translated subset of 500 high school-level math problems from the MATH dataset, including detailed solutions with LaTeX notation | math500 |
|
||||
|
||||
### Benchmark Results
|
||||
|
||||
| | **davidkim205<br>Hunminai<br>-1.0-12b** | google<br>gemma-3<br>-12b-it | unsloth<br>gemma-3<br>-12b-it | K-intelligence<br>Midm-2.0<br>-Base-Instruct | LGAI-EXAONE<br>EXAONE-3.5<br>-7.8B-Instruct |
|
||||
|---------|----------------------------------------:|-----------------------------:|------------------------------:|---------------------------------------------:|--------------------------------------------:|
|
||||
| Avg. | **7.80** | 7.75 | 7.71 | 7.54 | 7.31 |
|
||||
| bench | 7.96 | 8.00 | 7.83 | **8.01** | 7.70 |
|
||||
| ged | 8.65 | 8.61 | **8.73** | 8.10 | 8.25 |
|
||||
| ged:E | **9.72** | **9.72** | 9.51 | **9.72** | 9.65 |
|
||||
| ged:M | **9.63** | 9.55 | 9.39 | 9.31 | 9.10 |
|
||||
| ged:H | 9.32 | 9.36 | 9.24 | **9.48** | 9.00 |
|
||||
| gpqa | **3.18** | 2.88 | 2.98 | 2.68 | 3.13 |
|
||||
| math500 | 5.60 | 5.58 | **5.70** | 4.80 | 4.88 |
|
||||
| ifeval | **8.37** | 8.30 | 8.33 | 8.24 | 6.76 |
|
||||
|
||||
3
chat_template.json
Normal file
3
chat_template.json
Normal file
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"chat_template": "{{ bos_token }}\n{%- if messages[0]['role'] == 'system' -%}\n {%- if messages[0]['content'] is string -%}\n {%- set first_user_prefix = messages[0]['content'] + '\n\n' -%}\n {%- else -%}\n {%- set first_user_prefix = messages[0]['content'][0]['text'] + '\n\n' -%}\n {%- endif -%}\n {%- set loop_messages = messages[1:] -%}\n{%- else -%}\n {%- set first_user_prefix = \"\" -%}\n {%- set loop_messages = messages -%}\n{%- endif -%}\n{%- for message in loop_messages -%}\n {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}\n {{ raise_exception(\"Conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif -%}\n {%- if (message['role'] == 'assistant') -%}\n {%- set role = \"model\" -%}\n {%- else -%}\n {%- set role = message['role'] -%}\n {%- endif -%}\n {{ '<start_of_turn>' + role + '\n' + (first_user_prefix if loop.first else \"\") }}\n {%- if message['content'] is string -%}\n {{ message['content'] | trim }}\n {%- elif message['content'] is iterable -%}\n {%- for item in message['content'] -%}\n {%- if item['type'] == 'image' -%}\n {{ '<start_of_image>' }}\n {%- elif item['type'] == 'text' -%}\n {{ item['text'] | trim }}\n {%- endif -%}\n {%- endfor -%}\n {%- else -%}\n {{ raise_exception(\"Invalid content type\") }}\n {%- endif -%}\n {{ '<end_of_turn>\n' }}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n {{'<start_of_turn>model\n'}}\n{%- endif -%}\n"
|
||||
}
|
||||
64
config.json
Normal file
64
config.json
Normal file
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"architectures": [
|
||||
"Gemma3ForConditionalGeneration"
|
||||
],
|
||||
"boi_token_index": 255999,
|
||||
"eoi_token_index": 256000,
|
||||
"eos_token_id": [
|
||||
1,
|
||||
106
|
||||
],
|
||||
"hidden_size": 3840,
|
||||
"image_token_index": 262144,
|
||||
"initializer_range": 0.02,
|
||||
"mm_tokens_per_image": 256,
|
||||
"model_type": "gemma3",
|
||||
"text_config": {
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"attn_logit_softcapping": null,
|
||||
"cache_implementation": "hybrid",
|
||||
"final_logit_softcapping": null,
|
||||
"head_dim": 256,
|
||||
"hidden_activation": "gelu_pytorch_tanh",
|
||||
"hidden_size": 3840,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 15360,
|
||||
"max_position_embeddings": 131072,
|
||||
"model_type": "gemma3_text",
|
||||
"num_attention_heads": 16,
|
||||
"num_hidden_layers": 48,
|
||||
"num_key_value_heads": 8,
|
||||
"query_pre_attn_scalar": 256,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_local_base_freq": 10000.0,
|
||||
"rope_scaling": {
|
||||
"factor": 8.0,
|
||||
"rope_type": "linear"
|
||||
},
|
||||
"rope_theta": 1000000.0,
|
||||
"sliding_window": 1024,
|
||||
"sliding_window_pattern": 6,
|
||||
"torch_dtype": "bfloat16",
|
||||
"use_cache": true,
|
||||
"vocab_size": 262208
|
||||
},
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.51.3",
|
||||
"use_cache": true,
|
||||
"vision_config": {
|
||||
"attention_dropout": 0.0,
|
||||
"hidden_act": "gelu_pytorch_tanh",
|
||||
"hidden_size": 1152,
|
||||
"image_size": 896,
|
||||
"intermediate_size": 4304,
|
||||
"layer_norm_eps": 1e-06,
|
||||
"model_type": "siglip_vision_model",
|
||||
"num_attention_heads": 16,
|
||||
"num_channels": 3,
|
||||
"num_hidden_layers": 27,
|
||||
"patch_size": 14,
|
||||
"torch_dtype": "bfloat16",
|
||||
"vision_use_head": false
|
||||
}
|
||||
}
|
||||
13
generation_config.json
Normal file
13
generation_config.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"bos_token_id": 2,
|
||||
"cache_implementation": "hybrid",
|
||||
"do_sample": true,
|
||||
"eos_token_id": [
|
||||
1,
|
||||
106
|
||||
],
|
||||
"pad_token_id": 0,
|
||||
"top_k": 64,
|
||||
"top_p": 0.95,
|
||||
"transformers_version": "4.51.3"
|
||||
}
|
||||
3
model-00001-of-00005.safetensors
Normal file
3
model-00001-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e8e7ed9d99bf28ac43ff5a257656cbf5baa4f8123056d2d288c11824ec955e16
|
||||
size 4979902192
|
||||
3
model-00002-of-00005.safetensors
Normal file
3
model-00002-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3017794b1e167a855b1ba1d5491b9f62eede6dd501df2c9fed722b1af5623746
|
||||
size 4931296592
|
||||
3
model-00003-of-00005.safetensors
Normal file
3
model-00003-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c7e5826cbbf5ab8b91d3df45af2eaea779d5725d2fda2fbc03267f89fae739e2
|
||||
size 4931296656
|
||||
3
model-00004-of-00005.safetensors
Normal file
3
model-00004-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:780b8d6d931358f198caa8a93ecb21b8808ec1db5c80aa0fb06e5c602092f79a
|
||||
size 4931296656
|
||||
3
model-00005-of-00005.safetensors
Normal file
3
model-00005-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:89756d7ca117dc533e60d43c42950ac7aa6034ecc27580aa691e86f210b37294
|
||||
size 4601000928
|
||||
1072
model.safetensors.index.json
Normal file
1072
model.safetensors.index.json
Normal file
File diff suppressed because it is too large
Load Diff
29
preprocessor_config.json
Normal file
29
preprocessor_config.json
Normal file
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"do_convert_rgb": null,
|
||||
"do_normalize": true,
|
||||
"do_pan_and_scan": null,
|
||||
"do_rescale": true,
|
||||
"do_resize": true,
|
||||
"image_mean": [
|
||||
0.5,
|
||||
0.5,
|
||||
0.5
|
||||
],
|
||||
"image_processor_type": "Gemma3ImageProcessor",
|
||||
"image_seq_length": 256,
|
||||
"image_std": [
|
||||
0.5,
|
||||
0.5,
|
||||
0.5
|
||||
],
|
||||
"pan_and_scan_max_num_crops": null,
|
||||
"pan_and_scan_min_crop_size": null,
|
||||
"pan_and_scan_min_ratio_to_activate": null,
|
||||
"processor_class": "Gemma3Processor",
|
||||
"resample": 2,
|
||||
"rescale_factor": 0.00392156862745098,
|
||||
"size": {
|
||||
"height": 896,
|
||||
"width": 896
|
||||
}
|
||||
}
|
||||
4
processor_config.json
Normal file
4
processor_config.json
Normal file
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"image_seq_length": 256,
|
||||
"processor_class": "Gemma3Processor"
|
||||
}
|
||||
36
special_tokens_map.json
Normal file
36
special_tokens_map.json
Normal file
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<end_of_turn>"
|
||||
],
|
||||
"boi_token": "<start_of_image>",
|
||||
"bos_token": {
|
||||
"content": "<bos>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eoi_token": "<end_of_image>",
|
||||
"eos_token": {
|
||||
"content": "<eos>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"image_token": "<image_soft_token>",
|
||||
"pad_token": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
|
||||
size 33384568
|
||||
51351
tokenizer_config.json
Normal file
51351
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user