42dot/42dot_LLM-PLM-1.3B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: 42dot/42dot_LLM-PLM-1.3B
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-05-10 18:49:55 +08:00
commit 02baa65a97
14 changed files with 151094 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

110
README.md Normal file
View File

@@ -0,0 +1,110 @@
---
language:
- en
- ko
pipeline_tag: text-generation
tags:
- pytorch
- llama
- causal-lm
- 42dot_llm
license: cc-by-nc-4.0
---
# 42dot_LLM-PLM-1.3B
**42dot LLM-PLM** is a pre-trained language model (PLM) developed by [**42dot**](https://42dot.ai/) and is a part of **42dot LLM** (large language model). 42dot LLM-PLM is pre-trained using Korean and English text corpus and can be used as a foundation language model for several Korean and English natural language tasks. This repository contains a 1.3B-parameter version of the model.
## Model Description
### Hyperparameters
42dot LLM-PLM is built upon a Transformer decoder architecture similar to the [LLaMA 2](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/) and its hyperparameters are listed below.
| Params | Layers | Attention heads | Hidden size | FFN size |
| -- | -- | -- | -- | -- |
| 1.3B | 24 | 32 | 2,048 | 5,632 |
### Pre-training
Pre-training took about 49K GPU hours (NVIDIA A100). Related settings are listed below.
| Params | Global batch size\* | Initial learning rate | Train iter.\* | Max length\* | Weight decay |
| -- | -- | -- | -- | -- | -- |
| 1.3B | 4.0M | 4E-4 | 1.4T | 4,096 | 0.1 |
(\* unit: tokens)
### Pre-training datasets
We used a set of publicly available text corpus, including:
- Korean: including [Jikji project](http://jikji.duckdns.org/), [mC4-ko](https://huggingface.co/datasets/mc4), [LBox Open](https://github.com/lbox-kr/lbox-open), [KLUE](https://huggingface.co/datasets/klue), [Wikipedia (Korean)](https://ko.wikipedia.org/) and so on.
- English: including [The Pile](https://github.com/EleutherAI/the-pile), [RedPajama](https://github.com/togethercomputer/RedPajama-Data), [C4](https://huggingface.co/datasets/c4) and so on.
### Tokenizer
The tokenizer is based on the Byte-level BPE algorithm. We trained its vocabulary from scratch using a subset of the pre-training corpus. For constructing a subset, 10M and 10M documents are sampled from Korean and English corpus respectively. The resulting vocabulary sizes about 50K.
### Zero-shot evaluations
We evaluate 42dot LLM-PLM on a variety of academic benchmarks both in Korean and English. All the results are obtained using [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot) and models released on the Hugging Face Hub.
#### Korean (KOBEST)
<figure align="center">
<img src="https://huggingface.co/42dot/42dot_LLM-PLM-1.3B/resolve/main/asset/42dot_LLM_PLM_KO_score_background.png"/>
</figure>
|Tasks / Macro-F1|[KoGPT2](https://github.com/SKT-AI/KoGPT2) <br>1.2B|[Polyglot-Ko](https://github.com/EleutherAI/polyglot) <br>1.3B|[XGLM](https://huggingface.co/facebook/xglm-1.7B) <br>1.7B|[PolyLM](https://huggingface.co/DAMO-NLP-MT/polylm-1.7b) <br>1.7B|42dot LLM-PLM <br>1.3B|
|--------------|-----------|----------------|---------|-----------|------------------------|
|boolq |0.337 |0.355 |**0.502** |0.334 |0.369 |
|copa |0.67 |**0.721** |0.616 |0.513 |0.704 |
|hellaswag |0.404 |0.401 |0.374 |0.321 |**0.431** |
|sentineg |0.606 |0.679 |0.46 |0.382 |**0.69** |
|**average** |0.504 |0.539 |0.488 |0.388 |**0.549** |
#### English
<figure align="center">
<img src="https://huggingface.co/42dot/42dot_LLM-PLM-1.3B/resolve/main/asset/42dot_LLM_EN_score_white_background.png"/>
</figure>
| Tasks / Metric | MPT <br>1B | OPT <br>1.3B | XGLM <br>1.7B | PolyLM <br>1.7B | 42dot LLM-PLM <br>1.3B |
| ---------------------- | ------ | -------- | --------- | ----------- | ------------------------ |
| anli_r1/acc | 0.309 | **0.341** | 0.334 | 0.336 | 0.325 |
| anli_r2/acc | 0.334 | 0.339 | 0.331 | 0.314 | **0.34** |
| anli_r3/acc | 0.33 | 0.336 | 0.333 | **0.339** | 0.333 |
| arc_challenge/acc | 0.268 | 0.234 | 0.21 | 0.198 | **0.288** |
| arc_challenge/acc_norm | 0.291 | 0.295 | 0.243 | 0.256 | **0.317** |
| arc_easy/acc | 0.608 | 0.571 | 0.537 | 0.461 | **0.628** |
| arc_easy/acc_norm | 0.555 | 0.51 | 0.479 | 0.404 | **0.564** |
| boolq/acc | 0.517 | 0.578 | 0.585 | 0.617 | **0.624** |
| hellaswag/acc | 0.415 | 0.415 | 0.362 | 0.322 | **0.422** |
| hellaswag/acc_norm | 0.532 | 0.537 | 0.458 | 0.372 | **0.544** |
| openbookqa/acc | **0.238** | 0.234 | 0.17 | 0.166 | 0.222 |
| openbookqa/acc_norm | 0.334 | 0.334 | 0.298 | 0.334 | **0.34** |
| piqa/acc | 0.714 | 0.718 | 0.697 | 0.667 | **0.725** |
| piqa/acc_norm | 0.72 | 0.724 | 0.703 | 0.649 | **0.727** |
| record/f1 | 0.84 | **0.857** | 0.775 | 0.681 | 0.848 |
| record/em | 0.832 | **0.849** | 0.769 | 0.674 | 0.839 |
| rte/acc | 0.541 | 0.523 | **0.559** | 0.513 | 0.542 |
| truthfulqa_mc/mc1 | 0.224 | 0.237 | 0.215 | **0.251** | 0.236 |
| truthfulqa_mc/mc2 | 0.387 | 0.386 | 0.373 | **0.428** | 0.387 |
| wic/acc | 0.498 | **0.509** | 0.503 | 0.5 | 0.502 |
| winogrande/acc | 0.574 | **0.595** | 0.55 | 0.519 | 0.583 |
| **average** | 0.479 | 0.482 | 0.452 | 0.429 | **0.492** |
## Limitations and Ethical Considerations
42dot LLM-PLM shares a number of well-known limitations of other large language models (LLMs). For example, it may generate false and misinformative content since 42dot LLM-PLM is also subject to [hallucination](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)). In addition, 42dot LLM-PLM may generate toxic, harmful, and biased content due to the use of web-available training data. We strongly suggest that 42dot LLM-PLM users should be aware of those limitations and take necessary steps to mitigate those issues.
## Disclaimer
The contents generated by 42dot LLM series ("42dot LLM") do not necessarily reflect the views or opinions of 42dot Inc. ("42dot"). 42dot disclaims any and all liability to any part for any direct, indirect, implied, punitive, special, incidental, or other consequential damages arising from any use of the 42dot LLM and its generated contents.
## License
The 42dot LLM-PLM is licensed under the Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0).
## Citation
```
@misc{42dot2023llm,
title={42dot LLM: A Series of Large Language Model by 42dot},
author={42dot Inc.},
year={2023},
url = {https://github.com/42dot/42dot_LLM},
version = {1.0.0},
}
```

46
added_tokens.json Normal file
View File

@@ -0,0 +1,46 @@
{
"<||unused10||>": 50269,
"<||unused11||>": 50270,
"<||unused12||>": 50271,
"<||unused13||>": 50272,
"<||unused14||>": 50273,
"<||unused15||>": 50274,
"<||unused16||>": 50275,
"<||unused17||>": 50276,
"<||unused18||>": 50277,
"<||unused19||>": 50278,
"<||unused1||>": 50260,
"<||unused20||>": 50279,
"<||unused21||>": 50280,
"<||unused22||>": 50281,
"<||unused23||>": 50282,
"<||unused24||>": 50283,
"<||unused25||>": 50284,
"<||unused26||>": 50285,
"<||unused27||>": 50286,
"<||unused28||>": 50287,
"<||unused29||>": 50288,
"<||unused2||>": 50261,
"<||unused30||>": 50289,
"<||unused31||>": 50290,
"<||unused32||>": 50291,
"<||unused33||>": 50292,
"<||unused34||>": 50293,
"<||unused35||>": 50294,
"<||unused36||>": 50295,
"<||unused37||>": 50296,
"<||unused38||>": 50297,
"<||unused39||>": 50298,
"<||unused3||>": 50262,
"<||unused40||>": 50299,
"<||unused41||>": 50300,
"<||unused42||>": 50301,
"<||unused43||>": 50302,
"<||unused44||>": 50303,
"<||unused4||>": 50263,
"<||unused5||>": 50264,
"<||unused6||>": 50265,
"<||unused7||>": 50266,
"<||unused8||>": 50267,
"<||unused9||>": 50268
}

BIN
asset/42dot_LLM_EN_score_white_background.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 79 KiB

BIN
asset/42dot_LLM_PLM_KO_score_background.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 83 KiB

27
config.json Normal file
View File

@@ -0,0 +1,27 @@
{
"_name_or_path": "42dot-PLM-1.3B",
"architectures": [
"LlamaForCausalLM"
],
"bos_token_id": 50257,
"eos_token_id": 50256,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.01,
"intermediate_size": 5632,
"max_position_embeddings": 4096,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 24,
"num_key_value_heads": 32,
"pad_token_id": 50258,
"pretraining_tp": 1,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"tie_word_embeddings": false,
"tokenizer_class": "GPT2TokenizerFast",
"torch_dtype": "float32",
"transformers_version": "4.31.0",
"use_cache": true,
"vocab_size": 50304
}

7
generation_config.json Normal file
View File

@@ -0,0 +1,7 @@
{
"_from_model_config": true,
"bos_token_id": 50257,
"eos_token_id": 50256,
"pad_token_id": 50258,
"transformers_version": "4.31.0"
}

50001
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5887cb28a2368a6d45beb52450ada1b4829744c8c75fdaa5e1e1c25727fd1936
size 5757114640

3
pytorch_model.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9cbe77e96064b09b39c6a8ffbacd8fb0d83237119eb65dac5a8fed57cd325b54
size 5757165853

58
special_tokens_map.json Normal file
View File

@@ -0,0 +1,58 @@
{
"additional_special_tokens": [
"<||unused1||>",
"<||unused2||>",
"<||unused3||>",
"<||unused4||>",
"<||unused5||>",
"<||unused6||>",
"<||unused7||>",
"<||unused8||>",
"<||unused9||>",
"<||unused10||>",
"<||unused11||>",
"<||unused12||>",
"<||unused13||>",
"<||unused14||>",
"<||unused15||>",
"<||unused16||>",
"<||unused17||>",
"<||unused18||>",
"<||unused19||>",
"<||unused20||>",
"<||unused21||>",
"<||unused22||>",
"<||unused23||>",
"<||unused24||>",
"<||unused25||>",
"<||unused26||>",
"<||unused27||>",
"<||unused28||>",
"<||unused29||>",
"<||unused30||>",
"<||unused31||>",
"<||unused32||>",
"<||unused33||>",
"<||unused34||>",
"<||unused35||>",
"<||unused36||>",
"<||unused37||>",
"<||unused38||>",
"<||unused39||>",
"<||unused40||>",
"<||unused41||>",
"<||unused42||>",
"<||unused43||>",
"<||unused44||>"
],
"bos_token": "<||bos||>",
"eos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"pad_token": "<||pad||>",
"unk_token": "<||unk||>"
}

100769
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

34
tokenizer_config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"bos_token": {
"__type": "AddedToken",
"content": "<||bos||>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"clean_up_tokenization_spaces": true,
"eos_token": {
"__type": "AddedToken",
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"errors": "replace",
"model_max_length": 8192,
"pad_token": null,
"padding_side": "right",
"tokenizer_class": "GPT2Tokenizer",
"unk_token": {
"__type": "AddedToken",
"content": "<||unk||>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
}
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long