初始化项目，由ModelHub XC社区提供模型
Model: DedeProGames/Kiyo-135M Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
146
README.md
Normal file
146
README.md
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
---
|
||||||
|
license: apache-2.0
|
||||||
|
datasets:
|
||||||
|
- HuggingFaceFW/fineweb-edu
|
||||||
|
- mlfoundations/dclm-baseline-1.0
|
||||||
|
- HuggingFaceTB/finemath
|
||||||
|
- HuggingFaceCode/stack-v3-train
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
library_name: transformers
|
||||||
|
base_model: HuggingFaceTB/SmolLM2-135M
|
||||||
|
tags:
|
||||||
|
- causal-lm
|
||||||
|
- language-model
|
||||||
|
- base-model
|
||||||
|
- llama
|
||||||
|
- smollm
|
||||||
|
- pytorch
|
||||||
|
---
|
||||||
|
|
||||||
|
# Kiyo-135M
|
||||||
|
|
||||||
|
Kiyo-135M is a decoder-only language model pretrained from scratch on 200B tokens drawn from [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0), [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath), and [Stack-v3-train](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train).
|
||||||
|
|
||||||
|
The model follows the [SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) architecture: a Llama-style decoder with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings. SmolLM2 was chosen as a base architecture because it is specifically tuned for small-scale pretraining efficiency — narrow hidden size with a comparatively deep layer stack, and a large 49k-token vocabulary that keeps sequence lengths short for a model this size. Kiyo-135M reuses this architecture but is trained independently from a random initialization on its own data mixture, rather than starting from SmolLM2's own weights.
|
||||||
|
|
||||||
|
## Model Details
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|---|---:|
|
||||||
|
| Parameters | 134,515,008 |
|
||||||
|
| Architecture | Llama-style decoder (SmolLM2 architecture) |
|
||||||
|
| Layers | 30 |
|
||||||
|
| Hidden size | 576 |
|
||||||
|
| Intermediate size | 1,536 |
|
||||||
|
| Attention heads | 9 |
|
||||||
|
| KV heads | 3 |
|
||||||
|
| Attention type | Grouped query attention |
|
||||||
|
| Activation | SwiGLU |
|
||||||
|
| Normalization | RMSNorm |
|
||||||
|
| Positional encoding | RoPE (theta 100,000) |
|
||||||
|
| Vocabulary size | 49,152 |
|
||||||
|
| Context length | 8,192 |
|
||||||
|
| Embeddings | Tied input/output embeddings |
|
||||||
|
| Training tokens | 200,000,000,000 |
|
||||||
|
| Weight format | safetensors |
|
||||||
|
|
||||||
|
## Training Data
|
||||||
|
|
||||||
|
| Source | Domain |
|
||||||
|
|---|---|
|
||||||
|
| FineWeb-Edu | General web text, education-filtered |
|
||||||
|
| DCLM-Baseline | General web text, high-quality filtered |
|
||||||
|
| FineMath | Mathematical reasoning |
|
||||||
|
| Stack-v3-train | Source code |
|
||||||
|
|
||||||
|
## Benchmarks
|
||||||
|
|
||||||
|
Self-reported results from the official [BananaMind Base Bench 1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1) script, all measured with the same runner, dtype (bfloat16) and GPU.
|
||||||
|
|
||||||
|
| Model | Params | Overall Elo |
|
||||||
|
|---|---:|---:|
|
||||||
|
| **Kiyo-135M** | **134.5M** | **1,126** |
|
||||||
|
| BananaMind-2-Pro | 139.0M | 1,124 |
|
||||||
|
| Rose-Pro | 151.3M | 1,105 |
|
||||||
|
| GPT-2 | 124M | 990 |
|
||||||
|
|
||||||
|
Figures for BananaMind-2-Pro, Rose-Pro, and GPT-2 are as self-reported on their own model cards, all against the same BananaMind Base Bench 1.1 suite.
|
||||||
|
|
||||||
|
**Detailed Kiyo-135M result**
|
||||||
|
|
||||||
|
| Category | Accuracy | z vs. chance | Elo | Significant |
|
||||||
|
|---|---:|---:|---:|:---:|
|
||||||
|
| Language completion | 100.0% | +12.25 | 1,570 | * |
|
||||||
|
| Code completion | 86.0% | +9.96 | 1,420 | * |
|
||||||
|
| World knowledge | 80.0% | +8.98 | 1,151 | * |
|
||||||
|
| Commonsense | 74.0% | +8.00 | 1,110 | * |
|
||||||
|
| Logical reasoning | 58.0% | +5.39 | 1,118 | * |
|
||||||
|
| Context tracking | 44.0% | +3.10 | 914 | * |
|
||||||
|
| Quantitative | 32.0% | +1.14 | 913 | |
|
||||||
|
|
||||||
|
\* = passes 1.96σ vs. chance; n=50 per category
|
||||||
|
|
||||||
|
**By difficulty**
|
||||||
|
|
||||||
|
| Difficulty | Accuracy |
|
||||||
|
|---|---:|
|
||||||
|
| Easy | 76.9% |
|
||||||
|
| Medium | 69.2% |
|
||||||
|
| Hard | 56.9% |
|
||||||
|
|
||||||
|
**Summary**
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|---|---:|
|
||||||
|
| Parameters | 134,515,008 |
|
||||||
|
| Overall Elo | 1,126 |
|
||||||
|
| Chance floor | 805 |
|
||||||
|
| Above chance floor | +321 |
|
||||||
|
| Raw accuracy | 67.7% |
|
||||||
|
|
||||||
|
Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -U transformers safetensors torch
|
||||||
|
```
|
||||||
|
|
||||||
|
```python
|
||||||
|
import torch
|
||||||
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||||
|
|
||||||
|
model_id = "DedeProGames/Kiyo-135M"
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||||
|
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
model_id,
|
||||||
|
torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
|
||||||
|
).cuda().eval()
|
||||||
|
|
||||||
|
prompt = "The meaning of life is "
|
||||||
|
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
|
||||||
|
|
||||||
|
with torch.no_grad():
|
||||||
|
output = model.generate(
|
||||||
|
input_ids=input_ids,
|
||||||
|
max_new_tokens=64,
|
||||||
|
do_sample=False,
|
||||||
|
repetition_penalty=1.1,
|
||||||
|
pad_token_id=tokenizer.eos_token_id,
|
||||||
|
eos_token_id=tokenizer.eos_token_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
This is a base model, not instruction-tuned — it continues text rather than following instructions. At 135M parameters it produces fluent, well-structured text and is strong on language completion and code, but accuracy drops on quantitative and multi-step context-tracking tasks. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Apache 2.0
|
||||||
31
config.json
Normal file
31
config.json
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
{
|
||||||
|
"_name_or_path": "/fsx/elie_bakouch/hf-ckpt/135M-nanotron/240000",
|
||||||
|
"architectures": [
|
||||||
|
"LlamaForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 0,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 576,
|
||||||
|
"initializer_range": 0.041666666666666664,
|
||||||
|
"intermediate_size": 1536,
|
||||||
|
"is_llama_config": true,
|
||||||
|
"max_position_embeddings": 2048,
|
||||||
|
"mlp_bias": false,
|
||||||
|
"model_type": "llama",
|
||||||
|
"num_attention_heads": 9,
|
||||||
|
"num_hidden_layers": 30,
|
||||||
|
"num_key_value_heads": 3,
|
||||||
|
"pretraining_tp": 1,
|
||||||
|
"rms_norm_eps": 1e-05,
|
||||||
|
"rope_interleaved": false,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 10000.0,
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"transformers_version": "4.44.0",
|
||||||
|
"use_cache": true,
|
||||||
|
"vocab_size": 49152
|
||||||
|
}
|
||||||
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"_from_model_config": true,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 0,
|
||||||
|
"transformers_version": "4.44.0"
|
||||||
|
}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:9f87d85cd26011bd0c1b5f89abc53b359d6396e8c793e026768e0ad74c12c6df
|
||||||
|
size 269060552
|
||||||
42
special_tokens_map.json
Normal file
42
special_tokens_map.json
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
{
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|endoftext|>",
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<repo_name>",
|
||||||
|
"<reponame>",
|
||||||
|
"<file_sep>",
|
||||||
|
"<filename>",
|
||||||
|
"<gh_stars>",
|
||||||
|
"<issue_start>",
|
||||||
|
"<issue_comment>",
|
||||||
|
"<issue_closed>",
|
||||||
|
"<jupyter_start>",
|
||||||
|
"<jupyter_text>",
|
||||||
|
"<jupyter_code>",
|
||||||
|
"<jupyter_output>",
|
||||||
|
"<jupyter_script>",
|
||||||
|
"<empty_output>"
|
||||||
|
],
|
||||||
|
"bos_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"unk_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
98249
tokenizer.json
Normal file
98249
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
168
tokenizer_config.json
Normal file
168
tokenizer_config.json
Normal file
@@ -0,0 +1,168 @@
|
|||||||
|
{
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"0": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"1": {
|
||||||
|
"content": "<|im_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"2": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"3": {
|
||||||
|
"content": "<repo_name>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"4": {
|
||||||
|
"content": "<reponame>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"5": {
|
||||||
|
"content": "<file_sep>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"6": {
|
||||||
|
"content": "<filename>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"7": {
|
||||||
|
"content": "<gh_stars>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"8": {
|
||||||
|
"content": "<issue_start>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"9": {
|
||||||
|
"content": "<issue_comment>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"10": {
|
||||||
|
"content": "<issue_closed>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"11": {
|
||||||
|
"content": "<jupyter_start>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"12": {
|
||||||
|
"content": "<jupyter_text>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"13": {
|
||||||
|
"content": "<jupyter_code>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"14": {
|
||||||
|
"content": "<jupyter_output>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"15": {
|
||||||
|
"content": "<jupyter_script>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"16": {
|
||||||
|
"content": "<empty_output>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|endoftext|>",
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<repo_name>",
|
||||||
|
"<reponame>",
|
||||||
|
"<file_sep>",
|
||||||
|
"<filename>",
|
||||||
|
"<gh_stars>",
|
||||||
|
"<issue_start>",
|
||||||
|
"<issue_comment>",
|
||||||
|
"<issue_closed>",
|
||||||
|
"<jupyter_start>",
|
||||||
|
"<jupyter_text>",
|
||||||
|
"<jupyter_code>",
|
||||||
|
"<jupyter_output>",
|
||||||
|
"<jupyter_script>",
|
||||||
|
"<empty_output>"
|
||||||
|
],
|
||||||
|
"bos_token": "<|endoftext|>",
|
||||||
|
"chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|endoftext|>",
|
||||||
|
"model_max_length": 1000000000000000019884624838656,
|
||||||
|
"tokenizer_class": "GPT2Tokenizer",
|
||||||
|
"unk_token": "<|endoftext|>",
|
||||||
|
"vocab_size": 49152
|
||||||
|
}
|
||||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user