SupraLabs/Supra2-100M-Base
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: SupraLabs/Supra2-100M-Base
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-05 09:25:13 +08:00
commit 0e60131b4f
322 changed files with 337974 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
.gitattributes vendored Normal file
View File

@@ -0,0 +1,48 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
"/tokenizer.json" filter=lfs diff=lfs merge=lfs -text

209
README.md Normal file
View File

@@ -0,0 +1,209 @@
---
license: apache-2.0
viewer: false
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceFW/dclm_100BT-shuffled
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- small
- supra
- supra2
- qwen3
- sota
---
<h1 align="center">Supra2-100M Base</h1>
<p align="center">
Base model of new Supra2 family • 100M Parameters • 1K Context - extended to 2k
</p>
![supra2-_pro_-100m](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/zMBPPr0y634MNUvx2Krqy.png)
**Supra2-100M Base** is a 100M-parameter decoder-only language model pretrained from scratch by **SupraLabs** on 30B tokens of English web text. It uses the **Qwen3** architecture with a custom 32,768-token tokenizer.
This is a **base model**. It has *not* been instruction-tuned, chat-tuned, or aligned in any way.
---
## **Let the model speak - a sample**
Prompt: "Artificial intelligence is "
Completion:
```plaintext
100% efficient, yet it still requires a lot of resources.
We are in the midst of a major transition from the traditional computer and software to artificial intelligence.
It is very important to have a strategy in mind. This is not only important for the companies that use it, but also for individuals and organizations that use it.
There is a huge difference between the two approaches. The artificial intelligence approach is based on computers and the software.
It has been proven that the artificial intelligence approach is more efficient and has a much better performance.
The benefits of artificial intelligence are also known to be immense. It is a powerful tool that can improve productivity, increase efficiency, and reduce costs.
It can be used for a wide range of tasks, from customer service to product development.
It is also used for various other tasks, such as sales, marketing, and customer service.
It is also used for various other tasks, such as customer service and product development.
There are a number of different types of artificial intelligence. These include machine learning, natural language processing, and neural networks.
The main advantage of using artificial intelligence is that it is more efficient and can be used for a wide range of tasks.
It is also[...]
```
---
## **Evaluation & Benchmarks**
All benchmarks were evaluated using the EleutherAI LM-Eval Harness.
| Model | PIQA (acc_norm) | HellaSwag (acc_norm) | ARC-Easy (acc_norm) | ARC-Challenge (acc_norm) |
| :--- | :---: | :---: | :---: | :---: |
| Supra-50M-Base | 0.62 | 0.32 | 0.46 | 0.25 |
| Supra2-100M Base | 0.65 | 0.36 | 0.48 | 0.25 |
| BananaMind-2-Pro-Preview-EXP | 0.67 | 0.40 | 0.51 | 0.27 |
| GPT-X-125M | 0.65 | 0.37 | 0.51 | 0.25 |
| OpenAI GPT-2 Small | 0.62 | 0.31 | 0.39 | 0.22 |
![grafik](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/TfH9iBYB0nNvms2-M84lG.png)
---
## **Model Details**
| | |
| ----- | ----- |
| **Developed by** | SupraLabs |
| **Model type** | Causal decoder-only transformer (Qwen3) |
| **Language** | English |
| **Parameters** | 100.7M total / 75.5M non-embedding |
| **Training tokens** | 30B (300 tokens per parameter) |
| **Context length** | 1,024 (trained) / 2,048 (config maximum, untested) |
| **Precision** | bfloat16 |
| **License** | Apache 2.0 |
### **Architecture**
| Hyperparameter | Value |
| ----- | ----- |
| Hidden size | 768 |
| Layers | 12 |
| Attention heads | 12 (GQA, 4 KV heads) |
| Head dim | 64 |
| Intermediate size (SwiGLU) | 2048 |
| Vocab size | 32,768 |
| Positional encoding | RoPE θ=10,000 |
| Normalization | RMSNorm, ε=10\-6 (incl. QK-Norm) |
| Tied embeddings | Yes |
| Sliding window | Disabled |
| Attention implementation | SDPA |
---
## **Training Data**
| Source | Share | Approx. tokens |
| ----- | ----- | ----- |
| `HuggingFaceFW/fineweb-edu` (`sample-350BT`) | 70% | 21B |
| `HuggingFaceFW/dclm_100BT-shuffled` | 30% | 9B |
Documents were tokenized with the custom `supra2-tokenizer`, concatenated into a flat `uint16` token stream, and packed into contiguous 1,024-token chunks (no padding, no document masking — sequences may cross document boundaries).
---
## **Training Procedure**
| Setting | Value |
| ----- | ----- |
| Optimizer | AdamW (fused), β1=0.9, β2=0.95, ε=10−8 |
| Peak learning rate | 1×10-3 |
| LR schedule | WSD (Warmup–Stable–Decay), `1-sqrt` decay to 0 |
| Total steps | 114k |
| Micro batch size | 16 |
| Gradient accumulation | 16 |
| Effective batch | 256 sequences \= **262,144 tokens/step** |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Auxiliary loss | Router-free **z-loss**, coefficient 10\-4 |
| Compile | `torch.compile` enabled |
| Hardware | 1× RTX 5090 (32GB) |
---
## **Usage**
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SupraLabs/Supra2-100M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model.eval()
prompt = "Artificial intelligence is "
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model.generate(
**inputs,
max_new_tokens=256,
do_sample=True,
temperature=0.3,
top_p=0.85,
top_k=15,
no_repeat_ngram_size=3,
repetition_penalty=1.1,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```
### **Tokenizer notes**
The tokenizer ships with a **ChatML** template and reserved special tokens (`<|im_start|>`, `<|im_end|>`, `<think>`, `<tool_call>`, `<|reserved_0..15|>`, …). These exist so that downstream fine-tunes can adopt them without resizing embeddings — **the base model has never seen them in training** and will not behave sensibly if you apply the chat template.
---
## **Intended Use**
**Intended:**
* Research on small-scale pretraining, scaling behavior, and data mixtures
* A cheap starting point for SFT / instruction tuning / domain adaptation
* Educational use, ablations, and fast iteration on limited hardware
**Not intended:**
* Any production or user-facing deployment
* Factual question answering, advice, or decision support
* Non-English text (essentially untrained; only supports english\!)
---
## **Limitations and Bias**
* **Small.** At 100M parameters, factual recall, reasoning, arithmetic, and long-range coherence are weak. Expect frequent hallucination and topic drift.
* **Base model.** No RLHF, no safety tuning, no refusal behavior. It will continue whatever text you give it, including harmful or offensive prompts.
* **Web-derived data.** FineWeb-Edu and DCLM are filtered CommonCrawl derivatives and carry the biases, stereotypes, and factual errors of the open web.
* **Short context.** Trained exclusively at 1,024 tokens. The config allows 2,048, but extrapolation beyond 1,024 is untested and likely degraded.
* **No document masking.** Attention could cross document boundaries within a packed chunk, which slightly blurs document independence.
* **English only.**
---
## **What comes next?**
We are already training the next models in the Supra2-family:
- Supra2-Nano (800k params)
- Supra2-Small (\~1.3M params)
- Supra2-Medium (25M params)
- Supra2-IMG: our SOTA small text-to-image model
---
*© SupraLabs 2026*

4
chat_template.jinja Normal file
View File

@@ -0,0 +1,4 @@
{% for m in messages %}<|im_start|>{{ m['role'] }}
{{ m['content'] }}<|im_end|>
{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
{% endif %}

47
checkpoint-102000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-102000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-102000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:666939d838e019555de4b93d40d83d92f53ea1a1a05e181299da911d951e7370
size 402750928

3
checkpoint-102000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cb5da146a575ed84aaa9da2d1fd7b2ec16b54a0721b7aae3612554f4bced6753
size 805587275

3
checkpoint-102000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-102000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dcf805774b0aca59023e26be3dc9930fa1826ad1af6787de79d1def3b62786ca
size 1465

14586
checkpoint-102000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-102000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-105000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-105000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-105000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b0ae6fc52f37239f69ae4557242bb1283eedc03cab29edd60194f6e8af07957b
size 402750928

3
checkpoint-105000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:66e4fd60492031e93dc1c18572839199b447b5f3977b45aa7cd75f5cb55ae8fa
size 805587275

3
checkpoint-105000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-105000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1d8b13e24d7330f786fe6cfbe0273dcff98e71aca7d66e1d700ebba7d8fa3f19
size 1465

15014
checkpoint-105000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-105000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-108000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-108000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-108000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f6eb11a48287fbe6a9f1404f39f78737cf4a81fa7adec0be1cfcefba3a2eef1e
size 402750928

3
checkpoint-108000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c70f8892e1839c2267b184becdb777f90697abd41804d52550852b136df369d9
size 805587275

3
checkpoint-108000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-108000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eff82055a59c8f94dfaec820b5085944d1d32fad103c697d226375267eb7c667
size 1465

15442
checkpoint-108000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-108000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-111000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-111000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-111000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:002611cd815adbe9081ff0e83b44833fceb2133990954032c860d0d602832eb6
size 402750928

3
checkpoint-111000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:00981afec201e8533fb4213fd6bfbb96070ecbec5be1feb64307f140368b4605
size 805587275

3
checkpoint-111000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-111000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4c8cc4871e5d275372fd65e7538de15452c10f77e18ea536b46678cd5e873cd3
size 1465

15870
checkpoint-111000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-111000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-114000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-114000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-114000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cadcd53b732a8aa93e9dcc6141282796b73d6b150c7b9925758865fa497d8c7d
size 402750928

3
checkpoint-114000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:95f38cae973b14f8f51e2c8d542cce4096b0429bb550f3c48c1549271e4981fb
size 805587275

3
checkpoint-114000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-114000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:14fd58fd6bd7eabfa665a970a5b9aabba1b0be892267a1463cef0c917fcbde9b
size 1465

16298
checkpoint-114000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-114000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-114440/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-114440/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-114440/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:51a54c1527a2055a31c9b763b92b84ec4bdb0e2ba72ccb9fa019488d6b1ccb1b
size 402750928

3
checkpoint-114440/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc4f2f06a5cabf385ea1cb9007efa9c59e48e140055ad1328ea2a48e9d3b3662
size 805587275

3
checkpoint-114440/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-114440/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a7aa229bff4b10e69c49b1f383c70ee498554f2fefee8d94c9f67fd49f616a19
size 1465

16362
checkpoint-114440/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-114440/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-12000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-12000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-12000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a65b93a790a91861230004b9a198254dcdbc5596d7f1202f2f3ef130fb91f7fe
size 402750928

3
checkpoint-12000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1f1a34999cb4d1d7ced187d658f6e68d8f3d782ac2672f280fcbe8039b70c5fb
size 805587275

3
checkpoint-12000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-12000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:28f48b2cfb7582445448e1858bc0f56e835de96c151da0223c804526c0b7c524
size 1465

1746
checkpoint-12000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-12000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-15000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-15000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-15000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b85b48f21fad2b14cb14dff098b5cfc0719acb00da5eae134d264325ce0a64d3
size 402750928

3
checkpoint-15000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0c4700676939534f5a64999d0f1cd9c9429f5d1761aa7fa0b5e82aa5c6139d9a
size 805587275

3
checkpoint-15000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-15000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:880bf101481f4a589440fc23a56b5d99aba291c4c4d983d47dc6b34e5b1dbecf
size 1465

2174
checkpoint-15000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-15000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-18000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-18000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-18000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bfd681a39364e6440a0397c35fe1cd1cafd2b1e6a6fb116d5d4d1e7d60fa761a
size 402750928

3
checkpoint-18000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ab385357da73c3d26459b9a4ceeec7cc5da99697ff88ad036c857efc5780ab5a
size 805587275

3
checkpoint-18000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-18000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:033a150db651b20df4d58c899bfbbf307e0775cebeee0e9f35999aedf205f306
size 1465

2602
checkpoint-18000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-18000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-21000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-21000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-21000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:41b5d9e65952fca744caf7aa920dfda6a6832115d3aac738949c8874f94cee3a
size 402750928

3
checkpoint-21000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:06954d185bd1bf4ef313078f302c64fb05bf7ae8430367e27d584bdbdc50fab3
size 805587275

3
checkpoint-21000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-21000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:18a477e99a34445442195217e3dc83e6965bc2d98c245fc68871e8ad4f67f595
size 1465

3030
checkpoint-21000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-21000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-24000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-24000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-24000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7dd5553879b7b26f08deebee40468f8b878b55e31640ec6b12839ac4cd2ad174
size 402750928

3
checkpoint-24000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5dc316a2a6dda525d3ae772818e211f47c87d1cba0e28e0705e1442087830379
size 805587275

3
checkpoint-24000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-24000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0e2a476c50bb885e7c01075d7fe999513dfdd1af39a4231d10965d936ce48cf5
size 1465

3458
checkpoint-24000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-24000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-27000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

10
checkpoint-27000/generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.14.1",
"use_cache": true
}

3
checkpoint-27000/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:751f8524aee5c2724a5dad80dffd92ff1c4a1b0bf92cd4ccef171538bf49e4cf
size 402750928

3
checkpoint-27000/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:722f4aca78a9de7e3cb7b769ec3f55e4a3326aa66fef13e017043190b1f86c48
size 805587275

3
checkpoint-27000/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7ffffd41cb5466a23271a8c6303a885c8e75cf7a03496c0a7cb7f448827155d8
size 14645

3
checkpoint-27000/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ebebb0ce905b806baa502dd056e76cd609f22e5ebcaa298fad407ac03212d05d
size 1465

3886
checkpoint-27000/trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-27000/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b89574d4fb532cbff16922a1304201180637df9c21820d655203050012091374
size 5329

47
checkpoint-3000/config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 2048,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.14.1",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 32768
}

Some files were not shown because too many files have changed in this diff Show More