初始化项目，由ModelHub XC社区提供模型
Model: SupraLabs/Supra2-Medium-Base Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
217
README.md
Normal file
217
README.md
Normal file
@@ -0,0 +1,217 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
viewer: false
|
||||
datasets:
|
||||
- HuggingFaceFW/fineweb-edu
|
||||
language:
|
||||
- en
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
tags:
|
||||
- small
|
||||
- tiny
|
||||
- supra
|
||||
- supra2
|
||||
- qwen3
|
||||
- efficient
|
||||
---
|
||||
|
||||
<h1 align="center">Supra2-Medium Base</h1>
|
||||
|
||||
<p align="center">
|
||||
Ultra-efficient base model • 25M Parameters • 1K Context
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
**Supra2-Medium Base** is a 25M-parameter decoder-only language model pretrained from scratch by **SupraLabs** on 20B tokens of English web text. It uses the **Qwen3** architecture with a custom 16,384-token tokenizer.
|
||||
|
||||
This is a **base model**. It has *not* been instruction-tuned, chat-tuned, or aligned in any way.
|
||||
|
||||
At only 25 million parameters, Supra2-Medium demonstrates that meaningful language modeling can be achieved with extreme parameter efficiency—trained at ~800 tokens per parameter, which is significantly higher than typical pretraining ratios. This makes it ideal for research into data-efficient scaling and ultra-lightweight deployments.
|
||||
|
||||
---
|
||||
|
||||
## **Let the model speak - a sample**
|
||||
|
||||
Prompt: "Artificial intelligence (AI) is "
|
||||
Completion:
|
||||
```plaintext
|
||||
Artificial intelligence (AI) is espoused by the AI community.
|
||||
The AI community is a group of people who are interested in AI and are interested in the use of AI in the field of AI.
|
||||
The goal of AI is to improve the quality of life of people in the field.
|
||||
The aim of AI is the development of AI and the application of AI in a society.
|
||||
The purpose of AI is that it can be used to improve the performance of the society.
|
||||
It is a technology that is used to improve human intelligence.
|
||||
The technology is used to make the human intelligence.
|
||||
```
|
||||
|
||||
## **Evaluation & Benchmarks**
|
||||
|
||||
All benchmarks were evaluated using the EleutherAI LM-Eval Harness.
|
||||
|
||||
| Model | PIQA (acc\_norm) | HellaSwag (acc\_norm) | ARC-Easy (acc\_norm) | ARC-Challenge (acc\_norm) |
|
||||
| ----- | :---: | :---: | :---: | :---: |
|
||||
| Supra-50M-Base (50M) | 0.62 | 0.32 | 0.46 | 0.25 |
|
||||
| **Supra2-Medium-Base (25M)** | **59.14** | **29.29** | **41.84** | **23.72** |
|
||||
| Supra2-100M-Base (100M) | 0.65 | 0.36 | 0.48 | 0.25 |
|
||||
|
||||
**Final Train Loss: 3.2469** (no eval loss available)
|
||||
|
||||
*Note: The model shows strong performance relative to its size, particularly given the high token-per-parameter ratio. It even strongly competes with our previous 50M model!*
|
||||
|
||||
---
|
||||
|
||||
## **Model Details**
|
||||
|
||||
| | |
|
||||
| ----- | ----- |
|
||||
| **Developed by** | SupraLabs |
|
||||
| **Model type** | Causal decoder-only transformer (Qwen3) |
|
||||
| **Language** | English |
|
||||
| **Parameters** | 25.37M total / \~20M non-embedding |
|
||||
| **Training tokens** | 20B (~800 tokens per parameter) |
|
||||
| **Context length** | 1,024 |
|
||||
| **Precision** | bfloat16 |
|
||||
| **License** | Apache 2.0 |
|
||||
|
||||
### **Architecture**
|
||||
|
||||
| Hyperparameter | Value |
|
||||
| ----- | ----- |
|
||||
| Hidden size | 512 |
|
||||
| Layers | 7 |
|
||||
| Attention heads | 8 (MHA) |
|
||||
| Head dim | 64 |
|
||||
| Intermediate size (SwiGLU) | 896 |
|
||||
| Vocab size | 16,384 |
|
||||
| Positional encoding | RoPE θ=10,000 |
|
||||
| Normalization | RMSNorm, ε=10⁻⁶ |
|
||||
| Tied embeddings | Yes |
|
||||
| Attention implementation | SDPA |
|
||||
| Architecture ID | `qwen3-d07-h512-i0896` |
|
||||
|
||||
---
|
||||
|
||||
## **Training Data**
|
||||
|
||||
| Source | Share | Approx. tokens |
|
||||
| ----- | ----- | ----- |
|
||||
| `HuggingFaceFW/fineweb-edu` (`sample-100BT`) | 100% | 20B |
|
||||
|
||||
Documents were tokenized with the custom `tokenizer_16k`, concatenated into a flat `uint16` token stream, and packed into contiguous 1,024-token chunks (no padding, no document masking — sequences may cross document boundaries).
|
||||
|
||||
---
|
||||
|
||||
## **Training Procedure**
|
||||
|
||||
| Setting | Value |
|
||||
| ----- | ----- |
|
||||
| Optimizer | AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸ |
|
||||
| Peak learning rate | 3×10⁻³ |
|
||||
| LR schedule | Cosine with min LR (10% of peak) |
|
||||
| Warmup steps | 1,000 |
|
||||
| Micro batch size | 32 |
|
||||
| Gradient accumulation | 4 |
|
||||
| Effective batch | 256 sequences \= **262,144 tokens/step** |
|
||||
| Weight decay | 0.1 |
|
||||
| Gradient clipping | 1.0 |
|
||||
| Mixed precision | BF16 \+ TF32 |
|
||||
| Hardware | 2× GPU (DDP): RTX 5060 Ti 16GB + RTX 5060 8GB |
|
||||
| Training optimizations | Liger Kernel, DDP bucketing (10MB), gradient\_as\_bucket\_view |
|
||||
|
||||
### **Key Design Decisions**
|
||||
|
||||
* **High token-per-parameter ratio**: At 800 tokens/param, this model pushes the boundaries of data efficiency for small models
|
||||
* **Compact vocabulary**: 16K vocab reduces embedding overhead while maintaining coverage
|
||||
* **Tied embeddings**: Reduces parameter count and improves training stability
|
||||
* **Multi-head attention (MHA)**: Unlike larger Supra2 models using GQA, this architecture uses standard MHA for simplicity at this scale
|
||||
* **No sliding window**: Full attention within the 1K context
|
||||
|
||||
---
|
||||
|
||||
## **Usage**
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "SupraLabs/Supra2-Medium"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
)
|
||||
model.eval()
|
||||
|
||||
prompt = "The future of artificial intelligence"
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
out = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=512,
|
||||
do_sample=True,
|
||||
temperature=0.2,
|
||||
top_p=0.85,
|
||||
top_k=25,
|
||||
no_repeat_ngram_size=3,
|
||||
repetition_penalty=1.2,
|
||||
)
|
||||
|
||||
print(tokenizer.decode(out[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
### **Tokenizer notes**
|
||||
|
||||
The model uses a custom 16,384-token vocabulary optimized for English web text. The tokenizer is shared across the Supra2 family's smaller models for consistency and efficient multi-task fine-tuning.
|
||||
|
||||
---
|
||||
|
||||
## **Intended Use**
|
||||
|
||||
**Intended:**
|
||||
|
||||
* Research on extreme parameter efficiency and data-efficient pretraining
|
||||
* Ultra-lightweight edge deployments where memory is severely constrained
|
||||
* Educational use for understanding transformer architectures at minimal scale
|
||||
* Starting point for domain-specific fine-tuning when compute resources are limited
|
||||
* Ablation studies on small model behavior and scaling laws
|
||||
|
||||
**Not intended:**
|
||||
|
||||
* Production deployment for critical applications
|
||||
* Factual question answering or knowledge-intensive tasks
|
||||
* Long-form coherent generation beyond a few paragraphs
|
||||
* Non-English text (trained exclusively on English)
|
||||
* Any application requiring safety guarantees or alignment
|
||||
|
||||
---
|
||||
|
||||
## **Limitations and Bias**
|
||||
|
||||
* **Very small.** At 25M parameters, the model has extremely limited capacity. Expect frequent hallucinations, factual errors, repetitive outputs, and poor reasoning.
|
||||
* **Base model.** No RLHF, no safety tuning, no refusal behavior. It will continue whatever text you give it, including harmful or offensive prompts.
|
||||
* **Web-derived data.** FineWeb-Edu is a filtered CommonCrawl derivative and carries the biases, stereotypes, and factual errors of the open web.
|
||||
* **Short context.** Trained exclusively at 1,024 tokens. Extrapolation beyond this length is untested and likely degraded.
|
||||
* **No document masking.** Attention could cross document boundaries within a packed chunk, which slightly blurs document independence.
|
||||
* **English only.** Performance on other languages will be poor to non-existent.
|
||||
* **High token-per-parameter ratio.** While efficient, training at 800 tok/param means the model may be under-trained compared to models trained at lower ratios (e.g., 300 tok/param).
|
||||
|
||||
---
|
||||
|
||||
## **Performance Characteristics**
|
||||
|
||||
Despite its tiny size, Supra2-Medium achieves surprisingly coherent short-form generation. Key observations:
|
||||
|
||||
* **Strengths**: Basic grammar, simple factoids, short completions, pattern matching
|
||||
* **Weaknesses**: Complex reasoning, arithmetic, long-range coherence, factual accuracy, nuanced understanding
|
||||
* **Best use case**: Generating 1-3 sentence completions, simple text transformations, educational demonstrations
|
||||
|
||||
---
|
||||
|
||||
Future work includes instruction-tuned variants and exploration of even more efficient architectures at this scale.
|
||||
|
||||
*© SupraLabs 2026*
|
||||
42
config.json
Normal file
42
config.json
Normal file
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"architectures": [
|
||||
"Qwen3ForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"dtype": "float32",
|
||||
"eos_token_id": 2,
|
||||
"head_dim": 64,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 512,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 896,
|
||||
"layer_types": [
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention"
|
||||
],
|
||||
"max_position_embeddings": 1024,
|
||||
"max_window_layers": 28,
|
||||
"model_type": "qwen3",
|
||||
"num_attention_heads": 8,
|
||||
"num_hidden_layers": 7,
|
||||
"num_key_value_heads": 8,
|
||||
"pad_token_id": 0,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_parameters": {
|
||||
"rope_theta": 10000.0,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"sliding_window": null,
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "5.5.0",
|
||||
"use_cache": true,
|
||||
"use_sliding_window": false,
|
||||
"vocab_size": 16384
|
||||
}
|
||||
10
generation_config.json
Normal file
10
generation_config.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"pad_token_id": 0,
|
||||
"transformers_version": "5.5.0",
|
||||
"use_cache": false
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:add1d2a14151fabcdc3e582e300c52d30aa288e9c3856e95692c5864fc9b1412
|
||||
size 101492576
|
||||
81184
tokenizer.json
Normal file
81184
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
10
tokenizer_config.json
Normal file
10
tokenizer_config.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<s>",
|
||||
"eos_token": "</s>",
|
||||
"is_local": true,
|
||||
"model_max_length": 1024,
|
||||
"pad_token": "<pad>",
|
||||
"tokenizer_class": "TokenizersBackend",
|
||||
"unk_token": "<unk>"
|
||||
}
|
||||
Reference in New Issue
Block a user