初始化项目，由ModelHub XC社区提供模型
Model: SupraLabs/Supra-50M-Base Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
200
README.md
Normal file
200
README.md
Normal file
@@ -0,0 +1,200 @@
|
|||||||
|
---
|
||||||
|
license: apache-2.0
|
||||||
|
datasets:
|
||||||
|
- HuggingFaceFW/fineweb-edu
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
library_name: transformers
|
||||||
|
tags:
|
||||||
|
- supra
|
||||||
|
- chimera
|
||||||
|
- 50m
|
||||||
|
- llama
|
||||||
|
- small
|
||||||
|
- open
|
||||||
|
- open-source
|
||||||
|
- cpu
|
||||||
|
- tiny
|
||||||
|
- slm
|
||||||
|
---
|
||||||
|
|
||||||
|
# 🦅 Supra-50M BASE
|
||||||
|
|
||||||
|
**Supra-50M** is a compact 50M-parameter BASE causal language model built by SupraLabs, trained from scratch using a Llama-style architecture on 20 billion tokens of high-quality educational web text. Despite being significantly smaller than comparable open models, it achieves competitive or superior results on several key benchmarks. It's our first SupraLabs Scaling Up Plan model.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🏆 Benchmarks
|
||||||
|
|
||||||
|
| Benchmark | Supra-50M *(ours)* | GPT-2 (124M) | SmolLM-135M | OpenELM-270M |
|
||||||
|
| :--- | :--- | :--- | :--- | :--- |
|
||||||
|
| **Parameters** | **50M** | 124M *(2.5×)* | 135M *(2.7×)* | 270M *(5.4×)* |
|
||||||
|
| **BLiMP** (linguistics) | **76.3%** | 63.0% | 69.8% | *(N/A)* |
|
||||||
|
| **SciQ** (science) | 77.2% | 53.2% | 73.4% | **84.70%** |
|
||||||
|
| **ARC-Easy** (knowledge) | **52.2%** | 42.0% | 49.2% | 45.08% |
|
||||||
|
| **PIQA** (logic) | 62.2% | 63.0% | 67.3% | **69.75%** |
|
||||||
|
| **HellaSwag** (context) | 31.8% | 29.5% | 42.0% | **46.71%** |
|
||||||
|
|
||||||
|
|
||||||
|
| Task | Metric | Value |
|
||||||
|
| :--- | :--- | :---: |
|
||||||
|
| arc_easy | acc,none | 0.5185 |
|
||||||
|
| arc_easy | acc_stderr,none | 0.0103 |
|
||||||
|
| arc_easy | acc_norm,none | 0.4600 |
|
||||||
|
| arc_easy | acc_norm_stderr,none | 0.0102 |
|
||||||
|
| arc_challenge | acc,none | 0.2159 |
|
||||||
|
| arc_challenge | acc_stderr,none | 0.0120 |
|
||||||
|
| arc_challenge | acc_norm,none | 0.2517 |
|
||||||
|
| arc_challenge | acc_norm_stderr,none | 0.0127 |
|
||||||
|
| hellaswag | acc,none | 0.2903 |
|
||||||
|
| hellaswag | acc_stderr,none | 0.0045 |
|
||||||
|
| hellaswag | acc_norm,none | 0.3172 |
|
||||||
|
| hellaswag | acc_norm_stderr,none | 0.0046 |
|
||||||
|
| winogrande | acc,none | 0.5154 |
|
||||||
|
| winogrande | acc_stderr,none | 0.0140 |
|
||||||
|
| piqa | acc,none | 0.6251 |
|
||||||
|
| piqa | acc_stderr,none | 0.0113 |
|
||||||
|
| piqa | acc_norm,none | 0.6219 |
|
||||||
|
| piqa | acc_norm_stderr,none | 0.0113 |
|
||||||
|
| openbookqa | acc,none | 0.1860 |
|
||||||
|
| openbookqa | acc_stderr,none | 0.0174 |
|
||||||
|
| openbookqa | acc_norm,none | 0.3080 |
|
||||||
|
| openbookqa | acc_norm_stderr,none | 0.0207 |
|
||||||
|
| boolq | acc,none | 0.5303 |
|
||||||
|
| boolq | acc_stderr,none | 0.0087 |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧠 Model Architecture & Hyperparameters
|
||||||
|
|
||||||
|
Supra-50M is based on the `LlamaForCausalLM` architecture with the following configuration:
|
||||||
|
|
||||||
|
| Hyperparameter | Value |
|
||||||
|
|---|---|
|
||||||
|
| Architecture | Llama (decoder-only transformer) |
|
||||||
|
| Parameters | ~50M |
|
||||||
|
| `vocab_size` | 32,000 |
|
||||||
|
| `hidden_size` | 512 |
|
||||||
|
| `intermediate_size` | 1,408 |
|
||||||
|
| `num_hidden_layers` | 12 |
|
||||||
|
| `num_attention_heads` | 8 |
|
||||||
|
| `num_key_value_heads` | 4 (GQA) |
|
||||||
|
| `max_position_embeddings` | 1,024 |
|
||||||
|
| `rope_theta` | 10,000 |
|
||||||
|
| `tie_word_embeddings` | True |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📚 Training Data
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|---|---|
|
||||||
|
| Dataset | [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (`sample-100BT` split) |
|
||||||
|
| Total tokens | 20,000,000,000 (20B) |
|
||||||
|
| Sequence length | 1,024 tokens |
|
||||||
|
| Storage format | Memory-mapped binary (`uint16`, ~40 GB) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔤 Tokenizer
|
||||||
|
|
||||||
|
A custom **Byte-Level BPE** tokenizer was trained from scratch on 500,000 documents sampled from `fineweb-edu (sample-10BT)`.
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|---|---|
|
||||||
|
| Type | ByteLevelBPETokenizer |
|
||||||
|
| Vocabulary size | 32,000 |
|
||||||
|
| Min frequency | 2 |
|
||||||
|
| Special tokens | `<s>`, `<pad>`, `</s>`, `<unk>`, `<mask>` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚙️ Training Configuration
|
||||||
|
|
||||||
|
| Parameter | Value |
|
||||||
|
|---|---|
|
||||||
|
| Epochs | 1 |
|
||||||
|
| Per-device batch size | 32 |
|
||||||
|
| Gradient accumulation steps | 4 |
|
||||||
|
| Effective batch size | 128 × 1,024 tokens |
|
||||||
|
| Learning rate | 6e-4 |
|
||||||
|
| LR scheduler | Cosine |
|
||||||
|
| Warmup ratio | 2% |
|
||||||
|
| Optimizer | AdamW Fused (`adam_beta1=0.9`, `adam_beta2=0.95`) |
|
||||||
|
| Weight decay | 0.1 |
|
||||||
|
| Max grad norm | 1.0 |
|
||||||
|
| Precision | bfloat16 |
|
||||||
|
| `torch.compile` | Enabled |
|
||||||
|
| Hardware | Single GPU |
|
||||||
|
| Final loss | *3.259* |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Inference
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import pipeline
|
||||||
|
import torch
|
||||||
|
|
||||||
|
print("[*] Loading Supra-50M model from Hugging Face Hub...")
|
||||||
|
pipe = pipeline(
|
||||||
|
"text-generation",
|
||||||
|
model="SupraLabs/Supra-50M_BASE",
|
||||||
|
device_map="auto",
|
||||||
|
torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32
|
||||||
|
)
|
||||||
|
|
||||||
|
def generate_text(prompt, max_new_tokens=150):
|
||||||
|
result = pipe(
|
||||||
|
prompt,
|
||||||
|
max_new_tokens=max_new_tokens,
|
||||||
|
do_sample=True,
|
||||||
|
temperature=0.5,
|
||||||
|
top_k=25,
|
||||||
|
top_p=0.9,
|
||||||
|
repetition_penalty=1.2,
|
||||||
|
pad_token_id=pipe.tokenizer.pad_token_id,
|
||||||
|
eos_token_id=pipe.tokenizer.eos_token_id
|
||||||
|
)
|
||||||
|
return result[0]['generated_text']
|
||||||
|
|
||||||
|
# Example
|
||||||
|
prompt = "The importance of education is"
|
||||||
|
print(f"\nPrompt: {prompt}")
|
||||||
|
print("-" * 40)
|
||||||
|
print("\nOutput:\n" + generate_text(prompt))
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 💬 Sample Outputs
|
||||||
|
|
||||||
|
**Prompt:** `"The main concept of physics is "`
|
||||||
|
> The main concept of physics is iffy, and the idea that we can make things behave in a certain way. The most important part of physics is called quantum mechanics which states that all particles are made up of energy (energy) and matter (matter). In physics, there are two types of particles: elementary particles and exotic ones. These particles have properties like mass, speed or momentum but they don’t interact with each other to form new objects. This is because these particles do not exist independently from one another. In this case, an exotic particle might be created by adding more energy into its structure than it would take for a normal particle. However, when you add additional energy to an exotic particle, the new object will become smaller and larger until it becomes too large to fit within the existing structure. If you think about how light travels through space, it takes around 20 billion years before the light reaches our eyes. Light waves travel faster than light at high speeds so if we could create some kind of light wave, then we wouldn’t need any special equipment. It just needs a few hundred millionths of a second to produce light rays. So even though the light is moving along the same path as the current, the speed of light is different depending on where the light hits the
|
||||||
|
|
||||||
|
**Prompt:** `"Artificial intelligence is "`
|
||||||
|
> Artificial intelligence is iffy, it can be used to make intelligent machines that could take over the world. What does Artificial Intelligence mean? AI refers to artificial intelligence and machine learning technology which is a type of computer science (also known as artificial intelligence) in which computers are programmed with knowledge about their environment or other objects. The term AI comes from the Greek word "art" meaning "to create." The most common uses for AI include: - Machine Learning This means using algorithms like natural language processing systems to learn how words work together to form sentences such as “I am going to go to the store.” These programs will then use these rules to decide whether they should buy something or not so that you know what’s being sold on the internet. For example, if you purchase an ebook at Amazon, you may want to check its price first before purchasing it. If this happens, your shopping cart might look different than it did when purchased by someone else who bought it earlier. You can also think of AI as a way to help people understand themselves better through training and reasoning rather than simply seeing them doing things differently. In fact, we often see AI models working very well because of the way humans interact with our minds. This ability makes us more effective
|
||||||
|
|
||||||
|
**Prompt:** `"Once upon a time, "`
|
||||||
|
> Once upon a time, ...... I was so excited about the new school year and wanted to make some changes in my life. I had been looking for ways to help me become more self-aware. As an adult, I have always felt that there is no one way of doing things without thinking first. This has led me to start making small changes at home or at work. One such change was to create a space where I could be more mindful and aware of myself as well as other people around me. It’s important to remember that we all need our own personal growth and development. We can do this by taking responsibility for ourselves; being responsible for what happens outside us and keeping it within our control. By creating these smaller steps towards becoming more conscious of yourself, you will see how much better your future looks!The word "treaty" means something like "a treaty made with a king." The French word for "covenant," célèbre (French: cœle), comes from the Latin cecus ("to give up"). A covenant is not a binding agreement but rather an act of mutual understanding between two parties. In general terms, a contract is anything agreed on which someone agrees to agree to receive certain benefits. For example, if a person
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📄 License
|
||||||
|
|
||||||
|
This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Third-party validation benchmarks by Datdanboi25
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*© SupraLabs 2026 — Project Chimera*
|
||||||
90
benchmarks.md
Normal file
90
benchmarks.md
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
| Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr |
|
||||||
|
|------------------------------------------------------------|------:|------|-----:|---------------|---|-------:|---|-------|
|
||||||
|
|arc_easy | 1|none | 0|acc |↑ | 0.5223|± | 0.0102|
|
||||||
|
| | |none | 0|acc_norm |↑ | 0.4600|± | 0.0102|
|
||||||
|
|blimp | 2|none | |acc |↑ | 0.7631|± | 0.0014|
|
||||||
|
| - blimp_adjunct_island | 1|none | 0|acc |↑ | 0.8420|± | 0.0115|
|
||||||
|
| - blimp_anaphor_gender_agreement | 1|none | 0|acc |↑ | 0.8430|± | 0.0115|
|
||||||
|
| - blimp_anaphor_number_agreement | 1|none | 0|acc |↑ | 0.9620|± | 0.0060|
|
||||||
|
| - blimp_animate_subject_passive | 1|none | 0|acc |↑ | 0.7820|± | 0.0131|
|
||||||
|
| - blimp_animate_subject_trans | 1|none | 0|acc |↑ | 0.8040|± | 0.0126|
|
||||||
|
| - blimp_causative | 1|none | 0|acc |↑ | 0.6980|± | 0.0145|
|
||||||
|
| - blimp_complex_NP_island | 1|none | 0|acc |↑ | 0.4940|± | 0.0158|
|
||||||
|
| - blimp_coordinate_structure_constraint_complex_left_branch| 1|none | 0|acc |↑ | 0.7420|± | 0.0138|
|
||||||
|
| - blimp_coordinate_structure_constraint_object_extraction | 1|none | 0|acc |↑ | 0.7520|± | 0.0137|
|
||||||
|
| - blimp_determiner_noun_agreement_1 | 1|none | 0|acc |↑ | 0.9790|± | 0.0045|
|
||||||
|
| - blimp_determiner_noun_agreement_2 | 1|none | 0|acc |↑ | 0.9680|± | 0.0056|
|
||||||
|
| - blimp_determiner_noun_agreement_irregular_1 | 1|none | 0|acc |↑ | 0.8990|± | 0.0095|
|
||||||
|
| - blimp_determiner_noun_agreement_irregular_2 | 1|none | 0|acc |↑ | 0.9650|± | 0.0058|
|
||||||
|
| - blimp_determiner_noun_agreement_with_adj_2 | 1|none | 0|acc |↑ | 0.9340|± | 0.0079|
|
||||||
|
| - blimp_determiner_noun_agreement_with_adj_irregular_1 | 1|none | 0|acc |↑ | 0.8740|± | 0.0105|
|
||||||
|
| - blimp_determiner_noun_agreement_with_adj_irregular_2 | 1|none | 0|acc |↑ | 0.9270|± | 0.0082|
|
||||||
|
| - blimp_determiner_noun_agreement_with_adjective_1 | 1|none | 0|acc |↑ | 0.9410|± | 0.0075|
|
||||||
|
| - blimp_distractor_agreement_relational_noun | 1|none | 0|acc |↑ | 0.8780|± | 0.0104|
|
||||||
|
| - blimp_distractor_agreement_relative_clause | 1|none | 0|acc |↑ | 0.7210|± | 0.0142|
|
||||||
|
| - blimp_drop_argument | 1|none | 0|acc |↑ | 0.7500|± | 0.0137|
|
||||||
|
| - blimp_ellipsis_n_bar_1 | 1|none | 0|acc |↑ | 0.8060|± | 0.0125|
|
||||||
|
| - blimp_ellipsis_n_bar_2 | 1|none | 0|acc |↑ | 0.8820|± | 0.0102|
|
||||||
|
| - blimp_existential_there_object_raising | 1|none | 0|acc |↑ | 0.8750|± | 0.0105|
|
||||||
|
| - blimp_existential_there_quantifiers_1 | 1|none | 0|acc |↑ | 0.9730|± | 0.0051|
|
||||||
|
| - blimp_existential_there_quantifiers_2 | 1|none | 0|acc |↑ | 0.2070|± | 0.0128|
|
||||||
|
| - blimp_existential_there_subject_raising | 1|none | 0|acc |↑ | 0.8810|± | 0.0102|
|
||||||
|
| - blimp_expletive_it_object_raising | 1|none | 0|acc |↑ | 0.7830|± | 0.0130|
|
||||||
|
| - blimp_inchoative | 1|none | 0|acc |↑ | 0.6330|± | 0.0152|
|
||||||
|
| - blimp_intransitive | 1|none | 0|acc |↑ | 0.7310|± | 0.0140|
|
||||||
|
| - blimp_irregular_past_participle_adjectives | 1|none | 0|acc |↑ | 0.8620|± | 0.0109|
|
||||||
|
| - blimp_irregular_past_participle_verbs | 1|none | 0|acc |↑ | 0.8930|± | 0.0098|
|
||||||
|
| - blimp_irregular_plural_subject_verb_agreement_1 | 1|none | 0|acc |↑ | 0.8990|± | 0.0095|
|
||||||
|
| - blimp_irregular_plural_subject_verb_agreement_2 | 1|none | 0|acc |↑ | 0.9030|± | 0.0094|
|
||||||
|
| - blimp_left_branch_island_echo_question | 1|none | 0|acc |↑ | 0.3810|± | 0.0154|
|
||||||
|
| - blimp_left_branch_island_simple_question | 1|none | 0|acc |↑ | 0.6470|± | 0.0151|
|
||||||
|
| - blimp_matrix_question_npi_licensor_present | 1|none | 0|acc |↑ | 0.1260|± | 0.0105|
|
||||||
|
| - blimp_npi_present_1 | 1|none | 0|acc |↑ | 0.5710|± | 0.0157|
|
||||||
|
| - blimp_npi_present_2 | 1|none | 0|acc |↑ | 0.6190|± | 0.0154|
|
||||||
|
| - blimp_only_npi_licensor_present | 1|none | 0|acc |↑ | 0.6250|± | 0.0153|
|
||||||
|
| - blimp_only_npi_scope | 1|none | 0|acc |↑ | 0.5360|± | 0.0158|
|
||||||
|
| - blimp_passive_1 | 1|none | 0|acc |↑ | 0.8770|± | 0.0104|
|
||||||
|
| - blimp_passive_2 | 1|none | 0|acc |↑ | 0.8840|± | 0.0101|
|
||||||
|
| - blimp_principle_A_c_command | 1|none | 0|acc |↑ | 0.5560|± | 0.0157|
|
||||||
|
| - blimp_principle_A_case_1 | 1|none | 0|acc |↑ | 1.0000|± | 0|
|
||||||
|
| - blimp_principle_A_case_2 | 1|none | 0|acc |↑ | 0.9650|± | 0.0058|
|
||||||
|
| - blimp_principle_A_domain_1 | 1|none | 0|acc |↑ | 0.9430|± | 0.0073|
|
||||||
|
| - blimp_principle_A_domain_2 | 1|none | 0|acc |↑ | 0.8040|± | 0.0126|
|
||||||
|
| - blimp_principle_A_domain_3 | 1|none | 0|acc |↑ | 0.5200|± | 0.0158|
|
||||||
|
| - blimp_principle_A_reconstruction | 1|none | 0|acc |↑ | 0.2920|± | 0.0144|
|
||||||
|
| - blimp_regular_plural_subject_verb_agreement_1 | 1|none | 0|acc |↑ | 0.8930|± | 0.0098|
|
||||||
|
| - blimp_regular_plural_subject_verb_agreement_2 | 1|none | 0|acc |↑ | 0.9110|± | 0.0090|
|
||||||
|
| - blimp_sentential_negation_npi_licensor_present | 1|none | 0|acc |↑ | 0.9930|± | 0.0026|
|
||||||
|
| - blimp_sentential_negation_npi_scope | 1|none | 0|acc |↑ | 0.7100|± | 0.0144|
|
||||||
|
| - blimp_sentential_subject_island | 1|none | 0|acc |↑ | 0.3310|± | 0.0149|
|
||||||
|
| - blimp_superlative_quantifiers_1 | 1|none | 0|acc |↑ | 0.7800|± | 0.0131|
|
||||||
|
| - blimp_superlative_quantifiers_2 | 1|none | 0|acc |↑ | 0.7450|± | 0.0138|
|
||||||
|
| - blimp_tough_vs_raising_1 | 1|none | 0|acc |↑ | 0.5390|± | 0.0158|
|
||||||
|
| - blimp_tough_vs_raising_2 | 1|none | 0|acc |↑ | 0.8780|± | 0.0104|
|
||||||
|
| - blimp_transitive | 1|none | 0|acc |↑ | 0.8430|± | 0.0115|
|
||||||
|
| - blimp_wh_island | 1|none | 0|acc |↑ | 0.7190|± | 0.0142|
|
||||||
|
| - blimp_wh_questions_object_gap | 1|none | 0|acc |↑ | 0.7590|± | 0.0135|
|
||||||
|
| - blimp_wh_questions_subject_gap | 1|none | 0|acc |↑ | 0.9280|± | 0.0082|
|
||||||
|
| - blimp_wh_questions_subject_gap_long_distance | 1|none | 0|acc |↑ | 0.8550|± | 0.0111|
|
||||||
|
| - blimp_wh_vs_that_no_gap | 1|none | 0|acc |↑ | 0.9490|± | 0.0070|
|
||||||
|
| - blimp_wh_vs_that_no_gap_long_distance | 1|none | 0|acc |↑ | 0.9490|± | 0.0070|
|
||||||
|
| - blimp_wh_vs_that_with_gap | 1|none | 0|acc |↑ | 0.5920|± | 0.0155|
|
||||||
|
| - blimp_wh_vs_that_with_gap_long_distance | 1|none | 0|acc |↑ | 0.3280|± | 0.0149|
|
||||||
|
|hellaswag | 1|none | 0|acc |↑ | 0.2914|± | 0.0045|
|
||||||
|
| | |none | 0|acc_norm |↑ | 0.3178|± | 0.0046|
|
||||||
|
|lambada_openai | 1|none | 0|acc |↑ | 0.2591|± | 0.0061|
|
||||||
|
| | |none | 0|perplexity |↓ | 95.5121|± | 4.1325|
|
||||||
|
|lambada_standard | 1|none | 0|acc |↑ | 0.1716|± | 0.0053|
|
||||||
|
| | |none | 0|perplexity |↓ |488.2170|± |23.4634|
|
||||||
|
|piqa | 1|none | 0|acc |↑ | 0.6224|± | 0.0113|
|
||||||
|
| | |none | 0|acc_norm |↑ | 0.6208|± | 0.0113|
|
||||||
|
|sciq | 1|none | 0|acc |↑ | 0.7720|± | 0.0133|
|
||||||
|
| | |none | 0|acc_norm |↑ | 0.6810|± | 0.0147|
|
||||||
|
|wikitext | 2|none | 0|bits_per_byte |↓ | 1.0267|± | N/A |
|
||||||
|
| | |none | 0|byte_perplexity|↓ | 2.0374|± | N/A |
|
||||||
|
| | |none | 0|word_perplexity|↓ | 44.9548|± | N/A |
|
||||||
|
|winogrande | 1|none | 0|acc |↑ | 0.5099|± | 0.0140|
|
||||||
|
|
||||||
|
|Groups|Version|Filter|n-shot|Metric| |Value | |Stderr|
|
||||||
|
|------|------:|------|------|------|---|-----:|---|-----:|
|
||||||
|
|blimp | 2|none | |acc |↑ |0.7631|± |0.0014|
|
||||||
BIN
benchmarks.png
Normal file
BIN
benchmarks.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
32
config.json
Normal file
32
config.json
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"LlamaForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"dtype": "float32",
|
||||||
|
"eos_token_id": 2,
|
||||||
|
"head_dim": 64,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 512,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 1408,
|
||||||
|
"max_position_embeddings": 1024,
|
||||||
|
"mlp_bias": false,
|
||||||
|
"model_type": "llama",
|
||||||
|
"num_attention_heads": 8,
|
||||||
|
"num_hidden_layers": 12,
|
||||||
|
"num_key_value_heads": 4,
|
||||||
|
"pad_token_id": 1,
|
||||||
|
"pretraining_tp": 1,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_parameters": {
|
||||||
|
"rope_theta": 10000,
|
||||||
|
"rope_type": "default"
|
||||||
|
},
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"transformers_version": "5.8.1",
|
||||||
|
"use_cache": false,
|
||||||
|
"vocab_size": 32000
|
||||||
|
}
|
||||||
10
generation_config.json
Normal file
10
generation_config.json
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
{
|
||||||
|
"_from_model_config": true,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 2,
|
||||||
|
"output_attentions": false,
|
||||||
|
"output_hidden_states": false,
|
||||||
|
"pad_token_id": 1,
|
||||||
|
"transformers_version": "5.8.1",
|
||||||
|
"use_cache": true
|
||||||
|
}
|
||||||
33
inference.py
Normal file
33
inference.py
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
print("[*] Loading libraries...")
|
||||||
|
import torch
|
||||||
|
from transformers import LlamaForCausalLM, PreTrainedTokenizerFast
|
||||||
|
|
||||||
|
model_path = "./Chimera-FINAL"
|
||||||
|
|
||||||
|
print("[*] Loading tokenizer...")
|
||||||
|
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_path)
|
||||||
|
|
||||||
|
print("[*] Loading model...")
|
||||||
|
model = LlamaForCausalLM.from_pretrained(model_path)
|
||||||
|
model.eval()
|
||||||
|
|
||||||
|
prompt = "Artificial intelligence is " # "Artificial intelligence is " | "The main concept of physics is " | "Once upon a time, "
|
||||||
|
print(f"[*] Prompt: {prompt!r}")
|
||||||
|
|
||||||
|
inputs = tokenizer(prompt, return_tensors="pt")
|
||||||
|
|
||||||
|
with torch.no_grad():
|
||||||
|
outputs = model.generate(
|
||||||
|
input_ids=inputs["input_ids"],
|
||||||
|
attention_mask=inputs["attention_mask"],
|
||||||
|
max_new_tokens=256,
|
||||||
|
do_sample=True,
|
||||||
|
temperature=0.4,
|
||||||
|
top_p=0.85,
|
||||||
|
top_k=30,
|
||||||
|
repetition_penalty=1.1,
|
||||||
|
pad_token_id=tokenizer.pad_token_id,
|
||||||
|
eos_token_id=tokenizer.eos_token_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
print("[*] Output of Supra 50M Base:", tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:6b3287f381ae67127b09ac24939733792452cc7328925303b7ef7475f4f6f285
|
||||||
|
size 207157136
|
||||||
19
samples.txt
Normal file
19
samples.txt
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
[*] Prompt: 'The main concept of physics is '
|
||||||
|
[*] Output of Supra 50M Base: The main concept of physics is iffy, and the idea that we can make things behave in a certain way. The most important part of physics is called quantum mechanics which states that all particles are made up of energy (energy) and matter (matter).
|
||||||
|
In physics, there are two types of particles: elementary particles and exotic ones. These particles have properties like mass, speed or momentum but they don’t interact with each other to form new objects. This is because these particles do not exist independently from one another. In this case, an exotic particle might be created by adding more energy into its structure than it would take for a normal particle. However, when you add additional energy to an exotic particle, the new object will become smaller and larger until it becomes too large to fit within the existing structure.
|
||||||
|
If you think about how light travels through space, it takes around 20 billion years before the light reaches our eyes. Light waves travel faster than light at high speeds so if we could create some kind of light wave, then we wouldn’t need any special equipment. It just needs a few hundred millionths of a second to produce light rays. So even though the light is moving along the same path as the current, the speed of light is different depending on where the light hits the
|
||||||
|
|
||||||
|
[*] Prompt: 'Artificial intelligence is '
|
||||||
|
[*] Output of Supra 50M Base: Artificial intelligence is iffy, it can be used to make intelligent machines that could take over the world.
|
||||||
|
What does Artificial Intelligence mean?
|
||||||
|
AI refers to artificial intelligence and machine learning technology which is a type of computer science (also known as artificial intelligence) in which computers are programmed with knowledge about their environment or other objects. The term AI comes from the Greek word "art" meaning "to create."
|
||||||
|
The most common uses for AI include:
|
||||||
|
- Machine Learning
|
||||||
|
This means using algorithms like natural language processing systems to learn how words work together to form sentences such as “I am going to go to the store.”
|
||||||
|
These programs will then use these rules to decide whether they should buy something or not so that you know what’s being sold on the internet. For example, if you purchase an ebook at Amazon, you may want to check its price first before purchasing it. If this happens, your shopping cart might look different than it did when purchased by someone else who bought it earlier.
|
||||||
|
You can also think of AI as a way to help people understand themselves better through training and reasoning rather than simply seeing them doing things differently. In fact, we often see AI models working very well because of the way humans interact with our minds. This ability makes us more effective
|
||||||
|
|
||||||
|
[*] Prompt: 'Once upon a time, '
|
||||||
|
[*] Output of Supra 50M Base: Once upon a time, ...... I was so excited about the new school year and wanted to make some changes in my life.
|
||||||
|
I had been looking for ways to help me become more self-aware. As an adult, I have always felt that there is no one way of doing things without thinking first. This has led me to start making small changes at home or at work. One such change was to create a space where I could be more mindful and aware of myself as well as other people around me. It’s important to remember that we all need our own personal growth and development. We can do this by taking responsibility for ourselves; being responsible for what happens outside us and keeping it within our control. By creating these smaller steps towards becoming more conscious of yourself, you will see how much better your future looks!The word "treaty" means something like "a treaty made with a king." The French word for "covenant," célèbre (French: cœle), comes from the Latin cecus ("to give up"). A covenant is not a binding agreement but rather an act of mutual understanding between two parties. In general terms, a contract is anything agreed on which someone agrees to agree to receive certain benefits. For example, if a person
|
||||||
|
|
||||||
159032
tokenizer.json
Normal file
159032
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
9
tokenizer_config.json
Normal file
9
tokenizer_config.json
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"backend": "tokenizers",
|
||||||
|
"bos_token": "<s>",
|
||||||
|
"eos_token": "</s>",
|
||||||
|
"model_max_length": 1000000000000000019884624838656,
|
||||||
|
"pad_token": "<pad>",
|
||||||
|
"tokenizer_class": "TokenizersBackend",
|
||||||
|
"unk_token": "<unk>"
|
||||||
|
}
|
||||||
191
train.py
Normal file
191
train.py
Normal file
@@ -0,0 +1,191 @@
|
|||||||
|
"""
|
||||||
|
© SupraLabs 2026 - Official pretraining code for PROJECT CHIMERA - 50M Llama
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
os.environ["PYTORCH_ALLOC_CONF"] = "expandable_segments:True"
|
||||||
|
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
|
||||||
|
|
||||||
|
print("[*] Loading libraries...")
|
||||||
|
import torch
|
||||||
|
import math
|
||||||
|
import numpy as np
|
||||||
|
from datasets import load_dataset
|
||||||
|
from tokenizers import ByteLevelBPETokenizer
|
||||||
|
from transformers import (
|
||||||
|
LlamaConfig,
|
||||||
|
LlamaForCausalLM,
|
||||||
|
PreTrainedTokenizerFast,
|
||||||
|
Trainer,
|
||||||
|
TrainingArguments,
|
||||||
|
)
|
||||||
|
from torch.utils.data import Dataset
|
||||||
|
from tqdm import tqdm
|
||||||
|
|
||||||
|
print("[*] Loading tokenizer...")
|
||||||
|
fast_tokenizer = ByteLevelBPETokenizer(
|
||||||
|
"custom_llama_tokenizer-vocab.json",
|
||||||
|
"custom_llama_tokenizer-merges.txt"
|
||||||
|
)
|
||||||
|
tokenizer = PreTrainedTokenizerFast(
|
||||||
|
tokenizer_object=fast_tokenizer,
|
||||||
|
bos_token="<s>",
|
||||||
|
eos_token="</s>",
|
||||||
|
unk_token="<unk>",
|
||||||
|
pad_token="<pad>",
|
||||||
|
)
|
||||||
|
|
||||||
|
TOKEN_BIN = "tokens.bin"
|
||||||
|
TARGET_TOKENS = 20_000_000_000
|
||||||
|
SEQ_LEN = 1024
|
||||||
|
BATCH_TEXTS = 1000
|
||||||
|
FLUSH_EVERY = 1_000_000
|
||||||
|
|
||||||
|
|
||||||
|
def build_token_bin(fast_tokenizer, path=TOKEN_BIN, target_tokens=TARGET_TOKENS):
|
||||||
|
if os.path.exists(path) and os.path.getsize(path) >= target_tokens * 2:
|
||||||
|
print(f"[=] Reusing existing token file: {path}")
|
||||||
|
return
|
||||||
|
|
||||||
|
print(f"[*] Streaming + tokenizing {target_tokens:,} tokens → {path}")
|
||||||
|
mm = np.memmap(path, dtype=np.uint16, mode="w+", shape=(target_tokens,))
|
||||||
|
|
||||||
|
dataset = load_dataset(
|
||||||
|
"HuggingFaceFW/fineweb-edu", "sample-100BT",
|
||||||
|
split="train", streaming=True
|
||||||
|
)
|
||||||
|
|
||||||
|
written = 0
|
||||||
|
buf = []
|
||||||
|
texts = []
|
||||||
|
pbar = tqdm(total=target_tokens, desc="[*] Gathering tokens", unit="tok")
|
||||||
|
|
||||||
|
def flush_buf():
|
||||||
|
nonlocal written, buf
|
||||||
|
if not buf:
|
||||||
|
return False
|
||||||
|
n = min(len(buf), target_tokens - written)
|
||||||
|
mm[written:written + n] = np.asarray(buf[:n], dtype=np.uint16)
|
||||||
|
written += n
|
||||||
|
pbar.update(n)
|
||||||
|
del buf[:n]
|
||||||
|
return written >= target_tokens
|
||||||
|
|
||||||
|
for example in dataset:
|
||||||
|
texts.append(example["text"])
|
||||||
|
if len(texts) >= BATCH_TEXTS:
|
||||||
|
encs = fast_tokenizer.encode_batch(texts)
|
||||||
|
texts.clear()
|
||||||
|
for e in encs:
|
||||||
|
buf.extend(e.ids)
|
||||||
|
if len(buf) >= FLUSH_EVERY:
|
||||||
|
if flush_buf():
|
||||||
|
break
|
||||||
|
|
||||||
|
if written < target_tokens and texts:
|
||||||
|
encs = fast_tokenizer.encode_batch(texts)
|
||||||
|
for e in encs:
|
||||||
|
buf.extend(e.ids)
|
||||||
|
if written < target_tokens:
|
||||||
|
flush_buf()
|
||||||
|
|
||||||
|
pbar.close()
|
||||||
|
mm.flush()
|
||||||
|
del mm
|
||||||
|
print(f"[+] Wrote {written:,} tokens to {path} "
|
||||||
|
f"({os.path.getsize(path)/1e6:.1f} MB)")
|
||||||
|
|
||||||
|
|
||||||
|
class MemmapDataset(Dataset):
|
||||||
|
def __init__(self, path, total_tokens, seq_len=SEQ_LEN):
|
||||||
|
self.path = path
|
||||||
|
self.seq_len = seq_len
|
||||||
|
self.n_chunks = total_tokens // seq_len
|
||||||
|
self._data = None # lazy open (Multiprocessing-safe)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def data(self):
|
||||||
|
if self._data is None:
|
||||||
|
self._data = np.memmap(
|
||||||
|
self.path, dtype=np.uint16, mode="r",
|
||||||
|
shape=(self.n_chunks * self.seq_len,)
|
||||||
|
)
|
||||||
|
return self._data
|
||||||
|
|
||||||
|
def __len__(self):
|
||||||
|
return self.n_chunks
|
||||||
|
|
||||||
|
def __getitem__(self, idx):
|
||||||
|
s = idx * self.seq_len
|
||||||
|
arr = np.asarray(self.data[s:s + self.seq_len], dtype=np.int64)
|
||||||
|
ids = torch.from_numpy(arr)
|
||||||
|
return {"input_ids": ids, "labels": ids.clone()}
|
||||||
|
|
||||||
|
|
||||||
|
def collate_fn(batch):
|
||||||
|
input_ids = torch.stack([b["input_ids"] for b in batch])
|
||||||
|
labels = torch.stack([b["labels"] for b in batch])
|
||||||
|
return {"input_ids": input_ids, "labels": labels}
|
||||||
|
|
||||||
|
|
||||||
|
print(f"[*] Preparing {TARGET_TOKENS:,} tokens (streaming, memmap-backed)...")
|
||||||
|
build_token_bin(fast_tokenizer, TOKEN_BIN, TARGET_TOKENS)
|
||||||
|
dataset = MemmapDataset(TOKEN_BIN, TARGET_TOKENS, seq_len=SEQ_LEN)
|
||||||
|
print(f"[+] Dataset ready: {len(dataset):,} chunks of {SEQ_LEN} tokens")
|
||||||
|
|
||||||
|
print("[*] Setting up model...")
|
||||||
|
config = LlamaConfig(
|
||||||
|
vocab_size=32_000,
|
||||||
|
hidden_size=512,
|
||||||
|
intermediate_size=1408,
|
||||||
|
num_hidden_layers=12,
|
||||||
|
num_attention_heads=8,
|
||||||
|
num_key_value_heads=4,
|
||||||
|
max_position_embeddings=1024,
|
||||||
|
rope_theta=10_000,
|
||||||
|
tie_word_embeddings=True,
|
||||||
|
pad_token_id=tokenizer.pad_token_id,
|
||||||
|
bos_token_id=tokenizer.bos_token_id,
|
||||||
|
eos_token_id=tokenizer.eos_token_id,
|
||||||
|
)
|
||||||
|
model = LlamaForCausalLM(config)
|
||||||
|
print(f"[*] Model parameters: {model.num_parameters():,}")
|
||||||
|
|
||||||
|
print("[*] Defining training arguments...")
|
||||||
|
training_args = TrainingArguments(
|
||||||
|
output_dir="./Chimera",
|
||||||
|
num_train_epochs=1,
|
||||||
|
per_device_train_batch_size=32,
|
||||||
|
gradient_accumulation_steps=4,
|
||||||
|
save_steps=500,
|
||||||
|
save_total_limit=2,
|
||||||
|
logging_steps=100,
|
||||||
|
weight_decay=0.1,
|
||||||
|
fp16=False,
|
||||||
|
bf16=True,
|
||||||
|
push_to_hub=False,
|
||||||
|
report_to="none",
|
||||||
|
dataloader_num_workers=os.cpu_count() // 2,
|
||||||
|
dataloader_pin_memory=True,
|
||||||
|
learning_rate=6e-4,
|
||||||
|
lr_scheduler_type="cosine",
|
||||||
|
warmup_ratio=0.02,
|
||||||
|
max_grad_norm=1.0,
|
||||||
|
optim="adamw_torch_fused",
|
||||||
|
adam_beta1=0.9,
|
||||||
|
adam_beta2=0.95,
|
||||||
|
torch_compile=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
trainer = Trainer(
|
||||||
|
model=model,
|
||||||
|
args=training_args,
|
||||||
|
train_dataset=dataset,
|
||||||
|
data_collator=collate_fn,
|
||||||
|
)
|
||||||
|
|
||||||
|
print("[*] Starting training...")
|
||||||
|
trainer.train()
|
||||||
|
trainer.save_model("./Chimera-FINAL")
|
||||||
|
tokenizer.save_pretrained("./Chimera-FINAL")
|
||||||
|
print("[*] Training finished.")
|
||||||
25
train_tokenizer.py
Normal file
25
train_tokenizer.py
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
print("[*] Loading libraries...")
|
||||||
|
from datasets import load_dataset
|
||||||
|
from tokenizers import ByteLevelBPETokenizer
|
||||||
|
from tqdm import tqdm
|
||||||
|
|
||||||
|
dataset = load_dataset("HuggingFaceFW/fineweb-edu", "sample-10BT", split="train", streaming=True)
|
||||||
|
def get_training_corpus():
|
||||||
|
dataset_iter = iter(dataset)
|
||||||
|
for _ in tqdm(range(500_000), desc="Feeding data"):
|
||||||
|
yield next(dataset_iter)["text"]
|
||||||
|
|
||||||
|
tokenizer = ByteLevelBPETokenizer()
|
||||||
|
|
||||||
|
print("[*] Training tokenizer...")
|
||||||
|
|
||||||
|
tokenizer.train_from_iterator(
|
||||||
|
get_training_corpus(),
|
||||||
|
vocab_size=32_000,
|
||||||
|
min_frequency=2,
|
||||||
|
show_progress=True,
|
||||||
|
special_tokens=["<s>", "<pad>", "</s>", "<unk>", "<mask>"]
|
||||||
|
)
|
||||||
|
|
||||||
|
tokenizer.save_model(".", "custom_llama_tokenizer")
|
||||||
|
print("[*] Tokenizer training complete!")
|
||||||
1847
training.log
Normal file
1847
training.log
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user