SupraLabs/Supra-50M-Base
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: SupraLabs/Supra-50M-Base
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-25 21:59:04 +08:00
commit 3a121abd67
14 changed files with 161526 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

200
README.md Normal file
View File

@@ -0,0 +1,200 @@
---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- supra
- chimera
- 50m
- llama
- small
- open
- open-source
- cpu
- tiny
- slm
---
# 🦅 Supra-50M BASE
**Supra-50M** is a compact 50M-parameter BASE causal language model built by SupraLabs, trained from scratch using a Llama-style architecture on 20 billion tokens of high-quality educational web text. Despite being significantly smaller than comparable open models, it achieves competitive or superior results on several key benchmarks. It's our first SupraLabs Scaling Up Plan model.
---
## 🏆 Benchmarks
| Benchmark | Supra-50M *(ours)* | GPT-2 (124M) | SmolLM-135M | OpenELM-270M |
| :--- | :--- | :--- | :--- | :--- |
| **Parameters** | **50M** | 124M *(2.5×)* | 135M *(2.7×)* | 270M *(5.4×)* |
| **BLiMP** (linguistics) | **76.3%** | 63.0% | 69.8% | *(N/A)* |
| **SciQ** (science) | 77.2% | 53.2% | 73.4% | **84.70%** |
| **ARC-Easy** (knowledge) | **52.2%** | 42.0% | 49.2% | 45.08% |
| **PIQA** (logic) | 62.2% | 63.0% | 67.3% | **69.75%** |
| **HellaSwag** (context) | 31.8% | 29.5% | 42.0% | **46.71%** |
| Task | Metric | Value |
| :--- | :--- | :---: |
| arc_easy | acc,none | 0.5185 |
| arc_easy | acc_stderr,none | 0.0103 |
| arc_easy | acc_norm,none | 0.4600 |
| arc_easy | acc_norm_stderr,none | 0.0102 |
| arc_challenge | acc,none | 0.2159 |
| arc_challenge | acc_stderr,none | 0.0120 |
| arc_challenge | acc_norm,none | 0.2517 |
| arc_challenge | acc_norm_stderr,none | 0.0127 |
| hellaswag | acc,none | 0.2903 |
| hellaswag | acc_stderr,none | 0.0045 |
| hellaswag | acc_norm,none | 0.3172 |
| hellaswag | acc_norm_stderr,none | 0.0046 |
| winogrande | acc,none | 0.5154 |
| winogrande | acc_stderr,none | 0.0140 |
| piqa | acc,none | 0.6251 |
| piqa | acc_stderr,none | 0.0113 |
| piqa | acc_norm,none | 0.6219 |
| piqa | acc_norm_stderr,none | 0.0113 |
| openbookqa | acc,none | 0.1860 |
| openbookqa | acc_stderr,none | 0.0174 |
| openbookqa | acc_norm,none | 0.3080 |
| openbookqa | acc_norm_stderr,none | 0.0207 |
| boolq | acc,none | 0.5303 |
| boolq | acc_stderr,none | 0.0087 |
![Benchmarks](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/tTgsWh576T2sdVzetvZ8C.png)
![image](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/Jwa95LBR5UgZAuq7IZ4Uu.png)
---
## 🧠 Model Architecture & Hyperparameters
Supra-50M is based on the `LlamaForCausalLM` architecture with the following configuration:
| Hyperparameter | Value |
|---|---|
| Architecture | Llama (decoder-only transformer) |
| Parameters | ~50M |
| `vocab_size` | 32,000 |
| `hidden_size` | 512 |
| `intermediate_size` | 1,408 |
| `num_hidden_layers` | 12 |
| `num_attention_heads` | 8 |
| `num_key_value_heads` | 4 (GQA) |
| `max_position_embeddings` | 1,024 |
| `rope_theta` | 10,000 |
| `tie_word_embeddings` | True |
---
## 📚 Training Data
| Property | Value |
|---|---|
| Dataset | [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (`sample-100BT` split) |
| Total tokens | 20,000,000,000 (20B) |
| Sequence length | 1,024 tokens |
| Storage format | Memory-mapped binary (`uint16`, ~40 GB) |
---
## 🔤 Tokenizer
A custom **Byte-Level BPE** tokenizer was trained from scratch on 500,000 documents sampled from `fineweb-edu (sample-10BT)`.
| Property | Value |
|---|---|
| Type | ByteLevelBPETokenizer |
| Vocabulary size | 32,000 |
| Min frequency | 2 |
| Special tokens | `<s>`, `<pad>`, `</s>`, `<unk>`, `<mask>` |
---
## ⚙️ Training Configuration
| Parameter | Value |
|---|---|
| Epochs | 1 |
| Per-device batch size | 32 |
| Gradient accumulation steps | 4 |
| Effective batch size | 128 × 1,024 tokens |
| Learning rate | 6e-4 |
| LR scheduler | Cosine |
| Warmup ratio | 2% |
| Optimizer | AdamW Fused (`adam_beta1=0.9`, `adam_beta2=0.95`) |
| Weight decay | 0.1 |
| Max grad norm | 1.0 |
| Precision | bfloat16 |
| `torch.compile` | Enabled |
| Hardware | Single GPU |
| Final loss | *3.259* |
---
## 🚀 Inference
```python
from transformers import pipeline
import torch
print("[*] Loading Supra-50M model from Hugging Face Hub...")
pipe = pipeline(
"text-generation",
model="SupraLabs/Supra-50M_BASE",
device_map="auto",
torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32
)
def generate_text(prompt, max_new_tokens=150):
result = pipe(
prompt,
max_new_tokens=max_new_tokens,
do_sample=True,
temperature=0.5,
top_k=25,
top_p=0.9,
repetition_penalty=1.2,
pad_token_id=pipe.tokenizer.pad_token_id,
eos_token_id=pipe.tokenizer.eos_token_id
)
return result[0]['generated_text']
# Example
prompt = "The importance of education is"
print(f"\nPrompt: {prompt}")
print("-" * 40)
print("\nOutput:\n" + generate_text(prompt))
```
---
## 💬 Sample Outputs
**Prompt:** `"The main concept of physics is "`
> The main concept of physics is iffy, and the idea that we can make things behave in a certain way. The most important part of physics is called quantum mechanics which states that all particles are made up of energy (energy) and matter (matter). In physics, there are two types of particles: elementary particles and exotic ones. These particles have properties like mass, speed or momentum but they don’t interact with each other to form new objects. This is because these particles do not exist independently from one another. In this case, an exotic particle might be created by adding more energy into its structure than it would take for a normal particle. However, when you add additional energy to an exotic particle, the new object will become smaller and larger until it becomes too large to fit within the existing structure. If you think about how light travels through space, it takes around 20 billion years before the light reaches our eyes. Light waves travel faster than light at high speeds so if we could create some kind of light wave, then we wouldn’t need any special equipment. It just needs a few hundred millionths of a second to produce light rays. So even though the light is moving along the same path as the current, the speed of light is different depending on where the light hits the
**Prompt:** `"Artificial intelligence is "`
> Artificial intelligence is iffy, it can be used to make intelligent machines that could take over the world. What does Artificial Intelligence mean? AI refers to artificial intelligence and machine learning technology which is a type of computer science (also known as artificial intelligence) in which computers are programmed with knowledge about their environment or other objects. The term AI comes from the Greek word "art" meaning "to create." The most common uses for AI include: - Machine Learning This means using algorithms like natural language processing systems to learn how words work together to form sentences such as “I am going to go to the store.” These programs will then use these rules to decide whether they should buy something or not so that you know what’s being sold on the internet. For example, if you purchase an ebook at Amazon, you may want to check its price first before purchasing it. If this happens, your shopping cart might look different than it did when purchased by someone else who bought it earlier. You can also think of AI as a way to help people understand themselves better through training and reasoning rather than simply seeing them doing things differently. In fact, we often see AI models working very well because of the way humans interact with our minds. This ability makes us more effective
**Prompt:** `"Once upon a time, "`
> Once upon a time, ...... I was so excited about the new school year and wanted to make some changes in my life. I had been looking for ways to help me become more self-aware. As an adult, I have always felt that there is no one way of doing things without thinking first. This has led me to start making small changes at home or at work. One such change was to create a space where I could be more mindful and aware of myself as well as other people around me. It’s important to remember that we all need our own personal growth and development. We can do this by taking responsibility for ourselves; being responsible for what happens outside us and keeping it within our control. By creating these smaller steps towards becoming more conscious of yourself, you will see how much better your future looks!The word "treaty" means something like "a treaty made with a king." The French word for "covenant," célèbre (French: cœle), comes from the Latin cecus ("to give up"). A covenant is not a binding agreement but rather an act of mutual understanding between two parties. In general terms, a contract is anything agreed on which someone agrees to agree to receive certain benefits. For example, if a person
---
## 📄 License
This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
---
## Third-party validation benchmarks by Datdanboi25
![image](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/nB4fwHYhilGEyxwa9DWTD.png)
---
*© SupraLabs 2026 — Project Chimera*

90
benchmarks.md Normal file
View File

@@ -0,0 +1,90 @@
| Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr |
|------------------------------------------------------------|------:|------|-----:|---------------|---|-------:|---|-------|
|arc_easy | 1|none | 0|acc |↑ | 0.5223|± | 0.0102|
| | |none | 0|acc_norm |↑ | 0.4600|± | 0.0102|
|blimp | 2|none | |acc |↑ | 0.7631|± | 0.0014|
| - blimp_adjunct_island | 1|none | 0|acc |↑ | 0.8420|± | 0.0115|
| - blimp_anaphor_gender_agreement | 1|none | 0|acc |↑ | 0.8430|± | 0.0115|
| - blimp_anaphor_number_agreement | 1|none | 0|acc |↑ | 0.9620|± | 0.0060|
| - blimp_animate_subject_passive | 1|none | 0|acc |↑ | 0.7820|± | 0.0131|
| - blimp_animate_subject_trans | 1|none | 0|acc |↑ | 0.8040|± | 0.0126|
| - blimp_causative | 1|none | 0|acc |↑ | 0.6980|± | 0.0145|
| - blimp_complex_NP_island | 1|none | 0|acc |↑ | 0.4940|± | 0.0158|
| - blimp_coordinate_structure_constraint_complex_left_branch| 1|none | 0|acc |↑ | 0.7420|± | 0.0138|
| - blimp_coordinate_structure_constraint_object_extraction | 1|none | 0|acc |↑ | 0.7520|± | 0.0137|
| - blimp_determiner_noun_agreement_1 | 1|none | 0|acc |↑ | 0.9790|± | 0.0045|
| - blimp_determiner_noun_agreement_2 | 1|none | 0|acc |↑ | 0.9680|± | 0.0056|
| - blimp_determiner_noun_agreement_irregular_1 | 1|none | 0|acc |↑ | 0.8990|± | 0.0095|
| - blimp_determiner_noun_agreement_irregular_2 | 1|none | 0|acc |↑ | 0.9650|± | 0.0058|
| - blimp_determiner_noun_agreement_with_adj_2 | 1|none | 0|acc |↑ | 0.9340|± | 0.0079|
| - blimp_determiner_noun_agreement_with_adj_irregular_1 | 1|none | 0|acc |↑ | 0.8740|± | 0.0105|
| - blimp_determiner_noun_agreement_with_adj_irregular_2 | 1|none | 0|acc |↑ | 0.9270|± | 0.0082|
| - blimp_determiner_noun_agreement_with_adjective_1 | 1|none | 0|acc |↑ | 0.9410|± | 0.0075|
| - blimp_distractor_agreement_relational_noun | 1|none | 0|acc |↑ | 0.8780|± | 0.0104|
| - blimp_distractor_agreement_relative_clause | 1|none | 0|acc |↑ | 0.7210|± | 0.0142|
| - blimp_drop_argument | 1|none | 0|acc |↑ | 0.7500|± | 0.0137|
| - blimp_ellipsis_n_bar_1 | 1|none | 0|acc |↑ | 0.8060|± | 0.0125|
| - blimp_ellipsis_n_bar_2 | 1|none | 0|acc |↑ | 0.8820|± | 0.0102|
| - blimp_existential_there_object_raising | 1|none | 0|acc |↑ | 0.8750|± | 0.0105|
| - blimp_existential_there_quantifiers_1 | 1|none | 0|acc |↑ | 0.9730|± | 0.0051|
| - blimp_existential_there_quantifiers_2 | 1|none | 0|acc |↑ | 0.2070|± | 0.0128|
| - blimp_existential_there_subject_raising | 1|none | 0|acc |↑ | 0.8810|± | 0.0102|
| - blimp_expletive_it_object_raising | 1|none | 0|acc |↑ | 0.7830|± | 0.0130|
| - blimp_inchoative | 1|none | 0|acc |↑ | 0.6330|± | 0.0152|
| - blimp_intransitive | 1|none | 0|acc |↑ | 0.7310|± | 0.0140|
| - blimp_irregular_past_participle_adjectives | 1|none | 0|acc |↑ | 0.8620|± | 0.0109|
| - blimp_irregular_past_participle_verbs | 1|none | 0|acc |↑ | 0.8930|± | 0.0098|
| - blimp_irregular_plural_subject_verb_agreement_1 | 1|none | 0|acc |↑ | 0.8990|± | 0.0095|
| - blimp_irregular_plural_subject_verb_agreement_2 | 1|none | 0|acc |↑ | 0.9030|± | 0.0094|
| - blimp_left_branch_island_echo_question | 1|none | 0|acc |↑ | 0.3810|± | 0.0154|
| - blimp_left_branch_island_simple_question | 1|none | 0|acc |↑ | 0.6470|± | 0.0151|
| - blimp_matrix_question_npi_licensor_present | 1|none | 0|acc |↑ | 0.1260|± | 0.0105|
| - blimp_npi_present_1 | 1|none | 0|acc |↑ | 0.5710|± | 0.0157|
| - blimp_npi_present_2 | 1|none | 0|acc |↑ | 0.6190|± | 0.0154|
| - blimp_only_npi_licensor_present | 1|none | 0|acc |↑ | 0.6250|± | 0.0153|
| - blimp_only_npi_scope | 1|none | 0|acc |↑ | 0.5360|± | 0.0158|
| - blimp_passive_1 | 1|none | 0|acc |↑ | 0.8770|± | 0.0104|
| - blimp_passive_2 | 1|none | 0|acc |↑ | 0.8840|± | 0.0101|
| - blimp_principle_A_c_command | 1|none | 0|acc |↑ | 0.5560|± | 0.0157|
| - blimp_principle_A_case_1 | 1|none | 0|acc |↑ | 1.0000|± | 0|
| - blimp_principle_A_case_2 | 1|none | 0|acc |↑ | 0.9650|± | 0.0058|
| - blimp_principle_A_domain_1 | 1|none | 0|acc |↑ | 0.9430|± | 0.0073|
| - blimp_principle_A_domain_2 | 1|none | 0|acc |↑ | 0.8040|± | 0.0126|
| - blimp_principle_A_domain_3 | 1|none | 0|acc |↑ | 0.5200|± | 0.0158|
| - blimp_principle_A_reconstruction | 1|none | 0|acc |↑ | 0.2920|± | 0.0144|
| - blimp_regular_plural_subject_verb_agreement_1 | 1|none | 0|acc |↑ | 0.8930|± | 0.0098|
| - blimp_regular_plural_subject_verb_agreement_2 | 1|none | 0|acc |↑ | 0.9110|± | 0.0090|
| - blimp_sentential_negation_npi_licensor_present | 1|none | 0|acc |↑ | 0.9930|± | 0.0026|
| - blimp_sentential_negation_npi_scope | 1|none | 0|acc |↑ | 0.7100|± | 0.0144|
| - blimp_sentential_subject_island | 1|none | 0|acc |↑ | 0.3310|± | 0.0149|
| - blimp_superlative_quantifiers_1 | 1|none | 0|acc |↑ | 0.7800|± | 0.0131|
| - blimp_superlative_quantifiers_2 | 1|none | 0|acc |↑ | 0.7450|± | 0.0138|
| - blimp_tough_vs_raising_1 | 1|none | 0|acc |↑ | 0.5390|± | 0.0158|
| - blimp_tough_vs_raising_2 | 1|none | 0|acc |↑ | 0.8780|± | 0.0104|
| - blimp_transitive | 1|none | 0|acc |↑ | 0.8430|± | 0.0115|
| - blimp_wh_island | 1|none | 0|acc |↑ | 0.7190|± | 0.0142|
| - blimp_wh_questions_object_gap | 1|none | 0|acc |↑ | 0.7590|± | 0.0135|
| - blimp_wh_questions_subject_gap | 1|none | 0|acc |↑ | 0.9280|± | 0.0082|
| - blimp_wh_questions_subject_gap_long_distance | 1|none | 0|acc |↑ | 0.8550|± | 0.0111|
| - blimp_wh_vs_that_no_gap | 1|none | 0|acc |↑ | 0.9490|± | 0.0070|
| - blimp_wh_vs_that_no_gap_long_distance | 1|none | 0|acc |↑ | 0.9490|± | 0.0070|
| - blimp_wh_vs_that_with_gap | 1|none | 0|acc |↑ | 0.5920|± | 0.0155|
| - blimp_wh_vs_that_with_gap_long_distance | 1|none | 0|acc |↑ | 0.3280|± | 0.0149|
|hellaswag | 1|none | 0|acc |↑ | 0.2914|± | 0.0045|
| | |none | 0|acc_norm |↑ | 0.3178|± | 0.0046|
|lambada_openai | 1|none | 0|acc |↑ | 0.2591|± | 0.0061|
| | |none | 0|perplexity |↓ | 95.5121|± | 4.1325|
|lambada_standard | 1|none | 0|acc |↑ | 0.1716|± | 0.0053|
| | |none | 0|perplexity |↓ |488.2170|± |23.4634|
|piqa | 1|none | 0|acc |↑ | 0.6224|± | 0.0113|
| | |none | 0|acc_norm |↑ | 0.6208|± | 0.0113|
|sciq | 1|none | 0|acc |↑ | 0.7720|± | 0.0133|
| | |none | 0|acc_norm |↑ | 0.6810|± | 0.0147|
|wikitext | 2|none | 0|bits_per_byte |↓ | 1.0267|± | N/A |
| | |none | 0|byte_perplexity|↓ | 2.0374|± | N/A |
| | |none | 0|word_perplexity|↓ | 44.9548|± | N/A |
|winogrande | 1|none | 0|acc |↑ | 0.5099|± | 0.0140|
|Groups|Version|Filter|n-shot|Metric| |Value | |Stderr|
|------|------:|------|------|------|---|-----:|---|-----:|
|blimp | 2|none | |acc |↑ |0.7631|± |0.0014|

BIN
benchmarks.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

32
config.json Normal file
View File

@@ -0,0 +1,32 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 512,
"initializer_range": 0.02,
"intermediate_size": 1408,
"max_position_embeddings": 1024,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 8,
"num_hidden_layers": 12,
"num_key_value_heads": 4,
"pad_token_id": 1,
"pretraining_tp": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 10000,
"rope_type": "default"
},
"tie_word_embeddings": true,
"transformers_version": "5.8.1",
"use_cache": false,
"vocab_size": 32000
}

10
generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 1,
"transformers_version": "5.8.1",
"use_cache": true
}

33
inference.py Normal file
View File

@@ -0,0 +1,33 @@
print("[*] Loading libraries...")
import torch
from transformers import LlamaForCausalLM, PreTrainedTokenizerFast
model_path = "./Chimera-FINAL"
print("[*] Loading tokenizer...")
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_path)
print("[*] Loading model...")
model = LlamaForCausalLM.from_pretrained(model_path)
model.eval()
prompt = "Artificial intelligence is " # "Artificial intelligence is " | "The main concept of physics is " | "Once upon a time, "
print(f"[*] Prompt: {prompt!r}")
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
outputs = model.generate(
input_ids=inputs["input_ids"],
attention_mask=inputs["attention_mask"],
max_new_tokens=256,
do_sample=True,
temperature=0.4,
top_p=0.85,
top_k=30,
repetition_penalty=1.1,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print("[*] Output of Supra 50M Base:", tokenizer.decode(outputs[0], skip_special_tokens=True))

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6b3287f381ae67127b09ac24939733792452cc7328925303b7ef7475f4f6f285
size 207157136

19
samples.txt Normal file
View File

@@ -0,0 +1,19 @@
[*] Prompt: 'The main concept of physics is '
[*] Output of Supra 50M Base: The main concept of physics is iffy, and the idea that we can make things behave in a certain way. The most important part of physics is called quantum mechanics which states that all particles are made up of energy (energy) and matter (matter).
In physics, there are two types of particles: elementary particles and exotic ones. These particles have properties like mass, speed or momentum but they don’t interact with each other to form new objects. This is because these particles do not exist independently from one another. In this case, an exotic particle might be created by adding more energy into its structure than it would take for a normal particle. However, when you add additional energy to an exotic particle, the new object will become smaller and larger until it becomes too large to fit within the existing structure.
If you think about how light travels through space, it takes around 20 billion years before the light reaches our eyes. Light waves travel faster than light at high speeds so if we could create some kind of light wave, then we wouldn’t need any special equipment. It just needs a few hundred millionths of a second to produce light rays. So even though the light is moving along the same path as the current, the speed of light is different depending on where the light hits the
[*] Prompt: 'Artificial intelligence is '
[*] Output of Supra 50M Base: Artificial intelligence is iffy, it can be used to make intelligent machines that could take over the world.
What does Artificial Intelligence mean?
AI refers to artificial intelligence and machine learning technology which is a type of computer science (also known as artificial intelligence) in which computers are programmed with knowledge about their environment or other objects. The term AI comes from the Greek word "art" meaning "to create."
The most common uses for AI include:
- Machine Learning
This means using algorithms like natural language processing systems to learn how words work together to form sentences such as “I am going to go to the store.”
These programs will then use these rules to decide whether they should buy something or not so that you know what’s being sold on the internet. For example, if you purchase an ebook at Amazon, you may want to check its price first before purchasing it. If this happens, your shopping cart might look different than it did when purchased by someone else who bought it earlier.
You can also think of AI as a way to help people understand themselves better through training and reasoning rather than simply seeing them doing things differently. In fact, we often see AI models working very well because of the way humans interact with our minds. This ability makes us more effective
[*] Prompt: 'Once upon a time, '
[*] Output of Supra 50M Base: Once upon a time, ...... I was so excited about the new school year and wanted to make some changes in my life.
I had been looking for ways to help me become more self-aware. As an adult, I have always felt that there is no one way of doing things without thinking first. This has led me to start making small changes at home or at work. One such change was to create a space where I could be more mindful and aware of myself as well as other people around me. It’s important to remember that we all need our own personal growth and development. We can do this by taking responsibility for ourselves; being responsible for what happens outside us and keeping it within our control. By creating these smaller steps towards becoming more conscious of yourself, you will see how much better your future looks!The word "treaty" means something like "a treaty made with a king." The French word for "covenant," célèbre (French: cœle), comes from the Latin cecus ("to give up"). A covenant is not a binding agreement but rather an act of mutual understanding between two parties. In general terms, a contract is anything agreed on which someone agrees to agree to receive certain benefits. For example, if a person

159032
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

9
tokenizer_config.json Normal file
View File

@@ -0,0 +1,9 @@
{
"backend": "tokenizers",
"bos_token": "<s>",
"eos_token": "</s>",
"model_max_length": 1000000000000000019884624838656,
"pad_token": "<pad>",
"tokenizer_class": "TokenizersBackend",
"unk_token": "<unk>"
}

191
train.py Normal file
View File

@@ -0,0 +1,191 @@
"""
© SupraLabs 2026 - Official pretraining code for PROJECT CHIMERA - 50M Llama
"""
import os
os.environ["PYTORCH_ALLOC_CONF"] = "expandable_segments:True"
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
print("[*] Loading libraries...")
import torch
import math
import numpy as np
from datasets import load_dataset
from tokenizers import ByteLevelBPETokenizer
from transformers import (
LlamaConfig,
LlamaForCausalLM,
PreTrainedTokenizerFast,
Trainer,
TrainingArguments,
)
from torch.utils.data import Dataset
from tqdm import tqdm
print("[*] Loading tokenizer...")
fast_tokenizer = ByteLevelBPETokenizer(
"custom_llama_tokenizer-vocab.json",
"custom_llama_tokenizer-merges.txt"
)
tokenizer = PreTrainedTokenizerFast(
tokenizer_object=fast_tokenizer,
bos_token="<s>",
eos_token="</s>",
unk_token="<unk>",
pad_token="<pad>",
)
TOKEN_BIN = "tokens.bin"
TARGET_TOKENS = 20_000_000_000
SEQ_LEN = 1024
BATCH_TEXTS = 1000
FLUSH_EVERY = 1_000_000
def build_token_bin(fast_tokenizer, path=TOKEN_BIN, target_tokens=TARGET_TOKENS):
if os.path.exists(path) and os.path.getsize(path) >= target_tokens * 2:
print(f"[=] Reusing existing token file: {path}")
return
print(f"[*] Streaming + tokenizing {target_tokens:,} tokens → {path}")
mm = np.memmap(path, dtype=np.uint16, mode="w+", shape=(target_tokens,))
dataset = load_dataset(
"HuggingFaceFW/fineweb-edu", "sample-100BT",
split="train", streaming=True
)
written = 0
buf = []
texts = []
pbar = tqdm(total=target_tokens, desc="[*] Gathering tokens", unit="tok")
def flush_buf():
nonlocal written, buf
if not buf:
return False
n = min(len(buf), target_tokens - written)
mm[written:written + n] = np.asarray(buf[:n], dtype=np.uint16)
written += n
pbar.update(n)
del buf[:n]
return written >= target_tokens
for example in dataset:
texts.append(example["text"])
if len(texts) >= BATCH_TEXTS:
encs = fast_tokenizer.encode_batch(texts)
texts.clear()
for e in encs:
buf.extend(e.ids)
if len(buf) >= FLUSH_EVERY:
if flush_buf():
break
if written < target_tokens and texts:
encs = fast_tokenizer.encode_batch(texts)
for e in encs:
buf.extend(e.ids)
if written < target_tokens:
flush_buf()
pbar.close()
mm.flush()
del mm
print(f"[+] Wrote {written:,} tokens to {path} "
f"({os.path.getsize(path)/1e6:.1f} MB)")
class MemmapDataset(Dataset):
def __init__(self, path, total_tokens, seq_len=SEQ_LEN):
self.path = path
self.seq_len = seq_len
self.n_chunks = total_tokens // seq_len
self._data = None # lazy open (Multiprocessing-safe)
@property
def data(self):
if self._data is None:
self._data = np.memmap(
self.path, dtype=np.uint16, mode="r",
shape=(self.n_chunks * self.seq_len,)
)
return self._data
def __len__(self):
return self.n_chunks
def __getitem__(self, idx):
s = idx * self.seq_len
arr = np.asarray(self.data[s:s + self.seq_len], dtype=np.int64)
ids = torch.from_numpy(arr)
return {"input_ids": ids, "labels": ids.clone()}
def collate_fn(batch):
input_ids = torch.stack([b["input_ids"] for b in batch])
labels = torch.stack([b["labels"] for b in batch])
return {"input_ids": input_ids, "labels": labels}
print(f"[*] Preparing {TARGET_TOKENS:,} tokens (streaming, memmap-backed)...")
build_token_bin(fast_tokenizer, TOKEN_BIN, TARGET_TOKENS)
dataset = MemmapDataset(TOKEN_BIN, TARGET_TOKENS, seq_len=SEQ_LEN)
print(f"[+] Dataset ready: {len(dataset):,} chunks of {SEQ_LEN} tokens")
print("[*] Setting up model...")
config = LlamaConfig(
vocab_size=32_000,
hidden_size=512,
intermediate_size=1408,
num_hidden_layers=12,
num_attention_heads=8,
num_key_value_heads=4,
max_position_embeddings=1024,
rope_theta=10_000,
tie_word_embeddings=True,
pad_token_id=tokenizer.pad_token_id,
bos_token_id=tokenizer.bos_token_id,
eos_token_id=tokenizer.eos_token_id,
)
model = LlamaForCausalLM(config)
print(f"[*] Model parameters: {model.num_parameters():,}")
print("[*] Defining training arguments...")
training_args = TrainingArguments(
output_dir="./Chimera",
num_train_epochs=1,
per_device_train_batch_size=32,
gradient_accumulation_steps=4,
save_steps=500,
save_total_limit=2,
logging_steps=100,
weight_decay=0.1,
fp16=False,
bf16=True,
push_to_hub=False,
report_to="none",
dataloader_num_workers=os.cpu_count() // 2,
dataloader_pin_memory=True,
learning_rate=6e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.02,
max_grad_norm=1.0,
optim="adamw_torch_fused",
adam_beta1=0.9,
adam_beta2=0.95,
torch_compile=True,
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=dataset,
data_collator=collate_fn,
)
print("[*] Starting training...")
trainer.train()
trainer.save_model("./Chimera-FINAL")
tokenizer.save_pretrained("./Chimera-FINAL")
print("[*] Training finished.")

25
train_tokenizer.py Normal file
View File

@@ -0,0 +1,25 @@
print("[*] Loading libraries...")
from datasets import load_dataset
from tokenizers import ByteLevelBPETokenizer
from tqdm import tqdm
dataset = load_dataset("HuggingFaceFW/fineweb-edu", "sample-10BT", split="train", streaming=True)
def get_training_corpus():
dataset_iter = iter(dataset)
for _ in tqdm(range(500_000), desc="Feeding data"):
yield next(dataset_iter)["text"]
tokenizer = ByteLevelBPETokenizer()
print("[*] Training tokenizer...")
tokenizer.train_from_iterator(
get_training_corpus(),
vocab_size=32_000,
min_frequency=2,
show_progress=True,
special_tokens=["<s>", "<pad>", "</s>", "<unk>", "<mask>"]
)
tokenizer.save_model(".", "custom_llama_tokenizer")
print("[*] Tokenizer training complete!")

1847
training.log Normal file
View File

File diff suppressed because it is too large Load Diff