exnivo/tinybrain-100m-base
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: exnivo/tinybrain-100m-base
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-08 06:26:10 +08:00
commit 482fa250ba
10 changed files with 119626 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
assets/tinybrain-100m-base-banner.png filter=lfs diff=lfs merge=lfs -text

448
README.md Normal file
View File

@@ -0,0 +1,448 @@
---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- llama
- text-generation
- causal-lm
- tinybrain
- from-scratch
- 100m
- base-model
- small-language-model
- tiny-llm
- english
- pretraining
- transformers
datasets:
- exnivo/tinybrain-pretrain-corpus-2b
---
<p align="center">
<img
src="https://huggingface.co/exnivo/tinybrain-100m-base/resolve/main/assets/tinybrain-100m-base-banner.png"
alt="TinyBrain-100M Base — Base language model for small LLMs"
width="100%"
/>
</p>
# TinyBrain-100M Base
**A 103M parameter English causal language model trained from scratch.**
TinyBrain-100M Base is a small LLaMA-style causal language model trained from scratch on the [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) dataset.
This is a **base model**, not an instruct/chat model. It is intended for language modeling experiments, continued pretraining, supervised fine-tuning, and small-model research.
For chat or instruction-following behavior, use the instruction-tuned version:
[`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct)
## Quick Start
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "exnivo/tinybrain-100m-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "Photosynthesis is the process by which plants"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=80,
temperature=0.8,
top_p=0.9,
repetition_penalty=1.08,
do_sample=True,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## At a Glance
| Item | Details |
|---|---|
| Model type | Base causal language model |
| Parameters | 103,385,856 |
| Approx. size | 103.4M |
| Architecture | LLaMA-style causal transformer |
| Language | English |
| Context length | 2048 tokens |
| Vocabulary size | 24,000 |
| Tokenizer | Custom TinyBrain tokenizer |
| Training style | From scratch pretraining |
| Pretraining dataset | [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) |
| Instruct dataset | [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k) |
| Instruct model | [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct) |
## Model Details
| Item | Value |
|---|---|
| Parameters | 103.4M |
| Architecture | `llama` / `LlamaForCausalLM` |
| Vocabulary size | 24,000 |
| Context length | 2048 tokens |
| Hidden size | 768 |
| Intermediate size | 2048 |
| Layers | 12 |
| Attention heads | 12 |
| Key/value heads | 12 |
| Activation | SiLU |
| RMS norm epsilon | `1e-05` |
| Tied embeddings | true |
| BOS token | `<|bos|>` |
| EOS token | `<|eos|>` |
| PAD token | `<|pad|>` |
| Dataset | [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) |
## Intended Use
TinyBrain-100M Base is intended for:
- small language model research
- causal language modeling experiments
- continued pretraining
- supervised fine-tuning
- instruction tuning
- tokenizer/model experiments
- educational small-model projects
- comparing base vs instruct behavior
- lightweight local model experiments
This model is best used as a **base checkpoint** for further training.
## Not Intended For
This model is not intended to be used directly as a finished assistant.
Do not rely on the base model for:
- polished chat behavior
- instruction following
- safety-critical answers
- factual authority
- medical, legal, or financial advice
- live/current information
- advanced reasoning
- production use without evaluation
For assistant-style behavior, use [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct) instead.
## Training Data
TinyBrain-100M Base was trained on:
[`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b)
The pretraining corpus is a mixed-source English dataset containing factual text, educational text, math reasoning data, code data, conversation-style data, and clean web text.
The pretraining corpus scan found:
| Metric | Value |
|---|---:|
| Rows | 3,013,308 |
| Characters | 7,767,447,861 |
| Words | 1,249,832,587 |
| Approx. tokens | ~1.81B tokenizer-independent estimate |
The training run used an estimated **~2.1B training tokens**. Token counts may differ depending on tokenizer, packing, filtering, and training pipeline details.
## Dataset Mix
The pretraining corpus includes these broad categories:
| Category | Rows | Percent |
|---|---:|---:|
| `factual` | 773,492 | 25.67% |
| `educational` | 752,625 | 24.98% |
| `math_reasoning` | 633,341 | 21.02% |
| `code` | 326,019 | 10.82% |
| `conversation` | 296,728 | 9.85% |
| `clean_web` | 231,103 | 7.67% |
## Relationship to TinyBrain
TinyBrain is a small LLM project focused on compact datasets, small base models, and instruction-tuned models.
| Stage | Repository | Purpose |
|---|---|---|
| Pretraining corpus | [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) | Base language model training data |
| Base model | [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base) | Small causal LM trained from scratch |
| SFT dataset | [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k) | Instruction/chat fine-tuning data |
| Instruct model | [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct) | Chat/instruct model fine-tuned from the base model |
Pipeline:
```text
TinyBrain Pretrain Corpus 2B
↓
TinyBrain-100M Base
↓
TinyBrain Instruct 200K
↓
TinyBrain-100M Instruct
```
## Evaluation
A quick WikiText-2 evaluation was run on the base model.
| Metric | Value |
|---|---:|
| Eval tokens | 38,138 |
| Eval text chars | 159,791 |
| Loss | 3.7440 |
| Perplexity | 42.27 |
This is a lightweight evaluation, not a full benchmark suite. Results may vary depending on evaluation script, tokenizer settings, context length, and dataset preprocessing.
## Base Model Behavior
TinyBrain-100M Base is a raw pretrained model. It can complete text, but it is not tuned to follow instructions.
Example base prompt:
```text
Photosynthesis is the process by which plants
```
The base model may continue with partially useful text, but it can also repeat, drift, hallucinate, or produce broken completions. This is expected for a small base model and is one reason instruction tuning is needed.
For better chat behavior, use:
[`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct)
## Recommended Generation Settings
For raw base-model text completion:
```python
temperature = 0.8
top_p = 0.9
max_new_tokens = 80
repetition_penalty = 1.08
```
For more stable completions:
```python
temperature = 0.5
top_p = 0.85
max_new_tokens = 80
repetition_penalty = 1.1
```
For deterministic testing:
```python
do_sample = False
max_new_tokens = 80
```
## Example: Text Completion
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "exnivo/tinybrain-100m-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "Gravity is the force that"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=80,
do_sample=True,
temperature=0.8,
top_p=0.9,
repetition_penalty=1.08,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## Example: Fine-Tuning Starting Point
TinyBrain-100M Base can be fine-tuned on the TinyBrain SFT dataset:
```python
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import SFTTrainer, SFTConfig
base_model = "exnivo/tinybrain-100m-base"
dataset_id = "exnivo/tinybrain-instruct-sft-200k"
tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model)
ds = load_dataset(dataset_id, split="train")
def format_example(example):
text = ""
for message in example["messages"]:
role = message["role"]
content = message["content"].strip()
if role == "user":
text += f"User: {content}\n"
elif role == "assistant":
text += f"Assistant: {content}\n"
return {"text": text.strip()}
ds = ds.map(format_example)
config = SFTConfig(
output_dir="tinybrain-100m-instruct-sft",
dataset_text_field="text",
max_seq_length=512,
per_device_train_batch_size=8,
gradient_accumulation_steps=4,
learning_rate=2e-5,
num_train_epochs=1,
logging_steps=20,
save_steps=500,
)
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=ds,
args=config,
)
trainer.train()
```
## Training
TinyBrain-100M Base was trained from scratch on the TinyBrain pretraining corpus.
Training details:
| Item | Value |
|---|---|
| Training type | From-scratch causal language modeling |
| Dataset | TinyBrain Pretrain Corpus 2B |
| Approx. training tokens | ~2.1B |
| Reported best validation loss | 2.6779 |
| Training precision | bf16 |
| Hardware | NVIDIA RTX PRO 6000 Blackwell Server Edition |
## Strengths
TinyBrain-100M Base is useful because it is:
- small and lightweight
- trained from scratch
- easy to inspect
- easy to fine-tune
- based on an open TinyBrain data pipeline
- trained on a compact mixed-source corpus
- suitable for small-model experiments
- useful as a base checkpoint for SFT
## Limitations
TinyBrain-100M Base has important limitations.
The model may:
- hallucinate facts
- produce broken or repetitive text
- fail at math
- fail at instruction following
- misunderstand prompts
- generate incomplete code
- produce outdated or incorrect information
- drift off-topic
- repeat web/data artifacts
This is expected for a small **base** model. It has not been tuned to reliably follow user instructions.
For chat and assistant behavior, use the instruction-tuned model instead.
## Suggested Evaluation
Recommended checks:
- validation loss / perplexity
- text completion quality
- repetition behavior
- short factual completions
- simple math completions
- code completion sanity checks
- hallucination checks
- before/after SFT comparison
- downstream instruction-following after fine-tuning
Example base-model prompts:
```text
Paris is the capital city of
```
```text
The Netherlands is a country in
```
```text
A cat is an animal that
```
```text
One plus one equals
```
```text
Photosynthesis is the process by which plants
```
## Citation
If you use this model, you can cite it as:
```bibtex
@misc{tinybrain_100m_base,
title = {TinyBrain-100M Base},
author = {exnivo},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/exnivo/tinybrain-100m-base}}
}
```
## Related Repositories
- Pretraining corpus: [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b)
- Base model: [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base)
- SFT dataset: [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k)
- Instruct model: [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct)
## License
This model is released under the Apache 2.0 license.
The training dataset is mixed-source and currently listed under `license: other`. Users should review the upstream dataset licenses and source metadata before commercial use of models trained or fine-tuned from this checkpoint.
## Disclaimer
TinyBrain-100M Base is an experimental small base language model. It may produce incorrect, biased, unsafe, nonsensical, or misleading outputs.
Do not use it for high-stakes applications without additional training, filtering, evaluation, and safeguards.

0
assets/.gitkeep Normal file
View File

3
assets/tinybrain-100m-base-banner.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:026046aeca29069b800a10aa57ce460f13f6ab4b662b603ce401ccab1fbdf407
size 1094620

32
config.json Normal file
View File

@@ -0,0 +1,32 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 1,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 2048,
"max_position_embeddings": 2048,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_key_value_heads": 12,
"pad_token_id": 0,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_parameters": {
"rope_theta": 10000.0,
"rope_type": "default"
},
"tie_word_embeddings": true,
"transformers_version": "5.12.1",
"use_cache": false,
"vocab_size": 24000
}

10
generation_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"output_attentions": false,
"output_hidden_states": false,
"pad_token_id": 0,
"transformers_version": "5.12.1",
"use_cache": false
}

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1411777f034283d3dd6878ac969689eb6069be9e943878d14f1c5ea2377de8e5
size 413555632

14
special_tokens_map.json Normal file
View File

@@ -0,0 +1,14 @@
{
"pad_token": "<|pad|>",
"bos_token": "<|bos|>",
"eos_token": "<|eos|>",
"additional_special_tokens": [
"<|user|>",
"<|assistant|>",
"<|system|>",
"<|end|>",
"<|tool|>",
"<|tool_call|>",
"<|tool_result|>"
]
}

119061
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

19
tokenizer_config.json Normal file
View File

@@ -0,0 +1,19 @@
{
"tokenizer_class": "PreTrainedTokenizerFast",
"model_max_length": 2048,
"padding_side": "right",
"truncation_side": "right",
"clean_up_tokenization_spaces": false,
"bos_token": "<|bos|>",
"eos_token": "<|eos|>",
"pad_token": "<|pad|>",
"additional_special_tokens": [
"<|user|>",
"<|assistant|>",
"<|system|>",
"<|end|>",
"<|tool|>",
"<|tool_call|>",
"<|tool_result|>"
]
}