322 lines
7.5 KiB
Markdown
322 lines
7.5 KiB
Markdown
---
|
|
language:
|
|
- en
|
|
license: apache-2.0
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- causal-lm
|
|
- base-model
|
|
- transformers
|
|
- safetensors
|
|
- veyra
|
|
- small-language-model
|
|
model-index:
|
|
- name: Veyra2-Mango-30M-Base
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: SciCloze-900
|
|
type: veyra-ai/SciCloze-900
|
|
metrics:
|
|
- name: Accuracy
|
|
type: accuracy
|
|
value: 42.44
|
|
source:
|
|
name: Local evaluation
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
|
|
- task:
|
|
type: multiple-choice
|
|
name: Multiple Choice
|
|
dataset:
|
|
name: SciQ
|
|
type: sciq
|
|
metrics:
|
|
- name: Accuracy
|
|
type: accuracy
|
|
value: 65.10
|
|
- name: Normalized Accuracy
|
|
type: acc_norm
|
|
value: 58.30
|
|
source:
|
|
name: Local lm-evaluation-harness
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
|
|
- task:
|
|
type: multiple-choice
|
|
name: Multiple Choice
|
|
dataset:
|
|
name: PIQA
|
|
type: piqa
|
|
metrics:
|
|
- name: Normalized Accuracy
|
|
type: acc_norm
|
|
value: 59.52
|
|
source:
|
|
name: Local lm-evaluation-harness
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
|
|
- task:
|
|
type: multiple-choice
|
|
name: Multiple Choice
|
|
dataset:
|
|
name: ARC-Easy
|
|
type: ai2_arc
|
|
config: ARC-Easy
|
|
metrics:
|
|
- name: Normalized Accuracy
|
|
type: acc_norm
|
|
value: 37.88
|
|
source:
|
|
name: Local lm-evaluation-harness
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
|
|
- task:
|
|
type: multiple-choice
|
|
name: Multiple Choice
|
|
dataset:
|
|
name: ARC-Challenge
|
|
type: ai2_arc
|
|
config: ARC-Challenge
|
|
metrics:
|
|
- name: Normalized Accuracy
|
|
type: acc_norm
|
|
value: 23.29
|
|
source:
|
|
name: Local lm-evaluation-harness
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
|
|
- task:
|
|
type: multiple-choice
|
|
name: Multiple Choice
|
|
dataset:
|
|
name: HellaSwag
|
|
type: hellaswag
|
|
metrics:
|
|
- name: Normalized Accuracy
|
|
type: acc_norm
|
|
value: 28.76
|
|
source:
|
|
name: Local lm-evaluation-harness
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
|
|
- task:
|
|
type: multiple-choice
|
|
name: Multiple Choice
|
|
dataset:
|
|
name: Winogrande
|
|
type: winogrande
|
|
metrics:
|
|
- name: Accuracy
|
|
type: accuracy
|
|
value: 49.33
|
|
source:
|
|
name: Local lm-evaluation-harness
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
|
|
- task:
|
|
type: multiple-choice
|
|
name: Multiple Choice
|
|
dataset:
|
|
name: OpenBookQA
|
|
type: openbookqa
|
|
metrics:
|
|
- name: Accuracy
|
|
type: accuracy
|
|
value: 14.80
|
|
- name: Normalized Accuracy
|
|
type: acc_norm
|
|
value: 27.20
|
|
source:
|
|
name: Local lm-evaluation-harness
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
|
|
- task:
|
|
type: question-answering
|
|
name: Question Answering
|
|
dataset:
|
|
name: BoolQ
|
|
type: boolq
|
|
metrics:
|
|
- name: Accuracy
|
|
type: accuracy
|
|
value: 42.81
|
|
source:
|
|
name: Local lm-evaluation-harness
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
|
|
- task:
|
|
type: multiple-choice
|
|
name: Multiple Choice
|
|
dataset:
|
|
name: ArithMark-2.0
|
|
type: AxiomicLabs/ArithMark-2.0
|
|
split: train
|
|
metrics:
|
|
- name: Accuracy
|
|
type: accuracy
|
|
value: 27.96
|
|
source:
|
|
name: Local evaluation
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
|
|
- task:
|
|
type: multiple-choice
|
|
name: Multiple Choice
|
|
dataset:
|
|
name: ArithMark-3.0
|
|
type: AxiomicLabs/Arithmark-3.0
|
|
split: train
|
|
metrics:
|
|
- name: Accuracy
|
|
type: accuracy
|
|
value: 36.90
|
|
source:
|
|
name: Local evaluation
|
|
url: https://huggingface.co/veyra-ai/Veyra2-Mango-30M-Base
|
|
---
|
|
|
|
|
|
|
|
# Veyra2-Mango-30M-Base
|
|
|
|
Veyra2-Mango-30M-Base is a 30.7M-parameter Llama-like causal language model trained from scratch on approximately 30B tokens. It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation.
|
|
|
|
## Model Details
|
|
|
|
| Property | Value |
|
|
| :--- | :--- |
|
|
| **Parameters** | 30,683,520 |
|
|
| **Architecture** | LlamaForCausalLM |
|
|
| **Layers** | 16 |
|
|
| **Hidden size** | 384 |
|
|
| **Attention heads** | 6 |
|
|
| **KV heads** | 2 |
|
|
| **Head dim** | 64 |
|
|
| **Intermediate size** | 1152 |
|
|
| **Vocabulary size** | 8192 |
|
|
| **Context length used in training** | 3072 |
|
|
| **Activation** | SwiGLU / SiLU |
|
|
| **Normalization** | RMSNorm |
|
|
| **Attention** | GQA |
|
|
| **Positional encoding** | RoPE |
|
|
| **Weight tying** | Tied input embeddings and LM head |
|
|
| **Training tokens** | Approximately 30B |
|
|
| **Training precision** | bfloat16 |
|
|
| **Optimizer** | AdamW |
|
|
|
|
## Tokenizer
|
|
|
|
Special tokens:
|
|
|
|
- `<|endoftext|>`: 0
|
|
- `<|im_start|>`: 1
|
|
- `<|im_end|>`: 2
|
|
- `<|pad|>`: 3
|
|
|
|
## Training Data
|
|
|
|
The model was trained on a 30B-token pretraining mixture.
|
|
|
|
Stage 1 18,000,000,000 tokens 180 shards
|
|
|
|
Mixture:
|
|
|
|
dclm_baseline: 50%
|
|
finephrase: 20%
|
|
cosmopedia_v2: 10%
|
|
finemath_4plus: 10%
|
|
ultrafineweb_multistyle: 5%
|
|
ultrafineweb_qa: 5%
|
|
|
|
Stage 1.5 4,000,000,000 tokens 40 shards
|
|
|
|
This stage linearly transitions from the Stage 1 mixture to the Stage 2 mixture.
|
|
|
|
Stage 2 8,000,000,000 tokens 80 shards
|
|
|
|
Mixture:
|
|
|
|
finephrase: 30%
|
|
dclm_baseline: 30%
|
|
cosmopedia_v2: 18%
|
|
finemath_4plus: 10%
|
|
ultrafineweb_multistyle: 5%
|
|
ultrafineweb_qa: 5%
|
|
ultrachat: 2%
|
|
|
|
## Training Summary
|
|
|
|
- Final step: 25,432
|
|
- Tokens seen: 30,000,000,000
|
|
- Tokens per step: 1,179,648
|
|
- Sequence length: 3072
|
|
- Last train loss: 2.5062
|
|
|
|
## Usage
|
|
|
|
<pre><code>import torch
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
|
model_id = "veyra-ai/Veyra2-Mango-30M-Base"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(
|
|
model_id,
|
|
)
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_id,
|
|
torch_dtype=torch.float16,
|
|
device_map="auto",
|
|
)
|
|
|
|
prompt = "In the 19th century"
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|
|
|
with torch.no_grad():
|
|
output = model.generate(
|
|
**inputs,
|
|
max_new_tokens=120,
|
|
do_sample=True,
|
|
temperature=0.6,
|
|
top_p=0.9,
|
|
repetition_penalty=1.1,
|
|
use_cache=True,
|
|
pad_token_id=tokenizer.pad_token_id,
|
|
eos_token_id=tokenizer.eos_token_id,
|
|
)
|
|
|
|
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
|
</code></pre>
|
|
|
|
## Notes on Generation
|
|
|
|
Veyra2-Mango-30M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant.
|
|
Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.
|
|
|
|
## Intended Use
|
|
|
|
This model is intended for:
|
|
|
|
- small language model research
|
|
- continued pretraining
|
|
- benchmarking
|
|
- experimentation with compact causal LMs
|
|
|
|
## Limitations
|
|
|
|
- Not instruction tuned
|
|
- Not RLHF tuned
|
|
- Not safe for factual or high-stakes use without additional validation
|
|
- Can hallucinate names, citations, species, references, and technical claims
|
|
- Open-ended text may drift off-topic
|
|
- Context length during training was 3072 tokens
|
|
|
|
## Citation
|
|
|
|
If you use this model, please cite the model repository:
|
|
|
|
`veyra-ai/Veyra2-Mango-30M-Base` |