89 lines
3.5 KiB
Markdown
89 lines
3.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
tags:
|
|
- text-generation
|
|
- causal-lm
|
|
- chatml
|
|
- from-scratch
|
|
- hydrion
|
|
- opengcm
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
Hydrion is a 114M-parameter causal language model, pretrained from scratch and fine-tuned for chat, built on a single RTX 3060 plus a handful of rented A100 hours.
|
|
|
|
This repo (`OpenGCM/Hydrion-Base`) is the base model, non-chat ready version. The instruction model (chat formatting) is available at [`OpenGCM/Hydrion-SFT`](https://huggingface.co/OPENGCM/Hydrion-SFT).
|
|
## Model Details
|
|
|
|
- **Architecture:** Llama-style decoder-only transformer (RMSNorm, rotary position embeddings, SwiGLU MLP, grouped-query attention)
|
|
- **Parameters:** 114.1M
|
|
- **Layers:** 12
|
|
- **Hidden size:** 768
|
|
- **Attention heads:** 12 (4 KV heads, GQA)
|
|
- **Context length:** 1024 tokens
|
|
- **Tokenizer:** [`EleutherAI/gpt-neox-20b`](https://huggingface.co/EleutherAI/gpt-neox-20b)
|
|
- **License:** Apache 2.0
|
|
|
|
## Training
|
|
|
|
Hydrion was trained in two pretraining stages.
|
|
|
|
1. **Initial pretraining** — ~2B tokens on a FineWeb-Edu / Wikipedia mix, trained on a single RTX 3060 (12GB).
|
|
2. **Continued pretraining** — an additional ~0.5B tokens on a more diverse mix (FineWeb-Edu, Wikipedia, TinyStories, a code subset, and Dolly), run on a rented A100 to broaden register and topic coverage beyond pure web/encyclopedic text.
|
|
|
|
Total pretraining exposure: roughly **2.5 billion tokens**.
|
|
|
|
## Benchmarks
|
|
|
|
Evaluated with [`lm-evaluation-harness`](https://github.com/EleutherAI/lm-evaluation-harness) on the base (pre-SFT) checkpoint:
|
|
|
|
| Benchmark | Metric | Score |
|
|
|---|---|---|
|
|
| BLiMP | acc | 80.08% |
|
|
| ARC-Easy | acc | 47.26% |
|
|
| ARC-Easy | acc_norm | 43.39% |
|
|
| WikiText-2 | byte_perplexity | 2.04 |
|
|
| WikiText-2 | bits_per_byte | 1.03 |
|
|
| WikiText-2 | word_perplexity | 45.02 |
|
|
|
|
Grammatical judgment (BLiMP) is comparable to models trained on far larger token budgets; factual/reasoning performance (ARC-Easy) is meaningfully weaker, consistent with the relatively small pretraining corpus.
|
|
|
|
## Usage
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoTokenizer, LlamaForCausalLM
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("OPENGCM/Hydrion-Base")
|
|
model = LlamaForCausalLM.from_pretrained("OPENGCM/Hydrion-Base", torch_dtype=torch.bfloat16).cuda()
|
|
model.eval()
|
|
|
|
prompt = "What is the capital of"
|
|
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
|
|
|
|
with torch.no_grad():
|
|
output = model.generate(
|
|
**inputs,
|
|
max_new_tokens=150,
|
|
do_sample=True,
|
|
temperature=0.7,
|
|
top_p=0.9,
|
|
repetition_penalty=1.3,
|
|
no_repeat_ngram_size=3,
|
|
eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
|
|
)
|
|
|
|
response = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
|
|
print(response)
|
|
```
|
|
|
|
## Limitations
|
|
|
|
Hydrion is a small model trained on a modest token budget (~2.5B tokens, versus the trillions used by comparable production small models). It should **not** be relied on for factual accuracy. It reliably produces fluent, grammatically well-formed English and responds in a conversational chat format, but frequently states incorrect facts, fabricates names/dates/attributions, and performs poorly at arithmetic and multi-step reasoning. Treat outputs as unreliable by default — this model is best understood as a demonstration of a working from-scratch training pipeline rather than a usable knowledge source or assistant.
|