91 lines
3.0 KiB
Markdown
91 lines
3.0 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
license: mit
|
||
|
|
tags:
|
||
|
|
- learnllm
|
||
|
|
- fineweb
|
||
|
|
- pytorch
|
||
|
|
- llama
|
||
|
|
- bpe
|
||
|
|
- from-scratch
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
base_model: jaweed123/TinyJLLM
|
||
|
|
---
|
||
|
|
|
||
|
|
# TinyJLLM — 100M-parameter small language model built from scratch
|
||
|
|
|
||
|
|
A decoder-only Transformer (~102.5M parameters) pretrained **from random
|
||
|
|
initialization** on ~5 GB of FineWeb (`sample-10BT`), 3 epochs / 108,000
|
||
|
|
optimizer steps. Built as a fully educational pipeline (LearnLLM Run #2):
|
||
|
|
custom 32K byte-level BPE tokenizer, from-scratch Transformer, sharded
|
||
|
|
uint16 data pipeline, BF16 training, and verified exports.
|
||
|
|
|
||
|
|
**Final metrics:** validation loss 3.50 (perplexity 33.1); the best
|
||
|
|
checkpoint (step 89K) reached 3.48 / 32.6.
|
||
|
|
|
||
|
|
## Model details
|
||
|
|
|
||
|
|
| Property | Value |
|
||
|
|
|---|---|
|
||
|
|
| Parameters | 102,450,432 (~102.5M) |
|
||
|
|
| Architecture | Llama-style decoder-only: RMSNorm, RoPE (half-split), SwiGLU, tied embeddings, no biases |
|
||
|
|
| Layers / heads / head_dim | 11 / 12 / 64 |
|
||
|
|
| Context length | 512 |
|
||
|
|
| Vocabulary | 32,000 (custom byte-level BPE, `<pad> <unk> <bos> <eos>` = 0-3) |
|
||
|
|
| Pretraining data | FineWeb sample-10BT, ~5.37 GB raw, 1.75M documents |
|
||
|
|
| Tokens seen | 3.54B (3 epochs) |
|
||
|
|
| Hardware | RTX 4060 8 GB, ~30K tok/s (torch.compile) |
|
||
|
|
| Precision | BF16 mixed precision, FP32 master weights |
|
||
|
|
|
||
|
|
## Intended use
|
||
|
|
|
||
|
|
- Educational reference: inspect a small, complete, honest pretraining run.
|
||
|
|
- Qualitative experimentation: prompt it (it follows prompts *as text*; it
|
||
|
|
is a **base model** — no instruction tuning yet).
|
||
|
|
- A base for further stages (SFT, DPO, domain fine-tuning).
|
||
|
|
|
||
|
|
Known limitations: small scale ⇒ repetition in long generations, weak
|
||
|
|
instruction following, limited world knowledge.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("jaweed123/TinyJLLM")
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("jaweed123/TinyJLLM")
|
||
|
|
|
||
|
|
prompt = "The future of AI is"
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
|
out = model.generate(**inputs, max_new_tokens=50, temperature=0.8, top_k=50, top_p=0.95)
|
||
|
|
print(tokenizer.decode(out[0]))
|
||
|
|
```
|
||
|
|
|
||
|
|
### llama.cpp / GGUF
|
||
|
|
|
||
|
|
The repo also ships GGUF files under `gguf/` (F16, Q8_0, Q4_K_M) — load
|
||
|
|
directly with llama.cpp or `llama-cpp-python`.
|
||
|
|
|
||
|
|
## Training details
|
||
|
|
|
||
|
|
- Custom 32K byte-level BPE (trained on a 512 MB FineWeb sample).
|
||
|
|
- Tokens stored once as uint16 shards (591 train + 6 validation).
|
||
|
|
- AdamW (lr 3e-4, wd 0.1, decay/no-decay groups), warmup 1,000 + cosine
|
||
|
|
to 1e-5, effective batch 64 (32,768 tokens/step), gradient clipping 1.0.
|
||
|
|
- Full run: ~35 h on an RTX 4060.
|
||
|
|
|
||
|
|
## Files
|
||
|
|
|
||
|
|
- `config.json` — Llama-compatible config (`LlamaForCausalLM`)
|
||
|
|
- `model.safetensors` — FP32 weights
|
||
|
|
- `tokenizer.json` / `tokenizer_config.json` — custom BPE
|
||
|
|
- `generation_config.json` — decoding defaults
|
||
|
|
- `gguf/` — llama.cpp formats
|
||
|
|
|
||
|
|
## Acknowledgments
|
||
|
|
|
||
|
|
FineWeb (HuggingFaceFW), Hugging Face `tokenizers` / `datasets`,
|
||
|
|
PyTorch, llama.cpp. Built with the LearnLLM educational pipeline
|
||
|
|
(`src/learnllm` at the project repository).
|