初始化项目，由ModelHub XC社区提供模型

Model: jaweed123/TinyJLLM
Source: Original Platform

README.md

@@ -0,0 +1,91 @@
+---
+language:
+- en
+license: mit
+tags:
+- learnllm
+- fineweb
+- pytorch
+- llama
+- bpe
+- from-scratch
+library_name: transformers
+pipeline_tag: text-generation
+base_model: jaweed123/TinyJLLM
+---
+
+# TinyJLLM — 100M-parameter small language model built from scratch
+
+A decoder-only Transformer (~102.5M parameters) pretrained **from random
+initialization** on ~5 GB of FineWeb (`sample-10BT`), 3 epochs / 108,000
+optimizer steps. Built as a fully educational pipeline (LearnLLM Run #2):
+custom 32K byte-level BPE tokenizer, from-scratch Transformer, sharded
+uint16 data pipeline, BF16 training, and verified exports.
+
+**Final metrics:** validation loss 3.50 (perplexity 33.1); the best
+checkpoint (step 89K) reached 3.48 / 32.6.
+
+## Model details
+
+| Property | Value |
+|---|---|
+| Parameters | 102,450,432 (~102.5M) |
+| Architecture | Llama-style decoder-only: RMSNorm, RoPE (half-split), SwiGLU, tied embeddings, no biases |
+| Layers / heads / head_dim | 11 / 12 / 64 |
+| Context length | 512 |
+| Vocabulary | 32,000 (custom byte-level BPE, `<pad> <unk> <bos> <eos>` = 0-3) |
+| Pretraining data | FineWeb sample-10BT, ~5.37 GB raw, 1.75M documents |
+| Tokens seen | 3.54B (3 epochs) |
+| Hardware | RTX 4060 8 GB, ~30K tok/s (torch.compile) |
+| Precision | BF16 mixed precision, FP32 master weights |
+
+## Intended use
+
+- Educational reference: inspect a small, complete, honest pretraining run.
+- Qualitative experimentation: prompt it (it follows prompts *as text*; it
+  is a **base model** — no instruction tuning yet).
+- A base for further stages (SFT, DPO, domain fine-tuning).
+
+Known limitations: small scale ⇒ repetition in long generations, weak
+instruction following, limited world knowledge.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("jaweed123/TinyJLLM")
+tokenizer = AutoTokenizer.from_pretrained("jaweed123/TinyJLLM")
+
+prompt = "The future of AI is"
+inputs = tokenizer(prompt, return_tensors="pt")
+out = model.generate(**inputs, max_new_tokens=50, temperature=0.8, top_k=50, top_p=0.95)
+print(tokenizer.decode(out[0]))
+```
+
+### llama.cpp / GGUF
+
+The repo also ships GGUF files under `gguf/` (F16, Q8_0, Q4_K_M) — load
+directly with llama.cpp or `llama-cpp-python`.
+
+## Training details
+
+- Custom 32K byte-level BPE (trained on a 512 MB FineWeb sample).
+- Tokens stored once as uint16 shards (591 train + 6 validation).
+- AdamW (lr 3e-4, wd 0.1, decay/no-decay groups), warmup 1,000 + cosine
+  to 1e-5, effective batch 64 (32,768 tokens/step), gradient clipping 1.0.
+- Full run: ~35 h on an RTX 4060.
+
+## Files
+
+- `config.json` — Llama-compatible config (`LlamaForCausalLM`)
+- `model.safetensors` — FP32 weights
+- `tokenizer.json` / `tokenizer_config.json` — custom BPE
+- `generation_config.json` — decoding defaults
+- `gguf/` — llama.cpp formats
+
+## Acknowledgments
+
+FineWeb (HuggingFaceFW), Hugging Face `tokenizers` / `datasets`,
+PyTorch, llama.cpp. Built with the LearnLLM educational pipeline
+(`src/learnllm` at the project repository).