初始化项目，由ModelHub XC社区提供模型

Model: samueljayasingh/slimGPT
Source: Original Platform

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+images/eval_summary.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,200 @@
+---
+language:
+  - en
+license: mit
+tags:
+  - gpt2
+  - causal-lm
+  - text-generation
+  - slimgpt
+  - transformer
+  - from-scratch
+pipeline_tag: text-generation
+---
+
+# slimGPT — 124M Parameter GPT-Style Language Model
+
+**slimGPT** is a 124-million-parameter autoregressive language model built from scratch using a clean, modular PyTorch codebase. It follows the GPT-2 small architecture and was trained entirely on consumer-accessible hardware, demonstrating that capable language model training is achievable without large-scale infrastructure.
+
+---
+
+## Model Details
+
+| Property         | Value                    |
+|------------------|--------------------------|
+| **Architecture** | GPT-2 style (decoder-only Transformer) |
+| **Parameters**   | ~124 million             |
+| **Layers**       | 12                       |
+| **Attention Heads** | 12                    |
+| **Embedding Dim**| 768                      |
+| **Context Length**| 1024 tokens             |
+| **Vocabulary**   | GPT-2 BPE tokenizer (50,257 tokens) |
+| **Training Iters**| 5,000                   |
+| **Best Val Loss**| 3.3079                   |
+| **License**      | MIT                      |
+
+---
+
+## Training Infrastructure
+
+The model was trained on a single-GPU cloud instance with the following specifications:
+
+| Component        | Specification                        |
+|------------------|--------------------------------------|
+| **OS**           | Debian GNU/Linux 12 (Bookworm)       |
+| **CPU**          | Intel Xeon @ 2.20 GHz (4 vCPUs, 2 physical cores, 2 threads/core) |
+| **RAM**          | 16 GiB                               |
+| **Storage**      | 60 GB NVMe                           |
+| **GPU**          | NVIDIA L4                            |
+| **VRAM**         | 24 GB                                |
+| **NVIDIA Driver**| 550.54.15                            |
+
+Training was completed without any distributed setup, A single NVIDIA L4 GPU was sufficient for the full training run.
+
+---
+
+## Architecture Overview
+
+slimGPT follows the standard GPT-2 decoder-only Transformer architecture:
+
+- **Token + positional embeddings** — learned embeddings over the GPT-2 BPE vocabulary with 1024-token positional encodings
+- **12 Transformer blocks** — each with multi-head causal self-attention (12 heads) and a position-wise feed-forward network
+- **Pre-norm design** — LayerNorm applied before attention and MLP sub-layers
+- **Weight tying** — input embedding and output projection weights are tied
+- **Causal masking** — autoregressive, left-to-right generation
+
+---
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("samueljayasingh/slimGPT")
+model = AutoModelForCausalLM.from_pretrained("samueljayasingh/slimGPT")
+
+ids = tokenizer("The meaning of life is", return_tensors="pt").input_ids
+output = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.9)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+### Pipeline API
+
+```python
+from transformers import pipeline
+
+generator = pipeline("text-generation", model="samueljayasingh/slimGPT")
+result = generator("Once upon a time,", max_new_tokens=80, do_sample=True)
+print(result[0]["generated_text"])
+```
+
+### Serving with vLLM
+
+```bash
+pip install vllm
+vllm serve "samueljayasingh/slimGPT"
+
+curl -X POST "http://localhost:8000/v1/completions" \
+  -H "Content-Type: application/json" \
+  --data '{
+    "model": "samueljayasingh/slimGPT",
+    "prompt": "The future of AI is",
+    "max_tokens": 100,
+    "temperature": 0.7
+  }'
+```
+
+---
+
+## Intended Use
+
+This model is intended for:
+
+- **Research and experimentation** — studying language model behavior, attention patterns, and generation dynamics at the 124M scale
+- **Educational purposes** — understanding GPT-style architectures by working with a fully transparent, from-scratch implementation
+- **Prototyping** — lightweight text generation for downstream tasks, fine-tuning experiments, or benchmarking
+
+### Out-of-Scope Use
+
+- Production or safety-critical applications
+- Tasks requiring factual accuracy or up-to-date knowledge
+- Any use that relies on instruction-following or alignment — this is a base language model with no RLHF or instruction tuning
+
+---
+
+## Limitations
+
+- Trained for only **5,000 iterations** — the model is capable of coherent text continuation but has not converged to the quality of fully trained GPT-2
+- **No fine-tuning or alignment** — outputs are raw continuations and may be incoherent, biased, or off-topic
+- **English-only** — trained on English text; performance on other languages is not evaluated
+- **Context window of 1024 tokens** — longer documents are truncated
+
+---
+
+## Training Details
+
+The model was trained using a clean, readable PyTorch implementation with the following highlights:
+
+- **Optimizer**: AdamW with cosine learning rate decay and linear warmup
+- **Tokenizer**: GPT-2 BPE (via `tiktoken`)
+- **Data**: OpenWebText-style dataset sampled in token chunks of length 1024
+- **Mixed precision**: `torch.autocast` with `bfloat16` on the NVIDIA L4 GPU
+- **Gradient clipping**: Applied to stabilize training
+- **Checkpointing**: Best model saved based on validation loss
+
+---
+
+### Training Runtime
+
+- **Hardware**: NVIDIA L4 (24 GB VRAM), 4 vCPUs, 16 GB RAM
+- **Training iterations**: 5,000
+- **Total training time**: ~18 hours
+- **Average time per iteration**: ~13 seconds
+
+---
+
+## Evaluation
+
+| Metric         | Value   |
+|----------------|---------|
+| Best Val Loss  | 3.3079  |
+| Training Iters | 5,000   |
+
+Perplexity can be approximated as `exp(3.3079) ≈ 27.3`. For reference, a fully trained GPT-2 small achieves a perplexity of roughly 18–22 on OpenWebText; slimGPT sits in a reasonable range for its training budget.
+
+![Eval Summary](images/eval_summary.png)
+
+### Training loss
+![Loss Curve](images/loss_curve.png)
+
+### Perplexity comparison
+![Perplexity](images/perplexity_comparison.png)
+
+
+---
+
+## Citation
+
+If you use this model in your work, please credit:
+
+```
+@misc{slimgpt2026,
+  author       = {Samuel Jayasingh},
+  title        = {slimGPT: A 124M GPT-2-style language model trained from scratch},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/samueljayasingh/slimGPT}}
+}
+```
+---
+
+## Credits
+
+Inspired by Andrej Karpathy's "Let's reproduce GPT-2 (124M)" tutorial: https://www.youtube.com/watch?v=l8pRSuU81PU
+Special thanks to Andrej Karpathy for making modern LLM training and implementation accessible through open educational content.
+
+---
+
+## License
+
+This model is released under the [MIT License](https://opensource.org/licenses/MIT).

config.json

@@ -0,0 +1,36 @@
+{
+  "activation_function": "gelu_new",
+  "add_cross_attention": false,
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "dtype": "float32",
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_name": "slimGPT",
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "pad_token_id": null,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.11.0",
+  "use_cache": true,
+  "vocab_size": 50257
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.11.0",
+  "use_cache": true
+}

images/eval_summary.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ed00a25a43d45922360d9a245ec0776b4b7505e2595625279a40f7ebd4e89e2
+size 166558

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31f10d81bed2d292c3fc58b9c7d66fce386aa5bef8838fd41fc214a032feef00
+size 497774208

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 1024,
+  "pad_token": null,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}