初始化项目，由ModelHub XC社区提供模型

Model: MihaiPopa-1/CinnabarLM-1.5M-Base
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,87 @@
+---
+license: apache-2.0
+datasets:
+- HuggingFaceFW/fineweb
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- tiny-model
+- cinnabarlm
+- tiny-llm
+- tiny-lm
+- tinylm
+- tinyllm
+---
+
+# CinnabarLM 1.5M
+What happens if you take the CinnabarLM idea and push it further? You'll get this!
+
+CinnabarLM 1.5M is a tiny, 1.5M-parameter LLM trained for ~28 minutes on a T4 GPU (on Colab)! It's only 6 MB in size and now it's Llama-based!
+
+# Why?
+Because it's a good idea to make tiny LLMs. Some people already did with [MicroLM](https://huggingface.co/CromIA/MicroLM-1M), [Spark 4 5M](https://huggingface.co/LH-Tech-AI/Spark-5M-Base-v4) and [Tenete 8M](https://huggingface.co/Harley-ml/Tenete-8M), but not myself!
+
+# Model Configurations
+| Parameter | Value |
+|---|---|
+| Tokenizer | Llama 3's tokenizer (Tiktoken / BPE) |
+| Vocabulary Size | 4096 tokens |
+| Batch Size | 4 x 8 = 32 |
+| Context Window | Maybe 2048 tokens |
+| `hidden_size` | 128 |
+| `intermediate_size` | 256 |
+| `num_hidden_layers` | 4 |
+| `num_attention_heads` | 4 |
+| `max_position_embeddings` | 2048 |
+| `rms_norm_eps` | `1e-5` |
+| `initializer_range` | 0.02 |
+| `use_cache` | True
+| `tie_word_embeddings` | False
+| `rope_theta` | 10000.0
+
+# Training Configurations
+| Hyperparameter | Value |
+|---|---|
+| `output_dir` | "./cinnabarlm-v2" |
+| `max_steps` | 10000 |              
+| `per_device_train_batch_size` | 8 |
+| `gradient_accumulation_steps` | 4 |
+| `learning_rate` | 6e-4 |           
+| `weight_decay` | 0.01 |
+| `warmup_steps` | 500 |
+| `lr_scheduler_type` | "cosine" |
+| `logging_steps` | 100 |
+| `save_steps` | 2000 |
+| `fp16` | True |                    
+| `save_total_limit` | 2 |
+| `prediction_loss_only` | True |
+| `logging_first_step` | True |
+
+# Limitations
+* **Not Instruction-Tuned:** It's only a base model, so it only completes text.
+* **English-Only:** It's trained on English data (FineWeb), it's NOT multilingual.
+# Some other details
+* It's trained on 50 million tokens of [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) (CC-MAIN-2025-26 snapshot), and the knowledge cutoff is June 2025.
+* The name "CinnabarLM" that I picked was made by combining "Cinnabar" (the new block from the Chaos Cubed drop in Minecraft) + "LM" (Language Model)
+
+# Examples
+* "Take some time to show you how this LLM generates text:" gives:
+
+
+"Take some time to show you how this LLM generates text: love and fun.
+Your voice will be able to find out your style with a great fabric that will help your friend or family! I can't get more fun with us because we'll never give me the best idea why so I'm just happy to see all my favorite things at our time.
+Get a Curry Festival! Today, you've got the same gift of lucky cookies when it's been nice to take a look on. Please note: Google is not going to go to a lot of people looking for them. We're always happy to work"
+
+* "Mihai Popa, which is 15 years old, which the person likes" gives:
+
+
+"Mihai Popa, which is 15 years old, which the person likes a child is now being a bleeder who has been at the Time Summer.
+Presidents of Dragon's Day are the first time in the past six months to the school to the day of a hospital or office in San Francisville.
+The Best Times was also known as a family member. The school will be given for his life and family-friendly home, but it was a good place on a job to visit this site. They must have had a lot of moments from all around the world. However, there were some serious things about"
+
+* "Even though Striders also require a stick item" gives:
+
+
+"Even though Striders also require a stick item to create an outdoor space, but they are not just the perfect way to be in place. You can get a new look at the first time you’ll see how much the best thing we need is to make it easier for you!
+As always, our durable staff is incredibly easy and more than just one but for your newest team you will have to take some time in your day. The more you can give you everything in the best gift box? Let us know where you're going to share your style! From Cleaning Metaller to Western Distribution,"

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 32,
+  "hidden_act": "silu",
+  "hidden_size": 128,
+  "initializer_range": 0.02,
+  "intermediate_size": 256,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 4,
+  "num_key_value_heads": 4,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.0.0",
+  "use_cache": false,
+  "vocab_size": 4096
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.0.0",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77f063715b0ef70e871edf3b60dbac0e3eb00d9932a7b97b60c4ecacdf1c9c4a
+size 6824496

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": null,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "is_local": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3329e8bb54c7ab36542402c4bb2aac6494797bbe7975c998ad8fba185d2d4b6
+size 5137