Rain-100M/README.md

license, datasets, language, pipeline_tag, library_name

license	datasets	language	pipeline_tag	library_name
apache-2.0	HuggingFaceFW/fineweb-edu	en	text-generation	transformers

🩵 Rain-100M — Model Card

Rain-100M is an experimental language model trained from scratch based on the Qwen3 architecture.

🧠 Training & Data

• Training corpus: HuggingFaceFW/fineweb-edu
• Total tokens: ~3B
• Language: English only
• Tokenizer: Newly trained 16k BPE (optimized for small/compact models)
• Max sequence length: 4096

Sample training metrics:

train/grad_norm: 0.6640625
train/learning_rate: 0.00000000002171853813
train/loss: 3.4459

🏗️ Architecture (Qwen3-style)

• Parameters: ~100M
• Layers: 12 Transformer layers
• Hidden size: 768
• Attention heads: 12
• MLP dimension: 2048
• Activation: SiLU
• Weight dtype: bfloat16
• RMSNorm eps: 1e-6
• RoPE θ: 10000
• Inference framework: transformers

⚠️ Limitations

• Trained only on English data; weak or no capabilities in other languages, not suitable as a general-purpose chat model or for safety-critical use cases.
• No system-level alignment or safety fine-tuning has been applied.

📄 License

When using this model locally, please also comply with the licenses of the fineweb-edu dataset and the transformers / Qwen3-related components.

---

中文版本

🩵 Rain-100M — Model Card

Rain-100M 是一个基于 Qwen3 架构 从零训练的实验语言模型。

🧠 训练与数据

• 训练语料：HuggingFaceFW/fineweb-edu
• Tokens 数量：约 3B
• 语言：仅英语
• Tokenizer：全新训练的 16k BPE（面向轻量模型优化）
• 最大序列长度：4096

训练参数：

train/grad_norm:0.6640625
train/learning_rate:0.00000000002171853813
train/loss:3.4459

🏗️ 模型结构（Qwen3 规格）

• 参数量：约 100M
• 层数：12 层 Transformer
• 隐藏维度：768
• 注意力头数：12
• 中间层维度：2048
• 激活函数：SiLU
• 权重类型：bfloat16
• RMSNorm eps：1e-6
• RoPE θ：10000
• 推理框架：transformers

⚠️ 限制

• 仅使用英文语料，小语言能力有限，不适合作为通用聊天或安全敏感任务；
• 无系统对齐与安全强化。

📄 License

请在本地使用时遵循 fineweb-edu 数据集与 transformers/Qwen3 相关许可证。