初始化项目,由ModelHub XC社区提供模型
Model: Raghav-Singhal/tulu3sft-normal-smollm-1p7b-500B-30n-2048sl-960gbsz Source: Original Platform
This commit is contained in:
41
README.md
Normal file
41
README.md
Normal file
@@ -0,0 +1,41 @@
|
||||
---
|
||||
language: en
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- smollm
|
||||
- llama
|
||||
- causal-lm
|
||||
- sft
|
||||
- tulu
|
||||
model_type: llama
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# tulu3sft-normal-smollm-1p7b-500B-30n-2048sl-960gbsz
|
||||
|
||||
This is a supervised fine-tuned (SFT) checkpoint for a SmolLM2-style 1.7B model,
|
||||
trained on the `allenai/tulu-3-sft-mixture` dataset. It is based on the 500B-token
|
||||
pretrained base checkpoint and exported in Hugging Face `LlamaForCausalLM` format.
|
||||
|
||||
## Details
|
||||
|
||||
- Base model: `normal-smollm-1p7b-500B-30n-2048sl-960gbsz`
|
||||
- SFT dataset: `allenai/tulu-3-sft-mixture`
|
||||
- Context length: 2048
|
||||
- Vocab size: 49152
|
||||
- Architecture: Llama (RMSNorm, SwiGLU, RoPE)
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "REPLACE_WITH_OWNER/tulu3sft-normal-smollm-1p7b-500B-30n-2048sl-960gbsz"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
This is an SFT model intended for chat-style use. For preference tuning, run DPO
|
||||
on top of this checkpoint.
|
||||
Reference in New Issue
Block a user