初始化项目，由ModelHub XC社区提供模型
Model: SykoSLM/SykoLLM-v6.9 Source: Original Platform
This commit is contained in:
ModelHub XC
154
README.md
Normal file
154
README.md
Normal file
@@ -0,0 +1,154 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- causal-lm
|
||||
- text-generation
|
||||
- pretrained
|
||||
- tpu
|
||||
- sykollm
|
||||
base_model: SykoSLM/SykoLLM-V6.8
|
||||
---
|
||||
|
||||
# SykoLLM-V6.9
|
||||
|
||||
**The most powerful model in the SykoLLM family — trained on 8 billion tokens.**
|
||||
|
||||
SykoLLM-V6.9 is a 391M parameter causal language model, trained from scratch on a carefully curated mixture of high-quality English datasets. It is the latest and most capable model in the SykoLLM series, surpassing all previous versions in both token count and training quality.
|
||||
|
||||
---
|
||||
|
||||
## Model Details
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| **Architecture** | Causal Language Model (Phi-3 based) |
|
||||
| **Parameters** | 391,857,152 |
|
||||
| **Context Length** | 1,024 tokens |
|
||||
| **Vocabulary Size** | 50,000 |
|
||||
| **Hidden Size** | 1,024 |
|
||||
| **Intermediate Size** | 3,072 |
|
||||
| **Layers** | 24 |
|
||||
| **Attention Heads** | 8 (GQA: 2 KV heads) |
|
||||
| **Precision** | bfloat16 |
|
||||
| **Language** | English only |
|
||||
|
||||
---
|
||||
|
||||
## Training Details
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| **Total Tokens** | ~8 Billion |
|
||||
| **Training Steps** | 30,000 |
|
||||
| **Effective Batch Size** | 256 (16 × 2 × 8 cores) |
|
||||
| **Learning Rate** | 4e-4 (cosine decay) |
|
||||
| **Optimizer** | Adafactor |
|
||||
| **Hardware** | Google TPU v5e-8 |
|
||||
| **Precision** | bfloat16 (XLA native) |
|
||||
| **Weight Decay** | 0.05 |
|
||||
| **Warmup Steps** | 200 |
|
||||
|
||||
---
|
||||
|
||||
## Training Data
|
||||
|
||||
SykoLLM-V6.9 was trained on a curated mixture of 4 high-quality datasets, interleaved with carefully tuned sampling probabilities:
|
||||
|
||||
| Dataset | Sampling | Description |
|
||||
|---|---|---|
|
||||
| [openbmb/Ultra-FineWeb](https://huggingface.co/datasets/openbmb/Ultra-FineWeb) | 25% | High-quality web text, scored and filtered |
|
||||
| [openbmb/Ultra-FineWeb-L3](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3) | 40% | Multi-style synthetic English pretraining data |
|
||||
| [openbmb/UltraData-Math](https://huggingface.co/datasets/openbmb/UltraData-Math) | 20% | High-quality mathematical reasoning data |
|
||||
| [openbmb/UltraChat](https://huggingface.co/datasets/openbmb/UltraChat) | 15% | Multi-turn conversational data |
|
||||
|
||||
All datasets were filtered with a quality score threshold of ≥ 0.85 and additional heuristic filters to remove low-quality, noisy, or excessively long samples.
|
||||
|
||||
---
|
||||
|
||||
## Chat Format
|
||||
|
||||
SykoLLM-V6.9 uses the following chat template:
|
||||
|
||||
```
|
||||
<|user|>
|
||||
Your message here<|end|>
|
||||
<|assistant|>
|
||||
Model response here<|end|>
|
||||
```
|
||||
|
||||
For multi-turn conversations:
|
||||
|
||||
```
|
||||
<|user|>
|
||||
Hello, how are you?<|end|>
|
||||
<|assistant|>
|
||||
I'm doing great, thank you for asking!<|end|>
|
||||
<|user|>
|
||||
Can you help me with a math problem?<|end|>
|
||||
<|assistant|>
|
||||
Of course! What's the problem?<|end|>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
import torch
|
||||
|
||||
model_id = "SykoSLM/SykoLLM-V6.9"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
trust_remote_code=True,
|
||||
)
|
||||
|
||||
prompt = "<|user|>\nWhat is the capital of France?<|end|>\n<|assistant|>\n"
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=256,
|
||||
temperature=0.7,
|
||||
top_p=0.9,
|
||||
do_sample=True,
|
||||
)
|
||||
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SykoLLM Family
|
||||
|
||||
| Model | Tokens | Notes |
|
||||
|---|---|---|
|
||||
| SykoLLM-V6.9 | **~8B** | **Most powerful — current** |
|
||||
| SykoLLM-V6.8 | <8B | Previous version |
|
||||
| SykoLLM-V6.6 | <8B | Earlier version |
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
- **English only** — the model was trained exclusively on English data and does not support other languages.
|
||||
- **Context length** is limited to 1,024 tokens.
|
||||
- As a base pretrained model, it may produce outputs that are inaccurate, biased, or inappropriate. Use with appropriate safety measures.
|
||||
- Not instruction-tuned — for best results, use the chat format described above.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
|
||||
|
||||
---
|
||||
|
||||
*Trained with ❤️ by [SykoSLM](https://huggingface.co/SykoSLM)*
|
||||
Reference in New Issue
Block a user