80 lines
2.7 KiB
Markdown
80 lines
2.7 KiB
Markdown
|
---
|
||
|
|
license: llama2
|
||
|
|
---
|
||
|
|
|
||
|
|
# Toy LLaMA-39M
|
||
|
|
|
||
|
|
- This is a tiny LLaMA model pretrained on [Recag/Rp_C4_55](https://huggingface.co/datasets/Recag/Rp_C4_55), a small subset of C4 with `seq_len=512`.
|
||
|
|
- Model architecture
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"hidden_size": 512,
|
||
|
|
"intermediate_size": 2048,
|
||
|
|
"max_position_embeddings": 2048,
|
||
|
|
"num_attention_heads": 8,
|
||
|
|
"num_hidden_layers": 2,
|
||
|
|
"num_key_value_heads": 8
|
||
|
|
}
|
||
|
|
```
|
||
|
|
- Load model and tokenizer:
|
||
|
|
```python
|
||
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("Cheng98/llama-39m")
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("Cheng98/llama-39m")
|
||
|
|
```
|
||
|
|
- Training script: [huggingface/transformers/examples/pytorch/language-modeling/run_clm.py](https://github.com/huggingface/transformers/blob/e9476832942a19cf99354776ef112babc83c139a/examples/pytorch/language-modeling/run_clm.py)
|
||
|
|
```python
|
||
|
|
# "train" split is created from the last 95% samples of original "train" subset
|
||
|
|
raw_datasets["validation"] = load_dataset("Recag/Rp_C4_55", split="train[5%:]")
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
- Evaluation (`seq_len=512`):
|
||
|
|
|
||
|
|
| Dataset | Eval loss | Perplexity | Accuracy | block_size |
|
||
|
|
|----------------|-----------|------------|----------|------------|
|
||
|
|
| Recag/Rp_C4_55 | 3.63 | 37.78 | 0.3561 | 512 |
|
||
|
|
| Wikitext2 | 4.58 | 97.48 | 0.2719 | 512 |
|
||
|
|
|
||
|
|
- Evaluation command (Wikitext2):
|
||
|
|
```bash
|
||
|
|
# Evaluation command
|
||
|
|
python run_clm.py --model_name_or_path Cheng98/llama-39m \
|
||
|
|
--dataset_name wikitext \
|
||
|
|
--dataset_config_name wikitext-2-raw-v1 \
|
||
|
|
--block_size 512 \
|
||
|
|
--do_eval \
|
||
|
|
--output_dir ./results
|
||
|
|
```
|
||
|
|
|
||
|
|
- Evaluation on Recag/Rp_C4_55 (`seq_len=512`):
|
||
|
|
```python
|
||
|
|
# "validation" split is created from the first 5% samples of original "train" subset
|
||
|
|
raw_datasets["validation"] = load_dataset("Recag/Rp_C4_55", split="train[:5%]")
|
||
|
|
```
|
||
|
|
Results
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"eval_accuracy": 0.3561766818954313,
|
||
|
|
"eval_loss": 3.6318140029907227,
|
||
|
|
"eval_runtime": 190.8411,
|
||
|
|
"eval_samples": 19413,
|
||
|
|
"eval_samples_per_second": 101.723,
|
||
|
|
"eval_steps_per_second": 1.593,
|
||
|
|
"perplexity": 37.7812898658763
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
- Evaluation on Wikitext2 (`seq_len=512`):
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"eval_accuracy": 0.2718795201225219,
|
||
|
|
"eval_loss": 4.579628944396973,
|
||
|
|
"eval_runtime": 3.939,
|
||
|
|
"eval_samples": 575,
|
||
|
|
"eval_samples_per_second": 145.976,
|
||
|
|
"eval_steps_per_second": 0.762,
|
||
|
|
"perplexity": 97.47821765687856
|
||
|
|
}
|
||
|
|
```
|