babylm-anon/rta
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: babylm-anon/rta
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-05 15:06:17 +08:00
commit d1323322ae
13 changed files with 82824 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

79
README.md Normal file
View File

@@ -0,0 +1,79 @@
---
library_name: transformers
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: babylm-base5M-gpt2
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# babylm-base5M-gpt2 (Fork with chck_100M Checkpoint)
This repository is a fork of [alexandertam/babylm-base5m-gpt2](https://huggingface.co/alexandertam/babylm-base5m-gpt2), created for the [BabyLM Challenge 2025](https://babylm.github.io/) submission.
It extends the original model by including an additional checkpoint (`chck_100M`), adhering to the challenge's guidelines.
## Model Description
This model is a pre-trained version of the GPT-2 architecture.
It achieves the following results on the evaluation set:
- Loss: 3.0628
- Accuracy: 0.4521
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 190
- training_steps: 19000
- mixed_precision_training: Native AMP
### Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:------:|:-----:|:---------------:|:--------:|
| 5.579 | 0.1024 | 200 | 4.7677 | 0.3189 |
| 4.7716 | 0.2048 | 400 | 4.3385 | 0.3544 |
| 4.5162 | 0.3072 | 600 | 4.1772 | 0.3593 |
| 4.4056 | 0.4096 | 800 | 4.0754 | 0.3693 |
| 4.3138 | 0.5120 | 1000 | 4.0143 | 0.3626 |
| 4.2148 | 0.6144 | 1200 | 3.9601 | 0.3554 |
| 4.1925 | 0.7168 | 1400 | 3.9019 | 0.3723 |
| 4.0293 | 0.8193 | 1600 | 3.8579 | 0.3749 |
| 3.9407 | 0.9217 | 1800 | 3.8101 | 0.3782 |
| 3.8371 | 1.0241 | 2000 | 3.7870 | 0.3721 |
| 3.0659 | 2.0481 | 4000 | 3.4672 | 0.4085 |
| 2.6866 | 3.0722 | 6000 | 3.2850 | 0.4316 |
| 2.5063 | 4.0963 | 8000 | 3.1963 | 0.4372 |
| 2.4139 | 5.1203 | 10000 | 3.1406 | 0.4442 |
| 2.3246 | 6.1444 | 12000 | 3.1152 | 0.4484 |
| 2.3111 | 7.1685 | 14000 | 3.0879 | 0.4489 |
| 2.2761 | 8.1925 | 16000 | 3.0668 | 0.4542 |
| 2.2231 | 9.2166 | 18000 | 3.0695 | 0.4517 |
### Framework versions
- Transformers 4.50.3
- Pytorch 2.7.1+cu126
- Datasets 3.6.0
- Tokenizers 0.21.4

16
all_results.json Normal file
View File

@@ -0,0 +1,16 @@
{
"epoch": 9.728622631848438,
"eval_accuracy": 0.45214562944577746,
"eval_loss": 3.0628035068511963,
"eval_runtime": 95.1631,
"eval_samples": 33544,
"eval_samples_per_second": 352.49,
"eval_steps_per_second": 22.036,
"perplexity": 21.38743300660324,
"total_flos": 7.9413964701696e+16,
"train_loss": 2.775124670731394,
"train_runtime": 3271.669,
"train_samples": 31240,
"train_samples_per_second": 92.919,
"train_steps_per_second": 5.807
}

31
config.json Normal file
View File

@@ -0,0 +1,31 @@
{
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 1,
"embd_pdrop": 0.1,
"eos_token_id": 2,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_embd": 768,
"n_head": 12,
"n_inner": null,
"n_layer": 12,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"torch_dtype": "float32",
"transformers_version": "4.50.3",
"use_cache": true,
"vocab_size": 16384
}

10
eval_results.json Normal file
View File

@@ -0,0 +1,10 @@
{
"epoch": 9.728622631848438,
"eval_accuracy": 0.45214562944577746,
"eval_loss": 3.0628035068511963,
"eval_runtime": 95.1631,
"eval_samples": 33544,
"eval_samples_per_second": 352.49,
"eval_steps_per_second": 22.036,
"perplexity": 21.38743300660324
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"transformers_version": "4.50.3"
}

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f6ef61138a105c85ceffe6b6f43295e1227d2edb594f4b906933ab8302486203
size 393716352

30
special_tokens_map.json Normal file
View File

@@ -0,0 +1,30 @@
{
"bos_token": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<pad>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

81020
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

47
tokenizer_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"added_tokens_decoder": {
"0": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"3": {
"content": "<pad>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"bos_token": "<s>",
"clean_up_tokenization_spaces": false,
"cls_token": null,
"eos_token": "</s>",
"extra_special_tokens": {},
"mask_token": null,
"model_max_length": 1024,
"pad_token": "<pad>",
"sep_token": null,
"tokenizer_class": "PreTrainedTokenizer",
"unk_token": "<unk>"
}

9
train_results.json Normal file
View File

@@ -0,0 +1,9 @@
{
"epoch": 9.728622631848438,
"total_flos": 7.9413964701696e+16,
"train_loss": 2.775124670731394,
"train_runtime": 3271.669,
"train_samples": 31240,
"train_samples_per_second": 92.919,
"train_steps_per_second": 5.807
}

1535
trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d250537b2585c6d06b8b0dcfae2c77def32b4b994aeb860086cca09095186307
size 5841