tags, metrics, inference, widget, pipeline_tag, license, language
| tags |
metrics |
inference |
widget |
pipeline_tag |
license |
language |
|
|
|
| parameters |
| max_new_tokens |
do_sample |
repetition_penalty |
no_repeat_ngram_size |
guidance_scale |
eta_cutoff |
| 64 |
true |
1.1 |
5 |
1.01 |
0.001 |
|
|
| text |
example_title |
| My name is El Microondas the Wise and |
El Microondas |
|
| text |
example_title |
| A meme is |
meme |
|
| text |
example_title |
| Barack Obama nominated Hilary Clinton as his secretary of state on Monday. He chose her because she had |
Coreference resolution |
|
| text |
example_title |
| On a shelf, there are five books: a gray book, a red book, a purple book, a blue book, and a black book |
Logic puzzles |
|
| text |
example_title |
| The two men running to become New York City's next mayor will face off in their first debate Wednesday night |
Reading comprehension |
|
|
text-generation |
apache-2.0 |
|
pythia-31m-KI_v1-2048-scratch
Initialized from random weights based on config of EleutherAI/pythia-31m, 3 epochs bf16
It achieves the following results on the evaluation set:
- Loss: 4.6160
- Accuracy: 0.2448
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 2
- eval_batch_size: 2
- seed: 80085
- gradient_accumulation_steps: 64
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-07
- lr_scheduler_type: inverse_sqrt
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 3.0
Training results
| Training Loss |
Epoch |
Step |
Validation Loss |
Accuracy |
| 6.3874 |
0.16 |
100 |
6.4212 |
0.1487 |
| 5.7088 |
0.32 |
200 |
5.7926 |
0.1725 |
| 5.4575 |
0.48 |
300 |
5.5160 |
0.1903 |
| 5.2451 |
0.64 |
400 |
5.3429 |
0.1995 |
| 5.0954 |
0.8 |
500 |
5.2109 |
0.2059 |
| 5.0358 |
0.96 |
600 |
5.1068 |
0.2123 |
| 4.94 |
1.12 |
700 |
5.0321 |
0.2157 |
| 4.8532 |
1.28 |
800 |
4.9605 |
0.2202 |
| 4.7602 |
1.44 |
900 |
4.9047 |
0.224 |
| 4.6965 |
1.6 |
1000 |
4.8526 |
0.2276 |
| 4.6855 |
1.76 |
1100 |
4.8139 |
0.2300 |
| 4.6573 |
1.91 |
1200 |
4.7739 |
0.2327 |
| 4.5968 |
2.07 |
1300 |
4.7451 |
0.2346 |
| 4.5688 |
2.23 |
1400 |
4.7152 |
0.2370 |
| 4.5205 |
2.39 |
1500 |
4.6842 |
0.2396 |
| 4.5369 |
2.55 |
1600 |
4.6598 |
0.2410 |
| 4.5106 |
2.71 |
1700 |
4.6352 |
0.2433 |
| 4.4375 |
2.87 |
1800 |
4.6160 |
0.2448 |
Detailed results can be found here
| Metric |
Value |
| Avg. |
25.21 |
| ARC (25-shot) |
23.12 |
| HellaSwag (10-shot) |
25.23 |
| MMLU (5-shot) |
23.12 |
| TruthfulQA (0-shot) |
51.67 |
| Winogrande (5-shot) |
51.78 |
| GSM8K (5-shot) |
0.0 |
| DROP (3-shot) |
1.52 |
