初始化项目，由ModelHub XC社区提供模型
Model: antoniogr7/zagreus-0.4B-ita-elicit Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
98
README.md
Normal file
98
README.md
Normal file
@@ -0,0 +1,98 @@
|
||||
---
|
||||
language:
|
||||
- it
|
||||
base_model:
|
||||
- mii-llm/zagreus-0.4B-ita
|
||||
datasets:
|
||||
- antoniogr7/italic-lexical-elicitation
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
tags:
|
||||
- italian
|
||||
- italic
|
||||
- elicitation
|
||||
- multiple-choice
|
||||
---
|
||||
|
||||
# zagreus-0.4B-ita-elicit
|
||||
|
||||
An **elicitation** fine-tune of [`mii-llm/zagreus-0.4B-ita`](https://huggingface.co/mii-llm/zagreus-0.4B-ita)
|
||||
for the [ITALIC](https://github.com/Crisp-Unimib/ITALIC) benchmark. It does **not** inject new
|
||||
knowledge (the model is only 0.4B); it surfaces the lexical/semantic knowledge the base model
|
||||
*already* holds and routes it onto the answer-letter channel, while removing the base model's
|
||||
heavy answer-letter bias. **No ITALIC data is used for training** — the benchmark is only read
|
||||
to guard against leakage.
|
||||
|
||||
Net result: **ITALIC 0.280 → 0.3126** (fast, zero-shot, greedy, n=10,000).
|
||||
|
||||
## Results
|
||||
|
||||
ITALIC, **fast / zero-shot**, greedy, n=10,000 (0 unparsed). Base = `mii-llm/zagreus-0.4B-ita`.
|
||||
|
||||
| | base | **this model** | Δ |
|
||||
|---|---|---|---|
|
||||
| **overall** | 0.280 | **0.3126** | **+3.2** |
|
||||
| language capability (macro) | 0.2705 | 0.3133 | +4.3 |
|
||||
| culture and commonsense (macro) | 0.2868 | 0.3121 | +2.5 |
|
||||
| `lexicon` | 0.2584 | **0.3841** | **+12.6** |
|
||||
| `synonyms_and_antonyms` | 0.2915 | 0.3275 | +3.6 |
|
||||
|
||||
The intended target (`lexicon`) moves the most. The culture categories rise modestly too — not
|
||||
from injected facts but from repairing the shared letter-emission head; per-fact culture remains
|
||||
a wall for a 0.4B, so those gains are small.
|
||||
|
||||
## Method (brief)
|
||||
|
||||
Full fine-tune with an elicitation objective, per rendered option ordering:
|
||||
|
||||
1. **`CE(gold)`** — commit probability mass to the correct answer letter (peaking anchor, the
|
||||
primary driver).
|
||||
2. **`λ·KL(q ‖ p)`** — distill a **PMI-cloze teacher** computed over the option *text* (`q`) onto
|
||||
the answer-letter logits (`p`), down-weighted (`λ=0.3`) to avoid flattening.
|
||||
3. **format term** — keep probability mass on the option-letter tokens.
|
||||
|
||||
Options are re-permuted every epoch so the model must bind meaning to the letter it lands on, not
|
||||
to a position. Training data is a synthetic Italian lexical/semantic MCQ pool built from **Italian
|
||||
Wiktionary** (kaikki.org): [`antoniogr7/italic-lexical-elicitation`](https://huggingface.co/datasets/antoniogr7/italic-lexical-elicitation).
|
||||
Trained and evaluated in **bfloat16**.
|
||||
|
||||
## Usage
|
||||
|
||||
**Serve/evaluate zero-shot.** The model was trained and measured 0-shot; few-shot demos shift the
|
||||
emission distribution it was tuned for and regress the score. The ITALIC chat template is baked
|
||||
into the tokenizer.
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
name = "antoniogr7/zagreus-0.4B-ita-elicit"
|
||||
tok = AutoTokenizer.from_pretrained(name)
|
||||
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16).cuda().eval()
|
||||
|
||||
question = "Qual è il sinonimo di «celere»?"
|
||||
options = {"A": "lento", "B": "rapido", "C": "grande", "D": "scuro"}
|
||||
body = question + "\n\n" + "\n".join(f"{k}) {v}" for k, v in options.items()) + "\n\nRisposta:\n"
|
||||
msgs = [{"role": "system", "content": "Sei un assistente utile."},
|
||||
{"role": "user", "content": body}]
|
||||
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").cuda()
|
||||
out = model.generate(ids, max_new_tokens=2, do_sample=False)
|
||||
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).strip()) # -> "B"
|
||||
```
|
||||
|
||||
For faithful benchmark scoring use the official ITALIC harness (vLLM + `run_eval.py`, `fast: true`,
|
||||
`few_shot_file: null`).
|
||||
|
||||
## Limitations
|
||||
|
||||
- **0.4B model.** Only the lexical/semantic slice of ITALIC (~19.5%) is addressable by elicitation;
|
||||
culture is per-fact knowledge (a wall) and grammar is a capacity wall. The realistic ceiling of
|
||||
this approach is ~0.31–0.33 overall.
|
||||
- **Zero-shot only** (see Usage).
|
||||
- Italian only.
|
||||
|
||||
## License & attribution
|
||||
|
||||
The model inherits the license of the base model `mii-llm/zagreus-0.4B-ita`. The training dataset
|
||||
is derived from Italian Wiktionary via kaikki.org and released under **CC BY-SA 4.0** (attribution:
|
||||
Wiktionary contributors; extraction by wiktextract / kaikki.org).
|
||||
6
chat_template.jinja
Normal file
6
chat_template.jinja
Normal file
@@ -0,0 +1,6 @@
|
||||
{{ bos_token }}{% for m in messages %}{% if m['role'] == 'system' %}{{ m['content'] + '
|
||||
|
||||
' }}{% elif m['role'] == 'user' %}{{ m['content'] + '
|
||||
' }}{% elif m['role'] == 'assistant' %}{{ m['content'] + '
|
||||
|
||||
' }}{% endif %}{% endfor %}
|
||||
32
config.json
Normal file
32
config.json
Normal file
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 128000,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 128001,
|
||||
"head_dim": 64,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 960,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 2560,
|
||||
"max_position_embeddings": 4096,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 15,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 5,
|
||||
"pad_token_id": null,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_parameters": {
|
||||
"rope_theta": 10000.0,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "5.14.1",
|
||||
"use_cache": true,
|
||||
"vocab_size": 128256
|
||||
}
|
||||
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 128000,
|
||||
"eos_token_id": 128001,
|
||||
"transformers_version": "5.14.1"
|
||||
}
|
||||
304
metrics.jsonl
Normal file
304
metrics.jsonl
Normal file
@@ -0,0 +1,304 @@
|
||||
{"step": 0, "eval_acc": 0.27289719626168224, "eval_dist": {"A": 0.987, "B": 0.011, "C": 0.002}}
|
||||
{"step": 1, "loss": 2.167983055114746, "ema": 2.167983055114746, "kl": 1.4237451618537307, "ce": 1.676305491477251, "fmt": 0.6455380916595459, "cons": 0.0, "w": 1.0, "lr": 1.3333333333333334e-06}
|
||||
{"step": 10, "loss": 2.251310110092163, "ema": 2.2106624733852507, "kl": 1.4418754987418652, "ce": 1.7471868693828583, "fmt": 0.715605616569519, "cons": 0.0, "w": 1.0, "lr": 1.3333333333333333e-05}
|
||||
{"step": 20, "loss": 2.068622350692749, "ema": 2.2809185194476362, "kl": 1.2996590584516525, "ce": 1.621217731386423, "fmt": 0.575068473815918, "cons": 0.0, "w": 1.0, "lr": 2.6666666666666667e-05}
|
||||
{"step": 30, "loss": 1.7780365943908691, "ema": 2.107202739770574, "kl": 0.9026094879955053, "ce": 1.4629991985857487, "fmt": 0.44254589080810547, "cons": 0.0, "w": 1.0, "lr": 4e-05}
|
||||
{"step": 40, "loss": 1.7467033863067627, "ema": 1.860384845921967, "kl": 0.9607046134769917, "ce": 1.4298075139522552, "fmt": 0.2868441343307495, "cons": 0.0, "w": 1.0, "lr": 5.333333333333333e-05}
|
||||
{"step": 50, "loss": 1.7276952266693115, "ema": 1.7744285664633763, "kl": 1.053351016715169, "ce": 1.3967346921563148, "fmt": 0.14954936504364014, "cons": 0.0, "w": 1.0, "lr": 6.666666666666667e-05}
|
||||
{"step": 50, "eval_acc": 0.23925233644859814, "eval_dist": {"A": 0.047, "B": 0.23, "C": 0.234, "D": 0.49}}
|
||||
{"step": 60, "loss": 1.7059699296951294, "ema": 1.739356872693977, "kl": 0.9900747686624527, "ce": 1.3979401215910912, "fmt": 0.11007654666900635, "cons": 0.0, "w": 1.0, "lr": 8e-05}
|
||||
{"step": 70, "loss": 1.761826515197754, "ema": 1.7006063497536803, "kl": 1.000137597322464, "ce": 1.4546435549855232, "fmt": 0.07141733169555664, "cons": 0.0, "w": 1.0, "lr": 9.333333333333334e-05}
|
||||
{"step": 80, "loss": 1.6662564277648926, "ema": 1.7050200019755923, "kl": 0.8166952161118388, "ce": 1.414902850985527, "fmt": 0.06344902515411377, "cons": 0.0, "w": 1.0, "lr": 9.999897318630175e-05}
|
||||
{"step": 90, "loss": 1.6868399381637573, "ema": 1.6766598936858481, "kl": 0.9857213199138641, "ce": 1.3857138380408287, "fmt": 0.05409538745880127, "cons": 0.0, "w": 1.0, "lr": 9.999075892975724e-05}
|
||||
{"step": 100, "loss": 1.72280752658844, "ema": 1.698120279189482, "kl": 0.9759096149355173, "ce": 1.4259654507040977, "fmt": 0.04069399833679199, "cons": 0.0, "w": 1.0, "lr": 9.99743317661761e-05}
|
||||
{"step": 100, "eval_acc": 0.30093457943925234, "eval_dist": {"A": 0.811, "B": 0.006, "C": 0.069, "D": 0.114}}
|
||||
{"step": 110, "loss": 1.6505248546600342, "ema": 1.6696198120112289, "kl": 0.812773397192359, "ce": 1.4033105447888374, "fmt": 0.03382217884063721, "cons": 0.0, "w": 1.0, "lr": 9.994969439435249e-05}
|
||||
{"step": 120, "loss": 1.6577720642089844, "ema": 1.6713209918547107, "kl": 0.900005005300045, "ce": 1.3850033730268478, "fmt": 0.027672767639160156, "cons": 0.0, "w": 1.0, "lr": 9.991685086192341e-05}
|
||||
{"step": 130, "loss": 1.7046726942062378, "ema": 1.6682598172129683, "kl": 1.0312129324302077, "ce": 1.392556220293045, "fmt": 0.027527809143066406, "cons": 0.0, "w": 1.0, "lr": 9.987580656470367e-05}
|
||||
{"step": 140, "loss": 1.5828330516815186, "ema": 1.658354893808111, "kl": 0.7972859386354685, "ce": 1.3409376740455627, "fmt": 0.027096033096313477, "cons": 0.0, "w": 1.0, "lr": 9.98265682457995e-05}
|
||||
{"step": 150, "loss": 1.6544071435928345, "ema": 1.669921491624521, "kl": 0.8782475404441357, "ce": 1.3883126527071, "fmt": 0.026202917098999023, "cons": 0.0, "w": 1.0, "lr": 9.976914399450068e-05}
|
||||
{"step": 150, "eval_acc": 0.2897196261682243, "eval_dist": {"A": 0.591, "B": 0.112, "C": 0.105, "D": 0.193}}
|
||||
{"step": 160, "loss": 1.679589867591858, "ema": 1.6752550330450533, "kl": 0.9572035847231746, "ce": 1.3899546638131142, "fmt": 0.02473890781402588, "cons": 0.0, "w": 1.0, "lr": 9.97035432449516e-05}
|
||||
{"step": 170, "loss": 1.6180232763290405, "ema": 1.6633106223394174, "kl": 0.8204898843541741, "ce": 1.3697152212262154, "fmt": 0.02161097526550293, "cons": 0.0, "w": 1.0, "lr": 9.962977677460132e-05}
|
||||
{"step": 180, "loss": 1.7240402698516846, "ema": 1.6655457856338134, "kl": 1.0745898727327585, "ce": 1.3994789198040962, "fmt": 0.021843791007995605, "cons": 0.0, "w": 1.0, "lr": 9.954785670243297e-05}
|
||||
{"step": 190, "loss": 1.5884727239608765, "ema": 1.647398261929764, "kl": 0.8839151617139578, "ce": 1.3210551142692566, "fmt": 0.022429704666137695, "cons": 0.0, "w": 1.0, "lr": 9.945779648697267e-05}
|
||||
{"step": 200, "loss": 1.585273265838623, "ema": 1.6504643945043809, "kl": 0.8029458746314049, "ce": 1.3421472758054733, "fmt": 0.0224226713180542, "cons": 0.0, "w": 1.0, "lr": 9.93596109240786e-05}
|
||||
{"step": 200, "eval_acc": 0.2841121495327103, "eval_dist": {"A": 0.609, "B": 0.335, "D": 0.056}}
|
||||
{"step": 210, "loss": 1.7153626680374146, "ema": 1.670749775383617, "kl": 1.0312550459057093, "ce": 1.4041588306427002, "fmt": 0.018273472785949707, "cons": 0.0, "w": 1.0, "lr": 9.925331614451006e-05}
|
||||
{"step": 220, "loss": 1.7062081098556519, "ema": 1.669151705902786, "kl": 1.0577427074313164, "ce": 1.387254573404789, "fmt": 0.016306519508361816, "cons": 0.0, "w": 1.0, "lr": 9.913892961127744e-05}
|
||||
{"step": 230, "loss": 1.659885287284851, "ema": 1.669596948109604, "kl": 0.9082356281578541, "ce": 1.3858012706041336, "fmt": 0.016134142875671387, "cons": 0.0, "w": 1.0, "lr": 9.901647011677326e-05}
|
||||
{"step": 240, "loss": 1.6633973121643066, "ema": 1.6693820040458105, "kl": 0.8681450188159943, "ce": 1.4014161601662636, "fmt": 0.015373587608337402, "cons": 0.0, "w": 1.0, "lr": 9.888595777968479e-05}
|
||||
{"step": 250, "loss": 1.6765310764312744, "ema": 1.6591882664384094, "kl": 0.9062211904674768, "ce": 1.4031883105635643, "fmt": 0.014762639999389648, "cons": 0.0, "w": 1.0, "lr": 9.87474140416887e-05}
|
||||
{"step": 250, "eval_acc": 0.30093457943925234, "eval_dist": {"A": 0.37, "B": 0.331, "C": 0.271, "D": 0.028}}
|
||||
{"step": 260, "loss": 1.6081204414367676, "ema": 1.6646864512302164, "kl": 0.8488215692341328, "ce": 1.3520947098731995, "fmt": 0.013792872428894043, "cons": 0.0, "w": 1.0, "lr": 9.860086166392856e-05}
|
||||
{"step": 270, "loss": 1.609235167503357, "ema": 1.6577854913587657, "kl": 0.8923982325941324, "ce": 1.3403328955173492, "fmt": 0.011828780174255371, "cons": 0.0, "w": 1.0, "lr": 9.844632472327542e-05}
|
||||
{"step": 280, "loss": 1.6263422966003418, "ema": 1.652199054122782, "kl": 0.93196433596313, "ce": 1.3456725478172302, "fmt": 0.010801196098327637, "cons": 0.0, "w": 1.0, "lr": 9.828382860837215e-05}
|
||||
{"step": 290, "loss": 1.723608136177063, "ema": 1.6649705973313405, "kl": 1.1163921542465687, "ce": 1.3875135853886604, "fmt": 0.01177215576171875, "cons": 0.0, "w": 1.0, "lr": 9.811340001546251e-05}
|
||||
{"step": 300, "loss": 1.6239867210388184, "ema": 1.6535707896651044, "kl": 0.8131292732432485, "ce": 1.378871627151966, "fmt": 0.011764168739318848, "cons": 0.0, "w": 1.0, "lr": 9.793506694400523e-05}
|
||||
{"step": 300, "eval_acc": 0.29906542056074764, "eval_dist": {"A": 0.465, "B": 0.079, "C": 0.006, "D": 0.45}}
|
||||
{"step": 310, "loss": 1.616821527481079, "ema": 1.6436329457842365, "kl": 0.8220781218260527, "ce": 1.3688636347651482, "fmt": 0.013344407081604004, "cons": 0.0, "w": 1.0, "lr": 9.774885869207391e-05}
|
||||
{"step": 320, "loss": 1.662564754486084, "ema": 1.674324316758799, "kl": 0.8435377469286323, "ce": 1.408154234290123, "fmt": 0.013492107391357422, "cons": 0.0, "w": 1.0, "lr": 9.755480585154387e-05}
|
||||
{"step": 330, "loss": 1.6428256034851074, "ema": 1.6500681439266605, "kl": 0.9049816988408566, "ce": 1.3702363222837448, "fmt": 0.010949969291687012, "cons": 0.0, "w": 1.0, "lr": 9.735294030306611e-05}
|
||||
{"step": 340, "loss": 1.6991541385650635, "ema": 1.6743607892574144, "kl": 0.9861266203224659, "ce": 1.402275174856186, "fmt": 0.010409712791442871, "cons": 0.0, "w": 1.0, "lr": 9.714329521082977e-05}
|
||||
{"step": 350, "loss": 1.6739956140518188, "ema": 1.653873942038281, "kl": 0.8575408384203911, "ce": 1.4156309589743614, "fmt": 0.011023879051208496, "cons": 0.0, "w": 1.0, "lr": 9.692590501711359e-05}
|
||||
{"step": 350, "eval_acc": 0.2897196261682243, "eval_dist": {"A": 0.479, "B": 0.312, "C": 0.093, "D": 0.116}}
|
||||
{"step": 360, "loss": 1.6095995903015137, "ema": 1.6342276305281869, "kl": 0.9943408016115427, "ce": 1.3103295862674713, "fmt": 0.009677648544311523, "cons": 0.0, "w": 1.0, "lr": 9.67008054366274e-05}
|
||||
{"step": 370, "loss": 1.684261441230774, "ema": 1.6555154593347818, "kl": 0.9822064712643623, "ce": 1.388648934662342, "fmt": 0.009504556655883789, "cons": 0.0, "w": 1.0, "lr": 9.646803345064477e-05}
|
||||
{"step": 380, "loss": 1.6614688634872437, "ema": 1.6436089747797682, "kl": 0.9792606998234987, "ce": 1.3669275045394897, "fmt": 0.007628679275512695, "cons": 0.0, "w": 1.0, "lr": 9.622762730092716e-05}
|
||||
{"step": 390, "loss": 1.6163848638534546, "ema": 1.6374982116028562, "kl": 0.9154526013880968, "ce": 1.3409615233540535, "fmt": 0.007875442504882812, "cons": 0.0, "w": 1.0, "lr": 9.59796264834415e-05}
|
||||
{"step": 400, "loss": 1.615796685218811, "ema": 1.6338473539881218, "kl": 0.7971697305329144, "ce": 1.3758151307702065, "fmt": 0.008307218551635742, "cons": 0.0, "w": 1.0, "lr": 9.572407174187124e-05}
|
||||
{"step": 400, "eval_acc": 0.28598130841121494, "eval_dist": {"A": 0.286, "B": 0.536, "C": 0.155, "D": 0.022}}
|
||||
{"step": 410, "loss": 1.7082853317260742, "ema": 1.6380665523990754, "kl": 0.9597688755020499, "ce": 1.4193147644400597, "fmt": 0.010398507118225098, "cons": 0.0, "w": 1.0, "lr": 9.546100506092278e-05}
|
||||
{"step": 420, "loss": 1.7118034362792969, "ema": 1.6262135071858954, "kl": 1.0337830381467938, "ce": 1.4006408154964447, "fmt": 0.010278463363647461, "cons": 0.0, "w": 1.0, "lr": 9.519046965942776e-05}
|
||||
{"step": 430, "loss": 1.553517460823059, "ema": 1.6234308267228803, "kl": 0.7979789823293686, "ce": 1.312947817146778, "fmt": 0.011759161949157715, "cons": 0.0, "w": 1.0, "lr": 9.49125099832428e-05}
|
||||
{"step": 440, "loss": 1.530832290649414, "ema": 1.6168237819303908, "kl": 0.6955381603911519, "ce": 1.321100927889347, "fmt": 0.010699987411499023, "cons": 0.0, "w": 1.0, "lr": 9.462717169794749e-05}
|
||||
{"step": 450, "loss": 1.6425541639328003, "ema": 1.6149971831400682, "kl": 0.7962004002183676, "ce": 1.402600735425949, "fmt": 0.010931611061096191, "cons": 0.0, "w": 1.0, "lr": 9.433450168134204e-05}
|
||||
{"step": 450, "eval_acc": 0.34579439252336447, "eval_dist": {"A": 0.286, "B": 0.022, "C": 0.08, "D": 0.611}}
|
||||
{"step": 460, "loss": 1.63685941696167, "ema": 1.6091508803799393, "kl": 0.8922501457855105, "ce": 1.368178091943264, "fmt": 0.0100632905960083, "cons": 0.0, "w": 1.0, "lr": 9.403454801574589e-05}
|
||||
{"step": 470, "loss": 1.5748555660247803, "ema": 1.5927679915228659, "kl": 0.7466691629961133, "ce": 1.3500025793910027, "fmt": 0.008520960807800293, "cons": 0.0, "w": 1.0, "lr": 9.372735998009827e-05}
|
||||
{"step": 480, "loss": 1.7567726373672485, "ema": 1.6107811981064106, "kl": 1.046563514508307, "ce": 1.4418460987508297, "fmt": 0.00957345962524414, "cons": 0.0, "w": 1.0, "lr": 9.341298804186223e-05}
|
||||
{"step": 490, "loss": 1.5248265266418457, "ema": 1.5882808976957588, "kl": 0.7233081236481667, "ce": 1.306907705962658, "fmt": 0.009262919425964355, "cons": 0.0, "w": 1.0, "lr": 9.309148384873343e-05}
|
||||
{"step": 500, "loss": 1.5714470148086548, "ema": 1.5642790803795528, "kl": 0.982336463406682, "ce": 1.2759156003594398, "fmt": 0.00830221176147461, "cons": 0.0, "w": 1.0, "lr": 9.276290022015503e-05}
|
||||
{"step": 500, "eval_acc": 0.37009345794392523, "eval_dist": {"A": 0.069, "B": 0.338, "C": 0.523, "D": 0.069}}
|
||||
{"step": 510, "loss": 1.529497742652893, "ema": 1.5397515952059484, "kl": 0.9450295362621546, "ce": 1.2451013773679733, "fmt": 0.008875012397766113, "cons": 0.0, "w": 1.0, "lr": 9.24272911386401e-05}
|
||||
{"step": 520, "loss": 1.5119807720184326, "ema": 1.542756294274877, "kl": 0.7537515098229051, "ce": 1.284750372171402, "fmt": 0.011051654815673828, "cons": 0.0, "w": 1.0, "lr": 9.208471174090278e-05}
|
||||
{"step": 530, "loss": 1.37258780002594, "ema": 1.5396164076205108, "kl": 0.7841722946614027, "ce": 1.1362624764442444, "fmt": 0.010737419128417969, "cons": 0.0, "w": 1.0, "lr": 9.173521830880012e-05}
|
||||
{"step": 540, "loss": 1.5083152055740356, "ema": 1.5572631980694795, "kl": 0.849436828866601, "ce": 1.2526273876428604, "fmt": 0.008567571640014648, "cons": 0.0, "w": 1.0, "lr": 9.137886826008548e-05}
|
||||
{"step": 550, "loss": 1.482045292854309, "ema": 1.5322482649583002, "kl": 0.8789055906236172, "ce": 1.217356089502573, "fmt": 0.010174989700317383, "cons": 0.0, "w": 1.0, "lr": 9.101572013897555e-05}
|
||||
{"step": 550, "eval_acc": 0.4336448598130841, "eval_dist": {"A": 0.357, "B": 0.301, "C": 0.247, "D": 0.095}}
|
||||
{"step": 560, "loss": 1.2845016717910767, "ema": 1.5044184516769281, "kl": 0.8018833771348, "ce": 1.042845094576478, "fmt": 0.010916948318481445, "cons": 0.0, "w": 1.0, "lr": 9.064583360653216e-05}
|
||||
{"step": 570, "loss": 1.5194886922836304, "ema": 1.4963813565321178, "kl": 0.7835218599066138, "ce": 1.2833603713661432, "fmt": 0.010715246200561523, "cons": 0.0, "w": 1.0, "lr": 9.026926943086065e-05}
|
||||
{"step": 580, "loss": 1.4532886743545532, "ema": 1.470604117736626, "kl": 0.7623225310817361, "ce": 1.2234560940414667, "fmt": 0.0113600492477417, "cons": 0.0, "w": 1.0, "lr": 8.988608947712639e-05}
|
||||
{"step": 590, "loss": 1.3079419136047363, "ema": 1.4443783416862772, "kl": 0.8541441615670919, "ce": 1.05075572617352, "fmt": 0.009429335594177246, "cons": 0.0, "w": 1.0, "lr": 8.949635669739102e-05}
|
||||
{"step": 600, "loss": 1.222110629081726, "ema": 1.4429199524298184, "kl": 0.758715033531189, "ce": 0.9935558065772057, "fmt": 0.009403586387634277, "cons": 0.0, "w": 1.0, "lr": 8.910013512027022e-05}
|
||||
{"step": 600, "eval_acc": 0.48785046728971965, "eval_dist": {"A": 0.407, "B": 0.112, "C": 0.26, "D": 0.221}}
|
||||
{"step": 610, "loss": 1.1944189071655273, "ema": 1.348318335419306, "kl": 0.6720553506165743, "ce": 0.9919251594692469, "fmt": 0.008771777153015137, "cons": 0.0, "w": 1.0, "lr": 8.869748984041441e-05}
|
||||
{"step": 620, "loss": 1.2564548254013062, "ema": 1.4378891344582434, "kl": 0.9285712875425816, "ce": 0.9769554324448109, "fmt": 0.009276986122131348, "cons": 0.0, "w": 1.0, "lr": 8.828848700781459e-05}
|
||||
{"step": 630, "loss": 1.2642128467559814, "ema": 1.3474314117132518, "kl": 0.7418345184996724, "ce": 1.0406661182641983, "fmt": 0.009964346885681152, "cons": 0.0, "w": 1.0, "lr": 8.787319381693458e-05}
|
||||
{"step": 640, "loss": 1.5251730680465698, "ema": 1.3785072858554723, "kl": 1.0194712821394205, "ce": 1.2182167526334524, "fmt": 0.011149287223815918, "cons": 0.0, "w": 1.0, "lr": 8.74516784956717e-05}
|
||||
{"step": 650, "loss": 1.2936019897460938, "ema": 1.3564687170137542, "kl": 0.832710693590343, "ce": 1.0429086228832603, "fmt": 0.008801698684692383, "cons": 0.0, "w": 1.0, "lr": 8.702401029414787e-05}
|
||||
{"step": 650, "eval_acc": 0.5121495327102804, "eval_dist": {"A": 0.178, "B": 0.344, "C": 0.256, "D": 0.222}}
|
||||
{"step": 660, "loss": 1.3826504945755005, "ema": 1.3488963340084963, "kl": 0.7345772115513682, "ce": 1.1613542269915342, "fmt": 0.009232878684997559, "cons": 0.0, "w": 1.0, "lr": 8.659025947333244e-05}
|
||||
{"step": 670, "loss": 1.1447629928588867, "ema": 1.3634902201371446, "kl": 0.6411142586730421, "ce": 0.9514568275772035, "fmt": 0.009718060493469238, "cons": 0.0, "w": 1.0, "lr": 8.615049729349918e-05}
|
||||
{"step": 680, "loss": 1.337057113647461, "ema": 1.31159051019138, "kl": 0.8606938878074288, "ce": 1.0779208010062575, "fmt": 0.009281396865844727, "cons": 0.0, "w": 1.0, "lr": 8.57047960025191e-05}
|
||||
{"step": 690, "loss": 1.2063547372817993, "ema": 1.2884944764767676, "kl": 0.7169359140098095, "ce": 0.9904440399259329, "fmt": 0.008302092552185059, "cons": 0.0, "w": 1.0, "lr": 8.525322882399082e-05}
|
||||
{"step": 700, "loss": 1.3178514242172241, "ema": 1.3026567798629942, "kl": 0.8626838987693191, "ce": 1.0583276560064405, "fmt": 0.007184743881225586, "cons": 0.0, "w": 1.0, "lr": 8.479586994521097e-05}
|
||||
{"step": 700, "eval_acc": 0.5308411214953271, "eval_dist": {"A": 0.307, "B": 0.344, "C": 0.221, "D": 0.129}}
|
||||
{"step": 710, "loss": 1.4073556661605835, "ema": 1.3024136642150774, "kl": 0.6621118513867259, "ce": 1.2078693844377995, "fmt": 0.008525609970092773, "cons": 0.0, "w": 1.0, "lr": 8.433279450498581e-05}
|
||||
{"step": 720, "loss": 1.2158803939819336, "ema": 1.3387758304515611, "kl": 0.6567904673865996, "ce": 1.0179049195721745, "fmt": 0.00938403606414795, "cons": 0.0, "w": 1.0, "lr": 8.386407858128706e-05}
|
||||
{"step": 730, "loss": 1.0920017957687378, "ema": 1.309909415769093, "kl": 0.7025178072508425, "ce": 0.8803515592589974, "fmt": 0.008948087692260742, "cons": 0.0, "w": 1.0, "lr": 8.338979917875296e-05}
|
||||
{"step": 740, "loss": 0.9786521196365356, "ema": 1.220367131005402, "kl": 0.6228096783161163, "ce": 0.7910086326301098, "fmt": 0.008005499839782715, "cons": 0.0, "w": 1.0, "lr": 8.29100342160374e-05}
|
||||
{"step": 750, "loss": 1.5366700887680054, "ema": 1.3127164991012474, "kl": 0.6951618136372417, "ce": 1.3272929387167096, "fmt": 0.008286595344543457, "cons": 0.0, "w": 1.0, "lr": 8.242486251300876e-05}
|
||||
{"step": 750, "eval_acc": 0.5644859813084112, "eval_dist": {"A": 0.353, "B": 0.206, "C": 0.237, "D": 0.204}}
|
||||
{"step": 760, "loss": 1.5407609939575195, "ema": 1.37166104367332, "kl": 0.7837777566164732, "ce": 1.305006064940244, "fmt": 0.00621485710144043, "cons": 0.0, "w": 1.0, "lr": 8.193436377780065e-05}
|
||||
{"step": 770, "loss": 1.2966320514678955, "ema": 1.3098156349211283, "kl": 0.898317152634263, "ce": 1.0263306361157447, "fmt": 0.008064031600952148, "cons": 0.0, "w": 1.0, "lr": 8.143861859371689e-05}
|
||||
{"step": 780, "loss": 1.3509852886199951, "ema": 1.3021147152953862, "kl": 0.6898657637648284, "ce": 1.143305653706193, "fmt": 0.007198810577392578, "cons": 0.0, "w": 1.0, "lr": 8.09377084059925e-05}
|
||||
{"step": 790, "loss": 1.4421616792678833, "ema": 1.3231484016806894, "kl": 1.0315813897177577, "ce": 1.131993193179369, "fmt": 0.00694119930267334, "cons": 0.0, "w": 1.0, "lr": 8.043171550841326e-05}
|
||||
{"step": 800, "loss": 1.2897963523864746, "ema": 1.3118159067707014, "kl": 0.7542656407458708, "ce": 1.062730560079217, "fmt": 0.007862448692321777, "cons": 0.0, "w": 1.0, "lr": 7.992072302979574e-05}
|
||||
{"step": 800, "eval_acc": 0.5869158878504673, "eval_dist": {"A": 0.249, "B": 0.174, "C": 0.2, "D": 0.378}}
|
||||
{"step": 810, "loss": 1.0095988512039185, "ema": 1.2765115446352673, "kl": 0.5371714350767434, "ce": 0.8477514507248998, "fmt": 0.006960630416870117, "cons": 0.0, "w": 1.0, "lr": 7.940481492033018e-05}
|
||||
{"step": 820, "loss": 1.3918567895889282, "ema": 1.2294958382167303, "kl": 1.0506250832695514, "ce": 1.0760110719129443, "fmt": 0.006581664085388184, "cons": 0.0, "w": 1.0, "lr": 7.888407593778847e-05}
|
||||
{"step": 830, "loss": 1.3179770708084106, "ema": 1.1600642098875467, "kl": 0.8277733416762203, "ce": 1.0690186442807317, "fmt": 0.0062634944915771484, "cons": 0.0, "w": 1.0, "lr": 7.835859163359945e-05}
|
||||
{"step": 840, "loss": 1.294169545173645, "ema": 1.19731427847478, "kl": 0.7935073184780777, "ce": 1.055417166557163, "fmt": 0.007000923156738281, "cons": 0.0, "w": 1.0, "lr": 7.782844833879373e-05}
|
||||
{"step": 850, "loss": 0.958183765411377, "ema": 1.2119452778642625, "kl": 0.8300922764465213, "ce": 0.7084027072414756, "fmt": 0.007533550262451172, "cons": 0.0, "w": 1.0, "lr": 7.72937331498205e-05}
|
||||
{"step": 850, "eval_acc": 0.6130841121495327, "eval_dist": {"A": 0.245, "B": 0.247, "C": 0.321, "D": 0.187}}
|
||||
{"step": 860, "loss": 0.9957249164581299, "ema": 1.163827472810369, "kl": 0.592152432538569, "ce": 0.8173077967949212, "fmt": 0.007714271545410156, "cons": 0.0, "w": 1.0, "lr": 7.675453391423866e-05}
|
||||
{"step": 870, "loss": 0.9601846933364868, "ema": 1.1498456782472135, "kl": 0.5781636242754757, "ce": 0.7861383273266256, "fmt": 0.00597226619720459, "cons": 0.0, "w": 1.0, "lr": 7.62109392162844e-05}
|
||||
{"step": 880, "loss": 0.9828616380691528, "ema": 1.1624508294512435, "kl": 0.7073614438995719, "ce": 0.7701188195496798, "fmt": 0.005344390869140625, "cons": 0.0, "w": 1.0, "lr": 7.56630383623178e-05}
|
||||
{"step": 890, "loss": 1.06419837474823, "ema": 1.2363380827947623, "kl": 0.48982884525321424, "ce": 0.9166093168314546, "fmt": 0.006404280662536621, "cons": 0.0, "w": 1.0, "lr": 7.511092136615094e-05}
|
||||
{"step": 900, "loss": 1.247433066368103, "ema": 1.2160146550188133, "kl": 0.7773840294685215, "ce": 1.0135629302822053, "fmt": 0.006549239158630371, "cons": 0.0, "w": 1.0, "lr": 7.455467893425959e-05}
|
||||
{"step": 900, "eval_acc": 0.616822429906542, "eval_dist": {"A": 0.196, "B": 0.198, "C": 0.211, "D": 0.394}}
|
||||
{"step": 910, "loss": 1.4841305017471313, "ema": 1.1579864465169376, "kl": 0.9550966466777027, "ce": 1.1968963383696973, "fmt": 0.007053256034851074, "cons": 0.0, "w": 1.0, "lr": 7.399440245088116e-05}
|
||||
{"step": 920, "loss": 0.9815365672111511, "ema": 1.1460049987788845, "kl": 0.8983375523239374, "ce": 0.7115907515399158, "fmt": 0.004445195198059082, "cons": 0.0, "w": 1.0, "lr": 7.343018396300136e-05}
|
||||
{"step": 930, "loss": 1.0377646684646606, "ema": 1.1190009111685, "kl": 0.6418211553245783, "ce": 0.8445939151570201, "fmt": 0.0062465667724609375, "cons": 0.0, "w": 1.0, "lr": 7.286211616523193e-05}
|
||||
{"step": 940, "loss": 1.0549876689910889, "ema": 1.1516800137259264, "kl": 0.824024572968483, "ce": 0.8072100812569261, "fmt": 0.005701899528503418, "cons": 0.0, "w": 1.0, "lr": 7.229029238458196e-05}
|
||||
{"step": 950, "loss": 1.0970275402069092, "ema": 1.1685154434812026, "kl": 0.6433418949600309, "ce": 0.9033852405846119, "fmt": 0.006395578384399414, "cons": 0.0, "w": 1.0, "lr": 7.171480656512537e-05}
|
||||
{"step": 950, "eval_acc": 0.616822429906542, "eval_dist": {"A": 0.277, "B": 0.297, "C": 0.178, "D": 0.249}}
|
||||
{"step": 960, "loss": 1.2615981101989746, "ema": 1.1876187419887831, "kl": 0.6516558422008529, "ce": 1.0655653881840408, "fmt": 0.005361318588256836, "cons": 0.0, "w": 1.0, "lr": 7.113575325256694e-05}
|
||||
{"step": 970, "loss": 1.1278254985809326, "ema": 1.1488406860044864, "kl": 0.560641597956419, "ce": 0.9589674472808838, "fmt": 0.006655693054199219, "cons": 0.0, "w": 1.0, "lr": 7.05532275787096e-05}
|
||||
{"step": 980, "loss": 1.3618487119674683, "ema": 1.1288002920518445, "kl": 0.9162907432764769, "ce": 1.0866007783915848, "fmt": 0.0036079883575439453, "cons": 0.0, "w": 1.0, "lr": 6.996732524582524e-05}
|
||||
{"step": 990, "loss": 1.3181620836257935, "ema": 1.1729309608164158, "kl": 1.0712104293052107, "ce": 0.9962489539757371, "fmt": 0.00549924373626709, "cons": 0.0, "w": 1.0, "lr": 6.93781425109321e-05}
|
||||
{"step": 1000, "loss": 1.270733118057251, "ema": 1.1740302717321864, "kl": 0.8394742622040212, "ce": 1.0183514300733805, "fmt": 0.005393862724304199, "cons": 0.0, "w": 1.0, "lr": 6.878577616998074e-05}
|
||||
{"step": 1000, "eval_acc": 0.6429906542056075, "eval_dist": {"A": 0.176, "B": 0.221, "C": 0.288, "D": 0.316}}
|
||||
{"step": 1010, "loss": 1.2056329250335693, "ema": 1.0807281262750679, "kl": 0.865548599511385, "ce": 0.9454872822389007, "fmt": 0.0048111677169799805, "cons": 0.0, "w": 1.0, "lr": 6.819032354195158e-05}
|
||||
{"step": 1020, "loss": 1.0371966361999512, "ema": 1.0113326716788722, "kl": 0.8694827556610107, "ce": 0.7758266711607575, "fmt": 0.005250692367553711, "cons": 0.0, "w": 1.0, "lr": 6.759188245286653e-05}
|
||||
{"step": 1030, "loss": 1.0027440786361694, "ema": 1.0386774860394088, "kl": 0.8055644436972216, "ce": 0.7607043891912326, "fmt": 0.003701925277709961, "cons": 0.0, "w": 1.0, "lr": 6.69905512197173e-05}
|
||||
{"step": 1040, "loss": 1.2968944311141968, "ema": 1.0415529552896512, "kl": 0.553027649235446, "ce": 1.1305528648663312, "fmt": 0.004333019256591797, "cons": 0.0, "w": 1.0, "lr": 6.638642863431307e-05}
|
||||
{"step": 1050, "loss": 1.18927001953125, "ema": 1.1219717534277434, "kl": 0.6593470117077231, "ce": 0.9909649624023587, "fmt": 0.005010008811950684, "cons": 0.0, "w": 1.0, "lr": 6.577961394705008e-05}
|
||||
{"step": 1050, "eval_acc": 0.6747663551401869, "eval_dist": {"A": 0.273, "B": 0.202, "C": 0.245, "D": 0.28}}
|
||||
{"step": 1060, "loss": 1.068376064300537, "ema": 1.0995088991304547, "kl": 0.6498855254612863, "ce": 0.8729516011662781, "fmt": 0.004587411880493164, "cons": 0.0, "w": 1.0, "lr": 6.517020685060597e-05}
|
||||
{"step": 1070, "loss": 1.6381127834320068, "ema": 1.1847088108209882, "kl": 0.6937605009879917, "ce": 1.4294959674589336, "fmt": 0.004886269569396973, "cons": 0.0, "w": 1.0, "lr": 6.45583074635614e-05}
|
||||
{"step": 1080, "loss": 1.2349848747253418, "ema": 1.1249751341978407, "kl": 0.7798603690462187, "ce": 1.0006457371637225, "fmt": 0.0038106441497802734, "cons": 0.0, "w": 1.0, "lr": 6.394401631395175e-05}
|
||||
{"step": 1090, "loss": 1.248127818107605, "ema": 1.1542045236700924, "kl": 1.0297332960180938, "ce": 0.9387678118655458, "fmt": 0.004399418830871582, "cons": 0.0, "w": 1.0, "lr": 6.332743432275147e-05}
|
||||
{"step": 1100, "loss": 1.0184950828552246, "ema": 1.0455114694756928, "kl": 0.6638475921936333, "ce": 0.8188166990876198, "fmt": 0.005239725112915039, "cons": 0.0, "w": 1.0, "lr": 6.270866278729394e-05}
|
||||
{"step": 1100, "eval_acc": 0.6710280373831776, "eval_dist": {"A": 0.374, "B": 0.196, "C": 0.275, "D": 0.155}}
|
||||
{"step": 1110, "loss": 1.0805654525756836, "ema": 1.0640981016554782, "kl": 0.46230101119726896, "ce": 0.94126216461882, "fmt": 0.006130814552307129, "cons": 0.0, "w": 1.0, "lr": 6.208780336462956e-05}
|
||||
{"step": 1120, "loss": 1.1674941778182983, "ema": 1.0373850490469338, "kl": 0.644195671658963, "ce": 0.9737236117944121, "fmt": 0.005118131637573242, "cons": 0.0, "w": 1.0, "lr": 6.146495805482451e-05}
|
||||
{"step": 1130, "loss": 0.9401713609695435, "ema": 0.9939991935188188, "kl": 0.47861349931918085, "ce": 0.7959921651054174, "fmt": 0.005952119827270508, "cons": 0.0, "w": 1.0, "lr": 6.0840229184203435e-05}
|
||||
{"step": 1140, "loss": 1.0403752326965332, "ema": 1.0205399760570488, "kl": 1.1692930296994746, "ce": 0.688915295060724, "fmt": 0.006718754768371582, "cons": 0.0, "w": 1.0, "lr": 6.021371938853839e-05}
|
||||
{"step": 1150, "loss": 0.8085815906524658, "ema": 1.012362483676901, "kl": 0.4957937563303858, "ce": 0.6594793945550919, "fmt": 0.003641486167907715, "cons": 0.0, "w": 1.0, "lr": 5.958553159618693e-05}
|
||||
{"step": 1150, "eval_acc": 0.6672897196261682, "eval_dist": {"A": 0.312, "B": 0.269, "C": 0.256, "D": 0.163}}
|
||||
{"step": 1160, "loss": 0.9519131183624268, "ema": 1.0112859194632324, "kl": 0.5640503740869462, "ce": 0.7822483400814235, "fmt": 0.0044966936111450195, "cons": 0.0, "w": 1.0, "lr": 5.895576901118215e-05}
|
||||
{"step": 1170, "loss": 1.0350356101989746, "ema": 1.0673316728820255, "kl": 0.640855276491493, "ce": 0.8422510626260191, "fmt": 0.005279421806335449, "cons": 0.0, "w": 1.0, "lr": 5.832453509627751e-05}
|
||||
{"step": 1180, "loss": 1.0483644008636475, "ema": 1.0714138292162918, "kl": 0.48160202358849347, "ce": 0.9035150269046426, "fmt": 0.0036873817443847656, "cons": 0.0, "w": 1.0, "lr": 5.769193355594903e-05}
|
||||
{"step": 1190, "loss": 1.2572916746139526, "ema": 1.1133478553844387, "kl": 0.8425654373131692, "ce": 1.0039732069708407, "fmt": 0.005488157272338867, "cons": 0.0, "w": 1.0, "lr": 5.705806831935792e-05}
|
||||
{"step": 1200, "loss": 1.3446009159088135, "ema": 1.036993176677073, "kl": 0.8842297736555338, "ce": 1.0787488846108317, "fmt": 0.0058307647705078125, "cons": 0.0, "w": 1.0, "lr": 5.642304352327607e-05}
|
||||
{"step": 1200, "eval_acc": 0.6915887850467289, "eval_dist": {"A": 0.247, "B": 0.245, "C": 0.256, "D": 0.252}}
|
||||
{"step": 1210, "loss": 0.9616119265556335, "ema": 1.0692059738406823, "kl": 0.676732716470724, "ce": 0.7582431330229156, "fmt": 0.003488898277282715, "cons": 0.0, "w": 1.0, "lr": 5.5786963494977775e-05}
|
||||
{"step": 1220, "loss": 1.344433307647705, "ema": 1.052591388116776, "kl": 0.8140649802517146, "ce": 1.0998210341203958, "fmt": 0.0039272308349609375, "cons": 0.0, "w": 1.0, "lr": 5.514993273509975e-05}
|
||||
{"step": 1230, "loss": 1.102717399597168, "ema": 1.111109024851196, "kl": 0.6336296880617738, "ce": 0.9120676980819553, "fmt": 0.005607008934020996, "cons": 0.0, "w": 1.0, "lr": 5.4512055900473035e-05}
|
||||
{"step": 1240, "loss": 1.0275386571884155, "ema": 0.9801581125823959, "kl": 0.7510397983714938, "ce": 0.8017238226020709, "fmt": 0.0050286054611206055, "cons": 0.0, "w": 1.0, "lr": 5.387343778692903e-05}
|
||||
{"step": 1250, "loss": 0.941438615322113, "ema": 0.9750089561381994, "kl": 0.8936588485958055, "ce": 0.6728727999143302, "fmt": 0.004680037498474121, "cons": 0.0, "w": 1.0, "lr": 5.323418331208271e-05}
|
||||
{"step": 1250, "eval_acc": 0.697196261682243, "eval_dist": {"A": 0.273, "B": 0.222, "C": 0.279, "D": 0.226}}
|
||||
{"step": 1260, "loss": 1.0822031497955322, "ema": 1.051456755954492, "kl": 0.8507764476817101, "ce": 0.8263651341549121, "fmt": 0.0060520172119140625, "cons": 0.0, "w": 1.0, "lr": 5.2594397498095906e-05}
|
||||
{"step": 1270, "loss": 1.209940791130066, "ema": 0.9877108070217351, "kl": 0.6925017586909235, "ce": 1.001465406268835, "fmt": 0.007248044013977051, "cons": 0.0, "w": 1.0, "lr": 5.1954185454423375e-05}
|
||||
{"step": 1280, "loss": 1.3404643535614014, "ema": 0.9785492126974674, "kl": 0.8215758693404496, "ce": 1.0933225085027516, "fmt": 0.006689786911010742, "cons": 0.0, "w": 1.0, "lr": 5.1313652360544505e-05}
|
||||
{"step": 1290, "loss": 1.165497899055481, "ema": 0.9951176819392452, "kl": 0.8215993586927652, "ce": 0.9182843840681016, "fmt": 0.007337450981140137, "cons": 0.0, "w": 1.0, "lr": 5.067290344868355e-05}
|
||||
{"step": 1300, "loss": 1.1103487014770508, "ema": 1.0108268234602265, "kl": 0.8929044581018388, "ce": 0.8420266155153513, "fmt": 0.004508256912231445, "cons": 0.0, "w": 1.0, "lr": 5.003204398652115e-05}
|
||||
{"step": 1300, "eval_acc": 0.685981308411215, "eval_dist": {"A": 0.28, "B": 0.247, "C": 0.28, "D": 0.193}}
|
||||
{"step": 1310, "loss": 1.1685528755187988, "ema": 0.9719165751361445, "kl": 0.8154381016502157, "ce": 0.9230864702258259, "fmt": 0.008348941802978516, "cons": 0.0, "w": 1.0, "lr": 4.9391179259900125e-05}
|
||||
{"step": 1320, "loss": 0.8740050196647644, "ema": 1.0020695853221917, "kl": 0.7254291916033253, "ce": 0.6556908908532932, "fmt": 0.0068547725677490234, "cons": 0.0, "w": 1.0, "lr": 4.875041455552817e-05}
|
||||
{"step": 1330, "loss": 0.9236490726470947, "ema": 1.031332066826892, "kl": 0.6720662505831569, "ce": 0.7214050348848104, "fmt": 0.0062416791915893555, "cons": 0.0, "w": 1.0, "lr": 4.810985514368044e-05}
|
||||
{"step": 1340, "loss": 0.7425285577774048, "ema": 1.076738844403875, "kl": 0.5911096127238125, "ce": 0.5645549204200506, "fmt": 0.006407380104064941, "cons": 0.0, "w": 1.0, "lr": 4.7469606260904966e-05}
|
||||
{"step": 1350, "loss": 0.9287974834442139, "ema": 1.0318459489519378, "kl": 0.44651566119864583, "ce": 0.7942275749519467, "fmt": 0.006151795387268066, "cons": 0.0, "w": 1.0, "lr": 4.6829773092733326e-05}
|
||||
{"step": 1350, "eval_acc": 0.7065420560747664, "eval_dist": {"A": 0.333, "B": 0.228, "C": 0.258, "D": 0.181}}
|
||||
{"step": 1360, "loss": 0.9124118089675903, "ema": 1.0087702545781494, "kl": 0.8300607725977898, "ce": 0.6628535429481417, "fmt": 0.0054013729095458984, "cons": 0.0, "w": 1.0, "lr": 4.6190460756400044e-05}
|
||||
{"step": 1370, "loss": 0.9291731715202332, "ema": 1.011831822104197, "kl": 0.4356573737459257, "ce": 0.7978434537071735, "fmt": 0.00632476806640625, "cons": 0.0, "w": 1.0, "lr": 4.5551774283572965e-05}
|
||||
{"step": 1380, "loss": 1.3725370168685913, "ema": 0.9628048257359735, "kl": 0.9824748947285116, "ce": 1.0772789483889937, "fmt": 0.0051566362380981445, "cons": 0.0, "w": 1.0, "lr": 4.491381860309766e-05}
|
||||
{"step": 1390, "loss": 0.837517499923706, "ema": 0.9587338056798302, "kl": 1.0060886582359672, "ce": 0.5350183424307033, "fmt": 0.006725668907165527, "cons": 0.0, "w": 1.0, "lr": 4.427669852375904e-05}
|
||||
{"step": 1400, "loss": 1.0249205827713013, "ema": 1.0139870711481103, "kl": 1.1098054177127779, "ce": 0.6915423576720059, "fmt": 0.004366159439086914, "cons": 0.0, "w": 1.0, "lr": 4.364051871706225e-05}
|
||||
{"step": 1400, "eval_acc": 0.702803738317757, "eval_dist": {"A": 0.277, "B": 0.222, "C": 0.279, "D": 0.222}}
|
||||
{"step": 1410, "loss": 1.1874208450317383, "ema": 1.0031447203018065, "kl": 1.0229534655809402, "ce": 0.8800485127139837, "fmt": 0.004862427711486816, "cons": 0.0, "w": 1.0, "lr": 4.3005383700036525e-05}
|
||||
{"step": 1420, "loss": 0.9669800996780396, "ema": 0.9934954692727794, "kl": 0.9607930863567162, "ce": 0.6781033467268571, "fmt": 0.006389141082763672, "cons": 0.0, "w": 1.0, "lr": 4.2371397818064106e-05}
|
||||
{"step": 1430, "loss": 0.6093875169754028, "ema": 0.9245397012777923, "kl": 0.3511242351960391, "ce": 0.5035586644662544, "fmt": 0.00491642951965332, "cons": 0.0, "w": 1.0, "lr": 4.173866522773762e-05}
|
||||
{"step": 1440, "loss": 0.816233217716217, "ema": 0.8772316601951873, "kl": 0.8480414177756757, "ce": 0.5613298885873519, "fmt": 0.004908323287963867, "cons": 0.0, "w": 1.0, "lr": 4.11072898797482e-05}
|
||||
{"step": 1450, "loss": 0.9728041887283325, "ema": 0.9189439014529623, "kl": 0.6863559384364635, "ce": 0.7665571190882474, "fmt": 0.0034027099609375, "cons": 0.0, "w": 1.0, "lr": 4.047737550180776e-05}
|
||||
{"step": 1450, "eval_acc": 0.7009345794392523, "eval_dist": {"A": 0.254, "B": 0.256, "C": 0.299, "D": 0.191}}
|
||||
{"step": 1460, "loss": 0.7367637157440186, "ema": 0.9722746691530316, "kl": 0.6995962705113925, "ce": 0.5263955192640424, "fmt": 0.004892826080322266, "cons": 0.0, "w": 1.0, "lr": 3.984902558160755e-05}
|
||||
{"step": 1470, "loss": 0.9850180745124817, "ema": 0.9230869519399553, "kl": 0.5390156616922468, "ce": 0.8226226263213903, "fmt": 0.006907820701599121, "cons": 0.0, "w": 1.0, "lr": 3.9222343349816445e-05}
|
||||
{"step": 1480, "loss": 1.028172254562378, "ema": 0.9920299993847959, "kl": 0.6248799953609705, "ce": 0.840226300060749, "fmt": 0.00482022762298584, "cons": 0.0, "w": 1.0, "lr": 3.859743176312126e-05}
|
||||
{"step": 1490, "loss": 0.7680354118347168, "ema": 0.9911840112462185, "kl": 0.7879841467365623, "ce": 0.5311999259283766, "fmt": 0.004401445388793945, "cons": 0.0, "w": 1.0, "lr": 3.797439348731219e-05}
|
||||
{"step": 1500, "loss": 0.8396425843238831, "ema": 0.9959950341499336, "kl": 0.365481338230893, "ce": 0.7295759599655867, "fmt": 0.004221558570861816, "cons": 0.0, "w": 1.0, "lr": 3.735333088041596e-05}
|
||||
{"step": 1500, "eval_acc": 0.702803738317757, "eval_dist": {"A": 0.269, "B": 0.249, "C": 0.267, "D": 0.215}}
|
||||
{"step": 1510, "loss": 0.5954517722129822, "ema": 0.9195603294237067, "kl": 0.5180649870890193, "ce": 0.43932669698551763, "fmt": 0.007055401802062988, "cons": 0.0, "w": 1.0, "lr": 3.673434597587966e-05}
|
||||
{"step": 1520, "loss": 0.8354024291038513, "ema": 0.8997133800587871, "kl": 0.8623552332865074, "ce": 0.5759532073279843, "fmt": 0.007427096366882324, "cons": 0.0, "w": 1.0, "lr": 3.611754046580768e-05}
|
||||
{"step": 1530, "loss": 1.1594830751419067, "ema": 0.9973351980326085, "kl": 1.185442433692515, "ce": 0.802836678805761, "fmt": 0.010137796401977539, "cons": 0.0, "w": 1.0, "lr": 3.5503015684255044e-05}
|
||||
{"step": 1540, "loss": 0.9436646103858948, "ema": 0.9442165817931316, "kl": 0.8404226024867967, "ce": 0.6909072486450896, "fmt": 0.0063048601150512695, "cons": 0.0, "w": 1.0, "lr": 3.489087259057921e-05}
|
||||
{"step": 1550, "loss": 1.0413702726364136, "ema": 0.9958077792462452, "kl": 0.7227072507375851, "ce": 0.8239739493001252, "fmt": 0.005841732025146484, "cons": 0.0, "w": 1.0, "lr": 3.4281211752853796e-05}
|
||||
{"step": 1550, "eval_acc": 0.7121495327102804, "eval_dist": {"A": 0.286, "B": 0.237, "C": 0.282, "D": 0.194}}
|
||||
{"step": 1560, "loss": 0.7645087242126465, "ema": 0.9090210166517839, "kl": 0.6409536354476586, "ce": 0.5716069168411195, "fmt": 0.006156206130981445, "cons": 0.0, "w": 1.0, "lr": 3.367413333134619e-05}
|
||||
{"step": 1570, "loss": 1.1463549137115479, "ema": 0.9500677109135108, "kl": 0.547364437719807, "ce": 0.9815158644341864, "fmt": 0.006296992301940918, "cons": 0.0, "w": 1.0, "lr": 3.306973706206258e-05}
|
||||
{"step": 1580, "loss": 1.0882612466812134, "ema": 1.0125978101771078, "kl": 0.4699276631581597, "ce": 0.9466271434212103, "fmt": 0.006560921669006348, "cons": 0.0, "w": 1.0, "lr": 3.2468122240362284e-05}
|
||||
{"step": 1590, "loss": 1.0061726570129395, "ema": 1.0218242220483198, "kl": 0.5779011566191912, "ce": 0.8322631753981113, "fmt": 0.005391240119934082, "cons": 0.0, "w": 1.0, "lr": 3.186938770464486e-05}
|
||||
{"step": 1600, "loss": 1.0023213624954224, "ema": 1.0120906490292185, "kl": 0.8545904953498393, "ce": 0.7454528894741088, "fmt": 0.004912018775939941, "cons": 0.0, "w": 1.0, "lr": 3.127363182011187e-05}
|
||||
{"step": 1600, "eval_acc": 0.702803738317757, "eval_dist": {"A": 0.252, "B": 0.245, "C": 0.267, "D": 0.236}}
|
||||
{"step": 1610, "loss": 0.9858607649803162, "ema": 0.9860513521538863, "kl": 0.5981258838437498, "ce": 0.8058255887590349, "fmt": 0.005974173545837402, "cons": 0.0, "w": 1.0, "lr": 3.068095246260685e-05}
|
||||
{"step": 1620, "loss": 0.8384391665458679, "ema": 0.9354751343440064, "kl": 0.9068117213901132, "ce": 0.5659278226085007, "fmt": 0.004678130149841309, "cons": 0.0, "w": 1.0, "lr": 3.00914470025352e-05}
|
||||
{"step": 1630, "loss": 0.9945364594459534, "ema": 0.9647729615018963, "kl": 0.8235310260206461, "ce": 0.7468434432521462, "fmt": 0.006338715553283691, "cons": 0.0, "w": 1.0, "lr": 2.9505212288867545e-05}
|
||||
{"step": 1640, "loss": 1.4661955833435059, "ema": 0.9885254253614286, "kl": 0.7520663952454925, "ce": 1.2399946162477136, "fmt": 0.005808711051940918, "cons": 0.0, "w": 1.0, "lr": 2.8922344633228387e-05}
|
||||
{"step": 1650, "loss": 0.9997166991233826, "ema": 0.9335313386089195, "kl": 0.7769604045897722, "ce": 0.7659110380336642, "fmt": 0.007173776626586914, "cons": 0.0, "w": 1.0, "lr": 2.8342939794073408e-05}
|
||||
{"step": 1650, "eval_acc": 0.7177570093457943, "eval_dist": {"A": 0.271, "B": 0.226, "C": 0.269, "D": 0.234}}
|
||||
{"step": 1660, "loss": 0.8082342743873596, "ema": 0.8889085672568884, "kl": 0.8122424219036475, "ce": 0.5638759031426162, "fmt": 0.0068558454513549805, "cons": 0.0, "w": 1.0, "lr": 2.776709296095722e-05}
|
||||
{"step": 1670, "loss": 1.2839171886444092, "ema": 0.9445191496397075, "kl": 1.3028770750388503, "ce": 0.8922232801560313, "fmt": 0.00830686092376709, "cons": 0.0, "w": 1.0, "lr": 2.719489873889508e-05}
|
||||
{"step": 1680, "loss": 0.8055791854858398, "ema": 1.0267729351396306, "kl": 0.5285566317907069, "ce": 0.6465188422880601, "fmt": 0.004934072494506836, "cons": 0.0, "w": 1.0, "lr": 2.6626451132820085e-05}
|
||||
{"step": 1690, "loss": 0.9279726147651672, "ema": 0.9934742281278948, "kl": 0.8558181404368952, "ce": 0.6708394197048619, "fmt": 0.003877401351928711, "cons": 0.0, "w": 1.0, "lr": 2.606184353213956e-05}
|
||||
{"step": 1700, "loss": 0.8259214162826538, "ema": 0.9146790748311704, "kl": 0.47099627734860405, "ce": 0.684172242297791, "fmt": 0.004503488540649414, "cons": 0.0, "w": 1.0, "lr": 2.5501168695391913e-05}
|
||||
{"step": 1700, "eval_acc": 0.7102803738317757, "eval_dist": {"A": 0.273, "B": 0.25, "C": 0.273, "D": 0.204}}
|
||||
{"step": 1710, "loss": 0.9326278567314148, "ema": 0.9203634868209238, "kl": 0.7072904999367893, "ce": 0.7200655199121684, "fmt": 0.003751516342163086, "cons": 0.0, "w": 1.0, "lr": 2.4944518735007816e-05}
|
||||
{"step": 1720, "loss": 1.1014740467071533, "ema": 0.9232145928949558, "kl": 0.8776409064885229, "ce": 0.8377565275877714, "fmt": 0.004252076148986816, "cons": 0.0, "w": 1.0, "lr": 2.4391985102176858e-05}
|
||||
{"step": 1730, "loss": 0.9649674892425537, "ema": 0.9121305649734901, "kl": 1.0267186190467328, "ce": 0.6565678287297487, "fmt": 0.0038416385650634766, "cons": 0.0, "w": 1.0, "lr": 2.3843658571823464e-05}
|
||||
{"step": 1740, "loss": 0.9707590341567993, "ema": 0.9384691841237598, "kl": 1.0589740662835538, "ce": 0.6524089068407193, "fmt": 0.006579041481018066, "cons": 0.0, "w": 1.0, "lr": 2.3299629227693416e-05}
|
||||
{"step": 1750, "loss": 1.200559377670288, "ema": 1.0024224091485097, "kl": 1.2725751241669059, "ce": 0.8180888604838401, "fmt": 0.006978273391723633, "cons": 0.0, "w": 1.0, "lr": 2.27599864475542e-05}
|
||||
{"step": 1750, "eval_acc": 0.708411214953271, "eval_dist": {"A": 0.271, "B": 0.243, "C": 0.273, "D": 0.213}}
|
||||
{"step": 1760, "loss": 1.3998944759368896, "ema": 0.9844366264291169, "kl": 1.0431136505794711, "ce": 1.0864514328422956, "fmt": 0.005089402198791504, "cons": 0.0, "w": 1.0, "lr": 2.2224818888511273e-05}
|
||||
{"step": 1770, "loss": 0.9560684561729431, "ema": 0.9833534915389637, "kl": 0.6267587387701496, "ce": 0.7676386467646807, "fmt": 0.004021644592285156, "cons": 0.0, "w": 1.0, "lr": 2.169421447244272e-05}
|
||||
{"step": 1780, "loss": 0.6791104674339294, "ema": 0.9487188419529814, "kl": 0.4857394704595208, "ce": 0.5328691327013075, "fmt": 0.005195260047912598, "cons": 0.0, "w": 1.0, "lr": 2.1168260371554664e-05}
|
||||
{"step": 1790, "loss": 0.9634186625480652, "ema": 0.9248056417637206, "kl": 0.8961919426219538, "ce": 0.6940913322614506, "fmt": 0.004697561264038086, "cons": 0.0, "w": 1.0, "lr": 2.064704299405993e-05}
|
||||
{"step": 1800, "loss": 0.9071803092956543, "ema": 0.9125083075224173, "kl": 0.9975919472053647, "ce": 0.6073706520255655, "fmt": 0.0053217411041259766, "cons": 0.0, "w": 1.0, "lr": 2.013064796998211e-05}
|
||||
{"step": 1800, "eval_acc": 0.7065420560747664, "eval_dist": {"A": 0.264, "B": 0.249, "C": 0.249, "D": 0.239}}
|
||||
{"step": 1810, "loss": 1.0207257270812988, "ema": 0.9096538153057517, "kl": 0.9920973577536643, "ce": 0.7225487900432199, "fmt": 0.005478024482727051, "cons": 0.0, "w": 1.0, "lr": 1.9619160137087577e-05}
|
||||
{"step": 1820, "loss": 0.8027160167694092, "ema": 0.9960659715337437, "kl": 0.7457301907124929, "ce": 0.5784200591151603, "fmt": 0.005768299102783203, "cons": 0.0, "w": 1.0, "lr": 1.9112663526947616e-05}
|
||||
{"step": 1830, "loss": 1.0123579502105713, "ema": 0.9719874334953219, "kl": 0.9126169037190266, "ce": 0.7379798688343726, "fmt": 0.005930423736572266, "cons": 0.0, "w": 1.0, "lr": 1.8611241351132963e-05}
|
||||
{"step": 1840, "loss": 1.0784283876419067, "ema": 0.9921670938265894, "kl": 0.517675967188552, "ce": 0.922582981409505, "fmt": 0.005426645278930664, "cons": 0.0, "w": 1.0, "lr": 1.8114975987543143e-05}
|
||||
{"step": 1850, "loss": 0.9769431352615356, "ema": 0.959737728856075, "kl": 0.6641145059838891, "ce": 0.7771846940740943, "fmt": 0.005240321159362793, "cons": 0.0, "w": 1.0, "lr": 1.7623948966872704e-05}
|
||||
{"step": 1850, "eval_acc": 0.7046728971962617, "eval_dist": {"A": 0.245, "B": 0.254, "C": 0.26, "D": 0.241}}
|
||||
{"step": 1860, "loss": 0.9760332107543945, "ema": 1.040583912583064, "kl": 0.5914769279770553, "ce": 0.7979671023786068, "fmt": 0.006230473518371582, "cons": 0.0, "w": 1.0, "lr": 1.713824095921668e-05}
|
||||
{"step": 1870, "loss": 0.7889153361320496, "ema": 1.0011323242064032, "kl": 0.9840193664422259, "ce": 0.4934124517021701, "fmt": 0.0029702186584472656, "cons": 0.0, "w": 1.0, "lr": 1.6657931760817387e-05}
|
||||
{"step": 1880, "loss": 0.9433740377426147, "ema": 0.9413993802908995, "kl": 0.7823490665759891, "ce": 0.7080465398030356, "fmt": 0.0062253475189208984, "cons": 0.0, "w": 1.0, "lr": 1.618310028095486e-05}
|
||||
{"step": 1890, "loss": 0.9159164428710938, "ema": 0.9072716830557459, "kl": 1.2311747579369694, "ce": 0.5456981420284137, "fmt": 0.008657574653625488, "cons": 0.0, "w": 1.0, "lr": 1.5713824528982946e-05}
|
||||
{"step": 1900, "loss": 0.7910841107368469, "ema": 0.9087879187134632, "kl": 0.3266722357366234, "ce": 0.692682251567021, "fmt": 0.004001975059509277, "cons": 0.0, "w": 1.0, "lr": 1.5250181601513253e-05}
|
||||
{"step": 1900, "eval_acc": 0.719626168224299, "eval_dist": {"A": 0.249, "B": 0.249, "C": 0.265, "D": 0.237}}
|
||||
{"step": 1910, "loss": 0.9233184456825256, "ema": 0.9089404040140054, "kl": 0.713182887295261, "ce": 0.7086951271630824, "fmt": 0.006683945655822754, "cons": 0.0, "w": 1.0, "lr": 1.4792247669749083e-05}
|
||||
{"step": 1920, "loss": 1.0464320182800293, "ema": 0.9435030687561506, "kl": 0.7582038347609341, "ce": 0.8183762308908626, "fmt": 0.005946636199951172, "cons": 0.0, "w": 1.0, "lr": 1.4340097966971361e-05}
|
||||
{"step": 1930, "loss": 0.6877383589744568, "ema": 0.926892298885957, "kl": 0.3486161921173334, "ce": 0.5826828456483781, "fmt": 0.004706740379333496, "cons": 0.0, "w": 1.0, "lr": 1.3893806776178686e-05}
|
||||
{"step": 1940, "loss": 1.3030930757522583, "ema": 1.0261729767461727, "kl": 0.7437121914699674, "ce": 1.0794760838034563, "fmt": 0.005032062530517578, "cons": 0.0, "w": 1.0, "lr": 1.3453447417883492e-05}
|
||||
{"step": 1950, "loss": 0.7992661595344543, "ema": 0.9815865740451077, "kl": 0.4345882583875209, "ce": 0.6682264229748398, "fmt": 0.006632089614868164, "cons": 0.0, "w": 1.0, "lr": 1.3019092238066304e-05}
|
||||
{"step": 1950, "eval_acc": 0.7140186915887851, "eval_dist": {"A": 0.269, "B": 0.245, "C": 0.269, "D": 0.217}}
|
||||
{"step": 1960, "loss": 1.095784068107605, "ema": 1.06938609183854, "kl": 0.7741798964561895, "ce": 0.8631231454201043, "fmt": 0.004068851470947266, "cons": 0.0, "w": 1.0, "lr": 1.2590812596290157e-05}
|
||||
{"step": 1970, "loss": 0.8016267418861389, "ema": 0.9798996444047428, "kl": 0.9775568342301995, "ce": 0.5077287962194532, "fmt": 0.006309628486633301, "cons": 0.0, "w": 1.0, "lr": 1.2168678853977017e-05}
|
||||
{"step": 1980, "loss": 0.7650991082191467, "ema": 0.9265474714050568, "kl": 0.9010804165154696, "ce": 0.4943062919192016, "fmt": 0.004686832427978516, "cons": 0.0, "w": 1.0, "lr": 1.1752760362848214e-05}
|
||||
{"step": 1990, "loss": 1.1927258968353271, "ema": 0.9958834773080069, "kl": 0.777931785909459, "ce": 0.9587730136699975, "fmt": 0.0057337284088134766, "cons": 0.0, "w": 1.0, "lr": 1.1343125453530762e-05}
|
||||
{"step": 2000, "loss": 1.2737394571304321, "ema": 0.9932621430187492, "kl": 0.7573883819859475, "ce": 1.0458969343453646, "fmt": 0.006259918212890625, "cons": 0.0, "w": 1.0, "lr": 1.0939841424331415e-05}
|
||||
{"step": 2000, "eval_acc": 0.7121495327102804, "eval_dist": {"A": 0.254, "B": 0.236, "C": 0.265, "D": 0.245}}
|
||||
{"step": 2010, "loss": 0.7723740935325623, "ema": 0.8658843562277032, "kl": 0.6151916887611151, "ce": 0.5873064426705241, "fmt": 0.005101799964904785, "cons": 0.0, "w": 1.0, "lr": 1.0542974530180327e-05}
|
||||
{"step": 2020, "loss": 1.0792527198791504, "ema": 0.9452876702197724, "kl": 0.9160473882220685, "ce": 0.8040372690884396, "fmt": 0.004013419151306152, "cons": 0.0, "w": 1.0, "lr": 1.0152589971746124e-05}
|
||||
{"step": 2030, "loss": 0.9254496097564697, "ema": 0.9606781666462115, "kl": 0.7324816614855081, "ce": 0.7052433949429542, "fmt": 0.004615426063537598, "cons": 0.0, "w": 1.0, "lr": 9.768751884724192e-06}
|
||||
{"step": 2040, "loss": 0.6795163154602051, "ema": 0.9708486879686171, "kl": 0.5969290197826922, "ce": 0.49972442421130836, "fmt": 0.0071299076080322266, "cons": 0.0, "w": 1.0, "lr": 9.391523329299928e-06}
|
||||
{"step": 2050, "loss": 1.2292695045471191, "ema": 0.9721451262553212, "kl": 0.49461968743707985, "ce": 1.080217159469612, "fmt": 0.00666356086730957, "cons": 0.0, "w": 1.0, "lr": 9.020966279788651e-06}
|
||||
{"step": 2050, "eval_acc": 0.7158878504672898, "eval_dist": {"A": 0.265, "B": 0.232, "C": 0.271, "D": 0.232}}
|
||||
{"step": 2060, "loss": 0.9192420244216919, "ema": 0.8954377136520519, "kl": 0.3821327267214656, "ce": 0.8039505730848759, "fmt": 0.006516218185424805, "cons": 0.0, "w": 1.0, "lr": 8.657141614453956e-06}
|
||||
{"step": 2070, "loss": 1.2070016860961914, "ema": 0.9202144998703728, "kl": 1.048367981449701, "ce": 0.8919836867135018, "fmt": 0.005076885223388672, "cons": 0.0, "w": 1.0, "lr": 8.30010910550611e-06}
|
||||
{"step": 2080, "loss": 0.8865954279899597, "ema": 0.9186090946162863, "kl": 0.6215566895843949, "ce": 0.6995504659134895, "fmt": 0.00577998161315918, "cons": 0.0, "w": 1.0, "lr": 7.949927409282155e-06}
|
||||
{"step": 2090, "loss": 0.9601306319236755, "ema": 0.8560677083102648, "kl": 0.6373699095565826, "ce": 0.7685051697771996, "fmt": 0.004145264625549316, "cons": 0.0, "w": 1.0, "lr": 7.606654056609364e-06}
|
||||
{"step": 2100, "loss": 0.9444018006324768, "ema": 0.8555062821455622, "kl": 0.9645550095010549, "ce": 0.6543644024059176, "fmt": 0.006710052490234375, "cons": 0.0, "w": 1.0, "lr": 7.27034544335356e-06}
|
||||
{"step": 2100, "eval_acc": 0.7140186915887851, "eval_dist": {"A": 0.267, "B": 0.232, "C": 0.267, "D": 0.234}}
|
||||
{"step": 2110, "loss": 0.9914484620094299, "ema": 0.9107307572440096, "kl": 0.751109627984988, "ce": 0.7656216828909237, "fmt": 0.004938602447509766, "cons": 0.0, "w": 1.0, "lr": 6.941056821153991e-06}
|
||||
{"step": 2120, "loss": 1.3186700344085693, "ema": 1.0181701064942685, "kl": 0.9570318188052624, "ce": 1.0310127763077617, "fmt": 0.005477428436279297, "cons": 0.0, "w": 1.0, "lr": 6.6188422883459835e-06}
|
||||
{"step": 2130, "loss": 1.3172671794891357, "ema": 1.0014816625451608, "kl": 0.9856011709198356, "ce": 1.0211739898659289, "fmt": 0.004128813743591309, "cons": 0.0, "w": 1.0, "lr": 6.303754781073395e-06}
|
||||
{"step": 2140, "loss": 0.9871906042098999, "ema": 0.8972132209861351, "kl": 0.7918578053358942, "ce": 0.7490696646273136, "fmt": 0.0056362152099609375, "cons": 0.0, "w": 1.0, "lr": 5.99584606459162e-06}
|
||||
{"step": 2150, "loss": 0.879162073135376, "ema": 0.9202923486481982, "kl": 0.6989941848441958, "ce": 0.6688533751294017, "fmt": 0.0061043500900268555, "cons": 0.0, "w": 1.0, "lr": 5.695166724763334e-06}
|
||||
{"step": 2150, "eval_acc": 0.7140186915887851, "eval_dist": {"A": 0.265, "B": 0.239, "C": 0.264, "D": 0.232}}
|
||||
{"step": 2160, "loss": 1.047703742980957, "ema": 0.974380030501067, "kl": 0.8139725751243532, "ce": 0.8030364146688953, "fmt": 0.004754900932312012, "cons": 0.0, "w": 1.0, "lr": 5.401766159747651e-06}
|
||||
{"step": 2170, "loss": 1.0905312299728394, "ema": 0.9744166883019884, "kl": 0.8403065840248019, "ce": 0.8379731788299978, "fmt": 0.004659414291381836, "cons": 0.0, "w": 1.0, "lr": 5.115692571884734e-06}
|
||||
{"step": 2180, "loss": 1.2903591394424438, "ema": 0.9588457920363358, "kl": 1.218570981058292, "ce": 0.9242912942718249, "fmt": 0.004962563514709473, "cons": 0.0, "w": 1.0, "lr": 4.836992959776554e-06}
|
||||
{"step": 2190, "loss": 0.7809779644012451, "ema": 0.9446726971894375, "kl": 0.5265162892173976, "ce": 0.6224085642024875, "fmt": 0.006145358085632324, "cons": 0.0, "w": 1.0, "lr": 4.565713110565689e-06}
|
||||
{"step": 2200, "loss": 1.084097146987915, "ema": 0.9602498558602468, "kl": 0.8439770939876325, "ce": 0.8303462157782633, "fmt": 0.0055806636810302734, "cons": 0.0, "w": 1.0, "lr": 4.3018975924129065e-06}
|
||||
{"step": 2200, "eval_acc": 0.7102803738317757, "eval_dist": {"A": 0.258, "B": 0.247, "C": 0.269, "D": 0.226}}
|
||||
{"step": 2210, "loss": 0.7417587041854858, "ema": 0.9391238694816247, "kl": 0.7821864361176267, "ce": 0.5067348361480981, "fmt": 0.0036791563034057617, "cons": 0.0, "w": 1.0, "lr": 4.0455897471752545e-06}
|
||||
{"step": 2220, "loss": 0.6739327311515808, "ema": 0.8816609173330929, "kl": 0.5663016307516955, "ce": 0.5035055634798482, "fmt": 0.00536656379699707, "cons": 0.0, "w": 1.0, "lr": 3.7968316832853456e-06}
|
||||
{"step": 2230, "loss": 1.2715036869049072, "ema": 0.9117179555970254, "kl": 0.6165742238517851, "ce": 1.0860743067460135, "fmt": 0.0045691728591918945, "cons": 0.0, "w": 1.0, "lr": 3.555664268833592e-06}
|
||||
{"step": 2240, "loss": 1.3254295587539673, "ema": 0.9447743385266427, "kl": 0.8758065598085523, "ce": 1.062185560585931, "fmt": 0.0050199031829833984, "cons": 0.0, "w": 1.0, "lr": 3.322127124853863e-06}
|
||||
{"step": 2250, "loss": 0.7951018810272217, "ema": 0.9377185967590448, "kl": 0.5043160707573406, "ce": 0.6434189320425503, "fmt": 0.0038814544677734375, "cons": 0.0, "w": 1.0, "lr": 3.096258618814396e-06}
|
||||
{"step": 2250, "eval_acc": 0.7158878504672898, "eval_dist": {"A": 0.267, "B": 0.239, "C": 0.264, "D": 0.23}}
|
||||
{"step": 2260, "loss": 0.6965787410736084, "ema": 0.9399988547092827, "kl": 0.6976374002406374, "ce": 0.4867567084438633, "fmt": 0.005308032035827637, "cons": 0.0, "w": 1.0, "lr": 2.878095858314278e-06}
|
||||
{"step": 2270, "loss": 0.6609507203102112, "ema": 0.9352464757091019, "kl": 0.5147816789103672, "ce": 0.5059337674756534, "fmt": 0.005825519561767578, "cons": 0.0, "w": 1.0, "lr": 2.667674684987259e-06}
|
||||
{"step": 2280, "loss": 1.0974622964859009, "ema": 0.9796178238397801, "kl": 0.946076450869441, "ce": 0.8131310641765594, "fmt": 0.005083918571472168, "cons": 0.0, "w": 1.0, "lr": 2.4650296686132613e-06}
|
||||
{"step": 2290, "loss": 0.9705954790115356, "ema": 0.9281236952673709, "kl": 0.7632014972623438, "ce": 0.741292234393768, "fmt": 0.0034276247024536133, "cons": 0.0, "w": 1.0, "lr": 2.2701941014390273e-06}
|
||||
{"step": 2300, "loss": 1.0255206823349, "ema": 0.9443821191783195, "kl": 0.9686025697737932, "ce": 0.7345379674807191, "fmt": 0.004018664360046387, "cons": 0.0, "w": 1.0, "lr": 2.0831999927085623e-06}
|
||||
{"step": 2300, "eval_acc": 0.7121495327102804, "eval_dist": {"A": 0.269, "B": 0.236, "C": 0.267, "D": 0.228}}
|
||||
{"step": 2310, "loss": 0.6288042068481445, "ema": 0.8963163122460324, "kl": 0.7068061750615016, "ce": 0.4161701463162899, "fmt": 0.0059223175048828125, "cons": 0.0, "w": 1.0, "lr": 1.904078063404391e-06}
|
||||
{"step": 2320, "loss": 0.6623179316520691, "ema": 0.972015866234655, "kl": 0.6142447185702622, "ce": 0.4776258715428412, "fmt": 0.004186868667602539, "cons": 0.0, "w": 1.0, "lr": 1.7328577412004443e-06}
|
||||
{"step": 2330, "loss": 1.2985947132110596, "ema": 0.9485849922790563, "kl": 0.9858946092426777, "ce": 1.0023797734174877, "fmt": 0.004466652870178223, "cons": 0.0, "w": 1.0, "lr": 1.5695671556274493e-06}
|
||||
{"step": 2340, "loss": 0.9378584027290344, "ema": 0.9384516023798293, "kl": 0.8545124586962629, "ce": 0.6809805996599607, "fmt": 0.005239725112915039, "cons": 0.0, "w": 1.0, "lr": 1.4142331334515424e-06}
|
||||
{"step": 2350, "loss": 0.98108971118927, "ema": 0.9498064782139921, "kl": 0.5689580787438899, "ce": 0.8098169611766934, "fmt": 0.005851626396179199, "cons": 0.0, "w": 1.0, "lr": 1.2668811942669724e-06}
|
||||
{"step": 2350, "eval_acc": 0.7102803738317757, "eval_dist": {"A": 0.26, "B": 0.239, "C": 0.265, "D": 0.236}}
|
||||
{"step": 2360, "loss": 1.172850489616394, "ema": 0.9021311578298782, "kl": 0.9143929071724415, "ce": 0.8981210563360946, "fmt": 0.004113912582397461, "cons": 0.0, "w": 1.0, "lr": 1.1275355463035042e-06}
|
||||
{"step": 2370, "loss": 0.9173603653907776, "ema": 0.9026715753092143, "kl": 0.7964734907727689, "ce": 0.6776970899663866, "fmt": 0.007212281227111816, "cons": 0.0, "w": 1.0, "lr": 9.962190824492846e-07}
|
||||
{"step": 2380, "loss": 0.9468204379081726, "ema": 0.9606141279733248, "kl": 0.8673666850663722, "ce": 0.6861447677947581, "fmt": 0.004656791687011719, "cons": 0.0, "w": 1.0, "lr": 8.729533764898045e-07}
|
||||
{"step": 2390, "loss": 1.398486614227295, "ema": 0.9710086528448276, "kl": 0.7175046168267727, "ce": 1.182759227231145, "fmt": 0.004760622978210449, "cons": 0.0, "w": 1.0, "lr": 7.577586795635971e-07}
|
||||
{"step": 2400, "loss": 1.1366047859191895, "ema": 0.9976484497182544, "kl": 0.7124329882208258, "ce": 0.9222342260181904, "fmt": 0.006406188011169434, "cons": 0.0, "w": 1.0, "lr": 6.506539168351699e-07}
|
||||
{"step": 2400, "eval_acc": 0.7158878504672898, "eval_dist": {"A": 0.26, "B": 0.245, "C": 0.262, "D": 0.234}}
|
||||
{"step": 2410, "loss": 0.6948422789573669, "ema": 0.9312113191557123, "kl": 0.5436200489057228, "ce": 0.5311127313179895, "fmt": 0.006434917449951172, "cons": 0.0, "w": 1.0, "lr": 5.516566843858428e-07}
|
||||
{"step": 2420, "loss": 0.8132132291793823, "ema": 0.86831906380254, "kl": 0.6030269990442321, "ce": 0.6314782669069245, "fmt": 0.008269190788269043, "cons": 0.0, "w": 1.0, "lr": 4.607832463229156e-07}
|
||||
{"step": 2430, "loss": 0.8760694861412048, "ema": 0.9472253016881012, "kl": 0.9402947849594057, "ce": 0.5935059282928705, "fmt": 0.004752039909362793, "cons": 0.0, "w": 1.0, "lr": 3.7804853210764433e-07}
|
||||
{"step": 2440, "loss": 0.5788076519966125, "ema": 0.9047623289195721, "kl": 0.9866771816741675, "ce": 0.2822344549931586, "fmt": 0.005700826644897461, "cons": 0.0, "w": 1.0, "lr": 3.034661341025258e-07}
|
||||
{"step": 2450, "loss": 1.2353615760803223, "ema": 1.0609633090764563, "kl": 0.6228184146457352, "ce": 1.0478786234743893, "fmt": 0.006376504898071289, "cons": 0.0, "w": 1.0, "lr": 2.3704830533821108e-07}
|
||||
{"step": 2450, "eval_acc": 0.7177570093457943, "eval_dist": {"A": 0.267, "B": 0.239, "C": 0.26, "D": 0.234}}
|
||||
{"step": 2460, "loss": 0.7137166261672974, "ema": 1.0245788094599555, "kl": 0.8294416752178222, "ce": 0.4640772065613419, "fmt": 0.008069157600402832, "cons": 0.0, "w": 1.0, "lr": 1.788059575004719e-07}
|
||||
{"step": 2470, "loss": 0.8918067216873169, "ema": 0.9642659944225382, "kl": 0.6995815054979175, "ce": 0.6809590498451144, "fmt": 0.00973200798034668, "cons": 0.0, "w": 1.0, "lr": 1.2874865913755086e-07}
|
||||
{"step": 2480, "loss": 0.9602398872375488, "ema": 0.9436868921343524, "kl": 0.7225836296565831, "ce": 0.7429778887890279, "fmt": 0.004868865013122559, "cons": 0.0, "w": 1.0, "lr": 8.688463408814684e-08}
|
||||
{"step": 2490, "loss": 1.497164249420166, "ema": 1.0230468088810063, "kl": 1.011690714454744, "ce": 1.192863191710785, "fmt": 0.007938742637634277, "cons": 0.0, "w": 1.0, "lr": 5.322076013034027e-08}
|
||||
{"step": 2500, "loss": 0.5975190997123718, "ema": 0.8491393630979834, "kl": 0.343520506285131, "ce": 0.49410383123904467, "fmt": 0.003590703010559082, "cons": 0.0, "w": 1.0, "lr": 2.7762567851652433e-08}
|
||||
{"step": 2500, "eval_acc": 0.7102803738317757, "eval_dist": {"A": 0.267, "B": 0.239, "C": 0.26, "D": 0.234}}
|
||||
{"step": 2510, "loss": 0.8149818778038025, "ema": 0.939465894121901, "kl": 0.3221083856187761, "ce": 0.717774510383606, "fmt": 0.005747318267822266, "cons": 0.0, "w": 1.0, "lr": 1.0514239740416765e-08}
|
||||
{"step": 2520, "loss": 0.7888774871826172, "ema": 0.9203178586171168, "kl": 0.8741680020466447, "ce": 0.5258921015774831, "fmt": 0.007349729537963867, "cons": 0.0, "w": 1.0, "lr": 1.478609498678507e-09}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2024d4f2b8afa1f6d8e3f706e9df9463eb1ceb0d8d1e38ea9ccce4a2aa487c87
|
||||
size 875554600
|
||||
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
Binary file not shown.
15
tokenizer_config.json
Normal file
15
tokenizer_config.json
Normal file
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<|begin_of_text|>",
|
||||
"clean_up_tokenization_spaces": true,
|
||||
"eos_token": "<|end_of_text|>",
|
||||
"is_local": true,
|
||||
"local_files_only": false,
|
||||
"model_input_names": [
|
||||
"input_ids",
|
||||
"attention_mask"
|
||||
],
|
||||
"model_max_length": 131072,
|
||||
"pad_token": "<|end_of_text|>",
|
||||
"tokenizer_class": "TokenizersBackend"
|
||||
}
|
||||
37
train_args.json
Normal file
37
train_args.json
Normal file
@@ -0,0 +1,37 @@
|
||||
{
|
||||
"config": "config/baseline.yaml",
|
||||
"teacher": "results/teacher_model_lang_pool_v2.jsonl",
|
||||
"model": "model",
|
||||
"out": "runs/elicit_A2_cleanteacher",
|
||||
"epochs": 6,
|
||||
"lr": 0.0001,
|
||||
"weight_decay": 0.0,
|
||||
"warmup_ratio": 0.03,
|
||||
"batch_size": 16,
|
||||
"grad_accum": 2,
|
||||
"max_len": 1024,
|
||||
"temp": 1.0,
|
||||
"lambda_ce": 1.0,
|
||||
"kl_weight": 0.3,
|
||||
"ce_target": "gold",
|
||||
"mu_format": 0.1,
|
||||
"gate_gamma": 0,
|
||||
"permute": true,
|
||||
"perm_k": 1,
|
||||
"lambda_cons": 0.0,
|
||||
"cons_conf_gamma": 0.0,
|
||||
"shots": 0,
|
||||
"shot_file": "/workspace/mii-zagreus-post-training-challenge/data/raw/train_shots.jsonl",
|
||||
"dtype": "bfloat16",
|
||||
"grad_ckpt": false,
|
||||
"device": "cuda",
|
||||
"limit": null,
|
||||
"log_every": 10,
|
||||
"save_every": 300,
|
||||
"keep_last": 6,
|
||||
"eval_every": 50,
|
||||
"eval_data": "data/processed/lang_dev_v2.jsonl",
|
||||
"eval_n": 600,
|
||||
"eval_shots": null,
|
||||
"smoke": false
|
||||
}
|
||||
Reference in New Issue
Block a user