初始化项目，由ModelHub XC社区提供模型

Model: GODELEV/Ant-10M Source: Original Platform
2026-07-30 00:40:20 +08:00
commit 6f78d9ccc9
14 changed files with 20294 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+Code_Generated_Image[[:space:]](1).png filter=lfs diff=lfs merge=lfs -text
+graph.png filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,163 @@
+---
+language: en
+tags:
+- causal-lm
+- gqa
+- rope
+- swiglu
+license: apache-2.0
+datasets:
+- GODELEV/Archaea-5M-T
+pipeline_tag: text-generation
+---
+
+# Ant-10M
+
+Ant-10M is a 9.90-million parameter, decoder-only Llama-style transformer model. It was designed, configured, and trained from scratch as a pure engineering sandbox. The primary objectives of this project were to explore the empirical boundaries of Small Language Model (SLM) scaling laws, evaluate extreme tokenizer constraints, test ultra-compact hidden representation geometries, and validate structural training loop stability on highly constrained hardware footprints.
+
+This model serves as a direct technical continuation of its predecessor, Ant-5M, implementing critical structural changes to prevent the architectural collapse observed in that earlier iteration and pushing the boundaries of what a sub-10M parameter network can stabilize.
+
+---
+
+## Important Disclaimer and Evaluation Frame
+
+**Ant-10M outputs absolute gibberish and possesses no semantic coherency, conversational capacity, structural grammar, or factual reasoning.**
+
+When interacting with this model or interpreting its metrics, keep the following engineering constraints in mind:
+
+1. **The Vocabulary Suffocation:** The model is trained using a highly restricted custom vocabulary size of 4,096 tokens. This forces standard English text to be aggressively shattered into microscopic character fragments and syllables during tokenization.
+2. **Perplexity Interpretation Trap:** The low validation perplexity achieved during training (`12.57`) is a **byte/token-level perplexity**, not a standard word-level perplexity. Because the tokenizer space is highly compressed, the model is optimizing over a narrow probability distribution of tiny token shards. Standard word-level evaluations (like WikiText-2) will register massive, exploding perplexity values (`88,520,100.69`) because the evaluation frameworks attempt to calculate probabilities over traditional word boundaries that do not exist within this model's narrow dictionary maps.
+
+This model is not a functional assistant. It is a mathematical log of a successful optimization and convergence experiment.
+
+---
+
+## Technical Architecture Specification
+
+Ant-10M scales the internal hidden representation width of the network while maintaining an efficient attention execution path. It relies on a balanced width-to-depth ratio designed to maximize token processing speed on consumer-tier systems.
+
+* **Total Parameters:** 9.90 Million (`9,902,464`)
+* **Layers (`num_hidden_layers`):** 12
+* **Hidden Size (`hidden_size`):** 256
+* **Intermediate Size (`intermediate_size`):** 704
+* **Attention Heads (`num_attention_heads`):** 4
+* **Key-Value Heads (`num_key_value_heads`):** 2 (Grouped-Query Attention ratio of 2:1)
+* **Head Dimension (`head_dim`):** 64
+* **Max Sequence Length (`max_position_embeddings`):** 1,024 tokens
+* **Vocabulary Size (`vocab_size`):** 4,096 (Custom trained BPE tokenizer)
+* **Activation Function:** SiLU (SwiGLU variant without linear biases)
+* **Positional Embeddings:** Rotary Position Embeddings (RoPE) with a native base frequency ($\theta$) of 10,000.0
+* **Weight Tying:** `tie_word_embeddings: true` (Input embedding and final output projection share an identical tensor matrix to optimize parameter allocation)
+
+---
+
+## Hardware and Training Infrastructure Metadata
+
+The model was successfully pre-trained in a single continuous session lasting **9.63 hours (approx. 10 hours)**.
+
+* **Hardware Used:** 1x NVIDIA T4 GPU (16GB VRAM) via Kaggle Compute Engine
+* **Tokens Seen:** 2,979,215,382 (~3 Billion tokens)
+* **Engine Velocity:** Steady operational throughput of **81,520 to 83,000 tokens per second**
+* **Precision:** `torch.float16` Automatic Mixed Precision (AMP)
+* **Optimization Framework:** AdamW Optimizer with a Cosine Learning Rate Decay Schedule and a linear warmup phase peaking at step 200 ($4.0 \times 10^{-4}$)
+
+---
+   <img src="graph.png" alt="Ant-10M Pre-training Metrics Summary" width="1000"/>
+
+---
+
+## Training Dynamics and Convergence Curves
+
+The training loop executed flawlessly without gradient explosions, numerical underflow, or loss divergence. The training loss and validation loss tracked each other with near-zero variance, demonstrating excellent data regularization across the 3 Billion token dataset.
+
+| Metrics | Step 80 (Initialization) | Step 200 (Warmup Peak) | Step 600 (Mid-Run) | Step 1200 (Final Convergence) |
+| --- | --- | --- | --- | --- |
+| **Training Loss** | 5.0837 | 3.8214 | 2.7231 | **2.5303** |
+| **Validation Loss** | — | 3.8174 | 2.7217 | **2.5314** |
+| **Token Perplexity** | 161.37 | 45.49 | 15.22 | **12.57** |
+| **Learning Rate** | $2.46 \times 10^{-4}$ | $4.00 \times 10^{-4}$ | $2.31 \times 10^{-4}$ | $4.29 \times 10^{-5}$ |
+| **Gradient Norm** | 2.0964 | 0.8142 | 0.4431 | 0.3189 |
+
+---
+
+## Downstream Benchmarks: A Comparative Post-Mortem
+
+To understand the developmental step forward taken by Ant-10M, its zero-shot performance is compared below against its older sibling, [Ant-5M](https://huggingface.co/GODELEV/Ant-5M).
+
+Ant-5M suffered a catastrophic structural collapse due to severe architectural imbalances—specifically, a microscopic hidden size (128) forced into an overly deep structure (11 layers) combined with an excessive Grouped-Query Attention bottleneck. This caused Ant-5M to trap itself in endless degenerate loops, repeating singular words like "Sciences" or URL punctuation constantly.
+
+Ant-10M completely eliminates these degenerate loops. However, because its vocabulary is still heavily compressed down to 4,096 tokens, it remains choked during standard language evaluations that rely on whole-word assemblies.
+
+### Standard Language Benchmarks
+
+| Benchmark Dataset | Metric Type | Ant-5M (The Catastrophe) | Ant-10M (One Step Ahead) |
+| --- | --- | --- | --- |
+| **ARC-Challenge** | `acc_norm` | 0.2442 (Below Random Guess) | **0.2747** (Above Random Guess) |
+| **ARC-Easy** | `acc_norm` | 0.2319 | **0.2542** |
+| **PIQA** | `acc_norm` | 0.4951 | **0.5032** |
+| **WinoGrande** | `acc` | 0.4885 | **0.4964** |
+| **MMLU** | `acc` | 0.2412 | **0.2543** |
+| **SciQ** | `acc_norm` | 0.1980 | **0.2150** |
+| **BoolQ** | `acc` | 0.3621 | **0.3782** |
+| **HellaSwag** | `acc_norm` | 0.2514 | **0.2672** |
+| **WikiText-2** | `byte_perplexity` | 48.91 | **30.62** |
+| **WikiText-2** | `word_perplexity` | Run Crashed / Diverged | **88,520,100.69** (Token Splitting Artifact) |
+
+### Mathematical Reasoning Evaluation: Arithmark-2.0
+
+Arithmark-2.0 evaluates the latent computational capacity of tiny models by asking them to solve basic arithmetic strings containing varying numbers of operators. Because multiple choice contains 4 potential variations, the random baseline floor is 25.0%.
+
+| Arithmark-2.0 Slice | Ant-5M Score | Ant-10M Score |
+| --- | --- | --- |
+| **Overall Accuracy** | 22.10% (Fails Floor) | **25.44%** (Crosses Floor) |
+| **1 Operator (Easy)** | 23.40% | **26.40%** |
+| **2 Operators (Medium)** | 21.90% | **26.93%** |
+| **3 Operators (Hard)** | 20.10% | **20.80%** |
+
+### Key Takeaways from the Data
+
+* **ARC-Challenge Progression:** Ant-5M scored below the random multiple-choice baseline (25.0%). Ant-10M breaks past the baseline to achieve **27.47%**, proving that widening the hidden dimension to 256 allowed the attention heads to actively map structural positioning signals instead of outputting repetitive tokens.
+* **Arithmark Numerical Floor:** While Ant-5M failed to maintain stable positioning math during mathematical syntax, Ant-10M managed to clear the 25% guessing baseline on 1-operator and 2-operator strings. At 3 operators, the context requirements of tracking multi-step parenthesis tokens exceeded the model's 256 hidden dimension capabilities, dropping accuracy back down to 20.80%.
+* **Byte-Perplexity Improvement:** The compression performance on raw character patterns improved significantly, dropping from 48.91 down to **30.62**, confirming high computational density inside the 12 transformer layers.
+
+---
+
+## Verification and Weights Inspection
+
+To verify the weights of Ant-10M, explore its layers, or inspect its token-fragment distribution outputs, use the standard Hugging Face Transformers pipeline as written below.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "GODELEV/Ant-10M"
+
+# Load the custom tokenizer and model architecture
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, 
+    torch_dtype=torch.float16, 
+    device_map="auto"
+)
+
+# Set up raw text input
+prompt = "The basic principles of small language models require"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+# Generate using high repetition penalties to counter the narrow vocabulary space
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs, 
+        max_new_tokens=32,
+        do_sample=True,
+        temperature=0.7,
+        top_p=0.9,
+        repetition_penalty=1.5
+    )
+
+# Decode tokens back into structural text fragments
+generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print("Generated Output Fragments:")
+print(generated_text)
+
+```
--- a/config.json
+++ b/config.json
@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 704,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 2,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.40.0",
+  "use_cache": true,
+  "vocab_size": 4096
+}
--- a/eval_results.json
+++ b/eval_results.json
@@ -0,0 +1,164 @@
+[
+  {
+    "task": "hellaswag",
+    "benchmark": "HellaSwag",
+    "metric": "acc_norm",
+    "score": 0.2672774347739494,
+    "shots": 0,
+    "runtime_sec": 152.49,
+    "status": "success"
+  },
+  {
+    "task": "piqa",
+    "benchmark": "PIQA",
+    "metric": "acc_norm",
+    "score": 0.5032644178454843,
+    "shots": 0,
+    "runtime_sec": 29.83,
+    "status": "success"
+  },
+  {
+    "task": "winogrande",
+    "benchmark": "WinoGrande",
+    "metric": "acc",
+    "score": 0.4964483030781373,
+    "shots": 0,
+    "runtime_sec": 22.5,
+    "status": "success"
+  },
+  {
+    "task": "boolq",
+    "benchmark": "BoolQ",
+    "metric": "acc",
+    "score": 0.3782874617737003,
+    "shots": 0,
+    "runtime_sec": 54.69,
+    "status": "success"
+  },
+  {
+    "task": "arc_easy",
+    "benchmark": "ARC-Easy",
+    "metric": "acc_norm",
+    "score": 0.2769360269360269,
+    "shots": 0,
+    "runtime_sec": 38.91,
+    "status": "success"
+  },
+  {
+    "task": "arc_challenge",
+    "benchmark": "ARC-Challenge",
+    "metric": "acc_norm",
+    "score": 0.27474402730375425,
+    "shots": 0,
+    "runtime_sec": 29.25,
+    "status": "success"
+  },
+  {
+    "task": "openbookqa",
+    "benchmark": "OpenBookQA",
+    "metric": "acc_norm",
+    "score": 0.308,
+    "shots": 0,
+    "runtime_sec": 23.18,
+    "status": "success"
+  },
+  {
+    "task": "commonsense_qa",
+    "benchmark": "CommonsenseQA",
+    "metric": "acc",
+    "score": 0.18591318591318592,
+    "shots": 0,
+    "runtime_sec": 27.0,
+    "status": "success"
+  },
+  {
+    "task": "lambada_openai",
+    "benchmark": "LAMBADA",
+    "metric": "acc",
+    "score": 0.0,
+    "shots": 0,
+    "runtime_sec": 71.7,
+    "status": "success"
+  },
+  {
+    "task": "blimp",
+    "benchmark": "BLiMP",
+    "metric": "acc",
+    "score": 0.5428358208955224,
+    "shots": 0,
+    "runtime_sec": 367.52,
+    "status": "success"
+  },
+  {
+    "task": "mmlu",
+    "benchmark": "MMLU",
+    "metric": "acc",
+    "score": 0.2543797179888905,
+    "shots": 0,
+    "runtime_sec": 295.22,
+    "status": "success"
+  },
+  {
+    "task": "wikitext",
+    "benchmark": "WikiText-2",
+    "metric": "word_perplexity",
+    "score": 88520100.69650024,
+    "shots": 0,
+    "runtime_sec": 34.96,
+    "status": "success"
+  },
+  {
+    "task": "wikitext",
+    "benchmark": "WikiText-2",
+    "metric": "byte_perplexity",
+    "score": 30.629263941602346,
+    "shots": 0,
+    "runtime_sec": 31.49,
+    "status": "success"
+  },
+  {
+    "task": "sciq",
+    "benchmark": "SciQ",
+    "metric": "acc_norm",
+    "score": 0.215,
+    "shots": 0,
+    "runtime_sec": 41.48,
+    "status": "success"
+  },
+  {
+    "task": "copa",
+    "benchmark": "COPA",
+    "metric": "acc",
+    "score": 0.57,
+    "shots": 0,
+    "runtime_sec": 18.99,
+    "status": "success"
+  },
+  {
+    "task": "race",
+    "benchmark": "RACE",
+    "metric": "acc",
+    "score": 0.22775119617224882,
+    "shots": 0,
+    "runtime_sec": 100.93,
+    "status": "success"
+  },
+  {
+    "task": "swag",
+    "benchmark": "SWAG",
+    "metric": "acc_norm",
+    "score": 0.2575227431770469,
+    "shots": 0,
+    "runtime_sec": 153.63,
+    "status": "success"
+  },
+  {
+    "task": "truthfulqa_mc2",
+    "benchmark": "TruthfulQA MC2",
+    "metric": "acc",
+    "score": 0.4874513485881811,
+    "shots": 0,
+    "runtime_sec": 49.19,
+    "status": "success"
+  }
+]
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,4 @@
+{
+  "bos_token_id": 2,
+  "eos_token_id": 3
+}
--- a/graph.png
+++ b/graph.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0436c6d8239b7e272f3180abae0e559139d82bac910f77730862feebd5d2914
+size 287873
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3ca9fbcda9b5a5cda37b78cf0d7d7edecca719c4b7a1fc835359d6cf6d8716a
+size 39621248
--- a/resume/ckpt.pt
+++ b/resume/ckpt.pt
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20e00e9fc1202d17f2aaf8da8e481a3575c90d5f49d6217c5057fb79e3587059
+size 122094685
--- a/resume/latest_step.txt
+++ b/resume/latest_step.txt
@@ -0,0 +1 @@
+1264
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,13 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|bos|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|eos|>",
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "<|mask|>",
+  "model_max_length": 4096,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|unk|>"
+}
--- a/train_log.jsonl
+++ b/train_log.jsonl
@@ -0,0 +1,67 @@
+{"event": "run_start", "step": 60, "max_steps": 1264, "model_params_M": 9.902, "eff_batch": 2304, "amp_dtype": "torch.float16", "attn_backend": "chunked", "started_at": "2026-06-11T06:31:39.618880"}
+{"step": 80, "epoch": 0.063, "loss": 5.08373, "ppl": 161.375, "lr": 0.000246875, "lr_sched": "cosine", "grad_norm": 2.0964, "tokens": 188559360, "tok_s": 300995.5, "elapsed_s": 626.5, "vram_gb": 0.59, "ram_pct": 16.5, "disk_free_gb": 20.94, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 100, "epoch": 0.079, "loss": 4.72502, "ppl": 112.733, "lr": 0.000309375, "lr_sched": "cosine", "grad_norm": 1.062, "tokens": 235699200, "tok_s": 193022.6, "elapsed_s": 1221.1, "vram_gb": 0.59, "ram_pct": 16.5, "disk_free_gb": 20.94, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 120, "epoch": 0.095, "loss": 4.51705, "ppl": 91.565, "lr": 0.000371875, "lr_sched": "cosine", "grad_norm": 0.5691, "tokens": 282839040, "tok_s": 155225.5, "elapsed_s": 1822.1, "vram_gb": 0.59, "ram_pct": 16.5, "disk_free_gb": 20.94, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 140, "epoch": 0.111, "loss": 4.35544, "ppl": 77.901, "lr": 0.000399917, "lr_sched": "cosine", "grad_norm": 1.0289, "tokens": 329978880, "tok_s": 136127.2, "elapsed_s": 2424.0, "vram_gb": 0.59, "ram_pct": 16.5, "disk_free_gb": 20.94, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 160, "epoch": 0.127, "loss": 4.17712, "ppl": 65.178, "lr": 0.000399339, "lr_sched": "cosine", "grad_norm": 0.9668, "tokens": 377118720, "tok_s": 124668.0, "elapsed_s": 3025.0, "vram_gb": 0.59, "ram_pct": 16.5, "disk_free_gb": 20.94, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 180, "epoch": 0.142, "loss": 4.00303, "ppl": 54.764, "lr": 0.000398213, "lr_sched": "cosine", "grad_norm": 1.5329, "tokens": 424258560, "tok_s": 116944.5, "elapsed_s": 3627.9, "vram_gb": 0.59, "ram_pct": 16.4, "disk_free_gb": 20.94, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 200, "epoch": 0.158, "loss": 3.81103, "ppl": 45.197, "lr": 0.000396541, "lr_sched": "cosine", "grad_norm": 0.8057, "tokens": 471398400, "tok_s": 111401.8, "elapsed_s": 4231.5, "vram_gb": 0.59, "ram_pct": 16.5, "disk_free_gb": 20.94, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 200, "epoch": 0.158, "val_loss": 3.81748, "val_ppl": 45.489, "best_val_ppl": 45.489, "is_best": true}
+{"step": 220, "epoch": 0.174, "loss": 3.65134, "ppl": 38.526, "lr": 0.00039433, "lr_sched": "cosine", "grad_norm": 1.0934, "tokens": 518538240, "tok_s": 106264.9, "elapsed_s": 4879.7, "vram_gb": 0.59, "ram_pct": 18.4, "disk_free_gb": 20.74, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 240, "epoch": 0.19, "loss": 3.52499, "ppl": 33.953, "lr": 0.000391586, "lr_sched": "cosine", "grad_norm": 1.079, "tokens": 565678080, "tok_s": 103073.1, "elapsed_s": 5488.1, "vram_gb": 0.59, "ram_pct": 18.4, "disk_free_gb": 20.74, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 260, "epoch": 0.206, "loss": 3.42217, "ppl": 30.636, "lr": 0.000388316, "lr_sched": "cosine", "grad_norm": 1.1757, "tokens": 612817920, "tok_s": 100537.3, "elapsed_s": 6095.4, "vram_gb": 0.59, "ram_pct": 18.4, "disk_free_gb": 20.74, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 280, "epoch": 0.222, "loss": 3.33892, "ppl": 28.189, "lr": 0.000384533, "lr_sched": "cosine", "grad_norm": 1.1653, "tokens": 659957760, "tok_s": 98485.9, "elapsed_s": 6701.0, "vram_gb": 0.59, "ram_pct": 18.4, "disk_free_gb": 20.74, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 300, "epoch": 0.237, "loss": 3.24706, "ppl": 25.715, "lr": 0.000380245, "lr_sched": "cosine", "grad_norm": 1.0345, "tokens": 707097600, "tok_s": 96760.4, "elapsed_s": 7307.7, "vram_gb": 0.59, "ram_pct": 18.4, "disk_free_gb": 20.74, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 320, "epoch": 0.253, "loss": 3.18689, "ppl": 24.213, "lr": 0.000375468, "lr_sched": "cosine", "grad_norm": 1.0814, "tokens": 754237440, "tok_s": 95305.2, "elapsed_s": 7913.9, "vram_gb": 0.59, "ram_pct": 18.5, "disk_free_gb": 20.74, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 340, "epoch": 0.269, "loss": 3.10296, "ppl": 22.264, "lr": 0.000370215, "lr_sched": "cosine", "grad_norm": 1.1011, "tokens": 801377280, "tok_s": 94040.1, "elapsed_s": 8521.7, "vram_gb": 0.59, "ram_pct": 18.5, "disk_free_gb": 20.74, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 360, "epoch": 0.285, "loss": 3.05335, "ppl": 21.186, "lr": 0.000364503, "lr_sched": "cosine", "grad_norm": 1.0818, "tokens": 848517120, "tok_s": 92932.0, "elapsed_s": 9130.5, "vram_gb": 0.59, "ram_pct": 18.5, "disk_free_gb": 20.74, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 380, "epoch": 0.301, "loss": 3.00831, "ppl": 20.253, "lr": 0.000358349, "lr_sched": "cosine", "grad_norm": 0.9618, "tokens": 895656960, "tok_s": 91956.6, "elapsed_s": 9740.0, "vram_gb": 0.59, "ram_pct": 18.5, "disk_free_gb": 20.74, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 400, "epoch": 0.316, "loss": 2.95097, "ppl": 19.124, "lr": 0.000351772, "lr_sched": "cosine", "grad_norm": 0.8669, "tokens": 942796800, "tok_s": 91115.9, "elapsed_s": 10347.2, "vram_gb": 0.59, "ram_pct": 18.5, "disk_free_gb": 20.74, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 400, "epoch": 0.316, "val_loss": 2.96685, "val_ppl": 19.431, "best_val_ppl": 19.431, "is_best": true}
+{"step": 420, "epoch": 0.332, "loss": 2.92597, "ppl": 18.652, "lr": 0.000344792, "lr_sched": "cosine", "grad_norm": 1.256, "tokens": 989936640, "tok_s": 90238.0, "elapsed_s": 10970.3, "vram_gb": 0.59, "ram_pct": 18.6, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 440, "epoch": 0.348, "loss": 2.89415, "ppl": 18.068, "lr": 0.00033743, "lr_sched": "cosine", "grad_norm": 1.0429, "tokens": 1037076480, "tok_s": 89573.8, "elapsed_s": 11577.9, "vram_gb": 0.59, "ram_pct": 18.7, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 460, "epoch": 0.364, "loss": 2.8546, "ppl": 17.368, "lr": 0.000329709, "lr_sched": "cosine", "grad_norm": 1.1318, "tokens": 1084216320, "tok_s": 88970.2, "elapsed_s": 12186.3, "vram_gb": 0.59, "ram_pct": 18.6, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 480, "epoch": 0.38, "loss": 2.82379, "ppl": 16.841, "lr": 0.000321652, "lr_sched": "cosine", "grad_norm": 1.166, "tokens": 1131356160, "tok_s": 88407.9, "elapsed_s": 12797.0, "vram_gb": 0.59, "ram_pct": 18.7, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 500, "epoch": 0.396, "loss": 2.80073, "ppl": 16.457, "lr": 0.000313285, "lr_sched": "cosine", "grad_norm": 0.9405, "tokens": 1178496000, "tok_s": 87904.7, "elapsed_s": 13406.5, "vram_gb": 0.59, "ram_pct": 18.7, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 520, "epoch": 0.411, "loss": 2.77799, "ppl": 16.087, "lr": 0.000304632, "lr_sched": "cosine", "grad_norm": 1.0248, "tokens": 1225635840, "tok_s": 87461.3, "elapsed_s": 14013.5, "vram_gb": 0.59, "ram_pct": 18.7, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 540, "epoch": 0.427, "loss": 2.75765, "ppl": 15.763, "lr": 0.00029572, "lr_sched": "cosine", "grad_norm": 0.7223, "tokens": 1272775680, "tok_s": 87054.8, "elapsed_s": 14620.4, "vram_gb": 0.59, "ram_pct": 18.7, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 560, "epoch": 0.443, "loss": 2.75917, "ppl": 15.787, "lr": 0.000286577, "lr_sched": "cosine", "grad_norm": 1.0008, "tokens": 1319915520, "tok_s": 86692.0, "elapsed_s": 15225.3, "vram_gb": 0.59, "ram_pct": 18.7, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 580, "epoch": 0.459, "loss": 2.72373, "ppl": 15.237, "lr": 0.00027723, "lr_sched": "cosine", "grad_norm": 0.8632, "tokens": 1367055360, "tok_s": 86345.9, "elapsed_s": 15832.3, "vram_gb": 0.59, "ram_pct": 18.8, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 600, "epoch": 0.475, "loss": 2.71745, "ppl": 15.142, "lr": 0.000267708, "lr_sched": "cosine", "grad_norm": 1.0309, "tokens": 1414195200, "tok_s": 85995.5, "elapsed_s": 16445.0, "vram_gb": 0.59, "ram_pct": 18.8, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 600, "epoch": 0.475, "val_loss": 2.72172, "val_ppl": 15.206, "best_val_ppl": 15.206, "is_best": true}
+{"step": 620, "epoch": 0.491, "loss": 2.70782, "ppl": 14.997, "lr": 0.000258041, "lr_sched": "cosine", "grad_norm": 0.9506, "tokens": 1461335040, "tok_s": 85624.3, "elapsed_s": 17066.8, "vram_gb": 0.59, "ram_pct": 18.8, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 640, "epoch": 0.506, "loss": 2.69895, "ppl": 14.864, "lr": 0.000248257, "lr_sched": "cosine", "grad_norm": 1.1735, "tokens": 1508474880, "tok_s": 85357.9, "elapsed_s": 17672.4, "vram_gb": 0.59, "ram_pct": 18.8, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 660, "epoch": 0.522, "loss": 2.66671, "ppl": 14.392, "lr": 0.000238386, "lr_sched": "cosine", "grad_norm": 0.8704, "tokens": 1555614720, "tok_s": 85108.2, "elapsed_s": 18278.1, "vram_gb": 0.59, "ram_pct": 18.9, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 680, "epoch": 0.538, "loss": 2.6486, "ppl": 14.134, "lr": 0.000228459, "lr_sched": "cosine", "grad_norm": 0.638, "tokens": 1602754560, "tok_s": 84878.7, "elapsed_s": 18882.9, "vram_gb": 0.59, "ram_pct": 18.8, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 700, "epoch": 0.554, "loss": 2.65948, "ppl": 14.289, "lr": 0.000218507, "lr_sched": "cosine", "grad_norm": 0.8346, "tokens": 1649894400, "tok_s": 84661.6, "elapsed_s": 19488.1, "vram_gb": 0.59, "ram_pct": 18.8, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 720, "epoch": 0.57, "loss": 2.64185, "ppl": 14.039, "lr": 0.000208559, "lr_sched": "cosine", "grad_norm": 0.7506, "tokens": 1697034240, "tok_s": 84454.2, "elapsed_s": 20094.1, "vram_gb": 0.59, "ram_pct": 18.8, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 740, "epoch": 0.585, "loss": 2.63378, "ppl": 13.926, "lr": 0.000198646, "lr_sched": "cosine", "grad_norm": 0.7747, "tokens": 1744174080, "tok_s": 84261.2, "elapsed_s": 20699.6, "vram_gb": 0.59, "ram_pct": 18.9, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 760, "epoch": 0.601, "loss": 2.60751, "ppl": 13.565, "lr": 0.000188798, "lr_sched": "cosine", "grad_norm": 0.7297, "tokens": 1791313920, "tok_s": 84076.3, "elapsed_s": 21305.8, "vram_gb": 0.59, "ram_pct": 18.9, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 780, "epoch": 0.617, "loss": 2.61043, "ppl": 13.605, "lr": 0.000179045, "lr_sched": "cosine", "grad_norm": 0.578, "tokens": 1838453760, "tok_s": 83899.8, "elapsed_s": 21912.5, "vram_gb": 0.59, "ram_pct": 18.9, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 800, "epoch": 0.633, "loss": 2.60656, "ppl": 13.552, "lr": 0.000169418, "lr_sched": "cosine", "grad_norm": 0.807, "tokens": 1885593600, "tok_s": 83729.7, "elapsed_s": 22520.0, "vram_gb": 0.59, "ram_pct": 18.9, "disk_free_gb": 20.61, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 800, "epoch": 0.633, "val_loss": 2.612, "val_ppl": 13.626, "best_val_ppl": 13.626, "is_best": true}
+{"step": 820, "epoch": 0.649, "loss": 2.60156, "ppl": 13.485, "lr": 0.000159946, "lr_sched": "cosine", "grad_norm": 0.7742, "tokens": 1932733440, "tok_s": 83511.8, "elapsed_s": 23143.2, "vram_gb": 0.59, "ram_pct": 19.0, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 840, "epoch": 0.665, "loss": 2.58708, "ppl": 13.291, "lr": 0.000150657, "lr_sched": "cosine", "grad_norm": 0.7649, "tokens": 1979873280, "tok_s": 83361.5, "elapsed_s": 23750.4, "vram_gb": 0.59, "ram_pct": 19.0, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 860, "epoch": 0.68, "loss": 2.58372, "ppl": 13.246, "lr": 0.000141581, "lr_sched": "cosine", "grad_norm": 0.7413, "tokens": 2027013120, "tok_s": 83226.7, "elapsed_s": 24355.3, "vram_gb": 0.59, "ram_pct": 19.1, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 880, "epoch": 0.696, "loss": 2.58026, "ppl": 13.201, "lr": 0.000132744, "lr_sched": "cosine", "grad_norm": 0.8835, "tokens": 2074152960, "tok_s": 83104.9, "elapsed_s": 24958.2, "vram_gb": 0.59, "ram_pct": 19.0, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 900, "epoch": 0.712, "loss": 2.57021, "ppl": 13.069, "lr": 0.000124174, "lr_sched": "cosine", "grad_norm": 0.5684, "tokens": 2121292800, "tok_s": 82976.5, "elapsed_s": 25565.0, "vram_gb": 0.59, "ram_pct": 19.0, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 920, "epoch": 0.728, "loss": 2.57673, "ppl": 13.154, "lr": 0.000115897, "lr_sched": "cosine", "grad_norm": 0.5326, "tokens": 2168432640, "tok_s": 82856.0, "elapsed_s": 26171.1, "vram_gb": 0.59, "ram_pct": 19.1, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 940, "epoch": 0.744, "loss": 2.55775, "ppl": 12.907, "lr": 0.000107939, "lr_sched": "cosine", "grad_norm": 0.7495, "tokens": 2215572480, "tok_s": 82735.0, "elapsed_s": 26779.2, "vram_gb": 0.59, "ram_pct": 19.1, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 960, "epoch": 0.759, "loss": 2.56307, "ppl": 12.976, "lr": 0.000100323, "lr_sched": "cosine", "grad_norm": 0.5409, "tokens": 2262712320, "tok_s": 82604.2, "elapsed_s": 27392.2, "vram_gb": 0.59, "ram_pct": 19.1, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 980, "epoch": 0.775, "loss": 2.54577, "ppl": 12.753, "lr": 9.3073e-05, "lr_sched": "cosine", "grad_norm": 0.588, "tokens": 2309852160, "tok_s": 82500.8, "elapsed_s": 27997.9, "vram_gb": 0.59, "ram_pct": 19.1, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1000, "epoch": 0.791, "loss": 2.55359, "ppl": 12.853, "lr": 8.6212e-05, "lr_sched": "cosine", "grad_norm": 0.6079, "tokens": 2356992000, "tok_s": 82402.7, "elapsed_s": 28603.3, "vram_gb": 0.59, "ram_pct": 19.1, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1000, "epoch": 0.791, "val_loss": 2.55587, "val_ppl": 12.883, "best_val_ppl": 12.883, "is_best": true}
+{"step": 1020, "epoch": 0.807, "loss": 2.30778, "ppl": 10.052, "lr": 7.9759e-05, "lr_sched": "cosine", "grad_norm": 1.9184, "tokens": 2404131840, "tok_s": 82260.5, "elapsed_s": 29225.8, "vram_gb": 0.59, "ram_pct": 19.1, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1040, "epoch": 0.823, "loss": 2.2168, "ppl": 9.178, "lr": 7.3736e-05, "lr_sched": "cosine", "grad_norm": 0.7887, "tokens": 2451271680, "tok_s": 82169.8, "elapsed_s": 29831.8, "vram_gb": 0.59, "ram_pct": 19.2, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1060, "epoch": 0.839, "loss": 2.56972, "ppl": 13.062, "lr": 6.816e-05, "lr_sched": "cosine", "grad_norm": 0.5087, "tokens": 2498411520, "tok_s": 82073.2, "elapsed_s": 30441.3, "vram_gb": 0.59, "ram_pct": 19.1, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1080, "epoch": 0.854, "loss": 2.55791, "ppl": 12.909, "lr": 6.3048e-05, "lr_sched": "cosine", "grad_norm": 0.4667, "tokens": 2545551360, "tok_s": 81982.1, "elapsed_s": 31050.1, "vram_gb": 0.59, "ram_pct": 19.1, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1100, "epoch": 0.87, "loss": 2.54725, "ppl": 12.772, "lr": 5.8416e-05, "lr_sched": "cosine", "grad_norm": 0.4463, "tokens": 2592691200, "tok_s": 81900.2, "elapsed_s": 31656.7, "vram_gb": 0.59, "ram_pct": 19.2, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1120, "epoch": 0.886, "loss": 2.52993, "ppl": 12.553, "lr": 5.4279e-05, "lr_sched": "cosine", "grad_norm": 0.4327, "tokens": 2639831040, "tok_s": 81821.8, "elapsed_s": 32263.2, "vram_gb": 0.59, "ram_pct": 19.2, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1140, "epoch": 0.902, "loss": 2.53771, "ppl": 12.651, "lr": 5.0648e-05, "lr_sched": "cosine", "grad_norm": 0.3917, "tokens": 2686970880, "tok_s": 81743.1, "elapsed_s": 32870.9, "vram_gb": 0.59, "ram_pct": 19.2, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1160, "epoch": 0.918, "loss": 2.53772, "ppl": 12.651, "lr": 4.7535e-05, "lr_sched": "cosine", "grad_norm": 0.4292, "tokens": 2734110720, "tok_s": 81671.3, "elapsed_s": 33477.0, "vram_gb": 0.59, "ram_pct": 19.2, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1180, "epoch": 0.934, "loss": 2.53827, "ppl": 12.658, "lr": 4.495e-05, "lr_sched": "cosine", "grad_norm": 0.4724, "tokens": 2781250560, "tok_s": 81602.0, "elapsed_s": 34083.1, "vram_gb": 0.59, "ram_pct": 19.2, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1200, "epoch": 0.949, "loss": 2.53036, "ppl": 12.558, "lr": 4.29e-05, "lr_sched": "cosine", "grad_norm": 0.3189, "tokens": 2828390400, "tok_s": 81520.6, "elapsed_s": 34695.4, "vram_gb": 0.59, "ram_pct": 19.2, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1200, "epoch": 0.949, "val_loss": 2.53144, "val_ppl": 12.572, "best_val_ppl": 12.572, "is_best": true}
+{"step": 1220, "epoch": 0.965, "loss": 2.52293, "ppl": 12.465, "lr": 4.1392e-05, "lr_sched": "cosine", "grad_norm": 0.402, "tokens": 2875530240, "tok_s": 81412.0, "elapsed_s": 35320.7, "vram_gb": 0.59, "ram_pct": 19.3, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1240, "epoch": 0.981, "loss": 2.53116, "ppl": 12.568, "lr": 4.043e-05, "lr_sched": "cosine", "grad_norm": 0.3883, "tokens": 2922670080, "tok_s": 81346.2, "elapsed_s": 35928.8, "vram_gb": 0.59, "ram_pct": 19.3, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
+{"step": 1260, "epoch": 0.997, "loss": 2.52288, "ppl": 12.464, "lr": 4.0017e-05, "lr_sched": "cosine", "grad_norm": 0.4029, "tokens": 2969809920, "tok_s": 81286.3, "elapsed_s": 36535.2, "vram_gb": 0.59, "ram_pct": 19.3, "disk_free_gb": 20.49, "attn_backend": "chunked", "amp_dtype": "torch.float16"}
--- a/train_log.log
+++ b/train_log.log
@@ -0,0 +1,182 @@
+2026-06-11 06:29:54,615 | T4/P100 (Tesla T4 sm_75): Flash unavailable, using chunked attn
+2026-06-11 06:29:54,875 | HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK"
+2026-06-11 06:29:54,952 | HTTP Request: HEAD https://huggingface.co/GODELEV/TOK-4K/resolve/main/config.json "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:55,021 | HTTP Request: HEAD https://huggingface.co/GODELEV/TOK-4K/resolve/main/config.json "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:55,095 | HTTP Request: HEAD https://huggingface.co/GODELEV/TOK-4K/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
+2026-06-11 06:29:55,111 | HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/GODELEV/TOK-4K/0a93937fbb072e0b839a0ae1902127e0d22b872f/tokenizer_config.json "HTTP/1.1 200 OK"
+2026-06-11 06:29:55,129 | HTTP Request: GET https://huggingface.co/api/resolve-cache/models/GODELEV/TOK-4K/0a93937fbb072e0b839a0ae1902127e0d22b872f/tokenizer_config.json "HTTP/1.1 200 OK"
+2026-06-11 06:29:55,213 | HTTP Request: GET https://huggingface.co/api/models/GODELEV/TOK-4K/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:55,276 | HTTP Request: GET https://huggingface.co/api/models/GODELEV/TOK-4K/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
+2026-06-11 06:29:55,344 | HTTP Request: HEAD https://huggingface.co/GODELEV/TOK-4K/resolve/main/tokenizer.json "HTTP/1.1 307 Temporary Redirect"
+2026-06-11 06:29:55,361 | HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/GODELEV/TOK-4K/0a93937fbb072e0b839a0ae1902127e0d22b872f/tokenizer.json "HTTP/1.1 200 OK"
+2026-06-11 06:29:55,381 | HTTP Request: GET https://huggingface.co/api/resolve-cache/models/GODELEV/TOK-4K/0a93937fbb072e0b839a0ae1902127e0d22b872f/tokenizer.json "HTTP/1.1 200 OK"
+2026-06-11 06:29:55,461 | HTTP Request: HEAD https://huggingface.co/GODELEV/TOK-4K/resolve/main/tokenizer.model "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:55,541 | HTTP Request: HEAD https://huggingface.co/GODELEV/TOK-4K/resolve/main/added_tokens.json "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:55,610 | HTTP Request: HEAD https://huggingface.co/GODELEV/TOK-4K/resolve/main/special_tokens_map.json "HTTP/1.1 307 Temporary Redirect"
+2026-06-11 06:29:55,626 | HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/GODELEV/TOK-4K/0a93937fbb072e0b839a0ae1902127e0d22b872f/special_tokens_map.json "HTTP/1.1 200 OK"
+2026-06-11 06:29:55,644 | HTTP Request: GET https://huggingface.co/api/resolve-cache/models/GODELEV/TOK-4K/0a93937fbb072e0b839a0ae1902127e0d22b872f/special_tokens_map.json "HTTP/1.1 200 OK"
+2026-06-11 06:29:55,723 | HTTP Request: HEAD https://huggingface.co/GODELEV/TOK-4K/resolve/main/chat_template.jinja "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:55,838 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/main/README.md "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:56,059 | HTTP Request: GET https://huggingface.co/api/datasets/GODELEV/Ant-5M-V2-T "HTTP/1.1 200 OK"
+2026-06-11 06:29:56,127 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/Ant-5M-V2-T.py "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:56,250 | HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/GODELEV/Ant-5M-V2-T/GODELEV/Ant-5M-V2-T.py "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:56,320 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/README.md "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:56,387 | HTTP Request: GET https://huggingface.co/api/datasets/GODELEV/Ant-5M-V2-T/revision/bbdd76ffdd523212181b29971d8c6e834aa1c5ea "HTTP/1.1 200 OK"
+2026-06-11 06:29:56,452 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/.huggingface.yaml "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:56,609 | HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=GODELEV/Ant-5M-V2-T "HTTP/1.1 200 OK"
+2026-06-11 06:29:56,709 | HTTP Request: GET https://huggingface.co/api/datasets/GODELEV/Ant-5M-V2-T/tree/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/data?recursive=true&expand=false "HTTP/1.1 200 OK"
+2026-06-11 06:29:56,787 | HTTP Request: GET https://huggingface.co/api/datasets/GODELEV/Ant-5M-V2-T/tree/bbdd76ffdd523212181b29971d8c6e834aa1c5ea?recursive=false&expand=false "HTTP/1.1 200 OK"
+2026-06-11 06:29:56,859 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/dataset_infos.json "HTTP/1.1 404 Not Found"
+2026-06-11 06:29:56,970 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/data/train-00000-of-00006.parquet "HTTP/1.1 302 Found"
+2026-06-11 06:29:57,076 | HTTP Request: GET https://huggingface.co/api/datasets/GODELEV/Ant-5M-V2-T/xet-read-token/bbdd76ffdd523212181b29971d8c6e834aa1c5ea "HTTP/1.1 200 OK"
+2026-06-11 06:30:00,130 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/data/train-00001-of-00006.parquet "HTTP/1.1 302 Found"
+2026-06-11 06:30:04,826 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/data/train-00002-of-00006.parquet "HTTP/1.1 302 Found"
+2026-06-11 06:30:07,920 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/data/train-00003-of-00006.parquet "HTTP/1.1 302 Found"
+2026-06-11 06:30:12,862 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/data/train-00004-of-00006.parquet "HTTP/1.1 302 Found"
+2026-06-11 06:30:15,761 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/data/train-00005-of-00006.parquet "HTTP/1.1 302 Found"
+2026-06-11 06:30:19,862 | HTTP Request: HEAD https://huggingface.co/datasets/GODELEV/Ant-5M-V2-T/resolve/bbdd76ffdd523212181b29971d8c6e834aa1c5ea/data/val-00000-of-00001.parquet "HTTP/1.1 302 Found"
+2026-06-11 06:31:27,680 | Data loaded train=2,910,746 val=20,000 steps_per_epoch=1,264 total_epochs=1.00
+2026-06-11 06:31:31,643 | Model 9.902M params | dtype=float32 | amp=torch.float16
+2026-06-11 06:31:31,877 | HTTP Request: HEAD https://huggingface.co/GODELEV/Experimenting/resolve/main/resume/latest_step.txt "HTTP/1.1 307 Temporary Redirect"
+2026-06-11 06:31:32,027 | HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/GODELEV/Experimenting/73dbdd890e76f238232c2d5fed8e4927a5b510f4/resume%2Flatest_step.txt "HTTP/1.1 200 OK"
+2026-06-11 06:31:32,131 | HTTP Request: GET https://huggingface.co/api/resolve-cache/models/GODELEV/Experimenting/73dbdd890e76f238232c2d5fed8e4927a5b510f4/resume%2Flatest_step.txt "HTTP/1.1 200 OK"
+2026-06-11 06:31:32,217 | HTTP Request: HEAD https://huggingface.co/GODELEV/Experimenting/resolve/main/resume/ckpt.pt "HTTP/1.1 302 Found"
+2026-06-11 06:31:32,288 | HTTP Request: GET https://huggingface.co/api/models/GODELEV/Experimenting/xet-read-token/73dbdd890e76f238232c2d5fed8e4927a5b510f4 "HTTP/1.1 200 OK"
+2026-06-11 06:31:39,439 | Resumed step=60 tokens=141419520 samples=138240
+2026-06-11 06:42:05,961 | step=    80 | epoch=0.06 | loss=5.0837 | ppl=161.38 | lr=2.47e-04 | grad=2.096 | tok/s=300,996 | ETA=9:43:51 | VRAM=0.6GB | RAM=16%
+2026-06-11 06:52:00,604 | step=   100 | epoch=0.08 | loss=4.7250 | ppl=112.73 | lr=3.09e-04 | grad=1.062 | tok/s=193,023 | ETA=9:41:58 | VRAM=0.6GB | RAM=16%
+2026-06-11 07:02:01,626 | step=   120 | epoch=0.09 | loss=4.5171 | ppl=91.57 | lr=3.72e-04 | grad=0.569 | tok/s=155,225 | ETA=9:33:06 | VRAM=0.6GB | RAM=16%
+2026-06-11 07:12:03,556 | step=   140 | epoch=0.11 | loss=4.3554 | ppl=77.90 | lr=4.00e-04 | grad=1.029 | tok/s=136,127 | ETA=9:22:35 | VRAM=0.6GB | RAM=16%
+2026-06-11 07:22:04,493 | step=   160 | epoch=0.13 | loss=4.1771 | ppl=65.18 | lr=3.99e-04 | grad=0.967 | tok/s=124,668 | ETA=9:12:52 | VRAM=0.6GB | RAM=16%
+2026-06-11 07:32:07,371 | step=   180 | epoch=0.14 | loss=4.0030 | ppl=54.76 | lr=3.98e-04 | grad=1.533 | tok/s=116,944 | ETA=9:08:54 | VRAM=0.6GB | RAM=16%
+2026-06-11 07:42:11,025 | step=   200 | epoch=0.16 | loss=3.8110 | ppl=45.20 | lr=3.97e-04 | grad=0.806 | tok/s=111,402 | ETA=8:59:30 | VRAM=0.6GB | RAM=16%
+2026-06-11 07:42:55,395 | VAL step=200 epoch=0.16 loss=3.8175 ppl=45.49  BEST
+2026-06-11 07:42:55,474 | Saved safetensors: 110 tensors (tied=True, embed_key=model.embed_tokens.weight, lm_head omitted — HF will tie)
+2026-06-11 07:42:55,791 | HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict"
+2026-06-11 07:42:55,853 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 07:42:56,387 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 07:42:56,528 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/preupload/main "HTTP/1.1 200 OK"
+2026-06-11 07:42:56,641 | HTTP Request: POST https://huggingface.co/GODELEV/Experimenting.git/info/lfs/objects/batch "HTTP/1.1 200 OK"
+2026-06-11 07:42:56,710 | HTTP Request: GET https://huggingface.co/api/models/GODELEV/Experimenting/xet-write-token/main "HTTP/1.1 200 OK"
+2026-06-11 07:43:04,316 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/commit/main "HTTP/1.1 200 OK"
+2026-06-11 07:43:04,319 | Hub push step=200
+2026-06-11 07:52:59,184 | step=   220 | epoch=0.17 | loss=3.6513 | ppl=38.53 | lr=3.94e-04 | grad=1.093 | tok/s=106,265 | ETA=8:49:23 | VRAM=0.6GB | RAM=18%
+2026-06-11 08:03:07,634 | step=   240 | epoch=0.19 | loss=3.5250 | ppl=33.95 | lr=3.92e-04 | grad=1.079 | tok/s=103,073 | ETA=8:36:29 | VRAM=0.6GB | RAM=18%
+2026-06-11 08:13:14,936 | step=   260 | epoch=0.21 | loss=3.4222 | ppl=30.64 | lr=3.88e-04 | grad=1.176 | tok/s=100,537 | ETA=8:24:10 | VRAM=0.6GB | RAM=18%
+2026-06-11 08:23:20,546 | step=   280 | epoch=0.22 | loss=3.3389 | ppl=28.19 | lr=3.85e-04 | grad=1.165 | tok/s=98,486 | ETA=8:13:04 | VRAM=0.6GB | RAM=18%
+2026-06-11 08:33:27,224 | step=   300 | epoch=0.24 | loss=3.2471 | ppl=25.71 | lr=3.80e-04 | grad=1.034 | tok/s=96,760 | ETA=8:09:26 | VRAM=0.6GB | RAM=18%
+2026-06-11 08:43:33,425 | step=   320 | epoch=0.25 | loss=3.1869 | ppl=24.21 | lr=3.75e-04 | grad=1.081 | tok/s=95,305 | ETA=8:00:20 | VRAM=0.6GB | RAM=18%
+2026-06-11 08:53:41,161 | step=   340 | epoch=0.27 | loss=3.1030 | ppl=22.26 | lr=3.70e-04 | grad=1.101 | tok/s=94,040 | ETA=7:46:57 | VRAM=0.6GB | RAM=18%
+2026-06-11 09:03:50,023 | step=   360 | epoch=0.28 | loss=3.0533 | ppl=21.19 | lr=3.65e-04 | grad=1.082 | tok/s=92,932 | ETA=7:40:25 | VRAM=0.6GB | RAM=18%
+2026-06-11 09:13:59,503 | step=   380 | epoch=0.30 | loss=3.0083 | ppl=20.25 | lr=3.58e-04 | grad=0.962 | tok/s=91,957 | ETA=7:26:44 | VRAM=0.6GB | RAM=18%
+2026-06-11 09:24:06,735 | step=   400 | epoch=0.32 | loss=2.9510 | ppl=19.12 | lr=3.52e-04 | grad=0.867 | tok/s=91,116 | ETA=7:19:46 | VRAM=0.6GB | RAM=18%
+2026-06-11 09:24:17,996 | VAL step=400 epoch=0.32 loss=2.9669 ppl=19.43  BEST
+2026-06-11 09:24:18,177 | Checkpoint saved: step 400
+2026-06-11 09:24:18,234 | Saved safetensors: 110 tensors (tied=True, embed_key=model.embed_tokens.weight, lm_head omitted — HF will tie)
+2026-06-11 09:24:18,610 | HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict"
+2026-06-11 09:24:18,674 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 09:24:19,160 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 09:24:19,329 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/preupload/main "HTTP/1.1 200 OK"
+2026-06-11 09:24:19,407 | HTTP Request: POST https://huggingface.co/GODELEV/Experimenting.git/info/lfs/objects/batch "HTTP/1.1 200 OK"
+2026-06-11 09:24:19,468 | HTTP Request: GET https://huggingface.co/api/models/GODELEV/Experimenting/xet-write-token/main "HTTP/1.1 200 OK"
+2026-06-11 09:24:25,139 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/commit/main "HTTP/1.1 200 OK"
+2026-06-11 09:24:25,141 | Hub push step=400
+2026-06-11 09:34:29,791 | step=   420 | epoch=0.33 | loss=2.9260 | ppl=18.65 | lr=3.45e-04 | grad=1.256 | tok/s=90,238 | ETA=7:08:34 | VRAM=0.6GB | RAM=19%
+2026-06-11 09:44:37,407 | step=   440 | epoch=0.35 | loss=2.8942 | ppl=18.07 | lr=3.37e-04 | grad=1.043 | tok/s=89,574 | ETA=6:58:37 | VRAM=0.6GB | RAM=19%
+2026-06-11 09:54:45,798 | step=   460 | epoch=0.36 | loss=2.8546 | ppl=17.37 | lr=3.30e-04 | grad=1.132 | tok/s=88,970 | ETA=6:49:23 | VRAM=0.6GB | RAM=19%
+2026-06-11 10:04:56,517 | step=   480 | epoch=0.38 | loss=2.8238 | ppl=16.84 | lr=3.22e-04 | grad=1.166 | tok/s=88,408 | ETA=6:37:36 | VRAM=0.6GB | RAM=19%
+2026-06-11 10:15:06,029 | step=   500 | epoch=0.40 | loss=2.8007 | ppl=16.46 | lr=3.13e-04 | grad=0.940 | tok/s=87,905 | ETA=6:28:57 | VRAM=0.6GB | RAM=19%
+2026-06-11 10:25:12,965 | step=   520 | epoch=0.41 | loss=2.7780 | ppl=16.09 | lr=3.05e-04 | grad=1.025 | tok/s=87,461 | ETA=6:17:09 | VRAM=0.6GB | RAM=19%
+2026-06-11 10:35:19,899 | step=   540 | epoch=0.43 | loss=2.7576 | ppl=15.76 | lr=2.96e-04 | grad=0.722 | tok/s=87,055 | ETA=6:06:13 | VRAM=0.6GB | RAM=19%
+2026-06-11 10:45:24,846 | step=   560 | epoch=0.44 | loss=2.7592 | ppl=15.79 | lr=2.87e-04 | grad=1.001 | tok/s=86,692 | ETA=5:56:15 | VRAM=0.6GB | RAM=19%
+2026-06-11 10:55:31,823 | step=   580 | epoch=0.46 | loss=2.7237 | ppl=15.24 | lr=2.77e-04 | grad=0.863 | tok/s=86,346 | ETA=5:48:34 | VRAM=0.6GB | RAM=19%
+2026-06-11 11:05:44,507 | step=   600 | epoch=0.47 | loss=2.7174 | ppl=15.14 | lr=2.68e-04 | grad=1.031 | tok/s=85,995 | ETA=5:39:35 | VRAM=0.6GB | RAM=19%
+2026-06-11 11:05:55,800 | VAL step=600 epoch=0.47 loss=2.7217 ppl=15.21  BEST
+2026-06-11 11:05:55,855 | Saved safetensors: 110 tensors (tied=True, embed_key=model.embed_tokens.weight, lm_head omitted — HF will tie)
+2026-06-11 11:05:56,230 | HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict"
+2026-06-11 11:05:56,288 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 11:05:56,771 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 11:05:56,875 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/preupload/main "HTTP/1.1 200 OK"
+2026-06-11 11:05:56,950 | HTTP Request: POST https://huggingface.co/GODELEV/Experimenting.git/info/lfs/objects/batch "HTTP/1.1 200 OK"
+2026-06-11 11:05:57,011 | HTTP Request: GET https://huggingface.co/api/models/GODELEV/Experimenting/xet-write-token/main "HTTP/1.1 200 OK"
+2026-06-11 11:06:02,661 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/commit/main "HTTP/1.1 200 OK"
+2026-06-11 11:06:02,663 | Hub push step=600
+2026-06-11 11:16:06,339 | step=   620 | epoch=0.49 | loss=2.7078 | ppl=15.00 | lr=2.58e-04 | grad=0.951 | tok/s=85,624 | ETA=5:24:04 | VRAM=0.6GB | RAM=19%
+2026-06-11 11:26:11,866 | step=   640 | epoch=0.51 | loss=2.6989 | ppl=14.86 | lr=2.48e-04 | grad=1.173 | tok/s=85,358 | ETA=5:13:28 | VRAM=0.6GB | RAM=19%
+2026-06-11 11:36:17,585 | step=   660 | epoch=0.52 | loss=2.6667 | ppl=14.39 | lr=2.38e-04 | grad=0.870 | tok/s=85,108 | ETA=5:06:39 | VRAM=0.6GB | RAM=19%
+2026-06-11 11:46:22,399 | step=   680 | epoch=0.54 | loss=2.6486 | ppl=14.13 | lr=2.28e-04 | grad=0.638 | tok/s=84,879 | ETA=4:52:39 | VRAM=0.6GB | RAM=19%
+2026-06-11 11:56:27,607 | step=   700 | epoch=0.55 | loss=2.6595 | ppl=14.29 | lr=2.19e-04 | grad=0.835 | tok/s=84,662 | ETA=4:43:15 | VRAM=0.6GB | RAM=19%
+2026-06-11 12:06:33,643 | step=   720 | epoch=0.57 | loss=2.6418 | ppl=14.04 | lr=2.09e-04 | grad=0.751 | tok/s=84,454 | ETA=4:35:23 | VRAM=0.6GB | RAM=19%
+2026-06-11 12:16:39,113 | step=   740 | epoch=0.59 | loss=2.6338 | ppl=13.93 | lr=1.99e-04 | grad=0.775 | tok/s=84,261 | ETA=4:25:31 | VRAM=0.6GB | RAM=19%
+2026-06-11 12:26:45,316 | step=   760 | epoch=0.60 | loss=2.6075 | ppl=13.57 | lr=1.89e-04 | grad=0.730 | tok/s=84,076 | ETA=4:12:41 | VRAM=0.6GB | RAM=19%
+2026-06-11 12:36:51,989 | step=   780 | epoch=0.62 | loss=2.6104 | ppl=13.60 | lr=1.79e-04 | grad=0.578 | tok/s=83,900 | ETA=4:05:36 | VRAM=0.6GB | RAM=19%
+2026-06-11 12:46:59,512 | step=   800 | epoch=0.63 | loss=2.6066 | ppl=13.55 | lr=1.69e-04 | grad=0.807 | tok/s=83,730 | ETA=3:52:38 | VRAM=0.6GB | RAM=19%
+2026-06-11 12:47:10,645 | VAL step=800 epoch=0.63 loss=2.6120 ppl=13.63  BEST
+2026-06-11 12:47:10,814 | Checkpoint saved: step 800
+2026-06-11 12:47:10,869 | Saved safetensors: 110 tensors (tied=True, embed_key=model.embed_tokens.weight, lm_head omitted — HF will tie)
+2026-06-11 12:47:11,244 | HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict"
+2026-06-11 12:47:11,308 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 12:47:11,801 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 12:47:11,911 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/preupload/main "HTTP/1.1 200 OK"
+2026-06-11 12:47:12,016 | HTTP Request: POST https://huggingface.co/GODELEV/Experimenting.git/info/lfs/objects/batch "HTTP/1.1 200 OK"
+2026-06-11 12:47:12,079 | HTTP Request: GET https://huggingface.co/api/models/GODELEV/Experimenting/xet-write-token/main "HTTP/1.1 200 OK"
+2026-06-11 12:47:18,595 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/commit/main "HTTP/1.1 200 OK"
+2026-06-11 12:47:18,598 | Hub push step=800
+2026-06-11 12:57:22,752 | step=   820 | epoch=0.65 | loss=2.6016 | ppl=13.48 | lr=1.60e-04 | grad=0.774 | tok/s=83,512 | ETA=3:43:39 | VRAM=0.6GB | RAM=19%
+2026-06-11 13:07:29,946 | step=   840 | epoch=0.66 | loss=2.5871 | ppl=13.29 | lr=1.51e-04 | grad=0.765 | tok/s=83,362 | ETA=3:38:05 | VRAM=0.6GB | RAM=19%
+2026-06-11 13:17:34,836 | step=   860 | epoch=0.68 | loss=2.5837 | ppl=13.25 | lr=1.42e-04 | grad=0.741 | tok/s=83,227 | ETA=3:22:39 | VRAM=0.6GB | RAM=19%
+2026-06-11 13:27:37,756 | step=   880 | epoch=0.70 | loss=2.5803 | ppl=13.20 | lr=1.33e-04 | grad=0.883 | tok/s=83,105 | ETA=3:12:57 | VRAM=0.6GB | RAM=19%
+2026-06-11 13:37:44,484 | step=   900 | epoch=0.71 | loss=2.5702 | ppl=13.07 | lr=1.24e-04 | grad=0.568 | tok/s=82,977 | ETA=3:02:22 | VRAM=0.6GB | RAM=19%
+2026-06-11 13:47:50,604 | step=   920 | epoch=0.73 | loss=2.5767 | ppl=13.15 | lr=1.16e-04 | grad=0.533 | tok/s=82,856 | ETA=2:53:22 | VRAM=0.6GB | RAM=19%
+2026-06-11 13:57:58,661 | step=   940 | epoch=0.74 | loss=2.5578 | ppl=12.91 | lr=1.08e-04 | grad=0.750 | tok/s=82,735 | ETA=2:44:59 | VRAM=0.6GB | RAM=19%
+2026-06-11 14:08:11,735 | step=   960 | epoch=0.76 | loss=2.5631 | ppl=12.98 | lr=1.00e-04 | grad=0.541 | tok/s=82,604 | ETA=2:33:55 | VRAM=0.6GB | RAM=19%
+2026-06-11 14:18:17,443 | step=   980 | epoch=0.78 | loss=2.5458 | ppl=12.75 | lr=9.31e-05 | grad=0.588 | tok/s=82,501 | ETA=2:23:23 | VRAM=0.6GB | RAM=19%
+2026-06-11 14:28:22,853 | step=  1000 | epoch=0.79 | loss=2.5536 | ppl=12.85 | lr=8.62e-05 | grad=0.608 | tok/s=82,403 | ETA=2:12:14 | VRAM=0.6GB | RAM=19%
+2026-06-11 14:28:33,984 | VAL step=1000 epoch=0.79 loss=2.5559 ppl=12.88  BEST
+2026-06-11 14:28:34,058 | Saved safetensors: 110 tensors (tied=True, embed_key=model.embed_tokens.weight, lm_head omitted — HF will tie)
+2026-06-11 14:28:34,436 | HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict"
+2026-06-11 14:28:34,497 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 14:28:35,006 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 14:28:35,122 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/preupload/main "HTTP/1.1 200 OK"
+2026-06-11 14:28:35,213 | HTTP Request: POST https://huggingface.co/GODELEV/Experimenting.git/info/lfs/objects/batch "HTTP/1.1 200 OK"
+2026-06-11 14:28:35,276 | HTTP Request: GET https://huggingface.co/api/models/GODELEV/Experimenting/xet-write-token/main "HTTP/1.1 200 OK"
+2026-06-11 14:28:41,949 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/commit/main "HTTP/1.1 200 OK"
+2026-06-11 14:28:41,952 | Hub push step=1000
+2026-06-11 14:38:45,331 | step=  1020 | epoch=0.81 | loss=2.3078 | ppl=10.05 | lr=7.98e-05 | grad=1.918 | tok/s=82,261 | ETA=2:03:07 | VRAM=0.6GB | RAM=19%
+2026-06-11 14:48:51,304 | step=  1040 | epoch=0.82 | loss=2.2168 | ppl=9.18 | lr=7.37e-05 | grad=0.789 | tok/s=82,170 | ETA=1:53:16 | VRAM=0.6GB | RAM=19%
+2026-06-11 14:59:00,762 | step=  1060 | epoch=0.84 | loss=2.5697 | ppl=13.06 | lr=6.82e-05 | grad=0.509 | tok/s=82,073 | ETA=1:44:28 | VRAM=0.6GB | RAM=19%
+2026-06-11 15:09:09,586 | step=  1080 | epoch=0.85 | loss=2.5579 | ppl=12.91 | lr=6.30e-05 | grad=0.467 | tok/s=81,982 | ETA=1:33:40 | VRAM=0.6GB | RAM=19%
+2026-06-11 15:19:16,212 | step=  1100 | epoch=0.87 | loss=2.5473 | ppl=12.77 | lr=5.84e-05 | grad=0.446 | tok/s=81,900 | ETA=1:23:24 | VRAM=0.6GB | RAM=19%
+2026-06-11 15:29:22,702 | step=  1120 | epoch=0.89 | loss=2.5299 | ppl=12.55 | lr=5.43e-05 | grad=0.433 | tok/s=81,822 | ETA=1:12:55 | VRAM=0.6GB | RAM=19%
+2026-06-11 15:39:30,420 | step=  1140 | epoch=0.90 | loss=2.5377 | ppl=12.65 | lr=5.06e-05 | grad=0.392 | tok/s=81,743 | ETA=1:02:07 | VRAM=0.6GB | RAM=19%
+2026-06-11 15:49:36,534 | step=  1160 | epoch=0.92 | loss=2.5377 | ppl=12.65 | lr=4.75e-05 | grad=0.429 | tok/s=81,671 | ETA=0:52:05 | VRAM=0.6GB | RAM=19%
+2026-06-11 15:59:42,638 | step=  1180 | epoch=0.93 | loss=2.5383 | ppl=12.66 | lr=4.50e-05 | grad=0.472 | tok/s=81,602 | ETA=0:42:47 | VRAM=0.6GB | RAM=19%
+2026-06-11 16:09:54,897 | step=  1200 | epoch=0.95 | loss=2.5304 | ppl=12.56 | lr=4.29e-05 | grad=0.319 | tok/s=81,521 | ETA=0:32:44 | VRAM=0.6GB | RAM=19%
+2026-06-11 16:10:06,127 | VAL step=1200 epoch=0.95 loss=2.5314 ppl=12.57  BEST
+2026-06-11 16:10:06,313 | Checkpoint saved: step 1200
+2026-06-11 16:10:06,379 | Saved safetensors: 110 tensors (tied=True, embed_key=model.embed_tokens.weight, lm_head omitted — HF will tie)
+2026-06-11 16:10:06,753 | HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict"
+2026-06-11 16:10:06,814 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 16:10:07,303 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 16:10:07,409 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/preupload/main "HTTP/1.1 200 OK"
+2026-06-11 16:10:07,490 | HTTP Request: POST https://huggingface.co/GODELEV/Experimenting.git/info/lfs/objects/batch "HTTP/1.1 200 OK"
+2026-06-11 16:10:07,553 | HTTP Request: GET https://huggingface.co/api/models/GODELEV/Experimenting/xet-write-token/main "HTTP/1.1 200 OK"
+2026-06-11 16:10:16,698 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/commit/main "HTTP/1.1 200 OK"
+2026-06-11 16:10:16,701 | Hub push step=1200
+2026-06-11 16:20:20,220 | step=  1220 | epoch=0.97 | loss=2.5229 | ppl=12.47 | lr=4.14e-05 | grad=0.402 | tok/s=81,412 | ETA=0:22:03 | VRAM=0.6GB | RAM=19%
+2026-06-11 16:30:28,308 | step=  1240 | epoch=0.98 | loss=2.5312 | ppl=12.57 | lr=4.04e-05 | grad=0.388 | tok/s=81,346 | ETA=0:12:11 | VRAM=0.6GB | RAM=19%
+2026-06-11 16:40:34,691 | step=  1260 | epoch=1.00 | loss=2.5229 | ppl=12.46 | lr=4.00e-05 | grad=0.403 | tok/s=81,286 | ETA=0:02:00 | VRAM=0.6GB | RAM=19%
+2026-06-11 16:46:41,058 | Final eval loss=2.5222 ppl=12.46
+2026-06-11 16:46:41,255 | Checkpoint saved: step 1264
+2026-06-11 16:46:41,311 | Saved safetensors: 110 tensors (tied=True, embed_key=model.embed_tokens.weight, lm_head omitted — HF will tie)
+2026-06-11 16:46:41,699 | HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict"
+2026-06-11 16:46:41,759 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 16:46:42,253 | HTTP Request: POST https://huggingface.co/api/validate-yaml "HTTP/1.1 200 OK"
+2026-06-11 16:46:42,414 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/preupload/main "HTTP/1.1 200 OK"
+2026-06-11 16:46:42,503 | HTTP Request: POST https://huggingface.co/GODELEV/Experimenting.git/info/lfs/objects/batch "HTTP/1.1 200 OK"
+2026-06-11 16:46:42,569 | HTTP Request: GET https://huggingface.co/api/models/GODELEV/Experimenting/xet-write-token/main "HTTP/1.1 200 OK"
+2026-06-11 16:46:49,206 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/commit/main "HTTP/1.1 200 OK"
+2026-06-11 16:46:49,209 | Hub push step=1264
+2026-06-11 16:46:49,306 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/preupload/main "HTTP/1.1 200 OK"
+2026-06-11 16:46:50,104 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/commit/main "HTTP/1.1 200 OK"
+2026-06-11 16:46:50,203 | HTTP Request: POST https://huggingface.co/api/models/GODELEV/Experimenting/preupload/main "HTTP/1.1 200 OK"
--- a/training_meta.json
+++ b/training_meta.json
@@ -0,0 +1,8 @@
+{
+  "step": 1264,
+  "val_loss": 2.5222082792282103,
+  "val_ppl": 12.45607280175252,
+  "params_M": 9.902,
+  "pushed_at": "2026-06-11T16:46:41.317193",
+  "tokens_seen": 2979215382
+}