初始化项目，由ModelHub XC社区提供模型
Model: Pomilon/LEMA-llama-2-7b Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
132
README.md
Normal file
132
README.md
Normal file
@@ -0,0 +1,132 @@
|
||||
---
|
||||
license: mit
|
||||
language:
|
||||
- en
|
||||
base_model:
|
||||
- NousResearch/Llama-2-7b-hf
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- llm
|
||||
- llama
|
||||
- fine-tune
|
||||
- lema
|
||||
- vram-optimization
|
||||
- low-resource-computing
|
||||
- chat
|
||||
- merged
|
||||
---
|
||||
|
||||
# LEMA-Llama-2-7b (Proof of Concept)
|
||||
|
||||
This model is a demonstration of the **[LEMA (Layer-wise Efficient Memory Abstraction)](https://github.com/Pomilon/LEMA)** framework. It proves that large language models (7B+) can be fine-tuned on consumer-grade hardware with limited VRAM (e.g., 16GB Tesla P100) by virtualizing GPU memory.
|
||||
|
||||
**Key Achievement:**
|
||||
Fine-tuned Llama-2-7B using only **6.36 GB of VRAM** (standard LoRA typically requires ~14GB+ for this configuration).
|
||||
|
||||
> Training code is available over on the github repository: [**LEMA-llama**](https://github.com/Pomilon/LEMA-llama)
|
||||
|
||||
## Model Details
|
||||
|
||||
- **Base Model:** `NousResearch/Llama-2-7b-hf`
|
||||
- **Framework:** LEMA v1.0
|
||||
- **Fine-Tuning Method:** LoRA (Rank 16, Alpha 32)
|
||||
- **Memory Strategy:** Streaming (Triple-Buffer: Disk -> RAM -> VRAM)
|
||||
- **Precision:** FP16
|
||||
|
||||
## Training Configuration
|
||||
|
||||
The model was trained to learn a strict custom chat format (`[LEMA_REPLY]`) to verify that weight updates were successfully applied.
|
||||
|
||||
- **Hardware:** NVIDIA Tesla P100 (16GB VRAM)
|
||||
- **Batch Size:** 8 (Gradient Accumulation: 1)
|
||||
- **Sequence Length:** 512
|
||||
- **Steps:** 625 (1 Epoch over 5k examples)
|
||||
- **Optimizer:** AdamW (lr=1e-4)
|
||||
|
||||
### Memory Efficiency
|
||||
|
||||
| Metric | Standard PEFT/LoRA | **LEMA (This Run)** |
|
||||
| :--- | :--- | :--- |
|
||||
| **Peak VRAM** | **OOM** | **6.36 GB** |
|
||||
| **System RAM** | **OOM** | **2.40 GB** |
|
||||
|
||||
*Note: Standard PEFT typically OOMs at Batch Size 4-8 on 16GB cards with 512 context. LEMA held steady at <7GB.*
|
||||
|
||||
## Training Logs
|
||||
|
||||
The training loss converged smoothly, demonstrating stable learning despite the layer-wise streaming architecture.
|
||||
|
||||
```
|
||||
Step 10/625 | Loss: 2.1732 | VRAM: 6.36GB
|
||||
Step 100/625 | Loss: 0.0677 | VRAM: 6.36GB
|
||||
Step 200/625 | Loss: 0.0462 | VRAM: 6.36GB
|
||||
Step 300/625 | Loss: 0.0407 | VRAM: 6.36GB
|
||||
Step 400/625 | Loss: 0.0412 | VRAM: 6.36GB
|
||||
Step 500/625 | Loss: 0.0459 | VRAM: 6.36GB
|
||||
Step 600/625 | Loss: 0.0406 | VRAM: 6.36GB
|
||||
Final Step | Training Complete
|
||||
```
|
||||
|
||||
## Derived Metrics
|
||||
|
||||
* Total Training Time: 5h 40m
|
||||
* Average Step Time: 32.23s
|
||||
* Peak VRAM: 6.36GB (stable)
|
||||
* Peak RAM: 2.52GB
|
||||
|
||||
Full raw logs over [Here](training_logs.txt)
|
||||
|
||||
## Limitations & Known Issues
|
||||
|
||||
**⚠️ Warning: Experimental Proof-of-Concept**
|
||||
|
||||
This model was trained for only **1 epoch** as a mechanical stress test of the LEMA library. While it successfully learned the new vocabulary and special tags, it has not yet mastered the logical structure or grammar of the custom template.
|
||||
|
||||
- **Token Looping:** The model may repeat tags like `[LEMA_REPLY]` multiple times in a loop.
|
||||
- **Hallucinations:** It may invent creative definitions for terms it hasn't seen in its original pre-training (e.g., hallucinating an acronym for LEMA).
|
||||
- **Overfitting:** Due to the small, highly repetitive synthetic dataset and 1-epoch training, the model is likely overfit to the specific examples provided.
|
||||
- **Template Grammar:** It often skips the `Explanation:` and `Confidence:` fields.
|
||||
|
||||
To achieve production-grade results and make the model actually usable for general tasks, it is recommended to train for 3-5 epochs with a much larger, more diverse dataset (50k+ examples).
|
||||
|
||||
## Usage
|
||||
|
||||
This model uses a custom prompt format for testing purposes:
|
||||
|
||||
```text
|
||||
<|system|>
|
||||
You are a precise assistant trained using LEMA.
|
||||
|
||||
<|user|>
|
||||
What is LEMA?
|
||||
|
||||
<|assistant|>
|
||||
[LEMA_REPLY]
|
||||
Answer: ...
|
||||
Explanation: ...
|
||||
Confidence: High
|
||||
[/LEMA_REPLY]
|
||||
```
|
||||
|
||||
### Loading with Transformers
|
||||
|
||||
Since this model has been merged (LoRA adapter integrated into base), you can load it as a standard Llama model:
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained("Pomilon/LEMA-llama-2-7b")
|
||||
tokenizer = AutoTokenizer.from_pretrained("Pomilon/LEMA-llama-2-7b")
|
||||
|
||||
prompt = "<|system|>\nYou are a precise assistant trained using LEMA.\n\n<|user|>\nWhat is LEMA?\n\n<|assistant|>\n[LEMA_REPLY]\nAnswer:"
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
|
||||
|
||||
output = model.generate(**inputs, max_new_tokens=100)
|
||||
print(tokenizer.decode(output[0]))
|
||||
```
|
||||
|
||||
## About LEMA
|
||||
|
||||
LEMA is an experimental framework designed to democratize LLM fine-tuning. It treats model weights as a stream of data rather than a static block, allowing models to be processed layer-by-layer. This trades computation time (latency) for massive memory savings.
|
||||
|
||||
[Check out the GitHub Repository](https://github.com/Pomilon/LEMA)
|
||||
30
config.json
Normal file
30
config.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"dtype": "float16",
|
||||
"eos_token_id": 2,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 11008,
|
||||
"max_position_embeddings": 4096,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 32,
|
||||
"pad_token_id": 0,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 10000.0,
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "4.57.1",
|
||||
"use_cache": true,
|
||||
"vocab_size": 32000
|
||||
}
|
||||
3
final_checkpoint/adapter_model.bin
Normal file
3
final_checkpoint/adapter_model.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:fd3a4aac8000ca9619a3b1e90588f3f8b13676b3d854a7332bba2179b83a8b3c
|
||||
size 160047395
|
||||
30
final_checkpoint/lema_config.json
Normal file
30
final_checkpoint/lema_config.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"model_name_or_path": "NousResearch/Llama-2-7b-hf",
|
||||
"model_type": null,
|
||||
"gbi_path": "llama2_7b.safetensors",
|
||||
"device": "cuda",
|
||||
"strategy": "streaming",
|
||||
"ram_buffer_size": 2,
|
||||
"vram_buffer_size": 1,
|
||||
"use_lora": true,
|
||||
"lora_rank": 16,
|
||||
"lora_alpha": 32,
|
||||
"lora_target_modules": [
|
||||
"q_proj",
|
||||
"k_proj",
|
||||
"v_proj",
|
||||
"o_proj",
|
||||
"gate_proj",
|
||||
"up_proj",
|
||||
"down_proj"
|
||||
],
|
||||
"learning_rate": 0.0001,
|
||||
"batch_size": 8,
|
||||
"gradient_accumulation_steps": 1,
|
||||
"max_seq_length": 512,
|
||||
"gradient_checkpointing": true,
|
||||
"save_steps": 500,
|
||||
"output_dir": "checkpoints",
|
||||
"dtype": "float16",
|
||||
"attn_implementation": "eager"
|
||||
}
|
||||
3
final_checkpoint/optimizer.bin
Normal file
3
final_checkpoint/optimizer.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ab96dae4a5527346060ce770b39e12b54c79dd9e80d37460f5c1f32b639fac78
|
||||
size 320194275
|
||||
3
model-00001-of-00010.safetensors
Normal file
3
model-00001-of-00010.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b741b86fb7f60e2b0dcba249849d98c629072a8b3cc280179532f8fd48b6d4de
|
||||
size 262144104
|
||||
3
model-00002-of-00010.safetensors
Normal file
3
model-00002-of-00010.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6d354a6cb233e1999e743674b88ac73c3eb0f27724e6d20a26f1d4d1d6c3db05
|
||||
size 1619070960
|
||||
3
model-00003-of-00010.safetensors
Normal file
3
model-00003-of-00010.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:eaad078eec13a59035d03351f787f1f5a6b89d3761fd403c45d010aa784d2b03
|
||||
size 1619070960
|
||||
3
model-00004-of-00010.safetensors
Normal file
3
model-00004-of-00010.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1b16bd3d99f7b961be3258faa4ec7147399347acb92d4d8d5be701c2cb0abeda
|
||||
size 1619070976
|
||||
3
model-00005-of-00010.safetensors
Normal file
3
model-00005-of-00010.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ba498955e0cea8874f92cf6999483dda3ecf56ca1deff439b8c0b84f377c3b75
|
||||
size 1619071000
|
||||
3
model-00006-of-00010.safetensors
Normal file
3
model-00006-of-00010.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b02d7251b32191445aeb16d6acb8ac43014167fc1533017f1b7d08a6496f1492
|
||||
size 1619071000
|
||||
3
model-00007-of-00010.safetensors
Normal file
3
model-00007-of-00010.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:150c6c4e30f058ebcd6b5e9b159a23344e7fdc7e379134b319d0903a1769f0eb
|
||||
size 1619071000
|
||||
3
model-00008-of-00010.safetensors
Normal file
3
model-00008-of-00010.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3ac1d7a4ac862a470dd9064c2081d0b1fb0f05dd89c995b5337707427e0eeb6b
|
||||
size 1619071000
|
||||
3
model-00009-of-00010.safetensors
Normal file
3
model-00009-of-00010.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d1e29a4d9fcc0e3cdfebef489bb241e808495e1874b85360963cd3505d4f889b
|
||||
size 1619071000
|
||||
3
model-00010-of-00010.safetensors
Normal file
3
model-00010-of-00010.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:f429d40ee8632b1d3b903ee113a14c9b3137fbc46d63220888bcec58cf67964e
|
||||
size 262152376
|
||||
296
model.safetensors.index.json
Normal file
296
model.safetensors.index.json
Normal file
@@ -0,0 +1,296 @@
|
||||
{
|
||||
"metadata": {},
|
||||
"weight_map": {
|
||||
"model.embed_tokens.weight": "model-00001-of-00010.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.0.mlp.gate_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.1.mlp.gate_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.2.mlp.gate_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.3.mlp.gate_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.4.mlp.gate_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.5.mlp.gate_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.6.mlp.gate_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.7.mlp.gate_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.8.mlp.gate_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.9.mlp.gate_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.10.mlp.gate_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.11.mlp.gate_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.12.mlp.gate_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.13.mlp.gate_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.14.mlp.gate_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.15.mlp.gate_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.16.mlp.gate_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.17.mlp.gate_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.18.mlp.gate_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.19.mlp.gate_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.20.mlp.gate_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.21.mlp.gate_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.22.mlp.gate_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.23.mlp.gate_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.24.mlp.gate_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.25.mlp.gate_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.26.mlp.gate_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.27.mlp.gate_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
|
||||
"model.layers.28.input_layernorm.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.28.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.28.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.28.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.28.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.28.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.28.mlp.gate_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.28.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.28.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.29.input_layernorm.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.29.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.29.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.29.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.29.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.29.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.29.mlp.gate_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.29.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.29.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.30.input_layernorm.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.30.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.30.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.30.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.30.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.30.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.30.mlp.gate_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.30.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.30.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.31.input_layernorm.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.31.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.31.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.31.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.31.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.31.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.31.mlp.gate_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.31.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.layers.31.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
|
||||
"model.norm.weight": "model-00010-of-00010.safetensors",
|
||||
"lm_head.weight": "model-00010-of-00010.safetensors"
|
||||
}
|
||||
}
|
||||
30
special_tokens_map.json
Normal file
30
special_tokens_map.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
277139
tokenizer.json
Normal file
277139
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
Binary file not shown.
42
tokenizer_config.json
Normal file
42
tokenizer_config.json
Normal file
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"add_bos_token": true,
|
||||
"add_eos_token": false,
|
||||
"add_prefix_space": null,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"bos_token": "<s>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "</s>",
|
||||
"extra_special_tokens": {},
|
||||
"legacy": false,
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"pad_token": "<unk>",
|
||||
"sp_model_kwargs": {},
|
||||
"tokenizer_class": "LlamaTokenizer",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
129
training_logs.txt
Normal file
129
training_logs.txt
Normal file
@@ -0,0 +1,129 @@
|
||||
619.1s 534 2026-02-16 12:48:08,993 - __main__ - INFO - Step 10/625 | Loss: 2.1732 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
619.1s 535 INFO:__main__:Step 10/625 | Loss: 2.1732 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
941.4s 536 2026-02-16 12:53:31,274 - __main__ - INFO - Step 20/625 | Loss: 1.1008 | VRAM: 6.36GB | RAM: 2.40GB
|
||||
941.4s 537 INFO:__main__:Step 20/625 | Loss: 1.1008 | VRAM: 6.36GB | RAM: 2.40GB
|
||||
1263.7s 538 2026-02-16 12:58:53,631 - __main__ - INFO - Step 30/625 | Loss: 0.6457 | VRAM: 6.36GB | RAM: 2.42GB
|
||||
1263.7s 539 INFO:__main__:Step 30/625 | Loss: 0.6457 | VRAM: 6.36GB | RAM: 2.42GB
|
||||
1586.1s 540 2026-02-16 13:04:16,041 - __main__ - INFO - Step 40/625 | Loss: 0.4113 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
1586.1s 541 INFO:__main__:Step 40/625 | Loss: 0.4113 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
1908.6s 542 2026-02-16 13:09:38,514 - __main__ - INFO - Step 50/625 | Loss: 0.5867 | VRAM: 6.36GB | RAM: 2.40GB
|
||||
1908.6s 543 INFO:__main__:Step 50/625 | Loss: 0.5867 | VRAM: 6.36GB | RAM: 2.40GB
|
||||
2231.0s 544 2026-02-16 13:15:00,931 - __main__ - INFO - Step 60/625 | Loss: 0.2535 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
2231.0s 545 INFO:__main__:Step 60/625 | Loss: 0.2535 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
2553.5s 546 2026-02-16 13:20:23,403 - __main__ - INFO - Step 70/625 | Loss: 0.4313 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
2553.5s 547 INFO:__main__:Step 70/625 | Loss: 0.4313 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
2875.9s 548 2026-02-16 13:25:45,819 - __main__ - INFO - Step 80/625 | Loss: 0.1017 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
2875.9s 549 INFO:__main__:Step 80/625 | Loss: 0.1017 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
3198.3s 550 2026-02-16 13:31:08,168 - __main__ - INFO - Step 90/625 | Loss: 0.1827 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
3198.3s 551 INFO:__main__:Step 90/625 | Loss: 0.1827 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
3520.6s 552 2026-02-16 13:36:30,500 - __main__ - INFO - Step 100/625 | Loss: 0.0677 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
3520.6s 553 INFO:__main__:Step 100/625 | Loss: 0.0677 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
3843.0s 554 2026-02-16 13:41:52,842 - __main__ - INFO - Step 110/625 | Loss: 0.0511 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
3843.0s 555 INFO:__main__:Step 110/625 | Loss: 0.0511 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
4165.3s 556 2026-02-16 13:47:15,184 - __main__ - INFO - Step 120/625 | Loss: 0.0565 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
4165.3s 557 INFO:__main__:Step 120/625 | Loss: 0.0565 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
4487.6s 558 2026-02-16 13:52:37,493 - __main__ - INFO - Step 130/625 | Loss: 0.0511 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
4487.6s 559 INFO:__main__:Step 130/625 | Loss: 0.0511 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
4810.0s 560 2026-02-16 13:57:59,925 - __main__ - INFO - Step 140/625 | Loss: 0.0769 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
4810.0s 561 INFO:__main__:Step 140/625 | Loss: 0.0769 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
5132.4s 562 2026-02-16 14:03:22,277 - __main__ - INFO - Step 150/625 | Loss: 0.0484 | VRAM: 6.36GB | RAM: 2.35GB
|
||||
5132.4s 563 INFO:__main__:Step 150/625 | Loss: 0.0484 | VRAM: 6.36GB | RAM: 2.35GB
|
||||
5454.8s 564 2026-02-16 14:08:44,697 - __main__ - INFO - Step 160/625 | Loss: 0.0401 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
5454.8s 565 INFO:__main__:Step 160/625 | Loss: 0.0401 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
5777.2s 566 2026-02-16 14:14:07,061 - __main__ - INFO - Step 170/625 | Loss: 0.0533 | VRAM: 6.36GB | RAM: 2.35GB
|
||||
5777.2s 567 INFO:__main__:Step 170/625 | Loss: 0.0533 | VRAM: 6.36GB | RAM: 2.35GB
|
||||
6099.6s 568 2026-02-16 14:19:29,445 - __main__ - INFO - Step 180/625 | Loss: 0.0391 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
6099.6s 569 INFO:__main__:Step 180/625 | Loss: 0.0391 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
6421.9s 570 2026-02-16 14:24:51,789 - __main__ - INFO - Step 190/625 | Loss: 0.0464 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
6421.9s 571 INFO:__main__:Step 190/625 | Loss: 0.0464 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
6744.3s 572 2026-02-16 14:30:14,166 - __main__ - INFO - Step 200/625 | Loss: 0.0462 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
6744.3s 573 INFO:__main__:Step 200/625 | Loss: 0.0462 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
7066.7s 574 2026-02-16 14:35:36,596 - __main__ - INFO - Step 210/625 | Loss: 0.0450 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
7066.7s 575 INFO:__main__:Step 210/625 | Loss: 0.0450 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
7389.1s 576 2026-02-16 14:40:58,993 - __main__ - INFO - Step 220/625 | Loss: 0.0401 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
7389.1s 577 INFO:__main__:Step 220/625 | Loss: 0.0401 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
7711.5s 578 2026-02-16 14:46:21,447 - __main__ - INFO - Step 230/625 | Loss: 0.0433 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
7711.5s 579 INFO:__main__:Step 230/625 | Loss: 0.0433 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
8033.9s 580 2026-02-16 14:51:43,849 - __main__ - INFO - Step 240/625 | Loss: 0.0525 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
8033.9s 581 INFO:__main__:Step 240/625 | Loss: 0.0525 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
8356.3s 582 2026-02-16 14:57:06,160 - __main__ - INFO - Step 250/625 | Loss: 0.0464 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
8356.3s 583 INFO:__main__:Step 250/625 | Loss: 0.0464 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
8678.6s 584 2026-02-16 15:02:28,505 - __main__ - INFO - Step 260/625 | Loss: 0.0515 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
8678.6s 585 INFO:__main__:Step 260/625 | Loss: 0.0515 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
9000.9s 586 2026-02-16 15:07:50,785 - __main__ - INFO - Step 270/625 | Loss: 0.0405 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
9000.9s 587 INFO:__main__:Step 270/625 | Loss: 0.0405 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
9323.2s 588 2026-02-16 15:13:13,084 - __main__ - INFO - Step 280/625 | Loss: 0.0395 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
9323.2s 589 INFO:__main__:Step 280/625 | Loss: 0.0395 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
9645.5s 590 2026-02-16 15:18:35,374 - __main__ - INFO - Step 290/625 | Loss: 0.0413 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
9645.5s 591 INFO:__main__:Step 290/625 | Loss: 0.0413 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
9967.8s 592 2026-02-16 15:23:57,658 - __main__ - INFO - Step 300/625 | Loss: 0.0407 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
9967.8s 593 INFO:__main__:Step 300/625 | Loss: 0.0407 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
10290.0s 594 2026-02-16 15:29:19,910 - __main__ - INFO - Step 310/625 | Loss: 0.0462 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
10290.0s 595 INFO:__main__:Step 310/625 | Loss: 0.0462 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
10612.4s 596 2026-02-16 15:34:42,316 - __main__ - INFO - Step 320/625 | Loss: 0.0482 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
10612.4s 597 INFO:__main__:Step 320/625 | Loss: 0.0482 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
10934.8s 598 2026-02-16 15:40:04,673 - __main__ - INFO - Step 330/625 | Loss: 0.0432 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
10934.8s 599 INFO:__main__:Step 330/625 | Loss: 0.0432 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
11257.1s 600 2026-02-16 15:45:26,987 - __main__ - INFO - Step 340/625 | Loss: 0.0479 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
11257.1s 601 INFO:__main__:Step 340/625 | Loss: 0.0479 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
11579.4s 602 2026-02-16 15:50:49,273 - __main__ - INFO - Step 350/625 | Loss: 0.0427 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
11579.4s 603 INFO:__main__:Step 350/625 | Loss: 0.0427 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
11901.6s 604 2026-02-16 15:56:11,550 - __main__ - INFO - Step 360/625 | Loss: 0.0373 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
11901.6s 605 INFO:__main__:Step 360/625 | Loss: 0.0373 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
12224.0s 606 2026-02-16 16:01:33,889 - __main__ - INFO - Step 370/625 | Loss: 0.0440 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
12224.0s 607 INFO:__main__:Step 370/625 | Loss: 0.0440 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
12546.4s 608 2026-02-16 16:06:56,280 - __main__ - INFO - Step 380/625 | Loss: 0.0445 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
12546.4s 609 INFO:__main__:Step 380/625 | Loss: 0.0445 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
12868.7s 610 2026-02-16 16:12:18,600 - __main__ - INFO - Step 390/625 | Loss: 0.0427 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
12868.7s 611 INFO:__main__:Step 390/625 | Loss: 0.0427 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
13191.0s 612 2026-02-16 16:17:40,899 - __main__ - INFO - Step 400/625 | Loss: 0.0412 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
13191.0s 613 INFO:__main__:Step 400/625 | Loss: 0.0412 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
13513.3s 614 2026-02-16 16:23:03,243 - __main__ - INFO - Step 410/625 | Loss: 0.0431 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
13513.3s 615 INFO:__main__:Step 410/625 | Loss: 0.0431 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
13835.6s 616 2026-02-16 16:28:25,499 - __main__ - INFO - Step 420/625 | Loss: 0.0427 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
13835.6s 617 INFO:__main__:Step 420/625 | Loss: 0.0427 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
14158.0s 618 2026-02-16 16:33:47,864 - __main__ - INFO - Step 430/625 | Loss: 0.0464 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
14158.0s 619 INFO:__main__:Step 430/625 | Loss: 0.0464 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
14480.4s 620 2026-02-16 16:39:10,223 - __main__ - INFO - Step 440/625 | Loss: 0.0423 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
14480.4s 621 INFO:__main__:Step 440/625 | Loss: 0.0423 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
14802.6s 622 2026-02-16 16:44:32,512 - __main__ - INFO - Step 450/625 | Loss: 0.0435 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
14802.6s 623 INFO:__main__:Step 450/625 | Loss: 0.0435 | VRAM: 6.36GB | RAM: 2.37GB
|
||||
15124.9s 624 2026-02-16 16:49:54,804 - __main__ - INFO - Step 460/625 | Loss: 0.0372 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
15124.9s 625 INFO:__main__:Step 460/625 | Loss: 0.0372 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
15447.2s 626 2026-02-16 16:55:17,098 - __main__ - INFO - Step 470/625 | Loss: 0.0371 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
15447.2s 627 INFO:__main__:Step 470/625 | Loss: 0.0371 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
15769.5s 628 2026-02-16 17:00:39,362 - __main__ - INFO - Step 480/625 | Loss: 0.0410 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
15769.5s 629 INFO:__main__:Step 480/625 | Loss: 0.0410 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
16091.9s 630 2026-02-16 17:06:01,743 - __main__ - INFO - Step 490/625 | Loss: 0.0430 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
16091.9s 631 INFO:__main__:Step 490/625 | Loss: 0.0430 | VRAM: 6.36GB | RAM: 2.36GB
|
||||
16415.6s 632 2026-02-16 17:11:25,521 - __main__ - INFO - Step 500/625 | Loss: 0.0459 | VRAM: 6.36GB | RAM: 2.52GB
|
||||
16415.6s 633 INFO:__main__:Step 500/625 | Loss: 0.0459 | VRAM: 6.36GB | RAM: 2.52GB
|
||||
16737.4s 634 2026-02-16 17:16:47,226 - __main__ - INFO - Step 510/625 | Loss: 0.0408 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
16737.4s 635 INFO:__main__:Step 510/625 | Loss: 0.0408 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
17059.6s 636 2026-02-16 17:22:09,481 - __main__ - INFO - Step 520/625 | Loss: 0.0410 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
17059.6s 637 INFO:__main__:Step 520/625 | Loss: 0.0410 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
17381.8s 638 2026-02-16 17:27:31,724 - __main__ - INFO - Step 530/625 | Loss: 0.0432 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
17381.8s 639 INFO:__main__:Step 530/625 | Loss: 0.0432 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
17704.2s 640 2026-02-16 17:32:54,035 - __main__ - INFO - Step 540/625 | Loss: 0.0413 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
17704.2s 641 INFO:__main__:Step 540/625 | Loss: 0.0413 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
18026.4s 642 2026-02-16 17:38:16,344 - __main__ - INFO - Step 550/625 | Loss: 0.0386 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
18026.4s 643 INFO:__main__:Step 550/625 | Loss: 0.0386 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
18348.7s 644 2026-02-16 17:43:38,609 - __main__ - INFO - Step 560/625 | Loss: 0.0391 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
18348.7s 645 INFO:__main__:Step 560/625 | Loss: 0.0391 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
18671.0s 646 2026-02-16 17:49:00,843 - __main__ - INFO - Step 570/625 | Loss: 0.0387 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
18671.0s 647 INFO:__main__:Step 570/625 | Loss: 0.0387 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
18993.2s 648 2026-02-16 17:54:23,093 - __main__ - INFO - Step 580/625 | Loss: 0.0470 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
18993.2s 649 INFO:__main__:Step 580/625 | Loss: 0.0470 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
19315.4s 650 2026-02-16 17:59:45,309 - __main__ - INFO - Step 590/625 | Loss: 0.0377 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
19315.4s 651 INFO:__main__:Step 590/625 | Loss: 0.0377 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
19637.6s 652 2026-02-16 18:05:07,525 - __main__ - INFO - Step 600/625 | Loss: 0.0406 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
19637.6s 653 INFO:__main__:Step 600/625 | Loss: 0.0406 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
19959.8s 654 2026-02-16 18:10:29,713 - __main__ - INFO - Step 610/625 | Loss: 0.0399 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
19959.8s 655 INFO:__main__:Step 610/625 | Loss: 0.0399 | VRAM: 6.36GB | RAM: 2.39GB
|
||||
20282.0s 656 2026-02-16 18:15:51,872 - __main__ - INFO - Step 620/625 | Loss: 0.0388 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
20282.0s 657 INFO:__main__:Step 620/625 | Loss: 0.0388 | VRAM: 6.36GB | RAM: 2.38GB
|
||||
20443.1s 658 2026-02-16 18:18:32,947 - __main__ - INFO - Training complete.
|
||||
20443.1s 659 INFO:__main__:Training complete.
|
||||
20443.1s 660 2026-02-16 18:18:32,947 - __main__ - INFO - Saving final model to checkpoints/final
|
||||
20443.1s 661 INFO:__main__:Saving final model to checkpoints/final
|
||||
20444.6s 662 2026-02-16 18:18:34,524 - __main__ - INFO - Done!
|
||||
Reference in New Issue
Block a user