初始化项目，由ModelHub XC社区提供模型
Model: ducklingcodehouse/Finnish-DentalQA-v3 Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
310
README.md
Normal file
310
README.md
Normal file
@@ -0,0 +1,310 @@
|
|||||||
|
---
|
||||||
|
library_name: transformers
|
||||||
|
tags:
|
||||||
|
- transformers
|
||||||
|
- unsloth
|
||||||
|
- gemma3_text
|
||||||
|
- gemma
|
||||||
|
- finnish
|
||||||
|
- medical
|
||||||
|
- dental
|
||||||
|
- healthcare
|
||||||
|
- research-only
|
||||||
|
license: apache-2.0
|
||||||
|
language:
|
||||||
|
- fi
|
||||||
|
base_model: Finnish-NLP/Ahma-2-4B-Instruct
|
||||||
|
---
|
||||||
|
|
||||||
|
# Finnish-DentalQA-v3
|
||||||
|
|
||||||
|
**Finnish-language conversational AI assistant specialized for dental medicine.** This is a fine-tuned large language model (LLM) that generates structured text responses to dental consultation queries, designed to simulate professional dentist-to-dentist consultations with clinical reasoning and recommendations.
|
||||||
|
|
||||||
|
**RESEARCH USE ONLY - NOT FOR MEDICAL DECISIONS**
|
||||||
|
|
||||||
|
- **Developed by:** [Heikki Saxén](https://fi.linkedin.com/in/heikkisaxen) / [Duckling Codehouse Oy](https://www.linkedin.com/company/duckling-codehouse-oy/) ([ducklingcodehouse](https://huggingface.co/ducklingcodehouse))
|
||||||
|
- **Supported by:** [Entteri Professional Software Oy](https://www.entteri.com/) (makers of AssisDent dental PMS)
|
||||||
|
- **License:** apache-2.0
|
||||||
|
- **Base Model:** [Finnish-NLP/Ahma-2-4B-Instruct](https://huggingface.co/Finnish-NLP/Ahma-2-4B-Instruct) - Special thanks to Aapo Tanskanen and Rasmus Toivanen for creating this excellent Finnish LLM
|
||||||
|
- **Context Length:** Up to 128K input tokens, 8K output tokens (inherited from Ahma-2 base model)
|
||||||
|
- **Release Date:** January 7, 2026
|
||||||
|
|
||||||
|
Full fine-tuned model for Finnish dental medicine consultations between healthcare professionals. Trained on 50,132 synthetic dental conversations (approximately 80% expert clinical cases, 20% concept explanations) covering a broad range of different scenarios. Generated using pipeline with GPT-4.1.
|
||||||
|
|
||||||
|
**Version 3 Improvements:** This v3 release upgrades to the new Ahma-2-4B-Instruct base model, a significantly more capable 4B parameter Finnish LLM. The new base model features improved Finnish language understanding and generation quality through continued pretraining, supervised fine-tuning, and direct preference optimization. Training method changed from LoRA to full fine-tuning for maximum quality. Same proven dataset as v2.
|
||||||
|
|
||||||
|
**⚠️ Precision Requirement:** This model must be loaded in BF16 (`torch.bfloat16`). Ahma-2 is based on Gemma architecture which requires BF16 precision.
|
||||||
|
|
||||||
|
**Research Focus:** This model demonstrates domain-specific fine-tuning with accessible computational resources. The goal is to explore how specialized models can be trained and deployed on consumer hardware (including personal GPUs) rather than requiring high-end infrastructure.
|
||||||
|
|
||||||
|
**System Prompt Recommendation:** This model was trained with a specific system prompt. For best results, we recommend using the same prompt format shown in the examples below. While the system prompt can be modified, a 4B parameter model has limited ability to adapt to nuanced prompt changes — it will react to modifications, but don't expect the same prompt-following flexibility as larger models. Significant deviations from the training prompt may lead to inconsistent behavior.
|
||||||
|
|
||||||
|
**⚠️ Multi-Turn System Prompt Note:** Ahma-2 is based on Gemma architecture, which has **no native system role** — only `user` and `model`. When you pass `role: "system"`, the chat template silently prepends it to the first user message. This means:
|
||||||
|
- The model treats system instructions as regular user text
|
||||||
|
- In multi-turn conversations, the system prompt gets buried deeper in the context and its effect fades
|
||||||
|
- The model may start ignoring instructions or behaving inconsistently in longer chats
|
||||||
|
|
||||||
|
**Recommended fix for multi-turn use:** Prepend the system prompt to **every** user message, not just the first one. This keeps instructions in the strongest attention position regardless of conversation length. See the [Multi-Turn Example](#multi-turn-example) below.
|
||||||
|
|
||||||
|
**Response Format:** The model is trained to structure all clinical responses in three sections: "### Tausta" (Background), "### Arvio" (Assessment), "### Suositus" (Recommendation).
|
||||||
|
|
||||||
|
**Context Limit:** Supports up to 128K input tokens and 8K output tokens for inference (training used 2048 token sequences).
|
||||||
|
|
||||||
|
This model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's Transformers library.
|
||||||
|
|
||||||
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|
||||||
|
|
||||||
|
## Model Comparison
|
||||||
|
|
||||||
|
| Version | Base Model | Parameters | Training Method | Dataset | Notes |
|
||||||
|
|---------|-----------|------------|-----------------|---------|-------|
|
||||||
|
| **v3** | Ahma-2-4B-Instruct | 4B | Full fine-tuning | 50,132 | Latest, highest quality |
|
||||||
|
| v2 | Ahma-3B-Instruct | 3B | LoRA | 50,132 | Enhanced dataset, available as LoRA and merged |
|
||||||
|
| v1 | Ahma-3B-Instruct | 3B | LoRA | 30,908 | Original release, available as LoRA and merged |
|
||||||
|
|
||||||
|
**v3 (this model)**: Full fine-tuned on Ahma-2-4B-Instruct, highest quality
|
||||||
|
|
||||||
|
**v2 LoRA Version**: [ducklingcodehouse/Finnish-DentalQA-v2-lora](https://huggingface.co/ducklingcodehouse/Finnish-DentalQA-v2-lora) - Enhanced training (50,132 samples), separate adapter files, requires base model
|
||||||
|
|
||||||
|
**v2 Merged Version**: [ducklingcodehouse/Finnish-DentalQA-v2-merged](https://huggingface.co/ducklingcodehouse/Finnish-DentalQA-v2-merged) - Same as v2 LoRA but merged into standalone model
|
||||||
|
|
||||||
|
**v1 LoRA Version**: [ducklingcodehouse/Finnish-DentalQA-lora](https://huggingface.co/ducklingcodehouse/Finnish-DentalQA-lora) - Original training dataset (30,908 samples), separate adapter files
|
||||||
|
|
||||||
|
**v1 Merged Version**: [ducklingcodehouse/Finnish-DentalQA-merged](https://huggingface.co/ducklingcodehouse/Finnish-DentalQA-merged) - Original training dataset, merged into standalone model
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install transformers torch accelerate
|
||||||
|
```
|
||||||
|
|
||||||
|
## Loading the Model
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||||
|
import torch
|
||||||
|
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
"ducklingcodehouse/Finnish-DentalQA-v3",
|
||||||
|
torch_dtype=torch.bfloat16, # Required: BF16 for Ahma-2
|
||||||
|
device_map="auto"
|
||||||
|
)
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained("ducklingcodehouse/Finnish-DentalQA-v3")
|
||||||
|
```
|
||||||
|
|
||||||
|
## Generate Response
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Use exact system prompt from training
|
||||||
|
system_prompt = """Olet kokenut suomalainen hammaslääkäri. Vastaat ammattimaisesti kollegojesi
|
||||||
|
kysymyksiin käyttäen oikeaa hammaslääketieteellistä terminologiaa ja viittaat Käypä hoito
|
||||||
|
-suosituksiin kun relevanttia."""
|
||||||
|
|
||||||
|
user_question = """87-vuotias nainen, jolla on vaikea muistisairaus, asuu palvelutalossa ja on
|
||||||
|
kotihoidon piirissä. Potilaalla on hampaattomuus, yläleuan kokoproteesi ja alaleuassa 6
|
||||||
|
luonnollista hammasta. Lääkityksenä on mm. rivaroksabaani eteisvärinään, donepetsiili,
|
||||||
|
metoprololi, kalsium+D-vitamiini, sekä useita muita lääkkeitä (yhteensä 9). Suun limakalvot
|
||||||
|
ovat kuivat, kielessä fissuroitumista, ja proteesin alla lievää punoitusta. HbA1c 46 mmol/mol,
|
||||||
|
BMI 22. Kotihoito raportoi potilaan nielevän huonosti ja syljen erityksen olevan vähäistä.
|
||||||
|
Karies- ja parodontiittiriski arvioidaan korkeaksi. Miten suosittelisit suun hoidon
|
||||||
|
järjestämistä ja priorisointia tässä tilanteessa?"""
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": system_prompt},
|
||||||
|
{"role": "user", "content": user_question}
|
||||||
|
]
|
||||||
|
|
||||||
|
inputs = tokenizer.apply_chat_template(
|
||||||
|
messages,
|
||||||
|
tokenize=True,
|
||||||
|
add_generation_prompt=True,
|
||||||
|
return_tensors="pt"
|
||||||
|
).to(model.device)
|
||||||
|
|
||||||
|
# Recommended settings (model supports up to 128K input / 8K output context)
|
||||||
|
with torch.no_grad():
|
||||||
|
outputs = model.generate(
|
||||||
|
inputs,
|
||||||
|
max_new_tokens=800,
|
||||||
|
temperature=0.1,
|
||||||
|
do_sample=True,
|
||||||
|
top_p=0.9,
|
||||||
|
repetition_penalty=1.2
|
||||||
|
)
|
||||||
|
|
||||||
|
response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
|
||||||
|
print(response)
|
||||||
|
|
||||||
|
# Expected example response:
|
||||||
|
# ### Tausta
|
||||||
|
# Potilas on iäkäs, monisairas ja polyfarmasian vuoksi haavoittuvassa asemassa. Hampaattomuus
|
||||||
|
# ja proteesin käyttö yhdistettynä kuivaan suuhun lisäävät riskejä kuten sieni-infektioihin
|
||||||
|
# ja limakalvo-ongelmiin. Nielemisvaikeudet ja alentunut syljeneritys korostavat aspiraatioriskin
|
||||||
|
# hallinnan tarvetta. Antikoagulanttihoito (rivaroksabaani) tulee huomioida kaikissa
|
||||||
|
# invasiivisissa toimenpiteissä. Kotihoidon rooli on keskeinen päivittäisen suuhygienian
|
||||||
|
# toteutumisessa.
|
||||||
|
#
|
||||||
|
# ### Arvio
|
||||||
|
# Kliinisesti merkittäviä löydöksiä ovat kuivuminen, fissuroitunut kieli ja proteesistomatiitti.
|
||||||
|
# Korkea kariesriski liittyy sekä vähentyneeseen sylkeen että mahdolliseen heikentyneeseen
|
||||||
|
# omahoitoon. Parodontaalisairauksien riski kasvaa jäljellä olevien hampaiden määrän
|
||||||
|
# myötä. Muistisairaudet rajoittavat yhteistyökykyä ja edellyttävät yksinkertaisia
|
||||||
|
# ohjeita kotihoidolle. Käypä hoito -suositusten mukaan lyhennetyn hammaskaaren malli
|
||||||
|
# voi olla riittävä toiminnallisesti.
|
||||||
|
#
|
||||||
|
# ### Suositus
|
||||||
|
# Prioriteettina on kivuttoman ja infektiovapaan suun ylläpito: tehostettu mekaaninen
|
||||||
|
# puhdistus pehmeällä harjalla kahdesti päivässä, fluorilakkaus vastaanotolla neljännesvuosittain
|
||||||
|
# sekä klooriheksidiinigeeli paikallisesti riskialueille. Proteesi tulisi poistaa yöksi
|
||||||
|
# ja puhdistaa huolellisesti päivittäin. Syljenkorvikkeet tai keinotekoiset syljet
|
||||||
|
# voivat helpottaa oireita; ksylitolituotteet eivät ole ensisijaisia nielemishäiriön
|
||||||
|
# vuoksi. Säännölliset kontrollikäynnit suuhygienistin kanssa sekä tiivis yhteistyö
|
||||||
|
# kotihoidon henkilöstön kanssa ovat välttämättisiä. Tarvittaessa konsultoi erikoishammaslääkäriä
|
||||||
|
# mahdollisten sieni-infektiodiagnostiikan ja antifungaalisen hoidon osalta.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Multi-Turn Example
|
||||||
|
|
||||||
|
In multi-turn conversations, prepend the system prompt to every user message to prevent instruction drift. Store only the raw user messages in history — prepend the system prompt fresh for each request.
|
||||||
|
|
||||||
|
```python
|
||||||
|
system_prompt = """Olet kokenut suomalainen hammaslääkäri. Vastaat ammattimaisesti kollegojesi
|
||||||
|
kysymyksiin käyttäen oikeaa hammaslääketieteellistä terminologiaa ja viittaat Käypä hoito
|
||||||
|
-suosituksiin kun relevanttia."""
|
||||||
|
|
||||||
|
separator = "\n---\nYllä olevat ohjeet ovat pysyviä. Kollegan viesti:\n"
|
||||||
|
|
||||||
|
# Conversation history — stores raw user messages (no system prompt)
|
||||||
|
history = []
|
||||||
|
|
||||||
|
def generate_response(user_message):
|
||||||
|
# Prepend system prompt to this user message
|
||||||
|
augmented_message = system_prompt + separator + user_message
|
||||||
|
|
||||||
|
# Build messages for the model: history + current augmented message
|
||||||
|
messages = history.copy()
|
||||||
|
messages.append({"role": "user", "content": augmented_message})
|
||||||
|
|
||||||
|
inputs = tokenizer.apply_chat_template(
|
||||||
|
messages,
|
||||||
|
tokenize=True,
|
||||||
|
add_generation_prompt=True,
|
||||||
|
return_tensors="pt"
|
||||||
|
).to(model.device)
|
||||||
|
|
||||||
|
with torch.no_grad():
|
||||||
|
outputs = model.generate(
|
||||||
|
inputs,
|
||||||
|
max_new_tokens=800,
|
||||||
|
temperature=0.1,
|
||||||
|
do_sample=True,
|
||||||
|
top_p=0.9,
|
||||||
|
repetition_penalty=1.2
|
||||||
|
)
|
||||||
|
|
||||||
|
response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
|
||||||
|
|
||||||
|
# Store raw user message (not augmented) and response in history
|
||||||
|
history.append({"role": "user", "content": user_message})
|
||||||
|
history.append({"role": "assistant", "content": response})
|
||||||
|
|
||||||
|
return response
|
||||||
|
|
||||||
|
# Turn 1
|
||||||
|
print(generate_response("Mitä pitäisi huomioida viisaudenhampaan poistossa?"))
|
||||||
|
|
||||||
|
# Turn 2 — system prompt is still in the strongest attention position
|
||||||
|
print(generate_response("Entä jos potilas käyttää verenohennuslääkitystä?"))
|
||||||
|
```
|
||||||
|
|
||||||
|
The key insight: by prepending system instructions to every user message, they always appear right before the model's generation point — the highest-attention position in the context. Without this, the system prompt would be at the very beginning of a growing conversation and gradually lose influence.
|
||||||
|
|
||||||
|
## Example
|
||||||
|
|
||||||
|
**User:** 27-vuotias mies, jolla on voimakas kipu ja turvotus alaleuan oikealla puolella. Viisaudenhammas 48 on puoliksi puhkeamassa, ympärillä runsasta ienturvotusta ja märkäkertymä. Potilaalla on kuumetta 38,3°C ja suun avausrajoitus 22 mm. Mitä ensihoitoa suosittelet terveyskeskuksessa?
|
||||||
|
|
||||||
|
**Assistant:**
|
||||||
|
### Tausta
|
||||||
|
Potilas esittää tyypillisiä akuutin perikoroniitin oireita: paikallinen turvotus, märkäkertymä sekä yleisoireet kuten kuume ja trismus (avausrajoitus). Puoliksi puhjenneen alaviisaudenhampaan ympäristöön kehittynyt infektio voi levitä nopeasti pehmytkudoksiin tai aiheuttaa syvempiä komplikaatioita. Yleistilan heikkeneminen näkyy tässä tapauksessa selvästi kohonneena lämpötilana (>38 °C) sekä rajoittuneena suun avaamisena (<30 mm), mikä lisää riskiä vakavammille infektioille.
|
||||||
|
|
||||||
|
### Arvio
|
||||||
|
Kliinisesti kyseessä on vaikea akuutti perikoroniitti, jossa systeemiset oireet ovat jo ilmeisiä. Märkäkertymän koko ja potilaan yleisvointi edellyttävät välitöntä toimenpidettä infektion leviämisen estämiseksi. Suun avauksen rajoitus viittaa mahdolliseen lihasjäykkyyteen tai alkavaan flegmonaaliseen prosessiin submandibulaarialueella. Tilanne vaatii ripeää arviota mahdollisesta sairaalahoitoon ohjaamisesta.
|
||||||
|
|
||||||
|
### Suositus
|
||||||
|
Ensihoitona tulee tehdä huolellinen mekaaninen debridementti eli irrottaa tulehtunut limakalvo hampaan kruunun alta steriilein instrumentein. Mikrobiologisen viljelyn kannalta otetaan bakteerinäyte ennen antibioottihoidon aloittamista. Aloitetaan empiirinen mikrobilääkehoito esimerkiksi amoksisilliini + metronidatsoli; penisilliiniallergiselle klindamysiini. Kipulääkitys toteutetaan parasetamolilla ja/tai ibuprofeenilla annosteluohjeiden mukaisesti. Tarvittaessa konsultoidaan erikoissairaanhoitoa jatkohoidosta tilanteen mukaan.
|
||||||
|
|
||||||
|
**Follow-up:** Potilas palaa seuraavana päivänä. Kuume on noussut 39,2 °C:een, suun avaus onnistuu enää 10 mm, alaleuan oikea puoli on selvästi turvonnut ja nieleminen on kivuliasta. Hengitys on hieman rohisevaa. Mitä toimenpiteitä suosittelet tässä vaiheessa?
|
||||||
|
|
||||||
|
**Assistant:**
|
||||||
|
### Tausta
|
||||||
|
Tilanne on edennyt nopeasti: kuume nousee entisestään yli 39 °C:n tasolle, suun avaus vähenee merkittävästi vain noin 10 mm:iin sekä kasvoalueen turvotus laajenee myös kaulan alueelle. Nieleminen muuttuu kivuliaaksi ja hengityksessä ilmenee rohinaa – tämä viittaa mahdolliseen syvemmälle levinneeseen infektioprosessiin kuten submandibulaariseen absessiin tai Ludwig'n anginaan.
|
||||||
|
|
||||||
|
### Arvio
|
||||||
|
Kyseessä on henkeä uhkaavan infektion alkuvaihe: hengitysteiden ahtauma voi johtua pehmytkudosödeemasta tai varhaisesta mediastiniitista. Trismus yhdessä laaja-alaisen turvotuksen kanssa nostaa riskin hengitystiekomplikaatioihin erityisesti nuorilla aikuisilla. Tällaisissa tapauksissa pelkkä polikliininen seuranta ei riitä vaan tarvitaan välitöntä kirurgista interventiota.
|
||||||
|
|
||||||
|
### Suositus
|
||||||
|
Suosittelen kiireellistä lähettämistä erikoissairaanhoidon yksikköön suukirurgiaan tai korva-, nenä- ja kurkkutauteihin päivystyksellisesti. Sairaalaolosuhteissa voidaan tehdä tarvittavat kuvantamistutkimukset (esim. KKTT), aloittaa suonensisäinen antibioottihoito sekä harkita kirurgista dreneerausta anestesiassa. Hengitysteiden turvaaminen priorisoidaan aina ennen muita toimenpiteitä.
|
||||||
|
|
||||||
|
## Training Details
|
||||||
|
- **Method:** Full fine-tuning (all parameters trained)
|
||||||
|
- **Base Model:** Finnish-NLP/Ahma-2-4B-Instruct (4B parameters)
|
||||||
|
- **Learning Rate:** 5e-5
|
||||||
|
- **Epochs:** 2 (checkpoint at 2968 steps)
|
||||||
|
- **Quantization:** None (full precision training)
|
||||||
|
- **Batch Size:** 8, gradient_accumulation=4 (effective batch size 32)
|
||||||
|
- **Precision:** BF16 (native precision for Ahma-2)
|
||||||
|
- **Data:** 50,132 conversations covering a broad range of different scenarios
|
||||||
|
- **Training Context:** 2048 tokens (model supports up to 128K input / 8K output)
|
||||||
|
- **Hardware:** H100 GPU with Unsloth optimizations
|
||||||
|
- **Training Performance:** Eval loss 0.426, train loss 0.331 at release checkpoint
|
||||||
|
|
||||||
|
## Limitations and Important Notes
|
||||||
|
|
||||||
|
**Critical Medical Disclaimers:**
|
||||||
|
- Research purposes only - never use for actual medical decisions
|
||||||
|
- Model may hallucinate medical facts, treatments, or drug interactions
|
||||||
|
- Outputs have not been clinically validated and may contain medical misinformation
|
||||||
|
- All outputs require independent professional verification before any clinical application
|
||||||
|
- No liability is assumed for any consequences of model use
|
||||||
|
|
||||||
|
**Technical Limitations:**
|
||||||
|
- Primarily trained for dentist-to-dentist consultations - concept explanations included but not the main focus
|
||||||
|
- May particularly hallucinate on out-of-scope topics (further training could address this)
|
||||||
|
- Further fine-tuning may be needed for specific use cases
|
||||||
|
- Training used 2048 token sequences; inference supports full 128K/8K context but quality on very long sequences is untested
|
||||||
|
- No formal evaluation - quality assessment has been primarily subjective
|
||||||
|
|
||||||
|
**Bias and Fairness:**
|
||||||
|
- Training data consists of synthetic conversations which may contain inherent biases
|
||||||
|
- Model responses may reflect biases present in the GPT-4.1 generated training data
|
||||||
|
- Geographic and cultural biases toward Finnish dental practices and protocols
|
||||||
|
- Potential underrepresentation of certain patient demographics or clinical scenarios
|
||||||
|
|
||||||
|
## Related Models
|
||||||
|
- v2 LoRA version: [ducklingcodehouse/Finnish-DentalQA-v2-lora](https://huggingface.co/ducklingcodehouse/Finnish-DentalQA-v2-lora)
|
||||||
|
- v2 Merged version: [ducklingcodehouse/Finnish-DentalQA-v2-merged](https://huggingface.co/ducklingcodehouse/Finnish-DentalQA-v2-merged)
|
||||||
|
- v1 LoRA version: [ducklingcodehouse/Finnish-DentalQA-lora](https://huggingface.co/ducklingcodehouse/Finnish-DentalQA-lora)
|
||||||
|
- v1 Merged version: [ducklingcodehouse/Finnish-DentalQA-merged](https://huggingface.co/ducklingcodehouse/Finnish-DentalQA-merged)
|
||||||
|
- Base model: [Finnish-NLP/Ahma-2-4B-Instruct](https://huggingface.co/Finnish-NLP/Ahma-2-4B-Instruct)
|
||||||
|
|
||||||
|
## Citation
|
||||||
|
|
||||||
|
If you use this model, please cite both this work and the base Ahma model:
|
||||||
|
|
||||||
|
```bibtex
|
||||||
|
@misc{finnish-dentalqa-v3,
|
||||||
|
author = {Saxén, Heikki},
|
||||||
|
title = {Finnish-DentalQA-v3: Full Fine-tuned Model for Finnish Dental Medicine},
|
||||||
|
year = {2026},
|
||||||
|
publisher = {Hugging Face},
|
||||||
|
url = {https://huggingface.co/ducklingcodehouse/Finnish-DentalQA-v3}
|
||||||
|
}
|
||||||
|
|
||||||
|
@misc{ahma-2-4b-instruct,
|
||||||
|
author = {Tanskanen, Aapo and Toivanen, Rasmus},
|
||||||
|
title = {Ahma-2-4B-Instruct},
|
||||||
|
year = {2025},
|
||||||
|
publisher = {Hugging Face},
|
||||||
|
url = {https://huggingface.co/Finnish-NLP/Ahma-2-4B-Instruct}
|
||||||
|
}
|
||||||
|
```
|
||||||
3
added_tokens.json
Normal file
3
added_tokens.json
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
{
|
||||||
|
"<image_soft_token>": 262144
|
||||||
|
}
|
||||||
47
chat_template.jinja
Normal file
47
chat_template.jinja
Normal file
@@ -0,0 +1,47 @@
|
|||||||
|
{{ bos_token }}
|
||||||
|
{%- if messages[0]['role'] == 'system' -%}
|
||||||
|
{%- if messages[0]['content'] is string -%}
|
||||||
|
{%- set first_user_prefix = messages[0]['content'] + '
|
||||||
|
|
||||||
|
' -%}
|
||||||
|
{%- else -%}
|
||||||
|
{%- set first_user_prefix = messages[0]['content'][0]['text'] + '
|
||||||
|
|
||||||
|
' -%}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- set loop_messages = messages[1:] -%}
|
||||||
|
{%- else -%}
|
||||||
|
{%- set first_user_prefix = "" -%}
|
||||||
|
{%- set loop_messages = messages -%}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- for message in loop_messages -%}
|
||||||
|
{%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
|
||||||
|
{{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- if (message['role'] == 'assistant') -%}
|
||||||
|
{%- set role = "model" -%}
|
||||||
|
{%- else -%}
|
||||||
|
{%- set role = message['role'] -%}
|
||||||
|
{%- endif -%}
|
||||||
|
{{ '<start_of_turn>' + role + '
|
||||||
|
' + (first_user_prefix if loop.first else "") }}
|
||||||
|
{%- if message['content'] is string -%}
|
||||||
|
{{ message['content'] | trim }}
|
||||||
|
{%- elif message['content'] is iterable -%}
|
||||||
|
{%- for item in message['content'] -%}
|
||||||
|
{%- if item['type'] == 'image' -%}
|
||||||
|
{{ '<start_of_image>' }}
|
||||||
|
{%- elif item['type'] == 'text' -%}
|
||||||
|
{{ item['text'] | trim }}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- endfor -%}
|
||||||
|
{%- else -%}
|
||||||
|
{{ raise_exception("Invalid content type") }}
|
||||||
|
{%- endif -%}
|
||||||
|
{{ '<end_of_turn>
|
||||||
|
' }}
|
||||||
|
{%- endfor -%}
|
||||||
|
{%- if add_generation_prompt -%}
|
||||||
|
{{ '<start_of_turn>model
|
||||||
|
' }}
|
||||||
|
{%- endif -%}
|
||||||
74
config.json
Normal file
74
config.json
Normal file
@@ -0,0 +1,74 @@
|
|||||||
|
{
|
||||||
|
"_sliding_window_pattern": 6,
|
||||||
|
"architectures": [
|
||||||
|
"Gemma3ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"attn_logit_softcapping": null,
|
||||||
|
"bos_token_id": 2,
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"eos_token_id": 1,
|
||||||
|
"final_logit_softcapping": null,
|
||||||
|
"head_dim": 256,
|
||||||
|
"hidden_activation": "gelu_pytorch_tanh",
|
||||||
|
"hidden_size": 2560,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 10240,
|
||||||
|
"layer_types": [
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention"
|
||||||
|
],
|
||||||
|
"max_position_embeddings": 131072,
|
||||||
|
"model_type": "gemma3_text",
|
||||||
|
"num_attention_heads": 8,
|
||||||
|
"num_hidden_layers": 34,
|
||||||
|
"num_key_value_heads": 4,
|
||||||
|
"pad_token_id": 0,
|
||||||
|
"query_pre_attn_scalar": 256,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_local_base_freq": 10000.0,
|
||||||
|
"rope_scaling": {
|
||||||
|
"factor": 8.0,
|
||||||
|
"rope_type": "linear"
|
||||||
|
},
|
||||||
|
"rope_theta": 1000000.0,
|
||||||
|
"sliding_window": 1024,
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"transformers_version": "4.55.2",
|
||||||
|
"unsloth_version": "2025.9.6",
|
||||||
|
"use_cache": true,
|
||||||
|
"vocab_size": 262144
|
||||||
|
}
|
||||||
14
generation_config.json
Normal file
14
generation_config.json
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"bos_token_id": 2,
|
||||||
|
"cache_implementation": "hybrid",
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
1,
|
||||||
|
106
|
||||||
|
],
|
||||||
|
"max_length": 131072,
|
||||||
|
"pad_token_id": 0,
|
||||||
|
"top_k": 64,
|
||||||
|
"top_p": 0.95,
|
||||||
|
"transformers_version": "4.55.2"
|
||||||
|
}
|
||||||
3
model-00001-of-00002.safetensors
Normal file
3
model-00001-of-00002.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:9c5b3dda0f6214e8259ae7287ce624ab5b12d9b40a93f7ed58e8b9b177fad2a6
|
||||||
|
size 4960203664
|
||||||
3
model-00002-of-00002.safetensors
Normal file
3
model-00002-of-00002.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:1175819ff1af533e8e6a81de199ed71e89cbf18747f5b5b585a996b5b659174c
|
||||||
|
size 2800046672
|
||||||
452
model.safetensors.index.json
Normal file
452
model.safetensors.index.json
Normal file
@@ -0,0 +1,452 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"total_parameters": 3880099328,
|
||||||
|
"total_size": 7760198656
|
||||||
|
},
|
||||||
|
"weight_map": {
|
||||||
|
"model.embed_tokens.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.19.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.19.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.20.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.28.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.29.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.30.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.30.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.31.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.32.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.33.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.norm.weight": "model-00002-of-00002.safetensors"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:caa3cdc993dbadfd9dbd3046e888377edd5f0216c12a7330873defd9a1937c9b
|
||||||
|
size 29055
|
||||||
3
runs/scheduler.pt
Normal file
3
runs/scheduler.pt
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:dc46cf5cb0a3a5cb5924a3f63369af6e90e0fbedacba55a4ef259a0274916a70
|
||||||
|
size 1064
|
||||||
511
runs/trainer_state.json
Normal file
511
runs/trainer_state.json
Normal file
@@ -0,0 +1,511 @@
|
|||||||
|
{
|
||||||
|
"best_global_step": null,
|
||||||
|
"best_metric": null,
|
||||||
|
"best_model_checkpoint": null,
|
||||||
|
"epoch": 1.9976438909458096,
|
||||||
|
"eval_steps": 371,
|
||||||
|
"global_step": 2968,
|
||||||
|
"is_hyper_param_search": false,
|
||||||
|
"is_local_process_zero": true,
|
||||||
|
"is_world_process_zero": true,
|
||||||
|
"log_history": [
|
||||||
|
{
|
||||||
|
"epoch": 0.03365870077415012,
|
||||||
|
"grad_norm": 2.984375,
|
||||||
|
"learning_rate": 1.8421052631578947e-05,
|
||||||
|
"loss": 1.0907,
|
||||||
|
"step": 50
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.06731740154830024,
|
||||||
|
"grad_norm": 2.6875,
|
||||||
|
"learning_rate": 3.721804511278196e-05,
|
||||||
|
"loss": 0.7488,
|
||||||
|
"step": 100
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.10097610232245036,
|
||||||
|
"grad_norm": 1.9453125,
|
||||||
|
"learning_rate": 4.999831161062044e-05,
|
||||||
|
"loss": 0.7217,
|
||||||
|
"step": 150
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.13463480309660047,
|
||||||
|
"grad_norm": 1.734375,
|
||||||
|
"learning_rate": 4.997127617815812e-05,
|
||||||
|
"loss": 0.7079,
|
||||||
|
"step": 200
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.1682935038707506,
|
||||||
|
"grad_norm": 1.46875,
|
||||||
|
"learning_rate": 4.9911305528702e-05,
|
||||||
|
"loss": 0.6731,
|
||||||
|
"step": 250
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.2019522046449007,
|
||||||
|
"grad_norm": 1.4609375,
|
||||||
|
"learning_rate": 4.981847875898461e-05,
|
||||||
|
"loss": 0.6538,
|
||||||
|
"step": 300
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.23561090541905083,
|
||||||
|
"grad_norm": 1.40625,
|
||||||
|
"learning_rate": 4.9692918300466075e-05,
|
||||||
|
"loss": 0.6337,
|
||||||
|
"step": 350
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.24974755974419388,
|
||||||
|
"eval_loss": 0.6199322938919067,
|
||||||
|
"eval_runtime": 100.4004,
|
||||||
|
"eval_samples_per_second": 24.93,
|
||||||
|
"eval_steps_per_second": 6.235,
|
||||||
|
"step": 371
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.26926960619320095,
|
||||||
|
"grad_norm": 1.3203125,
|
||||||
|
"learning_rate": 4.953478975785629e-05,
|
||||||
|
"loss": 0.6167,
|
||||||
|
"step": 400
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.30292830696735107,
|
||||||
|
"grad_norm": 1.2421875,
|
||||||
|
"learning_rate": 4.934430169069488e-05,
|
||||||
|
"loss": 0.6001,
|
||||||
|
"step": 450
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.3365870077415012,
|
||||||
|
"grad_norm": 1.171875,
|
||||||
|
"learning_rate": 4.9121705338277035e-05,
|
||||||
|
"loss": 0.595,
|
||||||
|
"step": 500
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.3702457085156513,
|
||||||
|
"grad_norm": 1.140625,
|
||||||
|
"learning_rate": 4.886729428828791e-05,
|
||||||
|
"loss": 0.5824,
|
||||||
|
"step": 550
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.4039044092898014,
|
||||||
|
"grad_norm": 1.09375,
|
||||||
|
"learning_rate": 4.858140408958282e-05,
|
||||||
|
"loss": 0.5798,
|
||||||
|
"step": 600
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.43756311006395154,
|
||||||
|
"grad_norm": 1.0546875,
|
||||||
|
"learning_rate": 4.826441180962373e-05,
|
||||||
|
"loss": 0.563,
|
||||||
|
"step": 650
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.47122181083810166,
|
||||||
|
"grad_norm": 1.140625,
|
||||||
|
"learning_rate": 4.7916735537155956e-05,
|
||||||
|
"loss": 0.561,
|
||||||
|
"step": 700
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.49949511948838776,
|
||||||
|
"eval_loss": 0.563587486743927,
|
||||||
|
"eval_runtime": 100.0625,
|
||||||
|
"eval_samples_per_second": 25.014,
|
||||||
|
"eval_steps_per_second": 6.256,
|
||||||
|
"step": 742
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.5048805116122518,
|
||||||
|
"grad_norm": 1.0703125,
|
||||||
|
"learning_rate": 4.753883383078079e-05,
|
||||||
|
"loss": 0.5537,
|
||||||
|
"step": 750
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.5385392123864019,
|
||||||
|
"grad_norm": 1.0234375,
|
||||||
|
"learning_rate": 4.71312051141516e-05,
|
||||||
|
"loss": 0.5381,
|
||||||
|
"step": 800
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.572197913160552,
|
||||||
|
"grad_norm": 1.0390625,
|
||||||
|
"learning_rate": 4.669438701859086e-05,
|
||||||
|
"loss": 0.5376,
|
||||||
|
"step": 850
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.6058566139347021,
|
||||||
|
"grad_norm": 0.98046875,
|
||||||
|
"learning_rate": 4.6228955673995206e-05,
|
||||||
|
"loss": 0.5311,
|
||||||
|
"step": 900
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.6395153147088523,
|
||||||
|
"grad_norm": 0.94921875,
|
||||||
|
"learning_rate": 4.5735524948963945e-05,
|
||||||
|
"loss": 0.5254,
|
||||||
|
"step": 950
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.6731740154830024,
|
||||||
|
"grad_norm": 0.94921875,
|
||||||
|
"learning_rate": 4.521474564115293e-05,
|
||||||
|
"loss": 0.5235,
|
||||||
|
"step": 1000
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.7068327162571525,
|
||||||
|
"grad_norm": 1.0,
|
||||||
|
"learning_rate": 4.4667304618921875e-05,
|
||||||
|
"loss": 0.517,
|
||||||
|
"step": 1050
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.7404914170313026,
|
||||||
|
"grad_norm": 0.98828125,
|
||||||
|
"learning_rate": 4.4093923915407166e-05,
|
||||||
|
"loss": 0.5065,
|
||||||
|
"step": 1100
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.7492426792325816,
|
||||||
|
"eval_loss": 0.5226994156837463,
|
||||||
|
"eval_runtime": 99.9631,
|
||||||
|
"eval_samples_per_second": 25.039,
|
||||||
|
"eval_steps_per_second": 6.262,
|
||||||
|
"step": 1113
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.7741501178054527,
|
||||||
|
"grad_norm": 1.0,
|
||||||
|
"learning_rate": 4.349535977621498e-05,
|
||||||
|
"loss": 0.5059,
|
||||||
|
"step": 1150
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.8078088185796028,
|
||||||
|
"grad_norm": 0.95703125,
|
||||||
|
"learning_rate": 4.2872401661990755e-05,
|
||||||
|
"loss": 0.4953,
|
||||||
|
"step": 1200
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.841467519353753,
|
||||||
|
"grad_norm": 0.97265625,
|
||||||
|
"learning_rate": 4.222587120718056e-05,
|
||||||
|
"loss": 0.497,
|
||||||
|
"step": 1250
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.8751262201279031,
|
||||||
|
"grad_norm": 0.87890625,
|
||||||
|
"learning_rate": 4.155662113635761e-05,
|
||||||
|
"loss": 0.4887,
|
||||||
|
"step": 1300
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.9087849209020532,
|
||||||
|
"grad_norm": 0.93359375,
|
||||||
|
"learning_rate": 4.0865534139543396e-05,
|
||||||
|
"loss": 0.492,
|
||||||
|
"step": 1350
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.9424436216762033,
|
||||||
|
"grad_norm": 0.890625,
|
||||||
|
"learning_rate": 4.0153521708006474e-05,
|
||||||
|
"loss": 0.4841,
|
||||||
|
"step": 1400
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.9761023224503534,
|
||||||
|
"grad_norm": 0.96875,
|
||||||
|
"learning_rate": 3.9421522932074786e-05,
|
||||||
|
"loss": 0.4799,
|
||||||
|
"step": 1450
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 0.9989902389767755,
|
||||||
|
"eval_loss": 0.4874838888645172,
|
||||||
|
"eval_runtime": 98.6858,
|
||||||
|
"eval_samples_per_second": 25.363,
|
||||||
|
"eval_steps_per_second": 6.343,
|
||||||
|
"step": 1484
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.009424436216762,
|
||||||
|
"grad_norm": 0.7890625,
|
||||||
|
"learning_rate": 3.867050326254679e-05,
|
||||||
|
"loss": 0.4807,
|
||||||
|
"step": 1500
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.0430831369909122,
|
||||||
|
"grad_norm": 0.80078125,
|
||||||
|
"learning_rate": 3.790145323733514e-05,
|
||||||
|
"loss": 0.385,
|
||||||
|
"step": 1550
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.0767418377650624,
|
||||||
|
"grad_norm": 0.77734375,
|
||||||
|
"learning_rate": 3.71153871750225e-05,
|
||||||
|
"loss": 0.388,
|
||||||
|
"step": 1600
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.1104005385392124,
|
||||||
|
"grad_norm": 0.734375,
|
||||||
|
"learning_rate": 3.631334183705223e-05,
|
||||||
|
"loss": 0.3861,
|
||||||
|
"step": 1650
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.1440592393133624,
|
||||||
|
"grad_norm": 0.89453125,
|
||||||
|
"learning_rate": 3.5496375060318876e-05,
|
||||||
|
"loss": 0.3832,
|
||||||
|
"step": 1700
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.1777179400875126,
|
||||||
|
"grad_norm": 0.84375,
|
||||||
|
"learning_rate": 3.46655643619616e-05,
|
||||||
|
"loss": 0.3813,
|
||||||
|
"step": 1750
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.2113766408616629,
|
||||||
|
"grad_norm": 0.890625,
|
||||||
|
"learning_rate": 3.3822005518200875e-05,
|
||||||
|
"loss": 0.3817,
|
||||||
|
"step": 1800
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.2450353416358129,
|
||||||
|
"grad_norm": 0.72265625,
|
||||||
|
"learning_rate": 3.296681111909288e-05,
|
||||||
|
"loss": 0.3789,
|
||||||
|
"step": 1850
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.248401211713228,
|
||||||
|
"eval_loss": 0.46673256158828735,
|
||||||
|
"eval_runtime": 100.3992,
|
||||||
|
"eval_samples_per_second": 24.93,
|
||||||
|
"eval_steps_per_second": 6.235,
|
||||||
|
"step": 1855
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.2786940424099629,
|
||||||
|
"grad_norm": 0.921875,
|
||||||
|
"learning_rate": 3.210110910110775e-05,
|
||||||
|
"loss": 0.378,
|
||||||
|
"step": 1900
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.312352743184113,
|
||||||
|
"grad_norm": 0.89453125,
|
||||||
|
"learning_rate": 3.122604125946696e-05,
|
||||||
|
"loss": 0.3724,
|
||||||
|
"step": 1950
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.3460114439582633,
|
||||||
|
"grad_norm": 0.76171875,
|
||||||
|
"learning_rate": 3.0342761742202137e-05,
|
||||||
|
"loss": 0.3729,
|
||||||
|
"step": 2000
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.3796701447324133,
|
||||||
|
"grad_norm": 0.85546875,
|
||||||
|
"learning_rate": 2.9452435527921506e-05,
|
||||||
|
"loss": 0.3688,
|
||||||
|
"step": 2050
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.4133288455065633,
|
||||||
|
"grad_norm": 0.72265625,
|
||||||
|
"learning_rate": 2.855623688929152e-05,
|
||||||
|
"loss": 0.3681,
|
||||||
|
"step": 2100
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.4469875462807136,
|
||||||
|
"grad_norm": 0.90234375,
|
||||||
|
"learning_rate": 2.7655347844260417e-05,
|
||||||
|
"loss": 0.3629,
|
||||||
|
"step": 2150
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.4806462470548638,
|
||||||
|
"grad_norm": 0.82421875,
|
||||||
|
"learning_rate": 2.6750956597066286e-05,
|
||||||
|
"loss": 0.3637,
|
||||||
|
"step": 2200
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.4981487714574218,
|
||||||
|
"eval_loss": 0.4537140727043152,
|
||||||
|
"eval_runtime": 100.1826,
|
||||||
|
"eval_samples_per_second": 24.984,
|
||||||
|
"eval_steps_per_second": 6.249,
|
||||||
|
"step": 2226
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.5143049478290138,
|
||||||
|
"grad_norm": 0.91015625,
|
||||||
|
"learning_rate": 2.5844255971085913e-05,
|
||||||
|
"loss": 0.3652,
|
||||||
|
"step": 2250
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.5479636486031638,
|
||||||
|
"grad_norm": 0.7890625,
|
||||||
|
"learning_rate": 2.493644183559134e-05,
|
||||||
|
"loss": 0.357,
|
||||||
|
"step": 2300
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.581622349377314,
|
||||||
|
"grad_norm": 0.73046875,
|
||||||
|
"learning_rate": 2.4028711528489107e-05,
|
||||||
|
"loss": 0.3573,
|
||||||
|
"step": 2350
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.6152810501514643,
|
||||||
|
"grad_norm": 0.78515625,
|
||||||
|
"learning_rate": 2.312226227712245e-05,
|
||||||
|
"loss": 0.356,
|
||||||
|
"step": 2400
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.6489397509256143,
|
||||||
|
"grad_norm": 0.78125,
|
||||||
|
"learning_rate": 2.221828961921939e-05,
|
||||||
|
"loss": 0.3532,
|
||||||
|
"step": 2450
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.6825984516997643,
|
||||||
|
"grad_norm": 0.7890625,
|
||||||
|
"learning_rate": 2.1317985826069228e-05,
|
||||||
|
"loss": 0.3496,
|
||||||
|
"step": 2500
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.7162571524739145,
|
||||||
|
"grad_norm": 0.78515625,
|
||||||
|
"learning_rate": 2.042253833000726e-05,
|
||||||
|
"loss": 0.3479,
|
||||||
|
"step": 2550
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.7478963312016156,
|
||||||
|
"eval_loss": 0.4370940923690796,
|
||||||
|
"eval_runtime": 100.667,
|
||||||
|
"eval_samples_per_second": 24.864,
|
||||||
|
"eval_steps_per_second": 6.219,
|
||||||
|
"step": 2597
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.7499158532480648,
|
||||||
|
"grad_norm": 0.9609375,
|
||||||
|
"learning_rate": 1.9533128158281727e-05,
|
||||||
|
"loss": 0.3489,
|
||||||
|
"step": 2600
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.7835745540222148,
|
||||||
|
"grad_norm": 0.89453125,
|
||||||
|
"learning_rate": 1.865092837536851e-05,
|
||||||
|
"loss": 0.3438,
|
||||||
|
"step": 2650
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.8172332547963648,
|
||||||
|
"grad_norm": 0.859375,
|
||||||
|
"learning_rate": 1.777710253578821e-05,
|
||||||
|
"loss": 0.3439,
|
||||||
|
"step": 2700
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.850891955570515,
|
||||||
|
"grad_norm": 0.76953125,
|
||||||
|
"learning_rate": 1.6912803149466065e-05,
|
||||||
|
"loss": 0.3417,
|
||||||
|
"step": 2750
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.8845506563446652,
|
||||||
|
"grad_norm": 0.84765625,
|
||||||
|
"learning_rate": 1.605917016165891e-05,
|
||||||
|
"loss": 0.3398,
|
||||||
|
"step": 2800
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.9182093571188152,
|
||||||
|
"grad_norm": 0.8515625,
|
||||||
|
"learning_rate": 1.5217329449453915e-05,
|
||||||
|
"loss": 0.3401,
|
||||||
|
"step": 2850
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.9518680578929652,
|
||||||
|
"grad_norm": 0.8671875,
|
||||||
|
"learning_rate": 1.4388391336822216e-05,
|
||||||
|
"loss": 0.3389,
|
||||||
|
"step": 2900
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.9855267586671155,
|
||||||
|
"grad_norm": 0.890625,
|
||||||
|
"learning_rate": 1.3573449130185945e-05,
|
||||||
|
"loss": 0.3311,
|
||||||
|
"step": 2950
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"epoch": 1.9976438909458096,
|
||||||
|
"eval_loss": 0.4261794984340668,
|
||||||
|
"eval_runtime": 99.6602,
|
||||||
|
"eval_samples_per_second": 25.115,
|
||||||
|
"eval_steps_per_second": 6.281,
|
||||||
|
"step": 2968
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"logging_steps": 50,
|
||||||
|
"max_steps": 4458,
|
||||||
|
"num_input_tokens_seen": 0,
|
||||||
|
"num_train_epochs": 3,
|
||||||
|
"save_steps": 371,
|
||||||
|
"stateful_callbacks": {
|
||||||
|
"TrainerControl": {
|
||||||
|
"args": {
|
||||||
|
"should_epoch_stop": false,
|
||||||
|
"should_evaluate": false,
|
||||||
|
"should_log": false,
|
||||||
|
"should_save": true,
|
||||||
|
"should_training_stop": false
|
||||||
|
},
|
||||||
|
"attributes": {}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"total_flos": 1.7464408109253181e+18,
|
||||||
|
"train_batch_size": 8,
|
||||||
|
"trial_name": null,
|
||||||
|
"trial_params": null
|
||||||
|
}
|
||||||
3
runs/training_args.bin
Normal file
3
runs/training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:be3aedce61c0e1e2162640ad3c0164e6be1f9480239684b78b0929faae771af0
|
||||||
|
size 5816
|
||||||
33
special_tokens_map.json
Normal file
33
special_tokens_map.json
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
{
|
||||||
|
"boi_token": "<start_of_image>",
|
||||||
|
"bos_token": {
|
||||||
|
"content": "<bos>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"eoi_token": "<end_of_image>",
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<eos>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"image_token": "<image_soft_token>",
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<pad>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"unk_token": {
|
||||||
|
"content": "<unk>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
|
||||||
|
size 33384568
|
||||||
3
tokenizer.model
Normal file
3
tokenizer.model
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
|
||||||
|
size 4689074
|
||||||
51350
tokenizer_config.json
Normal file
51350
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user