97 lines
2.6 KiB
Markdown
97 lines
2.6 KiB
Markdown
---
|
||
license: gemma
|
||
language:
|
||
- en
|
||
base_model: kth8/gemma-3-270m-it-JSON-Fixer
|
||
datasets:
|
||
- kth8/json-fix-25000x
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- sft
|
||
- trl
|
||
- unsloth
|
||
- google
|
||
- gemma
|
||
- gemma3
|
||
- gemma3_text
|
||
---
|
||
|
||
A fine-tune of [unsloth/gemma-3-270m-it](https://huggingface.co/unsloth/gemma-3-270m-it) on the [kth8/json-fix-25000x](https://huggingface.co/datasets/kth8/json-fix-25000x) dataset.
|
||
|
||
## Usage example
|
||
**System prompt**
|
||
```
|
||
You are a JSON formatting specialist. Convert the provided JSON data into valid JSON format with 2 line indent and no additional commentary.
|
||
```
|
||
**User prompt**
|
||
```
|
||
The JSON is:\n[{\"name\":\"John Doe\", \"jobTitle\":Software Engineer, \"department\": \"Research and Development\"],, {\"name\"\"Jane Smith\", \"jobTitle\":\"Data Analyst', \"department\":\"Marketing and Sales\"}, ] //\" comment\n-- end --
|
||
```
|
||
**Assistant response**
|
||
```
|
||
[
|
||
{
|
||
"name": "John Doe",
|
||
"jobTitle": "Software Engineer",
|
||
"department": "Research and Development"
|
||
},
|
||
{
|
||
"name": "Jane Smith",
|
||
"jobTitle": "Data Analyst",
|
||
"department": "Marketing and Sales"
|
||
}
|
||
]
|
||
```
|
||
## Model Details
|
||
- Base Model: `unsloth/gemma-3-270m-it`
|
||
- Parameter Count: 268,098,176
|
||
- Precision: torch.bfloat16
|
||
|
||
### Hardware
|
||
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
|
||
- Announced: Mar 17th, 2025
|
||
- Release Date: Mar 18th, 2025
|
||
- Memory Type: GDDR7
|
||
- Bandwidth: 1.79 TB/s
|
||
- Memory Size: 96 GB
|
||
- Memory Bus: 512 bit
|
||
- Shading Units: 24064
|
||
- TDP: 600W
|
||
- FP16 (half): 126.0 TFLOPS (1:1)
|
||
|
||
## Training Settings
|
||
### PEFT
|
||
- Rank: 32
|
||
- LoRA alpha: 64
|
||
- Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
||
- Gradient checkpointing: unsloth
|
||
|
||
### SFT
|
||
- Epoch: 2
|
||
- Batch size: 32
|
||
- Gradient Accumulation steps: 1
|
||
- Warmup ratio: 0.05
|
||
- Learning rate: 0.0004
|
||
- Optimizer: adamw_torch_fused
|
||
- Learning rate scheduler: cosine
|
||
- Max seq length: 2048
|
||
|
||
## Training stats
|
||
- Date: 2026-03-23T04:39:38.019077
|
||
- Peak VRAM usage: 64.5 GB
|
||
- Global step: 1538
|
||
- Training runtime (seconds): 1142.9274
|
||
- Average training loss: 0.004019292104312295
|
||
- Final validation loss: 0.0014343492221087217
|
||
|
||
## Framework versions
|
||
- Unsloth: 2026.3.10
|
||
- TRL: 0.22.2
|
||
- Transformers: 4.56.2
|
||
- Pytorch: 2.10.0+cu128
|
||
- Datasets: 4.8.3
|
||
- Tokenizers: 0.22.2
|
||
|
||
## License
|
||
This model is released under the Gemma license. See the [Gemma Terms of Use](https://ai.google.dev/gemma/terms) and [Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy) regarding the use of Gemma-generated content. |