---
pipeline_tag: text-generation
library_name: transformers
tags:
- text-generation
- metadata-localization
- chat
- with-metadata
- sft
- lora-merged
---

# combined_with_metadata_chat

## Summary

This repo contains the merged chat model for the combined with metadata branch of the metadata localization project. It was produced by supervised fine-tuning on the project QA benchmark after project pretraining.

## Variant Metadata

- Stage: `sft_chat`
- Family: `chat`
- Metadata condition: `with_metadata`
- Base model lineage: `combined_with_metadata_1b`

## Weights & Biases Provenance

- Run name: `still-dew-1`
- Internal run URL: `https://wandb.ai/iamshnoo/huggingface/runs/x9ixdg0u`
- Note: the Weights & Biases workspace is private; public readers should use the summarized metrics and configuration below.
- State: `failed`
- Runtime: `3h 44m 45s`

## Run Summary

- `train/loss`: `1.2337`
- `train/global_step`: `7,764`
- `train/epoch`: `3`
- `train/learning_rate`: `0`
- `train/grad_norm`: `0.0725`

## Training Configuration

- `per_device_train_batch_size`: `2`
- `gradient_accumulation_steps`: `8`
- `learning_rate`: `0.0002`
- `num_train_epochs`: `3`
- `optim`: `adamw_bnb_8bit`
- `bf16`: `True`
- `gradient_checkpointing`: `True`
- `use_liger_kernel`: `True`

## SFT Notes

- Fine-tuning method: `PEFT / LoRA`
- Optimizer: `adamw_bnb_8bit`
- `bf16=True`, `gradient_checkpointing=True`, `use_liger_kernel=True`
- `per_device_train_batch_size=2`, `gradient_accumulation_steps=8`
- LoRA targets: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`

## Training Curves

Static plots below were exported from the private Weights & Biases run and embedded here for public access.

### Train Loss

![Train Loss](assets/train_loss.png)

### Learning Rate

![Learning Rate](assets/learning_rate.png)

### Gradient Norm

![Gradient Norm](assets/grad_norm.png)

## Project Context

This model is part of the metadata localization release. Related checkpoints and variants are grouped in the public Hugging Face collection [Metadata Conditioned LLMs](https://huggingface.co/collections/iamshnoo/metadata-conditioned-llms).
- Training data source: [News on the Web (NOW) Corpus](https://www.english-corpora.org/now/)
- Project repository: [https://github.com/iamshnoo/metadata_localization](https://github.com/iamshnoo/metadata_localization)
- Paper: [https://arxiv.org/abs/2601.15236](https://arxiv.org/abs/2601.15236)

Last synced: `2026-04-02 14:48:16 UTC`