---
library_name: transformers
pipeline_tag: text-generation
base_model:
- meta-llama/Meta-Llama-3-8B-Instruct
tags:
- peft
- lora
- merged
---
# Meta-Llama-3-8B-Instruct-fedavg-v0

This repository contains a full merged model produced by applying a PEFT LoRA adapter to its base model. It is intended for reproducible evaluation without requiring a separate adapter loading path.

## Quick Start

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "wvnvwn/Meta-Llama-3-8B-Instruct-fedavg-v0"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
```

Run the bundled FSL evaluation wrapper:

```bash
bash fsl/src/evaluation/overall_eval.sh \
    --model_name wvnvwn/Meta-Llama-3-8B-Instruct-fedavg-v0 \
    --model_type instruct \
    --max_samples 100 \
    --output_root fsl/results/evaluation/Meta-Llama-3-8B-Instruct-fedavg-v0
```

## Training Procedure

- Procedure: Federated LoRA fine-tuning followed by adapter aggregation. The resulting PEFT LoRA adapter was merged into the base model for reproducible evaluation.
- Algorithm: `fedavg`
- Training data: `data_hetero_with_4_tasks`
- Number of clients: `8`
- Communication round/checkpoint: `3`
- Local epochs: `3`
- Local batch size: `256`
- Local micro batch size: `16`
- Local learning rate: `0.0003`

Original training command:

Not specified

## Merge Metadata

- Base model: `meta-llama/Meta-Llama-3-8B-Instruct`
- Adapter source: `/NHNHOME/0226010080_A/BASE/jongbokwon/FLS/fsl/outputs/fedavg/8/2`
- PEFT type: `LORA`
- Task type: `CAUSAL_LM`
- LoRA rank: `16`
- LoRA alpha: `16`
- Target modules: `up_proj`, `v_proj`, `gate_proj`, `q_proj`, `k_proj`, `o_proj`, `down_proj`
- Merged at UTC: `2026-05-22T05:24:31.213169+00:00`

## Framework Versions

- python: `3.10.20`
- platform: `Linux-6.8.0-100-generic-x86_64-with-glibc2.39`
- torch: `2.11.0+cu130`
- cuda: `13.0`
- transformers: `4.57.1`
- peft: `0.19.1`
- accelerate: `1.13.0`
- huggingface_hub: `0.36.2`
- safetensors: `0.7.0`
- vllm: `0.20.2`
- wandb: `0.27.0`

## Notes

- This is a merged full-weight model, not an adapter-only checkpoint.
- Redistribution/access should follow the base model license and access policy.
- `merge_info.json` in this repository records the adapter and merge configuration.