86 lines
2.3 KiB
Markdown
86 lines
2.3 KiB
Markdown
---
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
base_model:
|
|
- meta-llama/Meta-Llama-3-8B-Instruct
|
|
tags:
|
|
- peft
|
|
- lora
|
|
- merged
|
|
---
|
|
# Meta-Llama-3-8B-Instruct-fedavg-v0
|
|
|
|
This repository contains a full merged model produced by applying a PEFT LoRA adapter to its base model. It is intended for reproducible evaluation without requiring a separate adapter loading path.
|
|
|
|
## Quick Start
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "wvnvwn/Meta-Llama-3-8B-Instruct-fedavg-v0"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_id,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto",
|
|
)
|
|
```
|
|
|
|
Run the bundled FSL evaluation wrapper:
|
|
|
|
```bash
|
|
bash fsl/src/evaluation/overall_eval.sh \
|
|
--model_name wvnvwn/Meta-Llama-3-8B-Instruct-fedavg-v0 \
|
|
--model_type instruct \
|
|
--max_samples 100 \
|
|
--output_root fsl/results/evaluation/Meta-Llama-3-8B-Instruct-fedavg-v0
|
|
```
|
|
|
|
## Training Procedure
|
|
|
|
- Procedure: Federated LoRA fine-tuning followed by adapter aggregation. The resulting PEFT LoRA adapter was merged into the base model for reproducible evaluation.
|
|
- Algorithm: `fedavg`
|
|
- Training data: `data_hetero_with_4_tasks`
|
|
- Number of clients: `8`
|
|
- Communication round/checkpoint: `3`
|
|
- Local epochs: `3`
|
|
- Local batch size: `256`
|
|
- Local micro batch size: `16`
|
|
- Local learning rate: `0.0003`
|
|
|
|
Original training command:
|
|
|
|
Not specified
|
|
|
|
## Merge Metadata
|
|
|
|
- Base model: `meta-llama/Meta-Llama-3-8B-Instruct`
|
|
- Adapter source: `/NHNHOME/0226010080_A/BASE/jongbokwon/FLS/fsl/outputs/fedavg/8/2`
|
|
- PEFT type: `LORA`
|
|
- Task type: `CAUSAL_LM`
|
|
- LoRA rank: `16`
|
|
- LoRA alpha: `16`
|
|
- Target modules: `up_proj`, `v_proj`, `gate_proj`, `q_proj`, `k_proj`, `o_proj`, `down_proj`
|
|
- Merged at UTC: `2026-05-22T05:24:31.213169+00:00`
|
|
|
|
## Framework Versions
|
|
|
|
- python: `3.10.20`
|
|
- platform: `Linux-6.8.0-100-generic-x86_64-with-glibc2.39`
|
|
- torch: `2.11.0+cu130`
|
|
- cuda: `13.0`
|
|
- transformers: `4.57.1`
|
|
- peft: `0.19.1`
|
|
- accelerate: `1.13.0`
|
|
- huggingface_hub: `0.36.2`
|
|
- safetensors: `0.7.0`
|
|
- vllm: `0.20.2`
|
|
- wandb: `0.27.0`
|
|
|
|
## Notes
|
|
|
|
- This is a merged full-weight model, not an adapter-only checkpoint.
|
|
- Redistribution/access should follow the base model license and access policy.
|
|
- `merge_info.json` in this repository records the adapter and merge configuration.
|