初始化项目,由ModelHub XC社区提供模型
Model: wvnvwn/Meta-Llama-3-8B-Instruct-fedavg-v0 Source: Original Platform
This commit is contained in:
85
README.md
Normal file
85
README.md
Normal file
@@ -0,0 +1,85 @@
|
||||
---
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
base_model:
|
||||
- meta-llama/Meta-Llama-3-8B-Instruct
|
||||
tags:
|
||||
- peft
|
||||
- lora
|
||||
- merged
|
||||
---
|
||||
# Meta-Llama-3-8B-Instruct-fedavg-v0
|
||||
|
||||
This repository contains a full merged model produced by applying a PEFT LoRA adapter to its base model. It is intended for reproducible evaluation without requiring a separate adapter loading path.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "wvnvwn/Meta-Llama-3-8B-Instruct-fedavg-v0"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
)
|
||||
```
|
||||
|
||||
Run the bundled FSL evaluation wrapper:
|
||||
|
||||
```bash
|
||||
bash fsl/src/evaluation/overall_eval.sh \
|
||||
--model_name wvnvwn/Meta-Llama-3-8B-Instruct-fedavg-v0 \
|
||||
--model_type instruct \
|
||||
--max_samples 100 \
|
||||
--output_root fsl/results/evaluation/Meta-Llama-3-8B-Instruct-fedavg-v0
|
||||
```
|
||||
|
||||
## Training Procedure
|
||||
|
||||
- Procedure: Federated LoRA fine-tuning followed by adapter aggregation. The resulting PEFT LoRA adapter was merged into the base model for reproducible evaluation.
|
||||
- Algorithm: `fedavg`
|
||||
- Training data: `data_hetero_with_4_tasks`
|
||||
- Number of clients: `8`
|
||||
- Communication round/checkpoint: `3`
|
||||
- Local epochs: `3`
|
||||
- Local batch size: `256`
|
||||
- Local micro batch size: `16`
|
||||
- Local learning rate: `0.0003`
|
||||
|
||||
Original training command:
|
||||
|
||||
Not specified
|
||||
|
||||
## Merge Metadata
|
||||
|
||||
- Base model: `meta-llama/Meta-Llama-3-8B-Instruct`
|
||||
- Adapter source: `/NHNHOME/0226010080_A/BASE/jongbokwon/FLS/fsl/outputs/fedavg/8/2`
|
||||
- PEFT type: `LORA`
|
||||
- Task type: `CAUSAL_LM`
|
||||
- LoRA rank: `16`
|
||||
- LoRA alpha: `16`
|
||||
- Target modules: `up_proj`, `v_proj`, `gate_proj`, `q_proj`, `k_proj`, `o_proj`, `down_proj`
|
||||
- Merged at UTC: `2026-05-22T05:24:31.213169+00:00`
|
||||
|
||||
## Framework Versions
|
||||
|
||||
- python: `3.10.20`
|
||||
- platform: `Linux-6.8.0-100-generic-x86_64-with-glibc2.39`
|
||||
- torch: `2.11.0+cu130`
|
||||
- cuda: `13.0`
|
||||
- transformers: `4.57.1`
|
||||
- peft: `0.19.1`
|
||||
- accelerate: `1.13.0`
|
||||
- huggingface_hub: `0.36.2`
|
||||
- safetensors: `0.7.0`
|
||||
- vllm: `0.20.2`
|
||||
- wandb: `0.27.0`
|
||||
|
||||
## Notes
|
||||
|
||||
- This is a merged full-weight model, not an adapter-only checkpoint.
|
||||
- Redistribution/access should follow the base model license and access policy.
|
||||
- `merge_info.json` in this repository records the adapter and merge configuration.
|
||||
Reference in New Issue
Block a user