70 lines
1.5 KiB
Markdown
70 lines
1.5 KiB
Markdown
---
|
|
language: en
|
|
license: apache-2.0
|
|
base_model: mistralai/Mistral-7B-Instruct-v0.3
|
|
library_name: transformers
|
|
tags:
|
|
- fine-tuned
|
|
- qlora
|
|
- lora
|
|
- llm
|
|
- instruction-tuning
|
|
- peft
|
|
---
|
|
|
|
# Mistral-7B LLM Architecture Expert
|
|
|
|
A fine-tuned version of Mistral-7B-Instruct-v0.3 trained using QLoRA on a custom dataset focused on LLM architecture concepts and internals.
|
|
|
|
Topics covered include:
|
|
|
|
- Attention mechanisms
|
|
- Transformers
|
|
- Training dynamics
|
|
- Scaling laws
|
|
- KV cache
|
|
- Tokenization
|
|
- Fine-tuning methods
|
|
- LLM evaluation
|
|
|
|
## Training Details
|
|
|
|
| Parameter | Value |
|
|
|-----------|-------|
|
|
| Base model | mistralai/Mistral-7B-Instruct-v0.3 |
|
|
| Method | QLoRA (NF4 4-bit + LoRA) |
|
|
| Dataset | 500 custom instruction examples |
|
|
| Domain | LLM Architecture |
|
|
| LoRA Rank | 64 |
|
|
| Trainable Parameters | 2.26% |
|
|
| Optimizer | Paged AdamW |
|
|
| Learning Rate Schedule | Cosine + 3% warmup |
|
|
| Final Training Loss | 1.2629 |
|
|
| Training Time | ~3.3 minutes |
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
import torch
|
|
|
|
model_id = "FazeFlynn/mistral-7b-llm-architecture-expert"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_id,
|
|
torch_dtype=torch.float16
|
|
)
|
|
|
|
prompt = "[INST] Explain how KV cache works in transformers [/INST]"
|
|
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
|
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=200
|
|
)
|
|
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
``` |