language, license, base_model, library_name, tags

language	license	base_model	library_name	tags
en	apache-2.0	mistralai/Mistral-7B-Instruct-v0.3	transformers	fine-tuned qlora lora llm instruction-tuning peft

Mistral-7B LLM Architecture Expert

A fine-tuned version of Mistral-7B-Instruct-v0.3 trained using QLoRA on a custom dataset focused on LLM architecture concepts and internals.

Topics covered include:

• Attention mechanisms
• Transformers
• Training dynamics
• Scaling laws
• KV cache
• Tokenization
• Fine-tuning methods
• LLM evaluation

Training Details

Parameter	Value
Base model	mistralai/Mistral-7B-Instruct-v0.3
Method	QLoRA (NF4 4-bit + LoRA)
Dataset	500 custom instruction examples
Domain	LLM Architecture
LoRA Rank	64
Trainable Parameters	2.26%
Optimizer	Paged AdamW
Learning Rate Schedule	Cosine + 3% warmup
Final Training Loss	1.2629
Training Time	~3.3 minutes

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "FazeFlynn/mistral-7b-llm-architecture-expert"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16
)

prompt = "[INST] Explain how KV cache works in transformers [/INST]"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=200
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))