---
library_name: transformers
tags:
- instruction-tuning
- sft
- lora
- tinyllama
- alpaca
---

# TinyLlama-SFT-Alpaca

TinyLlama-1.1B-Chat fine-tuned using Supervised Fine-Tuning (SFT) with LoRA on the yahma/alpaca-cleaned dataset as part of IBA Karachi NLP with Deep Learning Assignment 4.

## Model Details

- **Developed by:** Hadeeqa Al Islam
- **Model type:** Causal Language Model
- **Language:** English
- **Finetuned from:** TinyLlama/TinyLlama-1.1B-Chat-v1.0
- **Training data:** yahma/alpaca-cleaned (20,000 samples)
- **LoRA config:** r=8, lora_alpha=16, target_modules=[q_proj, v_proj]
- **Training:** lr=2e-4, batch=4, epochs=2, fp16=True

## How to Get Started

tokenizer = AutoTokenizer.from_pretrained("Haldi247/TinyLlama-SFT-Alpaca")
model = AutoModelForCausalLM.from_pretrained("Haldi247/TinyLlama-SFT-Alpaca")
messages = [{"role": "user", "content": "What is photosynthesis?"}]

## Training Details

- **Dataset:** yahma/alpaca-cleaned (20,000 samples after filtering)
- **Hardware:** NVIDIA RTX 5070 Ti (16GB VRAM) via WSL2
- **Training time:** ~14 minutes

### Training Procedure

#### Training Hyperparameters
- **Training regime:** fp16 mixed precision
- **Learning Rate:** 2e-4
- **Batch Size:** 4
- **Epochs:** 2

## Evaluation

### Testing Data, Factors & Metrics

#### Metrics
- **BLEU Score**: Used to evaluate the overlap of n-grams between the model output and the ground truth.
- **BERTScore**: Used to compute semantic similarity between the generated text and reference text using BERT embeddings.

### Results
- **Average BLEU Score**: 0.4303
- **Average BERTScore**: 0.7236

## Limitations
- **Tokenization Constraints**: The training process used sequence packing to manage memory effectively. Because the model was trained with packed sequences without special tokens (to prevent cross-contamination), it may exhibit formatting issues if the inference environment does not strictly adhere to the expected chat template.
- **Data Bias**: The model is fine-tuned on the Alpaca dataset, which may inherit biases present in the synthetic instruction data.
- **Inference Stability**: Due to the aggressive packing strategy, users may observe repetition or formatting artifacts if the prompt structure deviates significantly from the training format.

#### Summary



## Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]

## Technical Specifications [optional]

### Model Architecture and Objective

[More Information Needed]

### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

[More Information Needed]

## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Model Card Authors [optional]

[More Information Needed]

## Model Card Contact

[More Information Needed]