llama-3.2-1b-code-instruct/README.md

---
library_name: transformers
license: apache-2.0
datasets:
- sahil2801/CodeAlpaca-20k
base_model:
- meta-llama/Llama-3.2-1B
---


# 🧠 Llama-3.2-1B Code Solver (QLoRA Fine-Tuned)

A lightweight yet powerful **code-focused language model** fine-tuned from **Meta Llama-3.2-1B** using **QLoRA (4-bit)** on the **CodeAlpaca-20K** dataset.  
Designed for **efficient code generation, reasoning, and problem-solving** on limited GPU resources.

> 🚀 Trained on a single Tesla P100 GPU  
> ⚡ Optimized for Kaggle, Colab, and low-VRAM environments  
> 🧩 Ideal for research, education, and rapid prototyping

---

## 🔍 Model Overview

| Attribute | Value |
|---------|------|
| **Base Model** | `meta-llama/Llama-3.2-1B` |
| **Model Type** | Decoder-only causal language model |
| **Fine-Tuning Method** | QLoRA (4-bit quantization + LoRA) |
| **LoRA Rank** | 16 |
| **Task Domain** | Code generation & code reasoning |
| **Training Samples** | 10,000 |
| **Training Time** | ~5 hours |
| **Hardware** | NVIDIA Tesla P100 |
| **Precision** | 4-bit (NF4) |
| **Frameworks** | Hugging Face Transformers, PEFT, BitsAndBytes |

---

## 🎯 What This Model Is Good At

- 🧑‍💻 Code generation (Python-focused, but generalizable)
- 🧠 Step-by-step coding reasoning
- 🧪 Algorithmic problem solving
- 📘 Educational coding assistance
- ⚙️ Running efficiently on **low-VRAM GPUs**

---

## 📚 Training Dataset

### **CodeAlpaca-20K**

A high-quality instruction-tuning dataset derived from the Alpaca format and specialized for coding tasks.

- **Total dataset size**: 20,000 samples  
- **Used for training**: 10,000 samples (50%)
- **Data format**:
  ```json
  {
    "instruction": "Describe the coding task",
    "input": "Optional context or input code",
    "output": "Expected code solution"
  }
  ```

* **Task Types**:

  * Algorithm implementation
  * Code completion
  * Debugging
  * Function writing
  * Problem solving

---

## 🏗️ Training Methodology

This model was fine-tuned using **QLoRA**, enabling efficient adaptation of large language models on limited hardware.

### Key Techniques Used

* **4-bit Quantization (NF4)** via BitsAndBytes
* **LoRA adapters** applied to attention layers
* **Frozen base model weights**
* **Low-rank updates only**

### Why QLoRA?

* 🔻 Drastically reduces GPU memory usage
* ⚡ Enables training on consumer-grade GPUs
* 📈 Maintains strong downstream performance

---

## ⚙️ Training Configuration

| Parameter             | Value                   |
| --------------------- | ----------------------- |
| Max Sequence Length   | 1024                    |
| LoRA Rank (r)         | 16                      |
| LoRA Alpha            | 32                      |
| LoRA Dropout          | 0.05                    |
| Optimizer             | AdamW                   |
| Learning Rate         | 2e-4                    |
| Batch Size            | Small (GPU-constrained) |
| Gradient Accumulation | Enabled                 |
| Quantization          | 4-bit                   |

---

## 🚀 Usage

### Load the Model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_USERNAME/llama-3.2-1b-code-solver"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    load_in_4bit=True
)
```

### Example Inference

```python
prompt = "Write a Python function to check if a number is prime."

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```


## 🧪 Evaluation Notes

* This model is **instruction-tuned**, not benchmark-optimized
* No formal benchmarks (HumanEval / MBPP) were run
* Best evaluated through **qualitative code generation**

## ⚠️ Limitations

* 1B parameters → limited long-context reasoning
* Not optimized for natural language chat
* May hallucinate on complex or ambiguous prompts
* English-centric training data


## 🧭 Intended Use

✅ **Allowed**

* Research and experimentation
* Coding assistants
* Educational tools
* Prototyping LLM systems


## 🙏 Acknowledgements

* **Meta AI** for Llama 3.2
* **CodeAlpaca** dataset creators
* **Hugging Face** ecosystem
* **QLoRA & PEFT** authors