---
license: apache-2.0
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
tags:
  - text-generation
  - lora
  - peft
  - merged
  - medical
  - fine-tuned
datasets:
  - BI55/MedText
language:
  - en
library_name: transformers
pipeline_tag: text-generation
---

# TinyLlama-1.1B — Medical QLoRA (Merged)

A QLoRA fine-tuned variant of [TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T), trained on medical question-answering data. The LoRA adapter has been merged into the base weights, so this repo is a standalone model — no separate adapter loading required.

## ⚠️ Intended use

This model was trained as part of a **GenAI training lab** to demonstrate the fine-tuning → merge → deploy workflow. It is **not validated for real clinical or medical use** and should not be used to inform actual medical decisions. Treat its outputs as educational examples of what a small fine-tuned LLM can produce, not medical advice.

## Training details

- **Base model:** `TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T`
- **Method:** QLoRA (4-bit NF4 quantization during training) via `peft` + `trl.SFTTrainer`
- **LoRA config:** `r=8`, `lora_alpha=16`
- **Dataset:** [`BI55/MedText`](https://huggingface.co/datasets/BI55/MedText) — medical case/question-answer pairs
- **Steps:** 400
- **Merge:** adapter merged into the base model via `PeftModel.merge_and_unload()`, then saved as a full standalone model

## How to use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

repo_id = "tweijie/tinyllama-medical-merged"

model = AutoModelForCausalLM.from_pretrained(repo_id)
tokenizer = AutoTokenizer.from_pretrained(repo_id)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_length=200)
prompt = "<s>[INST] A patient presents with restless leg sensations at night. What could be the cause? [/INST]"
print(pipe(prompt)[0]["generated_text"])


Limitations
1.1B parameters — small model, prone to hallucination and factual errors, especially on medical content.
Fine-tuned on a single narrow dataset (400 training steps); not broadly evaluated.
No safety/alignment tuning beyond the base model's own training.