ResearchMate-Qwen2.5-3B/README.md

---
language:
- en

license: apache-2.0

base_model:
- Qwen/Qwen2.5-3B-Instruct

tags:
- research
- scientific-literature
- question-answering
- summarization
- instruction-tuning
- qlora
- unsloth
- peft
- lora
- qwen2.5
- arxiv
- pubmed
- academic-assistant

library_name: transformers

pipeline_tag: text-generation

datasets:
- qiaojin/PubMedQA
- arXiv

model-index:
- name: ResearchMate-Qwen2.5-3B
  results: []

---

# ResearchMate-Qwen2.5-3B

ResearchMate-Qwen2.5-3B is a domain-specialized Large Language Model designed to assist researchers, students, and practitioners with scientific literature.

The model is instruction fine-tuned from **Qwen2.5-3B-Instruct** using **QLoRA** and **Unsloth**. Instead of functioning as a general-purpose chatbot, ResearchMate focuses on understanding scientific papers and responding to research-oriented instructions.

This release represents **Version 1** of the ResearchMate project.

---

# Model Overview

| Property | Value |
|-----------|-------|
| Model | Qwen2.5-3B-Instruct |
| Fine-tuning | QLoRA |
| Framework | Unsloth |
| Parameter Count | 3B |
| Quantization | 4-bit |
| PEFT | LoRA |
| Primary Domain | Scientific Literature |
| Language | English |
| Version | 1.0 |

---

# Project Goal

ResearchMate aims to provide a lightweight, open-source scientific assistant capable of understanding research papers and responding to academic instructions.

The objective is not to replace Retrieval-Augmented Generation systems or search engines, but to improve a language model's understanding of scientific writing through supervised instruction fine-tuning.

Version 1 focuses on instruction tuning without retrieval.

---

# Supported Tasks

ResearchMate has been instruction-tuned for several scientific literature tasks including:

- Scientific Question Answering
- Paper Summarization
- Abstract Explanation
- Beginner-Friendly Concept Explanation
- Keyword Extraction
- Research Field Identification
- Scientific TL;DR Generation
- Contribution Identification
- Method Identification

---

# Training Method

The model was fine-tuned using:

- Unsloth
- QLoRA
- PEFT LoRA
- 4-bit Quantization

Training was performed on Kaggle GPUs to reduce memory requirements while maintaining strong instruction-following capabilities.

---

# Dataset Construction

ResearchMate does **not** train directly on raw datasets.

Instead, a dedicated dataset-building pipeline converts multiple scientific sources into a unified instruction dataset.

The preprocessing pipeline is independent of model training.

Dataset pipeline:

```
Scientific Dataset

↓

Validation

↓

Schema Conversion

↓

Cleaning

↓

Merge

↓

JSONL Export

↓

Training
```

---

# Dataset Schema

Every training example follows the same structure:

```json
{
    "instruction": "...",
    "input": "...",
    "output": "...",
    "source": "...",
    "task": "..."
}
```

This unified schema allows new scientific datasets to be added without modifying the training pipeline.

---

# Data Sources

Version 1 uses the following sources:

## PubMedQA

Purpose:

- Scientific Question Answering

Configuration:

```
pqa_labeled
```

Fields used:

- Question
- Context
- Long Answer

---

## arXiv

Scientific papers collected through the official arXiv API.

Paper metadata including titles, abstracts and categories were converted into instruction-response pairs for research-oriented tasks.

---

# Training Configuration

| Parameter | Value |
|-----------|-------|
| Fine-tuning Method | QLoRA |
| PEFT | LoRA |
| Quantization | 4-bit |
| Framework | Unsloth |
| Optimizer | AdamW |
| Epochs | 2 |
| Batch Size | 2 |
| Gradient Accumulation | 4 |
| Sequence Length | 2048 |

---

# Evaluation

The model was evaluated against the original Qwen2.5-3B-Instruct model.

Evaluation included:

- Scientific QA
- Response quality inspection
- ROUGE (where applicable)
- Latency comparison
- Hallucination inspection
- Task-wise qualitative comparison

---

# Example Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "igmoiiz/ResearchMate-Qwen2.5-3B"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto"
)

messages = [
    {
        "role": "user",
        "content": "Explain transfer learning in simple language."
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256
)

print(
    tokenizer.decode(
        outputs[0][inputs["input_ids"].shape[1]:],
        skip_special_tokens=True
    )
)
```

---

# Example Prompts

### Scientific Question Answering

```
Answer the scientific question using the provided context.

Question:
What is transfer learning?
```

---

### Summarization

```
Summarize the following scientific abstract.
```

---

### Beginner Explanation

```
Explain this abstract in simple language.
```

---

### Keyword Extraction

```
Extract the important keywords from this paper.
```

---

### Research Field Identification

```
Identify the research field of this paper.
```

---

# Intended Uses

Suitable for:

- Research assistants
- Literature exploration
- Educational tools
- Scientific tutoring
- Academic chatbots
- Research paper preprocessing
- Scientific writing assistance

---

# Limitations

ResearchMate Version 1 has several limitations.

- No Retrieval-Augmented Generation (RAG)
- No citation verification
- No PDF parsing
- No web search
- No factual verification beyond the model's learned parameters
- Performance depends on the quality and diversity of the instruction dataset
- May generate incorrect or outdated scientific information

Users should verify important scientific claims using authoritative sources.

---

# Future Work

Planned improvements include:

- Retrieval-Augmented Generation (RAG)
- Citation-aware responses
- PDF ingestion
- Larger scientific datasets
- Multi-turn research conversations
- Paper recommendation
- Local deployment using Ollama and vLLM
- Improved evaluation benchmarks
- Expanded scientific domains

---

# Repository Structure

```
Notebook 1
Dataset Builder

↓

Notebook 2
QLoRA Fine-tuning

↓

Notebook 3
Inference

↓

Notebook 4
Evaluation
```

---

# Citation

If you use this model in your work, please cite the repository.

```
@software{researchmate2026,
  title={ResearchMate-Qwen2.5-3B},
  author={Moiz Baloch},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/igmoiiz/ResearchMate-Qwen2.5-3B}
}
```

---

# Acknowledgements

This project builds upon the work of:

- Alibaba Qwen Team
- Unsloth AI
- Hugging Face
- PubMedQA
- arXiv

Their open-source contributions made this project possible.

---

# Contact

Author: **Moiz Baloch**

GitHub: https://github.com/igmoiiz

Hugging Face: https://huggingface.co/igmoiiz

---