422 lines
6.8 KiB
Markdown
422 lines
6.8 KiB
Markdown
---
|
|
language:
|
|
- en
|
|
|
|
license: apache-2.0
|
|
|
|
base_model:
|
|
- Qwen/Qwen2.5-3B-Instruct
|
|
|
|
tags:
|
|
- research
|
|
- scientific-literature
|
|
- question-answering
|
|
- summarization
|
|
- instruction-tuning
|
|
- qlora
|
|
- unsloth
|
|
- peft
|
|
- lora
|
|
- qwen2.5
|
|
- arxiv
|
|
- pubmed
|
|
- academic-assistant
|
|
|
|
library_name: transformers
|
|
|
|
pipeline_tag: text-generation
|
|
|
|
datasets:
|
|
- qiaojin/PubMedQA
|
|
- arXiv
|
|
|
|
model-index:
|
|
- name: ResearchMate-Qwen2.5-3B
|
|
results: []
|
|
|
|
---
|
|
|
|
# ResearchMate-Qwen2.5-3B
|
|
|
|
ResearchMate-Qwen2.5-3B is a domain-specialized Large Language Model designed to assist researchers, students, and practitioners with scientific literature.
|
|
|
|
The model is instruction fine-tuned from **Qwen2.5-3B-Instruct** using **QLoRA** and **Unsloth**. Instead of functioning as a general-purpose chatbot, ResearchMate focuses on understanding scientific papers and responding to research-oriented instructions.
|
|
|
|
This release represents **Version 1** of the ResearchMate project.
|
|
|
|
---
|
|
|
|
# Model Overview
|
|
|
|
| Property | Value |
|
|
|-----------|-------|
|
|
| Model | Qwen2.5-3B-Instruct |
|
|
| Fine-tuning | QLoRA |
|
|
| Framework | Unsloth |
|
|
| Parameter Count | 3B |
|
|
| Quantization | 4-bit |
|
|
| PEFT | LoRA |
|
|
| Primary Domain | Scientific Literature |
|
|
| Language | English |
|
|
| Version | 1.0 |
|
|
|
|
---
|
|
|
|
# Project Goal
|
|
|
|
ResearchMate aims to provide a lightweight, open-source scientific assistant capable of understanding research papers and responding to academic instructions.
|
|
|
|
The objective is not to replace Retrieval-Augmented Generation systems or search engines, but to improve a language model's understanding of scientific writing through supervised instruction fine-tuning.
|
|
|
|
Version 1 focuses on instruction tuning without retrieval.
|
|
|
|
---
|
|
|
|
# Supported Tasks
|
|
|
|
ResearchMate has been instruction-tuned for several scientific literature tasks including:
|
|
|
|
- Scientific Question Answering
|
|
- Paper Summarization
|
|
- Abstract Explanation
|
|
- Beginner-Friendly Concept Explanation
|
|
- Keyword Extraction
|
|
- Research Field Identification
|
|
- Scientific TL;DR Generation
|
|
- Contribution Identification
|
|
- Method Identification
|
|
|
|
---
|
|
|
|
# Training Method
|
|
|
|
The model was fine-tuned using:
|
|
|
|
- Unsloth
|
|
- QLoRA
|
|
- PEFT LoRA
|
|
- 4-bit Quantization
|
|
|
|
Training was performed on Kaggle GPUs to reduce memory requirements while maintaining strong instruction-following capabilities.
|
|
|
|
---
|
|
|
|
# Dataset Construction
|
|
|
|
ResearchMate does **not** train directly on raw datasets.
|
|
|
|
Instead, a dedicated dataset-building pipeline converts multiple scientific sources into a unified instruction dataset.
|
|
|
|
The preprocessing pipeline is independent of model training.
|
|
|
|
Dataset pipeline:
|
|
|
|
```
|
|
Scientific Dataset
|
|
|
|
↓
|
|
|
|
Validation
|
|
|
|
↓
|
|
|
|
Schema Conversion
|
|
|
|
↓
|
|
|
|
Cleaning
|
|
|
|
↓
|
|
|
|
Merge
|
|
|
|
↓
|
|
|
|
JSONL Export
|
|
|
|
↓
|
|
|
|
Training
|
|
```
|
|
|
|
---
|
|
|
|
# Dataset Schema
|
|
|
|
Every training example follows the same structure:
|
|
|
|
```json
|
|
{
|
|
"instruction": "...",
|
|
"input": "...",
|
|
"output": "...",
|
|
"source": "...",
|
|
"task": "..."
|
|
}
|
|
```
|
|
|
|
This unified schema allows new scientific datasets to be added without modifying the training pipeline.
|
|
|
|
---
|
|
|
|
# Data Sources
|
|
|
|
Version 1 uses the following sources:
|
|
|
|
## PubMedQA
|
|
|
|
Purpose:
|
|
|
|
- Scientific Question Answering
|
|
|
|
Configuration:
|
|
|
|
```
|
|
pqa_labeled
|
|
```
|
|
|
|
Fields used:
|
|
|
|
- Question
|
|
- Context
|
|
- Long Answer
|
|
|
|
---
|
|
|
|
## arXiv
|
|
|
|
Scientific papers collected through the official arXiv API.
|
|
|
|
Paper metadata including titles, abstracts and categories were converted into instruction-response pairs for research-oriented tasks.
|
|
|
|
---
|
|
|
|
# Training Configuration
|
|
|
|
| Parameter | Value |
|
|
|-----------|-------|
|
|
| Fine-tuning Method | QLoRA |
|
|
| PEFT | LoRA |
|
|
| Quantization | 4-bit |
|
|
| Framework | Unsloth |
|
|
| Optimizer | AdamW |
|
|
| Epochs | 2 |
|
|
| Batch Size | 2 |
|
|
| Gradient Accumulation | 4 |
|
|
| Sequence Length | 2048 |
|
|
|
|
---
|
|
|
|
# Evaluation
|
|
|
|
The model was evaluated against the original Qwen2.5-3B-Instruct model.
|
|
|
|
Evaluation included:
|
|
|
|
- Scientific QA
|
|
- Response quality inspection
|
|
- ROUGE (where applicable)
|
|
- Latency comparison
|
|
- Hallucination inspection
|
|
- Task-wise qualitative comparison
|
|
|
|
---
|
|
|
|
# Example Usage
|
|
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
|
model_name = "igmoiiz/ResearchMate-Qwen2.5-3B"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_name,
|
|
device_map="auto"
|
|
)
|
|
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": "Explain transfer learning in simple language."
|
|
}
|
|
]
|
|
|
|
prompt = tokenizer.apply_chat_template(
|
|
messages,
|
|
tokenize=False,
|
|
add_generation_prompt=True
|
|
)
|
|
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=256
|
|
)
|
|
|
|
print(
|
|
tokenizer.decode(
|
|
outputs[0][inputs["input_ids"].shape[1]:],
|
|
skip_special_tokens=True
|
|
)
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
# Example Prompts
|
|
|
|
### Scientific Question Answering
|
|
|
|
```
|
|
Answer the scientific question using the provided context.
|
|
|
|
Question:
|
|
What is transfer learning?
|
|
```
|
|
|
|
---
|
|
|
|
### Summarization
|
|
|
|
```
|
|
Summarize the following scientific abstract.
|
|
```
|
|
|
|
---
|
|
|
|
### Beginner Explanation
|
|
|
|
```
|
|
Explain this abstract in simple language.
|
|
```
|
|
|
|
---
|
|
|
|
### Keyword Extraction
|
|
|
|
```
|
|
Extract the important keywords from this paper.
|
|
```
|
|
|
|
---
|
|
|
|
### Research Field Identification
|
|
|
|
```
|
|
Identify the research field of this paper.
|
|
```
|
|
|
|
---
|
|
|
|
# Intended Uses
|
|
|
|
Suitable for:
|
|
|
|
- Research assistants
|
|
- Literature exploration
|
|
- Educational tools
|
|
- Scientific tutoring
|
|
- Academic chatbots
|
|
- Research paper preprocessing
|
|
- Scientific writing assistance
|
|
|
|
---
|
|
|
|
# Limitations
|
|
|
|
ResearchMate Version 1 has several limitations.
|
|
|
|
- No Retrieval-Augmented Generation (RAG)
|
|
- No citation verification
|
|
- No PDF parsing
|
|
- No web search
|
|
- No factual verification beyond the model's learned parameters
|
|
- Performance depends on the quality and diversity of the instruction dataset
|
|
- May generate incorrect or outdated scientific information
|
|
|
|
Users should verify important scientific claims using authoritative sources.
|
|
|
|
---
|
|
|
|
# Future Work
|
|
|
|
Planned improvements include:
|
|
|
|
- Retrieval-Augmented Generation (RAG)
|
|
- Citation-aware responses
|
|
- PDF ingestion
|
|
- Larger scientific datasets
|
|
- Multi-turn research conversations
|
|
- Paper recommendation
|
|
- Local deployment using Ollama and vLLM
|
|
- Improved evaluation benchmarks
|
|
- Expanded scientific domains
|
|
|
|
---
|
|
|
|
# Repository Structure
|
|
|
|
```
|
|
Notebook 1
|
|
Dataset Builder
|
|
|
|
↓
|
|
|
|
Notebook 2
|
|
QLoRA Fine-tuning
|
|
|
|
↓
|
|
|
|
Notebook 3
|
|
Inference
|
|
|
|
↓
|
|
|
|
Notebook 4
|
|
Evaluation
|
|
```
|
|
|
|
---
|
|
|
|
# Citation
|
|
|
|
If you use this model in your work, please cite the repository.
|
|
|
|
```
|
|
@software{researchmate2026,
|
|
title={ResearchMate-Qwen2.5-3B},
|
|
author={Moiz Baloch},
|
|
year={2026},
|
|
publisher={Hugging Face},
|
|
url={https://huggingface.co/igmoiiz/ResearchMate-Qwen2.5-3B}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
# Acknowledgements
|
|
|
|
This project builds upon the work of:
|
|
|
|
- Alibaba Qwen Team
|
|
- Unsloth AI
|
|
- Hugging Face
|
|
- PubMedQA
|
|
- arXiv
|
|
|
|
Their open-source contributions made this project possible.
|
|
|
|
---
|
|
|
|
# Contact
|
|
|
|
Author: **Moiz Baloch**
|
|
|
|
GitHub: https://github.com/igmoiiz
|
|
|
|
Hugging Face: https://huggingface.co/igmoiiz
|
|
|
|
--- |