--- language: - en license: apache-2.0 base_model: - Qwen/Qwen2.5-3B-Instruct tags: - research - scientific-literature - question-answering - summarization - instruction-tuning - qlora - unsloth - peft - lora - qwen2.5 - arxiv - pubmed - academic-assistant library_name: transformers pipeline_tag: text-generation datasets: - qiaojin/PubMedQA - arXiv model-index: - name: ResearchMate-Qwen2.5-3B results: [] --- # ResearchMate-Qwen2.5-3B ResearchMate-Qwen2.5-3B is a domain-specialized Large Language Model designed to assist researchers, students, and practitioners with scientific literature. The model is instruction fine-tuned from **Qwen2.5-3B-Instruct** using **QLoRA** and **Unsloth**. Instead of functioning as a general-purpose chatbot, ResearchMate focuses on understanding scientific papers and responding to research-oriented instructions. This release represents **Version 1** of the ResearchMate project. --- # Model Overview | Property | Value | |-----------|-------| | Model | Qwen2.5-3B-Instruct | | Fine-tuning | QLoRA | | Framework | Unsloth | | Parameter Count | 3B | | Quantization | 4-bit | | PEFT | LoRA | | Primary Domain | Scientific Literature | | Language | English | | Version | 1.0 | --- # Project Goal ResearchMate aims to provide a lightweight, open-source scientific assistant capable of understanding research papers and responding to academic instructions. The objective is not to replace Retrieval-Augmented Generation systems or search engines, but to improve a language model's understanding of scientific writing through supervised instruction fine-tuning. Version 1 focuses on instruction tuning without retrieval. --- # Supported Tasks ResearchMate has been instruction-tuned for several scientific literature tasks including: - Scientific Question Answering - Paper Summarization - Abstract Explanation - Beginner-Friendly Concept Explanation - Keyword Extraction - Research Field Identification - Scientific TL;DR Generation - Contribution Identification - Method Identification --- # Training Method The model was fine-tuned using: - Unsloth - QLoRA - PEFT LoRA - 4-bit Quantization Training was performed on Kaggle GPUs to reduce memory requirements while maintaining strong instruction-following capabilities. --- # Dataset Construction ResearchMate does **not** train directly on raw datasets. Instead, a dedicated dataset-building pipeline converts multiple scientific sources into a unified instruction dataset. The preprocessing pipeline is independent of model training. Dataset pipeline: ``` Scientific Dataset ↓ Validation ↓ Schema Conversion ↓ Cleaning ↓ Merge ↓ JSONL Export ↓ Training ``` --- # Dataset Schema Every training example follows the same structure: ```json { "instruction": "...", "input": "...", "output": "...", "source": "...", "task": "..." } ``` This unified schema allows new scientific datasets to be added without modifying the training pipeline. --- # Data Sources Version 1 uses the following sources: ## PubMedQA Purpose: - Scientific Question Answering Configuration: ``` pqa_labeled ``` Fields used: - Question - Context - Long Answer --- ## arXiv Scientific papers collected through the official arXiv API. Paper metadata including titles, abstracts and categories were converted into instruction-response pairs for research-oriented tasks. --- # Training Configuration | Parameter | Value | |-----------|-------| | Fine-tuning Method | QLoRA | | PEFT | LoRA | | Quantization | 4-bit | | Framework | Unsloth | | Optimizer | AdamW | | Epochs | 2 | | Batch Size | 2 | | Gradient Accumulation | 4 | | Sequence Length | 2048 | --- # Evaluation The model was evaluated against the original Qwen2.5-3B-Instruct model. Evaluation included: - Scientific QA - Response quality inspection - ROUGE (where applicable) - Latency comparison - Hallucination inspection - Task-wise qualitative comparison --- # Example Usage ```python from transformers import AutoTokenizer, AutoModelForCausalLM model_name = "igmoiiz/ResearchMate-Qwen2.5-3B" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, device_map="auto" ) messages = [ { "role": "user", "content": "Explain transfer learning in simple language." } ] prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) inputs = tokenizer(prompt, return_tensors="pt").to(model.device) outputs = model.generate( **inputs, max_new_tokens=256 ) print( tokenizer.decode( outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True ) ) ``` --- # Example Prompts ### Scientific Question Answering ``` Answer the scientific question using the provided context. Question: What is transfer learning? ``` --- ### Summarization ``` Summarize the following scientific abstract. ``` --- ### Beginner Explanation ``` Explain this abstract in simple language. ``` --- ### Keyword Extraction ``` Extract the important keywords from this paper. ``` --- ### Research Field Identification ``` Identify the research field of this paper. ``` --- # Intended Uses Suitable for: - Research assistants - Literature exploration - Educational tools - Scientific tutoring - Academic chatbots - Research paper preprocessing - Scientific writing assistance --- # Limitations ResearchMate Version 1 has several limitations. - No Retrieval-Augmented Generation (RAG) - No citation verification - No PDF parsing - No web search - No factual verification beyond the model's learned parameters - Performance depends on the quality and diversity of the instruction dataset - May generate incorrect or outdated scientific information Users should verify important scientific claims using authoritative sources. --- # Future Work Planned improvements include: - Retrieval-Augmented Generation (RAG) - Citation-aware responses - PDF ingestion - Larger scientific datasets - Multi-turn research conversations - Paper recommendation - Local deployment using Ollama and vLLM - Improved evaluation benchmarks - Expanded scientific domains --- # Repository Structure ``` Notebook 1 Dataset Builder ↓ Notebook 2 QLoRA Fine-tuning ↓ Notebook 3 Inference ↓ Notebook 4 Evaluation ``` --- # Citation If you use this model in your work, please cite the repository. ``` @software{researchmate2026, title={ResearchMate-Qwen2.5-3B}, author={Moiz Baloch}, year={2026}, publisher={Hugging Face}, url={https://huggingface.co/igmoiiz/ResearchMate-Qwen2.5-3B} } ``` --- # Acknowledgements This project builds upon the work of: - Alibaba Qwen Team - Unsloth AI - Hugging Face - PubMedQA - arXiv Their open-source contributions made this project possible. --- # Contact Author: **Moiz Baloch** GitHub: https://github.com/igmoiiz Hugging Face: https://huggingface.co/igmoiiz ---