87 lines
2.5 KiB
Markdown
87 lines
2.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: Qwen/Qwen2.5-0.5B-Instruct
|
|
tags:
|
|
- qwen
|
|
- qwen2
|
|
- gguf
|
|
- f16
|
|
- computer-science
|
|
- research
|
|
- instruction-tuning
|
|
model_creator: jalpan04
|
|
model_name: Qwen Research Assistant
|
|
pipeline_tag: text-generation
|
|
quantized_by: f16
|
|
---
|
|
|
|
# Qwen Researcher (0.5B - GGUF)
|
|
|
|
A specialized version of **Qwen2.5-0.5B-Instruct** fine-tuned for Computer Science research assistance. This model has been instruction-tuned on a curated subset of 2,000 arXiv Computer Science papers to provide academic summaries and technical insights.
|
|
|
|
## Model Details
|
|
|
|
- **Developed by:** Jalpan04
|
|
- **Model type:** Causal Language Model
|
|
- **Language(s):** English
|
|
- **License:** Apache-2.0
|
|
- **Fine-tuned from model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
|
|
- **Training Method:** QLoRA (Rank 16, Alpha 32)
|
|
- **Dataset:** arXiv Computer Science Metadata (Instruction-formatted)
|
|
|
|
## Usage
|
|
|
|
### Local Deployment (Ollama)
|
|
|
|
1. Create a `Modelfile`:
|
|
```dockerfile
|
|
FROM ./qwen-resercher.gguf
|
|
TEMPLATE """{{ if .System }}<|im_start|>system
|
|
{{ .System }}<|im_end|>
|
|
{{ end }}{{ if .User }}<|im_start|>user
|
|
{{ .User }}<|im_end|>
|
|
{{ end }}<|im_start|>assistant
|
|
{{ .Output }}<|im_end|>"""
|
|
PARAMETER stop "<|im_start|>"
|
|
PARAMETER stop "<|im_end|>"
|
|
SYSTEM "You are a professional computer science researcher. Provide academic, detailed information based on research abstracts."
|
|
```
|
|
|
|
2. Run in terminal:
|
|
```bash
|
|
ollama create qwen-researcher -f Modelfile
|
|
ollama run qwen-researcher
|
|
```
|
|
|
|
### Python (llama-cpp-python)
|
|
|
|
```python
|
|
from llama_cpp import Llama
|
|
|
|
llm = Llama.from_pretrained(
|
|
repo_id="jalpan04/qwen-researcher",
|
|
filename="qwen-resercher.gguf",
|
|
)
|
|
|
|
response = llm.create_chat_completion(
|
|
messages = [
|
|
{"role": "system", "content": "You are a CS researcher."},
|
|
{"role": "user", "content": "Summarize the latest trends in Neural Program Synthesis."}
|
|
]
|
|
)
|
|
print(response["choices"][0]["message"]["content"])
|
|
```
|
|
|
|
## Training Procedure
|
|
|
|
The model was trained on an **NVIDIA RTX 4060 (8GB)** using the following setup:
|
|
- **Optimization:** 4-bit NormalFloat (nf4) quantization.
|
|
- **Precision:** BFloat16 for stability on 40-series hardware.
|
|
- **Learning Rate:** 2e-4.
|
|
- **Batch Size:** 1 with Gradient Accumulation (16 steps).
|
|
- **Sequence Length:** 1024 tokens.
|
|
|
|
## Limitations
|
|
|
|
As a 0.5B parameter model, this is highly efficient but may exhibit hallucinations compared to larger models (7B+). It is best used for summarization and quick technical lookups rather than complex logical reasoning.
|