214 lines
4.1 KiB
Markdown
214 lines
4.1 KiB
Markdown
---
|
|
language:
|
|
- en
|
|
|
|
license: apache-2.0
|
|
|
|
base_model:
|
|
- unsloth/Qwen2.5-3B-Instruct-bnb-4bit
|
|
|
|
pipeline_tag: text-generation
|
|
|
|
library_name: transformers
|
|
|
|
tags:
|
|
- llm
|
|
- qwen2.5
|
|
- qlora
|
|
- unsloth
|
|
- fine-tuned
|
|
- chatbot
|
|
- instruction-tuned
|
|
- text-generation
|
|
- anmolgpt
|
|
---
|
|
|
|
# ANMOLGPT-3B-v0.1
|
|
|
|
> The first public release of the ANMOLGPT family of open-source language models.
|
|
|
|
## Overview
|
|
|
|
ANMOLGPT-3B-v0.1 is an instruction-following language model built by fine-tuning **Qwen2.5-3B-Instruct** using **QLoRA** with **Unsloth Studio**.
|
|
|
|
This release serves as the foundation of the ANMOLGPT project and demonstrates the complete workflow of dataset preparation, fine-tuning, evaluation, and deployment.
|
|
|
|
Although this is an early preview release, it establishes the baseline for future versions that will include larger datasets, stronger reasoning capabilities, and extensive benchmarking.
|
|
|
|
---
|
|
|
|
# Base Model
|
|
|
|
- **Model:** Qwen2.5-3B-Instruct
|
|
- **Framework:** Unsloth
|
|
- **Fine-tuning:** QLoRA (4-bit)
|
|
- **Export:** Hugging Face Transformers
|
|
|
|
---
|
|
|
|
# Training Details
|
|
|
|
## Dataset
|
|
|
|
- Databricks Dolly 25K
|
|
|
|
## Configuration
|
|
|
|
| Parameter | Value |
|
|
|-----------|------:|
|
|
| LoRA Rank | 8 |
|
|
| LoRA Alpha | 16 |
|
|
| LoRA Dropout | 0.05 |
|
|
| Context Length | 1024 |
|
|
| Learning Rate | 1e-4 |
|
|
| Batch Size | 2 |
|
|
| Gradient Accumulation | 8 |
|
|
| Effective Batch Size | 16 |
|
|
| Max Steps | 100 |
|
|
| Precision | 4-bit QLoRA |
|
|
|
|
---
|
|
## 📊 Benchmark Results
|
|
|
|
Evaluated using the EleutherAI LM Evaluation Harness.
|
|
|
|
| Benchmark | Metric | Score |
|
|
|-----------|--------|------:|
|
|
| HellaSwag | Accuracy | 54.29% |
|
|
| HellaSwag | Normalized Accuracy | **73.21%** |
|
|
| PIQA | Accuracy | **77.97%** |
|
|
| PIQA | Normalized Accuracy | **78.40%** |
|
|
| ARC-Easy | Accuracy | **77.95%** |
|
|
| ARC-Easy | Normalized Accuracy | **75.29%** |
|
|
| Winogrande | Accuracy | **70.24%** |
|
|
| TruthfulQA (MC2) | Accuracy | **42.52%** |
|
|
| MMLU | Accuracy | **65.59%** |
|
|
|
|
### MMLU Subject Breakdown
|
|
|
|
| Domain | Accuracy |
|
|
|---------|---------:|
|
|
| Social Sciences | **76.86%** |
|
|
| Other | **71.00%** |
|
|
| STEM | **61.37%** |
|
|
| Humanities | **57.47%** |
|
|
|
|
> These results represent the baseline performance of ANMOLGPT-3B-v0.1 and will be expanded in future releases.
|
|
|
|
---
|
|
|
|
# Example
|
|
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
|
model_name = "anmoldhandhania93/ANMOLGPT-3B-v0.1"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(model_name)
|
|
|
|
prompt = "Explain reinforcement learning."
|
|
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
|
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=200
|
|
)
|
|
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
---
|
|
|
|
# Intended Uses
|
|
|
|
ANMOLGPT is suitable for:
|
|
|
|
- Conversational AI
|
|
- Learning and experimentation
|
|
- Prompt engineering
|
|
- Text generation
|
|
- Educational projects
|
|
- Software development assistance
|
|
|
|
---
|
|
|
|
# Limitations
|
|
|
|
This is an **early proof-of-concept release**.
|
|
|
|
Current limitations include:
|
|
|
|
- Fine-tuned for only **100 optimization steps**
|
|
- Limited benchmark coverage
|
|
- General-purpose capabilities remain close to the base model
|
|
- Not intended for production or safety-critical applications
|
|
- May generate inaccurate or fabricated information
|
|
|
|
---
|
|
|
|
# Roadmap
|
|
|
|
## v0.2
|
|
|
|
- Improved instruction tuning
|
|
- Larger curated dataset
|
|
- More training steps
|
|
- Additional benchmarks
|
|
|
|
## v0.5
|
|
|
|
- Better reasoning
|
|
- Coding improvements
|
|
- Domain-specific datasets
|
|
- Human evaluation
|
|
|
|
## v1.0
|
|
|
|
- Comprehensive benchmark suite
|
|
- Optimized inference
|
|
- Expanded capabilities
|
|
- Stable production release
|
|
|
|
---
|
|
|
|
# Citation
|
|
|
|
```bibtex
|
|
@misc{anmolgpt2026,
|
|
title={ANMOLGPT-3B-v0.1},
|
|
author={Anmol Dhandhania},
|
|
year={2026},
|
|
publisher={Hugging Face},
|
|
url={https://huggingface.co/anmoldhandhania93/ANMOLGPT-3B-v0.1}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
# Acknowledgements
|
|
|
|
ANMOLGPT was built using:
|
|
|
|
- Qwen Team
|
|
- Unsloth AI
|
|
- Hugging Face
|
|
- Databricks Dolly Dataset
|
|
- EleutherAI LM Evaluation Harness
|
|
|
|
---
|
|
|
|
# Future Work
|
|
|
|
Future versions of ANMOLGPT will focus on:
|
|
|
|
- Better reasoning
|
|
- Improved instruction following
|
|
- Coding capabilities
|
|
- Domain-specialized models
|
|
- Efficient inference
|
|
- Comprehensive benchmarking
|
|
|
|
Feedback, issues, and contributions are welcome. |