初始化项目，由ModelHub XC社区提供模型
Model: anmoldhandhania93/ANMOLGPT-3B-v0.1 Source: Original Platform
This commit is contained in:
ModelHub XC
214
README.md
Normal file
214
README.md
Normal file
@@ -0,0 +1,214 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
|
||||
license: apache-2.0
|
||||
|
||||
base_model:
|
||||
- unsloth/Qwen2.5-3B-Instruct-bnb-4bit
|
||||
|
||||
pipeline_tag: text-generation
|
||||
|
||||
library_name: transformers
|
||||
|
||||
tags:
|
||||
- llm
|
||||
- qwen2.5
|
||||
- qlora
|
||||
- unsloth
|
||||
- fine-tuned
|
||||
- chatbot
|
||||
- instruction-tuned
|
||||
- text-generation
|
||||
- anmolgpt
|
||||
---
|
||||
|
||||
# ANMOLGPT-3B-v0.1
|
||||
|
||||
> The first public release of the ANMOLGPT family of open-source language models.
|
||||
|
||||
## Overview
|
||||
|
||||
ANMOLGPT-3B-v0.1 is an instruction-following language model built by fine-tuning **Qwen2.5-3B-Instruct** using **QLoRA** with **Unsloth Studio**.
|
||||
|
||||
This release serves as the foundation of the ANMOLGPT project and demonstrates the complete workflow of dataset preparation, fine-tuning, evaluation, and deployment.
|
||||
|
||||
Although this is an early preview release, it establishes the baseline for future versions that will include larger datasets, stronger reasoning capabilities, and extensive benchmarking.
|
||||
|
||||
---
|
||||
|
||||
# Base Model
|
||||
|
||||
- **Model:** Qwen2.5-3B-Instruct
|
||||
- **Framework:** Unsloth
|
||||
- **Fine-tuning:** QLoRA (4-bit)
|
||||
- **Export:** Hugging Face Transformers
|
||||
|
||||
---
|
||||
|
||||
# Training Details
|
||||
|
||||
## Dataset
|
||||
|
||||
- Databricks Dolly 25K
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|------:|
|
||||
| LoRA Rank | 8 |
|
||||
| LoRA Alpha | 16 |
|
||||
| LoRA Dropout | 0.05 |
|
||||
| Context Length | 1024 |
|
||||
| Learning Rate | 1e-4 |
|
||||
| Batch Size | 2 |
|
||||
| Gradient Accumulation | 8 |
|
||||
| Effective Batch Size | 16 |
|
||||
| Max Steps | 100 |
|
||||
| Precision | 4-bit QLoRA |
|
||||
|
||||
---
|
||||
## 📊 Benchmark Results
|
||||
|
||||
Evaluated using the EleutherAI LM Evaluation Harness.
|
||||
|
||||
| Benchmark | Metric | Score |
|
||||
|-----------|--------|------:|
|
||||
| HellaSwag | Accuracy | 54.29% |
|
||||
| HellaSwag | Normalized Accuracy | **73.21%** |
|
||||
| PIQA | Accuracy | **77.97%** |
|
||||
| PIQA | Normalized Accuracy | **78.40%** |
|
||||
| ARC-Easy | Accuracy | **77.95%** |
|
||||
| ARC-Easy | Normalized Accuracy | **75.29%** |
|
||||
| Winogrande | Accuracy | **70.24%** |
|
||||
| TruthfulQA (MC2) | Accuracy | **42.52%** |
|
||||
| MMLU | Accuracy | **65.59%** |
|
||||
|
||||
### MMLU Subject Breakdown
|
||||
|
||||
| Domain | Accuracy |
|
||||
|---------|---------:|
|
||||
| Social Sciences | **76.86%** |
|
||||
| Other | **71.00%** |
|
||||
| STEM | **61.37%** |
|
||||
| Humanities | **57.47%** |
|
||||
|
||||
> These results represent the baseline performance of ANMOLGPT-3B-v0.1 and will be expanded in future releases.
|
||||
|
||||
---
|
||||
|
||||
# Example
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
model_name = "anmoldhandhania93/ANMOLGPT-3B-v0.1"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(model_name)
|
||||
|
||||
prompt = "Explain reinforcement learning."
|
||||
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=200
|
||||
)
|
||||
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Intended Uses
|
||||
|
||||
ANMOLGPT is suitable for:
|
||||
|
||||
- Conversational AI
|
||||
- Learning and experimentation
|
||||
- Prompt engineering
|
||||
- Text generation
|
||||
- Educational projects
|
||||
- Software development assistance
|
||||
|
||||
---
|
||||
|
||||
# Limitations
|
||||
|
||||
This is an **early proof-of-concept release**.
|
||||
|
||||
Current limitations include:
|
||||
|
||||
- Fine-tuned for only **100 optimization steps**
|
||||
- Limited benchmark coverage
|
||||
- General-purpose capabilities remain close to the base model
|
||||
- Not intended for production or safety-critical applications
|
||||
- May generate inaccurate or fabricated information
|
||||
|
||||
---
|
||||
|
||||
# Roadmap
|
||||
|
||||
## v0.2
|
||||
|
||||
- Improved instruction tuning
|
||||
- Larger curated dataset
|
||||
- More training steps
|
||||
- Additional benchmarks
|
||||
|
||||
## v0.5
|
||||
|
||||
- Better reasoning
|
||||
- Coding improvements
|
||||
- Domain-specific datasets
|
||||
- Human evaluation
|
||||
|
||||
## v1.0
|
||||
|
||||
- Comprehensive benchmark suite
|
||||
- Optimized inference
|
||||
- Expanded capabilities
|
||||
- Stable production release
|
||||
|
||||
---
|
||||
|
||||
# Citation
|
||||
|
||||
```bibtex
|
||||
@misc{anmolgpt2026,
|
||||
title={ANMOLGPT-3B-v0.1},
|
||||
author={Anmol Dhandhania},
|
||||
year={2026},
|
||||
publisher={Hugging Face},
|
||||
url={https://huggingface.co/anmoldhandhania93/ANMOLGPT-3B-v0.1}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Acknowledgements
|
||||
|
||||
ANMOLGPT was built using:
|
||||
|
||||
- Qwen Team
|
||||
- Unsloth AI
|
||||
- Hugging Face
|
||||
- Databricks Dolly Dataset
|
||||
- EleutherAI LM Evaluation Harness
|
||||
|
||||
---
|
||||
|
||||
# Future Work
|
||||
|
||||
Future versions of ANMOLGPT will focus on:
|
||||
|
||||
- Better reasoning
|
||||
- Improved instruction following
|
||||
- Coding capabilities
|
||||
- Domain-specialized models
|
||||
- Efficient inference
|
||||
- Comprehensive benchmarking
|
||||
|
||||
Feedback, issues, and contributions are welcome.
|
||||
Reference in New Issue
Block a user