331 lines
7.1 KiB
Markdown
331 lines
7.1 KiB
Markdown
---
|
|
library_name: transformers
|
|
model_name: qwen3_0.6b_fft
|
|
tags:
|
|
- generated_from_trainer
|
|
- trl
|
|
- sft
|
|
licence: license
|
|
---
|
|
|
|
|
|
|
|
# Qwen3-0.6B Fine-Tuning Project - Tech3Space + Spiritual Knowledge
|
|
|
|
This repository contains the complete pipeline for fine-tuning **Qwen3-0.6B** on https://ollama.com/ankitkushwahahacker9910921/tech3space-pro custom knowledge about https://tech3space.com/ **AnkitKushwaha90**, **Tech3Space**, and **Kundalini Spiritual Dataset**.
|
|
|
|
## 📋 Table of Contents
|
|
- [Project Overview](#project-overview)
|
|
- [Prerequisites](#prerequisites)
|
|
- [1. Dataset Generation](#1-dataset-generation)
|
|
- [2. Full Fine-Tuning](#2-full-fine-tuning)
|
|
- [3. Model Conversion to GGUF](#3-model-conversion-to-gguf)
|
|
- [4. Inference / Usage](#4-inference--usage)
|
|
- [Directory Structure](#directory-structure)
|
|
- [Troubleshooting](#troubleshooting)
|
|
|
|
---
|
|
|
|
## Project Overview
|
|
|
|
- **Base Model**: Qwen3-0.6B
|
|
- **Knowledge Domains**:
|
|
- AnkitKushwaha90 (Cybersecurity, AI Researcher)
|
|
- Tech3Space Platform
|
|
- Hugging Face Profile
|
|
- **New**: In-depth Kundalini Energy, 7 Chakras, 27 Nakshatras, 12 Rashis, Hindi Months, Awakening Practices
|
|
|
|
The model is trained to respond accurately about these topics using SFT (Supervised Fine-Tuning).
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
```bash
|
|
conda create -n safetensors python=3.11
|
|
conda activate safetensors
|
|
|
|
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
|
pip install transformers datasets trl accelerate
|
|
pip install huggingface_hub
|
|
```
|
|
# For GGUF:
|
|
```bash
|
|
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && make clean && LLAMA_CUDA=1 make -j
|
|
```
|
|
|
|
# 🚀 Tech3Space-FT
|
|
|
|
### Full Fine-Tuned Large Language Model by Tech3Space
|
|
|
|
[]()
|
|
[]()
|
|
[]()
|
|
[]()
|
|
|
|
---
|
|
|
|
## 🌟 Overview
|
|
|
|
**Tech3Space-FT** is a fully fine-tuned Large Language Model designed to demonstrate the power of domain adaptation, instruction following, reasoning, coding assistance, and knowledge enhancement through Full Fine-Tuning (FFT).
|
|
|
|
Unlike parameter-efficient methods such as LoRA, this model has been trained by updating **all model parameters**, enabling deeper adaptation to the target dataset and learning objectives.
|
|
|
|
The goal of this project is not only to build a capable AI assistant but also to inspire researchers, students, developers, and AI enthusiasts to explore the complete lifecycle of modern LLM training.
|
|
|
|
---
|
|
|
|
## ✨ Why This Model?
|
|
|
|
Building an AI model is more than training weights.
|
|
|
|
It is about:
|
|
|
|
* Understanding Transformer architectures
|
|
* Learning tokenization strategies
|
|
* Managing datasets at scale
|
|
* Optimizing GPU resources
|
|
* Evaluating model behavior
|
|
* Contributing to the open-source AI ecosystem
|
|
|
|
Tech3Space-FT represents that journey.
|
|
|
|
---
|
|
|
|
## 🎯 Key Features
|
|
|
|
✅ Full Fine-Tuning (All Parameters Updated)
|
|
|
|
✅ Instruction Following
|
|
|
|
✅ Natural Language Understanding
|
|
|
|
✅ Code Generation Support
|
|
|
|
✅ Research Assistance
|
|
|
|
✅ Educational Use Cases
|
|
|
|
✅ Hugging Face Transformers Compatible
|
|
|
|
✅ PyTorch Compatible
|
|
|
|
---
|
|
|
|
## 🏗 Training Method
|
|
|
|
### Full Fine-Tuning (FFT)
|
|
|
|
During training:
|
|
|
|
* All model weights were updated.
|
|
* No LoRA adapters were used.
|
|
* No PEFT layers were used.
|
|
* The complete model learned from the training dataset.
|
|
|
|
This allows the model to:
|
|
|
|
* Adapt more deeply to specialized domains.
|
|
* Improve consistency.
|
|
* Learn new patterns directly within base weights.
|
|
* Achieve stronger domain-specific performance.
|
|
|
|
---
|
|
|
|
## 📊 Training Configuration
|
|
|
|
| Parameter | Value |
|
|
| --------------- | ------------------------ |
|
|
| Training Method | Full Fine-Tuning |
|
|
| Framework | PyTorch |
|
|
| Trainer | TRL SFTTrainer |
|
|
| Precision | BF16 / FP16 |
|
|
| Architecture | Transformer |
|
|
| Optimizer | AdamW |
|
|
| Learning Rate | Custom |
|
|
| Dataset Format | JSON Instruction Dataset |
|
|
|
|
---
|
|
|
|
## 📚 Dataset
|
|
|
|
The model was trained on carefully curated instruction-response examples.
|
|
|
|
Dataset characteristics:
|
|
|
|
* Instruction Tuning
|
|
* Question Answering
|
|
* Educational Content
|
|
* Technical Discussions
|
|
* Coding Tasks
|
|
* Reasoning Tasks
|
|
|
|
Example:
|
|
|
|
```json
|
|
{
|
|
"instruction": "Explain machine learning.",
|
|
"output": "Machine learning is a branch of AI that enables systems to learn from data."
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Installation
|
|
|
|
```bash
|
|
pip install transformers torch accelerate
|
|
```
|
|
|
|
### Load Model
|
|
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
|
model_path = "Tech3Space/tech3space3-0.6B"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_path,
|
|
torch_dtype="auto",
|
|
device_map="auto"
|
|
)
|
|
|
|
prompt = "Explain neural networks."
|
|
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=256
|
|
)
|
|
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
---
|
|
|
|
## 💻 Example Usage
|
|
|
|
### Coding Assistant
|
|
|
|
```text
|
|
User:
|
|
Write a Python function to reverse a string.
|
|
|
|
Assistant:
|
|
def reverse_string(text):
|
|
return text[::-1]
|
|
```
|
|
|
|
### Educational Assistant
|
|
|
|
```text
|
|
User:
|
|
What is Kubernetes?
|
|
|
|
Assistant:
|
|
Kubernetes is an open-source container orchestration platform...
|
|
```
|
|
|
|
### Research Assistant
|
|
|
|
```text
|
|
User:
|
|
Explain transformer architecture.
|
|
|
|
Assistant:
|
|
Transformers use self-attention mechanisms to process sequences...
|
|
```
|
|
|
|
---
|
|
|
|
## 📈 Potential Applications
|
|
|
|
* AI Research
|
|
* Education
|
|
* Coding Assistance
|
|
* Knowledge Retrieval
|
|
* Chatbots
|
|
* Automation
|
|
* Experimentation
|
|
* Fine-Tuning Research
|
|
|
|
---
|
|
|
|
## ⚠️ Limitations
|
|
|
|
Like all language models:
|
|
|
|
* Responses may contain inaccuracies.
|
|
* Generated content should be verified.
|
|
* Performance depends on training data quality.
|
|
* Not intended for high-risk decision making.
|
|
|
|
---
|
|
|
|
## 🤝 Contributing
|
|
|
|
Contributions are welcome.
|
|
|
|
Ways to contribute:
|
|
|
|
* Improve datasets
|
|
* Enhance evaluation methods
|
|
* Report issues
|
|
* Submit pull requests
|
|
* Share benchmarks
|
|
|
|
Together we can build stronger open-source AI systems.
|
|
|
|
---
|
|
|
|
## 🔬 Research Philosophy
|
|
|
|
We believe that understanding AI comes from building it.
|
|
|
|
Every experiment, dataset, training run, and failure contributes to deeper knowledge.
|
|
|
|
Tech3Space-FT is part of that learning journey.
|
|
|
|
If this project helps you start your own LLM research, then it has already achieved its purpose.
|
|
|
|
---
|
|
|
|
## 🙏 Acknowledgements
|
|
|
|
Special thanks to:
|
|
|
|
* Hugging Face
|
|
* PyTorch Community
|
|
* TRL
|
|
* Transformers Library
|
|
* Open Source AI Researchers
|
|
|
|
for making modern AI development accessible to everyone.
|
|
|
|
---
|
|
|
|
## 📜 License
|
|
|
|
Released under the Apache 2.0 License.
|
|
|
|
Please verify the license before commercial deployment.
|
|
|
|
---
|
|
|
|
# ⭐ Support the Project
|
|
|
|
If you find this model useful:
|
|
|
|
* Star the repository
|
|
* Share your results
|
|
* Contribute improvements
|
|
* Build something amazing
|
|
|
|
### "The future of AI belongs to those who learn by building."
|
|
|
|
— Tech3Space
|