OpceanAI/Yuuki-3.7
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
f6217ffe84060d9b9a38d841291ded8b3ac9d010
Yuuki-3.7/README.md

713 lines
16 KiB
Markdown
Raw Normal View History

初始化项目，由ModelHub XC社区提供模型 Model: OpceanAI/Yuuki-3.7 Source: Original Platform
2026-07-21 10:56:11 +08:00
---
license: apache-2.0
datasets:
- bigcode/the-stack
language:
- en
- es
base_model:
- openai-community/gpt2
pipeline_tag: text-generation
library_name: pytorch
tags:
- code
- transformers
metrics:
- perplexity
- code_eval
new_version: OpceanAI/Yuuki-v0.1
---
<div align="center">
<br>
<img src="https://img.shields.io/badge/%E2%9C%A6-YUUKI--3.7-000000?style=for-the-badge&labelColor=000000" alt="Yuuki 3.7" height="50">
<br><br>
# Code Generation Model Trained on a Phone
**Multilingual code LLM trained entirely on a smartphone with zero budget.**<br>
**GPT-2 architecture. Snapdragon 685. CPU only. $0.00.**
<br>
<a href="#features"><img src="https://img.shields.io/badge/FEATURES-000000?style=for-the-badge" alt="Features"></a>
&nbsp;&nbsp;
<a href="https://huggingface.co/spaces/OpceanAI/Yuuki"><img src="https://img.shields.io/badge/LIVE_DEMO-000000?style=for-the-badge" alt="Demo"></a>
&nbsp;&nbsp;
<a href="https://github.com/sponsors/aguitauwu"><img src="https://img.shields.io/badge/SPONSOR-000000?style=for-the-badge" alt="Sponsor"></a>
<br><br>
[![License](https://img.shields.io/badge/Apache_2.0-222222?style=flat-square&logo=opensourceinitiative&logoColor=white)](LICENSE)
&nbsp;
[![Model](https://img.shields.io/badge/GPT--2-222222?style=flat-square&logo=openai&logoColor=white)](https://huggingface.co/openai-community/gpt2)
&nbsp;
[![Dataset](https://img.shields.io/badge/The_Stack-222222?style=flat-square&logo=databricks&logoColor=white)](https://huggingface.co/datasets/bigcode/the-stack)
&nbsp;
[![HuggingFace](https://img.shields.io/badge/Transformers-222222?style=flat-square&logo=huggingface&logoColor=white)](https://huggingface.co/docs/transformers)
&nbsp;
[![Training](https://img.shields.io/badge/Mobile_CPU-222222?style=flat-square&logo=qualcomm&logoColor=white)](https://www.qualcomm.com/products/mobile/snapdragon/smartphones/snapdragon-6-series-mobile-platforms/snapdragon-685-mobile-platform)
<br>
---
<br>
</div>
## ⚠️ Early Development Notice
**Yuuki 3.7** represents **~3.7%** of the total training planned for version v0.1. This is an **early snapshot** of the model during active training.
- ⏳ Training progress: **1,417 / 37,500 steps (3.78%)**
- 📅 Estimated completion: **Mid-March 2026**
- 🧪 Performance assessments on this checkpoint **do not reflect** the final v0.1 model
<br>
---
<br>
<div align="center">
## What is Yuuki 3.7?
</div>
<br>
**Yuuki 3.7** is a multilingual code generation language model being trained entirely on a **Redmi 12 smartphone** (Snapdragon 685, CPU only) with **zero cloud budget**. This project proves that meaningful AI training doesn't require expensive GPU infrastructure -- just patience, determination, and time.
The model is based on **GPT-2 architecture** (82M parameters) and is being trained on **The Stack** dataset with 75,000 carefully selected code examples across multiple programming languages.
<br>
---
<br>
<div align="center">
## Features
</div>
<br>
<table>
<tr>
<td width="50%" valign="top">
**Zero-Budget Training**
Trained entirely on a $150 Android phone with no cloud compute, no GPU acceleration, and no data center infrastructure. Pure CPU grinding over 6+ weeks of continuous training.
<br>
**Multilingual Code Generation**
Trained on code from 80+ programming languages including Agda, C, Assembly, JavaScript, Python, and more. Dataset ordered alphabetically, with early languages showing strongest performance.
<br>
**Multiple Quantized Formats**
Available in GGUF format with multiple quantization levels (Q4_0, Q4_K_M, Q5_K_M, Q8_0, F32) for efficient inference on CPU and mobile devices.
<br>
**Open Source & Commercial Use**
Licensed under Apache 2.0. Use commercially, modify, distribute -- just give credit. Full transparency with training code, dataset details, and loss curves.
</td>
<td width="50%" valign="top">
**Real-Time Training Progress**
Follow the training journey with regular updates, loss curves, and example outputs. Currently at step 1,417 with best loss of 1.7053 achieved.
<br>
**Early Language Specialization**
Due to alphabetical dataset ordering, the model shows strong early performance in Agda (85/100 quality), with C, Assembly, and other languages progressively learning.
<br>
**Proof of Concept**
Demonstrates that LLM training is accessible to anyone with consumer hardware. Breaking the myth that AI development requires expensive infrastructure.
<br>
**Complete Ecosystem**
Part of the Yuuki project with CLI tools (yuy, yuy-chat), web interfaces (Yuuki-chat, Yuuki-web), and comprehensive documentation.
</td>
</tr>
</table>
<br>
---
<br>
<div align="center">
## Training Status
</div>
<br>
### Current Progress
| Metric | Value |
|:-------|:------|
| **Training Steps** | 1,417 / 37,500 (3.78%) |
| **Epoch** | 0.08 / 2.0 |
| **Current Loss** | 1.70 - 2.23 (oscillating) |
| **Best Loss** | 1.7053 ⭐ |
| **Training Time** | ~3 days |
| **Time Remaining** | ~39 days |
| **Speed** | ~100 seconds/step |
| **ETA** | Mid-March 2026 |
<br>
### Loss Progression
```
Step 0: Loss 3.35 (baseline)
Step 500: Loss 2.50 ↓ -25%
Step 1000: Loss 2.00 ↓ -40%
Step 1265: Loss 1.83 ↓ -45%
Step 1292: Loss 1.71 ↓ -49% ⭐ RECORD
Step 1417: Loss 2.23 (current)
```
<br>
### Language Quality (Current Checkpoint)
| Language | Exposure | Quality Score | Status |
|:---------|:---------|:-------------|:-------|
| **Agda** | High | 85 / 100 | ✅ Excellent |
| **C** | Starting | 30 / 100 | ⏳ Learning |
| **Assembly** | Low | 5 / 100 | 🌱 Minimal |
| **Python** | None | 0 / 100 | ❌ Not reached yet |
<br>
---
<br>
<div align="center">
## Example Output
</div>
<br>
### Agda Code Generation (Step 1,300)
**Prompt:** `module Main where`
```agda
module Main where (x, f) in a
open import Cubical.Sigma
open import Cubical.Sigma.Core
open import Cubical.Foundations.H
```
✅ **Real Agda libraries!** The model has learned actual Cubical type theory modules and imports, demonstrating genuine understanding of the language structure.
<br>
---
<br>
<div align="center">
## Hardware & Setup
</div>
<br>
<table>
<tr>
<td width="50%" valign="top">
**Training Hardware**
| Component | Specification |
|:----------|:--------------|
| Device | Redmi 12 (Android phone) |
| CPU | Snapdragon 685 (8-core ARM) |
| RAM | 6 GB |
| Storage | 128 GB |
| NPU | Hexagon 686 (1 TOPS) |
| GPU | Adreno 610 (243 GFLOPS) |
| GPU Usage | **NOT USED** for training |
| Cost | **$0.00** in compute |
</td>
<td width="50%" valign="top">
**Training Configuration**
| Parameter | Value |
|:----------|:------|
| Base Model | DistilGPT-2 (82M parameters) |
| Dataset | The Stack (75,000 examples) |
| Batch Size | 1 |
| Gradient Accumulation | 4 |
| Effective Batch Size | 4 |
| Learning Rate | 5e-5 |
| Max Sequence Length | 256 tokens |
| Optimizer | AdamW |
| Epochs | 2.0 |
| Total Tokens | ~30M |
</td>
</tr>
</table>
<br>
### Why So Slow?
```
100 seconds/step × 37,500 steps = 3,750,000 seconds
= 1,042 hours
= 43.4 days
= ~6 weeks of continuous training
```
**No GPU acceleration. Pure CPU grinding.** 💪
<br>
### Training Breakdown (per step)
- **Forward pass:** 40 seconds
- **Backward pass:** 40 seconds
- **Optimizer step:** 20 seconds
- **Total:** ~100 seconds
<br>
### CPU vs GPU Comparison
| Method | Speed | Cost (42 days) | Verdict |
|:-------|:------|:---------------|:--------|
| **Mobile CPU** | 100 sec/step | **$0.00** | FREE but SLOW |
| **Cloud GPU** | 0.5 sec/step (200x faster) | $500 - $2,000 | FAST but EXPENSIVE |
For proof of concept: **Mobile wins.** 🏆
<br>
---
<br>
<div align="center">
## Available Formats
</div>
<br>
### GGUF Quantized Models
Optimized for CPU inference with llama.cpp and Ollama.
| Format | Size | Use Case | Quality |
|:-------|:-----|:---------|:--------|
| **yuuki-3.7-f32.gguf** | ~328 MB | Full precision baseline | Best |
| **yuuki-3.7-q8_0.gguf** | ~87 MB | High quality, smaller size | Excellent |
| **yuuki-3.7-q5_k_m.gguf** | ~56 MB | Balanced quality/size | Very Good |
| **yuuki-3.7-q4_k_m.gguf** | ~47 MB | Good quality, efficient | Good |
| **yuuki-3.7-q4_0.gguf** | ~46 MB | Most efficient, fast | Good |
<br>
---
<br>
<div align="center">
## Usage
</div>
<br>
### With Transformers (PyTorch)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model
model = AutoModelForCausalLM.from_pretrained("OpceanAI/Yuuki-3.7")
tokenizer = AutoTokenizer.from_pretrained("OpceanAI/Yuuki-3.7")
# Generate code
prompt = "def fibonacci(n):"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100, temperature=0.7)
code = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(code)
```
<br>
### With llama.cpp (GGUF)
```bash
# Run inference
./llama.cpp/main -m yuuki-3.7-q4_k_m.gguf \
-p "def hello_world():" \
-n 50 \
-t 4
```
<br>
### With Ollama
```bash
# Create Modelfile
cat > Modelfile << EOF
FROM ./yuuki-3.7-q4_k_m.gguf
TEMPLATE """{{ .Prompt }}"""
PARAMETER temperature 0.7
PARAMETER top_p 0.9
EOF
# Import and run
ollama create yuuki-3.7 -f Modelfile
ollama run yuuki-3.7 "module Main where"
```
<br>
---
<br>
<div align="center">
## Roadmap
</div>
<br>
### v0.1 (Current - Proof of Concept)
- [x] Setup training pipeline
- [x] Start training (Step 0)
- [x] Reach Step 1,000
- [x] Break loss 2.0 barrier
- [x] Break loss 1.8 barrier ⭐
- [ ] Checkpoint 2,500 (7%)
- [ ] Checkpoint 5,000 (13%)
- [ ] Checkpoint 10,000 (27%)
- [ ] Checkpoint 18,750 (50% - Epoch 1 complete)
- [ ] Checkpoint 37,500 (100% - DONE)
- [x] Quantize to GGUF formats
- [ ] Convert to ONNX
- [ ] Publish final model
**ETA:** Mid-March 2026
<br>
### v0.2 (The Full Dataset)
- **Dataset:** 786,387 examples (full Stack)
- **Duration:** 418 days (~14 months)
- **Epochs:** 2.0
- **Total tokens:** ~314M
- **Dataset fix:** SHUFFLED (not alphabetical)
- **Languages:** All 80+ languages balanced
- **Start:** March 2026
- **End:** May 2027
<br>
### v0.3+ (PC Era)
- **Hardware upgrade:** RTX 4060/4070
- **Larger models:** 350M-1B parameters
- **Faster training:** ~30x speedup
- **Advanced techniques:** LoRA, QLoRA, etc.
<br>
---
<br>
<div align="center">
## Philosophy
</div>
<br>
> **"The barrier to AI isn't money. It's mindset."**
This project demonstrates:
- ✅ You **CAN** train LLMs without GPUs
- ✅ Patience > Hardware
- ✅ $0 budget is enough to start
- ✅ Limited resources inspire creativity
- ✅ Anyone can contribute to AI
<br>
---
<br>
<div align="center">
## Known Limitations
</div>
<br>
⚠️ **Dataset Ordering:** Alphabetical (not shuffled) - learns early languages best
⚠️ **Token Count:** Only ~30M tokens (vs GPT-2's 40B)
⚠️ **Training Speed:** Very slow (~100 sec/step)
⚠️ **Model Size:** Small (82M parameters)
⚠️ **Language Coverage:** Incomplete due to alphabetical ordering
**These will be addressed in v0.2 with shuffled dataset and extended training.**
<br>
---
<br>
<div align="center">
## Benchmarks
</div>
<br>
### Expected Performance (Post-Training)
Coming soon after training completes (~March 2026).
| Language | Expected Score | Notes |
|:---------|:--------------|:------|
| Agda | 85-95 / 100 | Primary language, highest exposure |
| C | 85-92 / 100 | Secondary language, good coverage |
| Assembly | 75-85 / 100 | Tertiary language, moderate coverage |
| Python | 10-20 / 100 | Minimal due to alphabet order |
<br>
---
<br>
<div align="center">
## Related Projects
</div>
<br>
| Project | Description |
|:--------|:------------|
| [Yuuki-best](https://huggingface.co/OpceanAI/Yuuki-best) | Best checkpoint model with highest quality |
| [Yuuki-v0.1](https://huggingface.co/OpceanAI/Yuuki-v0.1) | Latest release version |
| [yuy](https://github.com/YuuKi-OS/yuy) | CLI for downloading, managing, and running Yuuki models |
| [yuy-chat](https://github.com/YuuKi-OS/yuy-chat) | TUI chat interface for local AI conversations |
| [Yuuki-chat](https://github.com/YuuKi-OS/Yuuki-chat) | Web-based chat interface with research modes |
| [Yuuki-web](https://github.com/YuuKi-OS/Yuuki-web) | Official landing page and project showcase |
| [yuuki-training](https://github.com/YuuKi-OS/yuuki-training) | Training code and scripts |
| [Yuuki Space](https://huggingface.co/spaces/OpceanAI/Yuuki) | Web-based interactive demo |
<br>
---
<br>
<div align="center">
## Links
</div>
<br>
<div align="center">
[![Model Weights](https://img.shields.io/badge/Model_Weights-Hugging_Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/OpceanAI/Yuuki-3.7)
&nbsp;
[![Live Demo](https://img.shields.io/badge/Live_Demo-Spaces-ffd21e?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/spaces/OpceanAI/Yuuki)
&nbsp;
[![YUY CLI](https://img.shields.io/badge/Yuy_CLI-GitHub-181717?style=for-the-badge&logo=github&logoColor=white)](https://github.com/YuuKi-OS/yuy)
<br>
[![YUY Chat](https://img.shields.io/badge/Yuy_Chat-GitHub-181717?style=for-the-badge&logo=github&logoColor=white)](https://github.com/YuuKi-OS/yuy-chat)
&nbsp;
[![Sponsor](https://img.shields.io/badge/Sponsor-GitHub_Sponsors-ea4aaa?style=for-the-badge&logo=githubsponsors&logoColor=white)](https://github.com/sponsors/aguitauwu)
&nbsp;
[![Discord](https://img.shields.io/badge/Discord-Community-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/j8zV2u8k)
</div>
<br>
---
<br>
<div align="center">
## Community
</div>
<br>
Join the Yuuki community:
- 💬 [Discord Server](https://discord.gg/j8zV2u8k) - Chat with other users and contributors
- 🐦 [Twitter Updates](https://twitter.com/aguitauwu) - Follow development progress
- 📺 [GitHub](https://github.com/aguitauwu) - Star repos and contribute
- 💖 [GitHub Sponsors](https://github.com/sponsors/aguitauwu) - Support the project
- 🦙 .[ollama](https://ollama.com/aguitachan3/yuuki-3.7)
<br>
---
<br>
<div align="center">
## Acknowledgments
</div>
<br>
- **HuggingFace** - Infrastructure and transformers library
- **BigCode** - The Stack dataset
- **The ML community** - For saying "you need GPUs" (best motivation 😏)
- **Everyone who believed** - This is for you
<br>
---
<br>
<div align="center">
## License
</div>
<br>
```
Apache License 2.0
Copyright (c) 2026 Yuuki Project
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```
**You can use Yuuki commercially, modify it, distribute it. Just give credit.** ✅
<br>
---
<br>
<div align="center">
## Updates
</div>
<br>
| Date | Milestone |
|:-----|:----------|
| **2026-01-29** | Training started |
| **2026-01-29** | Step 1,000 reached - Loss 2.00 |
| **2026-01-29** | Step 1,292 - NEW RECORD Loss 1.7053 |
| **2026-01-29** | Repository created on HuggingFace |
| **2026-02-15** | GGUF quantized models released |
**Last updated:** 2026-02-15
<br>
---
<br>
<div align="center">
**Built with patience, a phone, and zero budget.**
<br>
[![Yuuki Project](https://img.shields.io/badge/Yuuki_Project-2026-000000?style=for-the-badge)](https://huggingface.co/OpceanAI)
<br>
*Follow the journey of training an LLM with $0 budget. One step at a time.* 🌸
</div>
Reference in New Issue Copy Permalink