133 lines
5.3 KiB
Markdown
133 lines
5.3 KiB
Markdown
|
||
---
|
||
|
||
license: apache-2.0
|
||
datasets:
|
||
- nvidia/ChatQA-Training-Data
|
||
language:
|
||
- en
|
||
base_model:
|
||
- meta-llama/Llama-3.2-3B
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
|
||
---
|
||
|
||
[](https://hf.co/QuantFactory)
|
||
|
||
|
||
# QuantFactory/OneLLM-Doey-V1-Llama-3.2-3B-GGUF
|
||
This is quantized version of [DoeyLLM/OneLLM-Doey-V1-Llama-3.2-3B](https://huggingface.co/DoeyLLM/OneLLM-Doey-V1-Llama-3.2-3B) created using llama.cpp
|
||
|
||
# Original Model Card
|
||
|
||
## **Model Summary**
|
||
This model is a fine-tuned version of **LLaMA 3.2-3B**, optimized using **LoRA (Low-Rank Adaptation)** on the [NVIDIA ChatQA-Training-Data](https://huggingface.co/datasets/nvidia/ChatQA-Training-Data). It is tailored for conversational AI, question answering, and other instruction-following tasks, with support for sequences up to 1024 tokens.
|
||
|
||
---
|
||
|
||
## **Key Features**
|
||
- **Base Model**: LLaMA 3.2-3B
|
||
- **Fine-Tuning Framework**: LoRA
|
||
- **Dataset**: NVIDIA ChatQA-Training-Data
|
||
- **Max Sequence Length**: 1024 tokens
|
||
- **Use Case**: Instruction-based tasks, question answering, conversational AI.
|
||
|
||
## **Model Usage**
|
||
This fine-tuned model is suitable for:
|
||
- **Conversational AI**: Chatbots and dialogue agents with improved contextual understanding.
|
||
- **Question Answering**: Generating concise and accurate answers to user queries.
|
||
- **Instruction Following**: Responding to structured prompts.
|
||
- **Long-Context Tasks**: Processing sequences up to 1024 tokens for long-text reasoning.
|
||
|
||
# **How to Use DoeyLLM / OneLLM-Doey-V1-Llama-3.2-3B-Instruct**
|
||
|
||
This guide explains how to use the **DoeyLLM** model on both app (iOS) and PC platforms.
|
||
|
||
---
|
||
|
||
## **App (iOS): Use with OneLLM**
|
||
|
||
OneLLM brings versatile large language models (LLMs) to your device—Llama, Gemma, Qwen, Mistral, and more. Enjoy private, offline GPT and AI tools tailored to your needs.
|
||
|
||
With OneLLM, experience the capabilities of leading-edge language models directly on your device, all without an internet connection. Get fast, reliable, and intelligent responses, while keeping your data secure with local processing.
|
||
|
||
### **Quick Start for iOS**
|
||
|
||
Follow these steps to integrate the **DoeyLLM** model using the OneLLM app:
|
||
|
||
1. **Download OneLLM**
|
||
Get the app from the [App Store](https://apps.apple.com/us/app/onellm-private-ai-gpt-llm/id6737907910) and install it on your iOS device.
|
||
|
||
2. **Load the DoeyLLM Model**
|
||
Use the OneLLM interface to load the DoeyLLM model directly into the app:
|
||
- Navigate to the **Model Library**.
|
||
- Search for `DoeyLLM`.
|
||
- Select the model and tap **Download** to store it locally on your device.
|
||
3. **Start Conversing**
|
||
Once the model is loaded, you can begin interacting with it through the app's chat interface. For example:
|
||
- Tap the **Chat** tab.
|
||
- Type your question or prompt, such as:
|
||
> "Explain the significance of AI in education."
|
||
- Receive real-time, intelligent responses generated locally.
|
||
|
||
### **Key Features of OneLLM**
|
||
- **Versatile Models**: Supports various LLMs, including Llama, Gemma, and Qwen.
|
||
- **Private & Secure**: All processing occurs locally on your device, ensuring data privacy.
|
||
- **Offline Capability**: Use the app without requiring an internet connection.
|
||
- **Fast Performance**: Optimized for mobile devices, delivering low-latency responses.
|
||
|
||
For more details or support, visit the [OneLLM App Store page](https://apps.apple.com/us/app/onellm-private-ai-gpt-llm/id6737907910).
|
||
|
||
## **PC: Use with Transformers**
|
||
|
||
The DoeyLLM model can also be used on PC platforms through the `transformers` library, enabling robust and scalable inference for various NLP tasks.
|
||
|
||
### **Quick Start for PC**
|
||
Follow these steps to use the model with Transformers:
|
||
|
||
1. **Install Transformers**
|
||
Ensure you have `transformers >= 4.43.0` installed. Update or install it via pip:
|
||
|
||
```bash
|
||
pip install --upgrade transformers
|
||
|
||
2. **Load the Model**
|
||
Use the transformers library to load the model and tokenizer:
|
||
|
||
Starting with `transformers >= 4.43.0` onward, you can run conversational inference using the Transformers `pipeline` abstraction or by leveraging the Auto classes with the `generate()` function.
|
||
|
||
Make sure to update your transformers installation via `pip install --upgrade transformers`.
|
||
|
||
```python
|
||
import torch
|
||
from transformers import pipeline
|
||
|
||
model_id = "OneLLM-Doey-V1-Llama-3.2-3B"
|
||
pipe = pipeline(
|
||
"text-generation",
|
||
model=model_id,
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
)
|
||
messages = [
|
||
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
|
||
{"role": "user", "content": "Who are you?"},
|
||
]
|
||
outputs = pipe(
|
||
messages,
|
||
max_new_tokens=256,
|
||
)
|
||
print(outputs[0]["generated_text"][-1])
|
||
```
|
||
|
||
|
||
|
||
## Responsibility & Safety
|
||
|
||
As part of our responsible release strategy, we adopted a three-pronged approach to managing trust and safety risks:
|
||
|
||
Enable developers to deploy helpful, safe, and flexible experiences for their target audience and the use cases supported by the model.
|
||
Protect developers from adversarial users attempting to exploit the model’s capabilities to potentially cause harm.
|
||
Provide safeguards for the community to help prevent the misuse of the model.
|