256 lines
8.5 KiB
Markdown
256 lines
8.5 KiB
Markdown
|
---
|
||
|
|
license: cc-by-nc-4.0
|
||
|
|
tags:
|
||
|
|
- moe
|
||
|
|
- merge
|
||
|
|
- mergekit
|
||
|
|
model-index:
|
||
|
|
- name: TinyUltra-4x1.1B-Base-Alpha
|
||
|
|
results:
|
||
|
|
- task:
|
||
|
|
type: text-generation
|
||
|
|
name: Text Generation
|
||
|
|
dataset:
|
||
|
|
name: AI2 Reasoning Challenge (25-Shot)
|
||
|
|
type: ai2_arc
|
||
|
|
config: ARC-Challenge
|
||
|
|
split: test
|
||
|
|
args:
|
||
|
|
num_few_shot: 25
|
||
|
|
metrics:
|
||
|
|
- type: acc_norm
|
||
|
|
value: 34.9
|
||
|
|
name: normalized accuracy
|
||
|
|
source:
|
||
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=indischepartij/TinyUltra-4x1.1B-Base-Alpha
|
||
|
|
name: Open LLM Leaderboard
|
||
|
|
- task:
|
||
|
|
type: text-generation
|
||
|
|
name: Text Generation
|
||
|
|
dataset:
|
||
|
|
name: HellaSwag (10-Shot)
|
||
|
|
type: hellaswag
|
||
|
|
split: validation
|
||
|
|
args:
|
||
|
|
num_few_shot: 10
|
||
|
|
metrics:
|
||
|
|
- type: acc_norm
|
||
|
|
value: 61.42
|
||
|
|
name: normalized accuracy
|
||
|
|
source:
|
||
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=indischepartij/TinyUltra-4x1.1B-Base-Alpha
|
||
|
|
name: Open LLM Leaderboard
|
||
|
|
- task:
|
||
|
|
type: text-generation
|
||
|
|
name: Text Generation
|
||
|
|
dataset:
|
||
|
|
name: MMLU (5-Shot)
|
||
|
|
type: cais/mmlu
|
||
|
|
config: all
|
||
|
|
split: test
|
||
|
|
args:
|
||
|
|
num_few_shot: 5
|
||
|
|
metrics:
|
||
|
|
- type: acc
|
||
|
|
value: 25.42
|
||
|
|
name: accuracy
|
||
|
|
source:
|
||
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=indischepartij/TinyUltra-4x1.1B-Base-Alpha
|
||
|
|
name: Open LLM Leaderboard
|
||
|
|
- task:
|
||
|
|
type: text-generation
|
||
|
|
name: Text Generation
|
||
|
|
dataset:
|
||
|
|
name: TruthfulQA (0-shot)
|
||
|
|
type: truthful_qa
|
||
|
|
config: multiple_choice
|
||
|
|
split: validation
|
||
|
|
args:
|
||
|
|
num_few_shot: 0
|
||
|
|
metrics:
|
||
|
|
- type: mc2
|
||
|
|
value: 37.59
|
||
|
|
source:
|
||
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=indischepartij/TinyUltra-4x1.1B-Base-Alpha
|
||
|
|
name: Open LLM Leaderboard
|
||
|
|
- task:
|
||
|
|
type: text-generation
|
||
|
|
name: Text Generation
|
||
|
|
dataset:
|
||
|
|
name: Winogrande (5-shot)
|
||
|
|
type: winogrande
|
||
|
|
config: winogrande_xl
|
||
|
|
split: validation
|
||
|
|
args:
|
||
|
|
num_few_shot: 5
|
||
|
|
metrics:
|
||
|
|
- type: acc
|
||
|
|
value: 65.75
|
||
|
|
name: accuracy
|
||
|
|
source:
|
||
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=indischepartij/TinyUltra-4x1.1B-Base-Alpha
|
||
|
|
name: Open LLM Leaderboard
|
||
|
|
- task:
|
||
|
|
type: text-generation
|
||
|
|
name: Text Generation
|
||
|
|
dataset:
|
||
|
|
name: GSM8k (5-shot)
|
||
|
|
type: gsm8k
|
||
|
|
config: main
|
||
|
|
split: test
|
||
|
|
args:
|
||
|
|
num_few_shot: 5
|
||
|
|
metrics:
|
||
|
|
- type: acc
|
||
|
|
value: 2.58
|
||
|
|
name: accuracy
|
||
|
|
source:
|
||
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=indischepartij/TinyUltra-4x1.1B-Base-Alpha
|
||
|
|
name: Open LLM Leaderboard
|
||
|
|
widget:
|
||
|
|
- example_title: Pirate!
|
||
|
|
messages:
|
||
|
|
- role: system
|
||
|
|
content: You are a pirate chatbot who always responds with Arr!
|
||
|
|
- role: user
|
||
|
|
content: "There's a llama on my lawn, how can I get rid of him?"
|
||
|
|
output:
|
||
|
|
text: >-
|
||
|
|
Arr! 'Tis a puzzlin' matter, me hearty! A llama on yer lawn be a rare
|
||
|
|
sight, but I've got a plan that might help ye get rid of 'im. Ye'll need
|
||
|
|
to gather some carrots and hay, and then lure the llama away with the
|
||
|
|
promise of a tasty treat. Once he's gone, ye can clean up yer lawn and
|
||
|
|
enjoy the peace and quiet once again. But beware, me hearty, for there
|
||
|
|
may be more llamas where that one came from! Arr!
|
||
|
|
---
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
# TinyUltra-4x1.1B-Base-Alpha
|
||
|
|
|
||
|
|
TinyUltra-4x1.1B-Base-Alpha is a Mixure of Experts (MoE) made with the following models using MergeKit:
|
||
|
|
* [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)
|
||
|
|
* [vihangd/DopeyTinyLlama-1.1B-v1](https://huggingface.co/vihangd/DopeyTinyLlama-1.1B-v1)
|
||
|
|
* [cognitivecomputations/TinyDolphin-2.8.1-1.1b](https://huggingface.co/cognitivecomputations/TinyDolphin-2.8.1-1.1b)
|
||
|
|
* [Josephgflowers/Tinyllama-Cinder-1.3B-Reason-Test](https://huggingface.co/Josephgflowers/Tinyllama-Cinder-1.3B-Reason-Test)
|
||
|
|
|
||
|
|
|
||
|
|
# Modelfile/Prompt format
|
||
|
|
```markdown
|
||
|
|
SYSTEM You are a TinyUltra, helpful and lovely AI assistant.
|
||
|
|
|
||
|
|
TEMPLATE <|system|> {{ .System }}</s> <|user|> {{ .Prompt }}</s> <|assistant|>
|
||
|
|
|
||
|
|
PARAMETER stop <|system|>
|
||
|
|
PARAMETER stop <|user|>
|
||
|
|
PARAMETER stop <|assistant|>
|
||
|
|
PARAMETER stop </s>
|
||
|
|
```
|
||
|
|
|
||
|
|
## 🧩 Configuration
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||
|
|
gate_mode: hidden
|
||
|
|
dtype: float16
|
||
|
|
experts:
|
||
|
|
- source_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||
|
|
positive_prompts:
|
||
|
|
- "Help me debug this code."
|
||
|
|
- "Rewrite this function in Python."
|
||
|
|
- "Optimize this C# script."
|
||
|
|
- "Implement this feature using JavaScript."
|
||
|
|
- "Convert this HTML structure into a more efficient design."
|
||
|
|
- "Assist me with writing a program that"
|
||
|
|
- source_model: vihangd/DopeyTinyLlama-1.1B-v1
|
||
|
|
positive_prompts:
|
||
|
|
- "How do you"
|
||
|
|
- "Explain the concept of"
|
||
|
|
- "Give an overview of"
|
||
|
|
- "Compare and contrast between"
|
||
|
|
- "Provide information about"
|
||
|
|
- "Help me understand"
|
||
|
|
- "Summarize"
|
||
|
|
- "Make a recommendation on"
|
||
|
|
- "Answer this question"
|
||
|
|
- source_model: cognitivecomputations/TinyDolphin-2.8.1-1.1b
|
||
|
|
positive_prompts:
|
||
|
|
- "Write a program to solve this problem"
|
||
|
|
- "Modify this function to improve its performance"
|
||
|
|
- "Refactor this code to enhance readability"
|
||
|
|
- "Create a custom function for this specific use case"
|
||
|
|
- "Optimize this algorithm to reduce computational complexity"
|
||
|
|
- "Implement this feature by extending existing codebase"
|
||
|
|
- "Integrate this API call into the application"
|
||
|
|
- "Help me troubleshoot and fix this bug"
|
||
|
|
- "Review and test this code snippet before deployment"
|
||
|
|
- "Analyze this error log to identify potential issues"
|
||
|
|
- "Generate a set of unit tests for this module"
|
||
|
|
- "Evaluate different approaches to solving this problem"
|
||
|
|
- "Do a web search for"
|
||
|
|
- "Use the plugin to"
|
||
|
|
- source_model: Josephgflowers/Tinyllama-Cinder-1.3B-Reason-Test
|
||
|
|
positive_prompts:
|
||
|
|
- "add these numbers"
|
||
|
|
- "whats 2+2"
|
||
|
|
- "subtraction"
|
||
|
|
- "division"
|
||
|
|
- "multiplication"
|
||
|
|
- "addition"
|
||
|
|
- "I need help with a math problem"
|
||
|
|
- "Solve for x"
|
||
|
|
- "Add these two numbers together: 4 + 3 = 7"
|
||
|
|
- "Multiply 5 by 6: 5 * 6 = 30"
|
||
|
|
- "Divide 8 by 2: 8 / 2 = 4"
|
||
|
|
- "Find the remainder when 9 is divided by 3: 9 % 3 = 0"
|
||
|
|
- "Calculate the square root of 16: sqrt(16) = 4"
|
||
|
|
- "Simplify the expression (a+b)/(c-d): (a+b)/(c-d)"
|
||
|
|
- "Factor out the common factor of 2 from 4x + 6y: 2(2x + 3y)"
|
||
|
|
- "Solve for x in the equation 3x - 7 = 2x + 5: x = 12"
|
||
|
|
- "Graph the line y = 2x + 3"
|
||
|
|
- "Approximate pi to three decimal places: 3.142"
|
||
|
|
- "Find the derivative of f(x) = sin(x): f'(x) = cos(x)"
|
||
|
|
- "Integrate g(x) = x^2 over the interval [0, 1]: g(1) - g(0) = 1/3"
|
||
|
|
- "Calculate the determinant of the matrix A = [[2, 3], [4, 5]]: det(A) = 2*5 - 3*4 = -2"
|
||
|
|
- "Solve the system of equations Ax = b: x = [-5, 10]"
|
||
|
|
- "Calculate the sum of the first n natural numbers using the formula Sn = n*(n+1)/2: sum(n=1 to 5) = 15"
|
||
|
|
```
|
||
|
|
|
||
|
|
## 💻 Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
!pip install -qU transformers bitsandbytes accelerate
|
||
|
|
|
||
|
|
from transformers import AutoTokenizer
|
||
|
|
import transformers
|
||
|
|
import torch
|
||
|
|
|
||
|
|
model = "gmonsoon/TinyUltra-4x1.1B-Base-Alpha"
|
||
|
|
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(model)
|
||
|
|
pipeline = transformers.pipeline(
|
||
|
|
"text-generation",
|
||
|
|
model=model,
|
||
|
|
model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
|
||
|
|
)
|
||
|
|
|
||
|
|
messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
|
||
|
|
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
|
|
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
|
||
|
|
print(outputs[0]["generated_text"])
|
||
|
|
```
|
||
|
|
GGUF: https://huggingface.co/indischepartij/TinyUltra-4x1.1B-Base-Alpha-GGUF
|
||
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_indischepartij__TinyUltra-4x1.1B-Base-Alpha)
|
||
|
|
|
||
|
|
| Metric |Value|
|
||
|
|
|---------------------------------|----:|
|
||
|
|
|Avg. |37.94|
|
||
|
|
|AI2 Reasoning Challenge (25-Shot)|34.90|
|
||
|
|
|HellaSwag (10-Shot) |61.42|
|
||
|
|
|MMLU (5-Shot) |25.42|
|
||
|
|
|TruthfulQA (0-shot) |37.59|
|
||
|
|
|Winogrande (5-shot) |65.75|
|
||
|
|
|GSM8k (5-shot) | 2.58|
|
||
|
|
|