Felladrin/Llama-160M-Chat-v1
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
1 Commit 1 Branch 0 Tags
main
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE

README.md

language, license, tags, base_model, datasets, widget, inference, model-index
language license tags base_model datasets widget inference model-index
en
apache-2.0
text-generation
JackFram/llama-160m
ehartford/wizard_vicuna_70k_unfiltered
totally-not-an-llm/EverythingLM-data-V3
Open-Orca/SlimOrca-Dedup
databricks/databricks-dolly-15k
THUDM/webglm-qa
messages
role content
system You are a helpful assistant, who answers with empathy.
role content
user Got a question for you!
role content
assistant Sure! What's it?
role content
user Why do you love cats so much!? 🐈
messages
role content
system You are a helpful assistant who answers user's questions with empathy.
role content
user Who is Mona Lisa?
messages
role content
system You are a helpful assistant who provides concise responses.
role content
user Heya!
role content
assistant Hi! How may I help you today?
role content
user I need to build a simple website. Where should I start learning about web development?
messages
role content
user Invited some friends to come home today. Give me some ideas for games to play with them!
messages
role content
system You are a helpful assistant who answers user's questions with details and curiosity.
role content
user What are some potential applications for quantum computing?
messages
role content
system You are a helpful assistant who gives creative responses.
role content
user Write the specs of a game about mages in a fantasy world.
messages
role content
system You are a helpful assistant who answers user's questions with details.
role content
user Tell me about the pros and cons of social media.
messages
role content
system You are a helpful assistant who answers user's questions with confidence.
role content
user What is a dog?
role content
assistant A dog is a four-legged, domesticated animal that is a member of the class Mammalia, which includes all mammals. Dogs are known for their loyalty, playfulness, and ability to be trained for various tasks. They are also used for hunting, herding, and as service animals.
role content
user What is the color of an apple?
parameters
max_new_tokens penalty_alpha top_k repetition_penalty
250 0.5 4 1.01
name results
Llama-160M-Chat-v1
task dataset metrics source
type name
text-generation Text Generation
name type config split args
AI2 Reasoning Challenge (25-Shot) ai2_arc ARC-Challenge test
num_few_shot
25
type value name
acc_norm 24.74 normalized accuracy
url name
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type split args
HellaSwag (10-Shot) hellaswag validation
num_few_shot
10
type value name
acc_norm 35.29 normalized accuracy
url name
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type config split args
MMLU (5-Shot) cais/mmlu all test
num_few_shot
5
type value name
acc 26.13 accuracy
url name
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type config split args
TruthfulQA (0-shot) truthful_qa multiple_choice validation
num_few_shot
0
type value
mc2 44.16
url name
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type config split args
Winogrande (5-shot) winogrande winogrande_xl validation
num_few_shot
5
type value name
acc 51.3 accuracy
url name
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type config split args
GSM8k (5-shot) gsm8k main test
num_few_shot
5
type value name
acc 0.0 accuracy
url name
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type args
IFEval (0-Shot) HuggingFaceH4/ifeval
num_few_shot
0
type value name
inst_level_strict_acc and prompt_level_strict_acc 15.75 strict accuracy
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type args
BBH (3-Shot) BBH
num_few_shot
3
type value name
acc_norm 3.17 normalized accuracy
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type args
MATH Lvl 5 (4-Shot) hendrycks/competition_math
num_few_shot
4
type value name
exact_match 0.0 exact match
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type args
GPQA (0-shot) Idavidrein/gpqa
num_few_shot
0
type value name
acc_norm 1.01 acc_norm
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type args
MuSR (0-shot) TAUR-Lab/MuSR
num_few_shot
0
type value name
acc_norm 3.17 acc_norm
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard
task dataset metrics source
type name
text-generation Text Generation
name type config split args
MMLU-PRO (5-shot) TIGER-Lab/MMLU-Pro main test
num_few_shot
5
type value name
acc 1.51 accuracy
url name
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard

A Llama Chat Model of 160M Parameters

Recommended Prompt Format

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant

Recommended Inference Parameters

penalty_alpha: 0.5
top_k: 4
repetition_penalty: 1.01

Usage Example

from transformers import pipeline

generate = pipeline("text-generation", "Felladrin/Llama-160M-Chat-v1")

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant who answers user's questions with details and curiosity.",
    },
    {
        "role": "user",
        "content": "What are some potential applications for quantum computing?",
    },
]

prompt = generate.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

output = generate(
    prompt,
    max_new_tokens=1024,
    penalty_alpha=0.5,
    top_k=4,
    repetition_penalty=1.01,
)

print(output[0]["generated_text"])

Old Open LLM Leaderboard Evaluation Results

Metric Value
Avg. 30.27
AI2 Reasoning Challenge (25-Shot) 24.74
HellaSwag (10-Shot) 35.29
MMLU (5-Shot) 26.13
TruthfulQA (0-shot) 44.16
Winogrande (5-shot) 51.30
GSM8k (5-shot) 0.00

New Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 4.10
IFEval (0-Shot) 15.75
BBH (3-Shot) 3.17
MATH Lvl 5 (4-Shot) 0.00
GPQA (0-shot) 1.01
MuSR (0-shot) 3.17
MMLU-PRO (5-shot) 1.51
Description
Model synced from source: Felladrin/Llama-160M-Chat-v1
Readme 581 KiB