Llama-160M-Chat-v1/README.md

language, license, tags, base_model, datasets, widget, inference, model-index

language	license	tags	base_model	datasets	widget	inference	model-index
en	apache-2.0	text-generation	JackFram/llama-160m	ehartford/wizard_vicuna_70k_unfiltered totally-not-an-llm/EverythingLM-data-V3 Open-Orca/SlimOrca-Dedup databricks/databricks-dolly-15k THUDM/webglm-qa	messages role content system You are a helpful assistant, who answers with empathy. role content user Got a question for you! role content assistant Sure! What's it? role content user Why do you love cats so much!? 🐈 messages role content system You are a helpful assistant who answers user's questions with empathy. role content user Who is Mona Lisa? messages role content system You are a helpful assistant who provides concise responses. role content user Heya! role content assistant Hi! How may I help you today? role content user I need to build a simple website. Where should I start learning about web development? messages role content user Invited some friends to come home today. Give me some ideas for games to play with them! messages role content system You are a helpful assistant who answers user's questions with details and curiosity. role content user What are some potential applications for quantum computing? messages role content system You are a helpful assistant who gives creative responses. role content user Write the specs of a game about mages in a fantasy world. messages role content system You are a helpful assistant who answers user's questions with details. role content user Tell me about the pros and cons of social media. messages role content system You are a helpful assistant who answers user's questions with confidence. role content user What is a dog? role content assistant A dog is a four-legged, domesticated animal that is a member of the class Mammalia, which includes all mammals. Dogs are known for their loyalty, playfulness, and ability to be trained for various tasks. They are also used for hunting, herding, and as service animals. role content user What is the color of an apple?	parameters max_new_tokens penalty_alpha top_k repetition_penalty 250 0.5 4 1.01	name results Llama-160M-Chat-v1 task dataset metrics source type name text-generation Text Generation name type config split args AI2 Reasoning Challenge (25-Shot) ai2_arc ARC-Challenge test num_few_shot 25 type value name acc_norm 24.74 normalized accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args HellaSwag (10-Shot) hellaswag validation num_few_shot 10 type value name acc_norm 35.29 normalized accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU (5-Shot) cais/mmlu all test num_few_shot 5 type value name acc 26.13 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args TruthfulQA (0-shot) truthful_qa multiple_choice validation num_few_shot 0 type value mc2 44.16 url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args Winogrande (5-shot) winogrande winogrande_xl validation num_few_shot 5 type value name acc 51.3 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args GSM8k (5-shot) gsm8k main test num_few_shot 5 type value name acc 0.0 accuracy url name https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args IFEval (0-Shot) HuggingFaceH4/ifeval num_few_shot 0 type value name inst_level_strict_acc and prompt_level_strict_acc 15.75 strict accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args BBH (3-Shot) BBH num_few_shot 3 type value name acc_norm 3.17 normalized accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MATH Lvl 5 (4-Shot) hendrycks/competition_math num_few_shot 4 type value name exact_match 0.0 exact match url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args GPQA (0-shot) Idavidrein/gpqa num_few_shot 0 type value name acc_norm 1.01 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MuSR (0-shot) TAUR-Lab/MuSR num_few_shot 0 type value name acc_norm 3.17 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU-PRO (5-shot) TIGER-Lab/MMLU-Pro main test num_few_shot 5 type value name acc 1.51 accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Felladrin/Llama-160M-Chat-v1 Open LLM Leaderboard

A Llama Chat Model of 160M Parameters

• Base model: JackFram/llama-160m
• Datasets:
  • ehartford/wizard_vicuna_70k_unfiltered
  • totally-not-an-llm/EverythingLM-data-V3
  • Open-Orca/SlimOrca-Dedup
  • databricks/databricks-dolly-15k
  • THUDM/webglm-qa
• Availability in other ML formats:
  • GGUF: Felladrin/gguf-Llama-160M-Chat-v1
  • ONNX: Felladrin/onnx-Llama-160M-Chat-v1
  • MLC: Felladrin/mlc-q4f16-Llama-160M-Chat-v1
  • MLX: mlx-community/Llama-160M-Chat-v1-4bit-mlx

Recommended Prompt Format

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant

Recommended Inference Parameters

penalty_alpha: 0.5
top_k: 4
repetition_penalty: 1.01

Usage Example

from transformers import pipeline

generate = pipeline("text-generation", "Felladrin/Llama-160M-Chat-v1")

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant who answers user's questions with details and curiosity.",
    },
    {
        "role": "user",
        "content": "What are some potential applications for quantum computing?",
    },
]

prompt = generate.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

output = generate(
    prompt,
    max_new_tokens=1024,
    penalty_alpha=0.5,
    top_k=4,
    repetition_penalty=1.01,
)

print(output[0]["generated_text"])

Old Open LLM Leaderboard Evaluation Results

Metric	Value
Avg.	30.27
AI2 Reasoning Challenge (25-Shot)	24.74
HellaSwag (10-Shot)	35.29
MMLU (5-Shot)	26.13
TruthfulQA (0-shot)	44.16
Winogrande (5-shot)	51.30
GSM8k (5-shot)	0.00

New Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	4.10
IFEval (0-Shot)	15.75
BBH (3-Shot)	3.17
MATH Lvl 5 (4-Shot)	0.00
GPQA (0-shot)	1.01
MuSR (0-shot)	3.17
MMLU-PRO (5-shot)	1.51