license, language, base_model, pipeline_tag, library_name, tags, model-index

license	language	base_model	pipeline_tag	library_name	tags	model-index
apache-2.0	en zh de	Qwen/Qwen2.5-14B-Instruct	text-generation	transformers	text-generation-inference StreamlinedMemory code Math	name results Sombrero-Opus-14B-Sm5 task dataset metrics source type name text-generation Text Generation name type split args IFEval (0-Shot) wis-k/instruction-following-eval train num_few_shot 0 type value name inst_level_strict_acc and prompt_level_strict_acc 68.52 averaged accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FSombrero-Opus-14B-Sm5 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args BBH (3-Shot) SaylorTwift/bbh test num_few_shot 3 type value name acc_norm 50.6 normalized accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FSombrero-Opus-14B-Sm5 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args MATH Lvl 5 (4-Shot) lighteval/MATH-Hard test num_few_shot 4 type value name exact_match 40.94 exact match url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FSombrero-Opus-14B-Sm5 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type split args GPQA (0-shot) Idavidrein/gpqa train num_few_shot 0 type value name acc_norm 18.23 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FSombrero-Opus-14B-Sm5 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MuSR (0-shot) TAUR-Lab/MuSR num_few_shot 0 type value name acc_norm 19.51 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FSombrero-Opus-14B-Sm5 Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU-PRO (5-shot) TIGER-Lab/MMLU-Pro main test num_few_shot 5 type value name acc 48.89 accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FSombrero-Opus-14B-Sm5 Open LLM Leaderboard

Sombrero-Opus-14B-Sm5

Sombrero-Opus-14B-Sm5 is based on the Qwen 2.5 14B modality architecture, designed to enhance coding efficiency and computational reasoning. This model is optimized for streamlined memory usage, avoiding unwanted textual token generation, and excelling in coding, explanatory reasoning, mathematical problem-solving, and technical tasks. It has been fine-tuned using specialized datasets to improve code generation, structured programming logic, and problem-solving capabilities.

Key Improvements

1. Optimized for Coding: The model specializes in generating high-quality, structured code with minimal redundant tokens, ensuring efficient execution.
2. Enhanced Memory Utilization: Implements streamlined memory optimization to reduce computational overhead and improve performance.
3. Superior Reasoning Capabilities: Excels in solving complex mathematical and algorithmic problems with logical and structured explanations.
4. Long-Context Support: Supports up to 128K tokens for input context and can generate up to 8K tokens in a single output, making it ideal for detailed coding responses.
5. Reduced Unwanted Textual Tokens: Ensures a more focused output for coding tasks by minimizing excessive textual responses.

Quickstart with transformers

Here is a code snippet with apply_chat_template to show you how to load the tokenizer and model and generate content:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "prithivMLmods/Sombrero-Opus-14B-Sm5"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Write a Python function to find the Fibonacci sequence."
messages = [
    {"role": "system", "content": "You are an advanced coding assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Intended Use

1. Code Generation & Optimization:
   Designed for developers, assisting in writing, refactoring, and optimizing code across multiple programming languages.

2. Algorithm & Mathematical Problem Solving:
   Provides precise explanations and solutions for computational and mathematical problems.

3. Technical Explanations & Documentation:
   Generates clear and structured explanations for coding concepts, libraries, and APIs.

4. Debugging Assistance:
   Helps analyze code snippets, detect errors, and suggest corrections.

5. Educational Use:
   Assists students and learners by breaking down complex programming topics into easily understandable sections.

6. Structured Data Processing:
   Capable of analyzing and generating structured outputs, such as JSON, XML, and tables, making it ideal for data science applications.

Limitations

1. Hardware Requirements:
   Requires high-memory GPUs or TPUs due to its large parameter size and long-context support.

2. Potential Bias in Responses:
   While designed to be neutral, outputs may still reflect biases present in training data.

3. Inconsistent Outputs in Creative Tasks:
   May produce variable results in storytelling and non-technical topics.

4. Limited Real-World Awareness:
   Does not have access to real-time events beyond its training cutoff.

5. Error Propagation in Extended Outputs:
   Minor errors in early responses may affect overall coherence in long-form code outputs.

6. Prompt Sensitivity:
   The effectiveness of responses may depend on how well the input prompt is structured.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here! Summarized results can be found here!

Metric	Value (%)
Average	41.12
IFEval (0-Shot)	68.52
BBH (3-Shot)	50.60
MATH Lvl 5 (4-Shot)	40.94
GPQA (0-shot)	18.23
MuSR (0-shot)	19.51
MMLU-PRO (5-shot)	48.89