225 lines
8.4 KiB
Markdown
225 lines
8.4 KiB
Markdown
|
---
|
|||
|
|
license: llama3.1
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
base_model:
|
|||
|
|
- microsoft/phi-4
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
library_name: transformers
|
|||
|
|
tags:
|
|||
|
|
- text-generation-inference
|
|||
|
|
- llama3.1
|
|||
|
|
- phi4
|
|||
|
|
- LlamaForCausalLM
|
|||
|
|
- Corpus
|
|||
|
|
- trl
|
|||
|
|
model-index:
|
|||
|
|
- name: Megatron-Opus-14B-2.0
|
|||
|
|
results:
|
|||
|
|
- task:
|
|||
|
|
type: text-generation
|
|||
|
|
name: Text Generation
|
|||
|
|
dataset:
|
|||
|
|
name: IFEval (0-Shot)
|
|||
|
|
type: wis-k/instruction-following-eval
|
|||
|
|
split: train
|
|||
|
|
args:
|
|||
|
|
num_few_shot: 0
|
|||
|
|
metrics:
|
|||
|
|
- type: inst_level_strict_acc and prompt_level_strict_acc
|
|||
|
|
value: 66.94
|
|||
|
|
name: averaged accuracy
|
|||
|
|
source:
|
|||
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FMegatron-Opus-14B-2.0
|
|||
|
|
name: Open LLM Leaderboard
|
|||
|
|
- task:
|
|||
|
|
type: text-generation
|
|||
|
|
name: Text Generation
|
|||
|
|
dataset:
|
|||
|
|
name: BBH (3-Shot)
|
|||
|
|
type: SaylorTwift/bbh
|
|||
|
|
split: test
|
|||
|
|
args:
|
|||
|
|
num_few_shot: 3
|
|||
|
|
metrics:
|
|||
|
|
- type: acc_norm
|
|||
|
|
value: 54.7
|
|||
|
|
name: normalized accuracy
|
|||
|
|
source:
|
|||
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FMegatron-Opus-14B-2.0
|
|||
|
|
name: Open LLM Leaderboard
|
|||
|
|
- task:
|
|||
|
|
type: text-generation
|
|||
|
|
name: Text Generation
|
|||
|
|
dataset:
|
|||
|
|
name: MATH Lvl 5 (4-Shot)
|
|||
|
|
type: lighteval/MATH-Hard
|
|||
|
|
split: test
|
|||
|
|
args:
|
|||
|
|
num_few_shot: 4
|
|||
|
|
metrics:
|
|||
|
|
- type: exact_match
|
|||
|
|
value: 27.79
|
|||
|
|
name: exact match
|
|||
|
|
source:
|
|||
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FMegatron-Opus-14B-2.0
|
|||
|
|
name: Open LLM Leaderboard
|
|||
|
|
- task:
|
|||
|
|
type: text-generation
|
|||
|
|
name: Text Generation
|
|||
|
|
dataset:
|
|||
|
|
name: GPQA (0-shot)
|
|||
|
|
type: Idavidrein/gpqa
|
|||
|
|
split: train
|
|||
|
|
args:
|
|||
|
|
num_few_shot: 0
|
|||
|
|
metrics:
|
|||
|
|
- type: acc_norm
|
|||
|
|
value: 14.54
|
|||
|
|
name: acc_norm
|
|||
|
|
source:
|
|||
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FMegatron-Opus-14B-2.0
|
|||
|
|
name: Open LLM Leaderboard
|
|||
|
|
- task:
|
|||
|
|
type: text-generation
|
|||
|
|
name: Text Generation
|
|||
|
|
dataset:
|
|||
|
|
name: MuSR (0-shot)
|
|||
|
|
type: TAUR-Lab/MuSR
|
|||
|
|
args:
|
|||
|
|
num_few_shot: 0
|
|||
|
|
metrics:
|
|||
|
|
- type: acc_norm
|
|||
|
|
value: 10.52
|
|||
|
|
name: acc_norm
|
|||
|
|
source:
|
|||
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FMegatron-Opus-14B-2.0
|
|||
|
|
name: Open LLM Leaderboard
|
|||
|
|
- task:
|
|||
|
|
type: text-generation
|
|||
|
|
name: Text Generation
|
|||
|
|
dataset:
|
|||
|
|
name: MMLU-PRO (5-shot)
|
|||
|
|
type: TIGER-Lab/MMLU-Pro
|
|||
|
|
config: main
|
|||
|
|
split: test
|
|||
|
|
args:
|
|||
|
|
num_few_shot: 5
|
|||
|
|
metrics:
|
|||
|
|
- type: acc
|
|||
|
|
value: 46.34
|
|||
|
|
name: accuracy
|
|||
|
|
source:
|
|||
|
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FMegatron-Opus-14B-2.0
|
|||
|
|
name: Open LLM Leaderboard
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
# **Megatron-Opus-14B-2.0 [ Exp ]**
|
|||
|
|
|
|||
|
|
[Megatron-Opus-14B-2.0 ] Exp finetuned from Microsoft's Phi-4 is a state-of-the-art open model developed with a focus on responsible problem solving and advanced reasoning capabilities. Built upon a diverse blend of synthetic datasets, carefully filtered public domain websites, and high-quality academic books and Q&A datasets, Megatron-Opus-14B-2.0 ensures that small, capable models are trained with datasets of exceptional depth and precision.
|
|||
|
|
|
|||
|
|
Megatron-Opus-14B-2.0 adopts a robust safety post-training approach using open-source and in-house synthetic datasets. This involves a combination of SFT (Supervised Fine-Tuning) and iterative DPO (Direct Preference Optimization) techniques, ensuring helpful and harmless outputs across various safety categories.
|
|||
|
|
|
|||
|
|
|
|||
|
|
# **Dataset Info**
|
|||
|
|
|
|||
|
|
Megatron-Opus-14B-2.0 is fine-tuned on a carefully curated synthetic dataset generated using an advanced pipeline optimized for Chain of Thought (CoT) reasoning and Responsible Problem Breakdown (RPB) methodologies. This ensures that the model excels at:
|
|||
|
|
|
|||
|
|
- **Logical reasoning**
|
|||
|
|
- **Step-by-step problem-solving**
|
|||
|
|
- **Breaking down complex tasks into manageable parts**
|
|||
|
|
|
|||
|
|
The dataset also emphasizes responsible decision-making and fairness in generating solutions.
|
|||
|
|
|
|||
|
|
# **Run with Transformers**
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
# pip install accelerate
|
|||
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained("prithivMLmods/Megatron-Opus-14B-2.0")
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
"prithivMLmods/Megatron-Opus-14B-2.0",
|
|||
|
|
device_map="auto",
|
|||
|
|
torch_dtype=torch.bfloat16,
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
input_text = "Explain the concept of black holes."
|
|||
|
|
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
|
|||
|
|
|
|||
|
|
outputs = model.generate(**input_ids, max_new_tokens=64)
|
|||
|
|
print(tokenizer.decode(outputs[0]))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
For chat-style interactions, use `tokenizer.apply_chat_template`:
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
messages = [
|
|||
|
|
{"role": "user", "content": "Explain the concept of black holes."},
|
|||
|
|
]
|
|||
|
|
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda")
|
|||
|
|
|
|||
|
|
outputs = model.generate(**input_ids, max_new_tokens=256)
|
|||
|
|
print(tokenizer.decode(outputs[0]))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
# **Intended Use**
|
|||
|
|
|
|||
|
|
Megatron-Opus-14B-2.0 is tailored for a wide range of applications, especially those involving **advanced reasoning**, **multilingual capabilities**, and **responsible problem-solving**. Its primary use cases include:
|
|||
|
|
|
|||
|
|
1. **Responsible Problem Solving**
|
|||
|
|
- Breaking down complex problems into logical, actionable steps.
|
|||
|
|
- Offering ethical, well-rounded solutions in academic and professional contexts.
|
|||
|
|
|
|||
|
|
2. **Advanced Reasoning Tasks**
|
|||
|
|
- Excelling in mathematics, logic, and scientific reasoning.
|
|||
|
|
- Providing detailed explanations and systematic answers.
|
|||
|
|
|
|||
|
|
3. **Content Generation**
|
|||
|
|
- Assisting in generating high-quality content for various domains, including creative writing and technical documentation.
|
|||
|
|
- Supporting marketers, writers, and educators with detailed and well-structured outputs.
|
|||
|
|
|
|||
|
|
4. **Educational Support**
|
|||
|
|
- Acting as a virtual tutor for students by generating practice questions, answers, and detailed explanations.
|
|||
|
|
- Helping educators design learning material that promotes critical thinking and step-by-step problem-solving.
|
|||
|
|
|
|||
|
|
5. **Customer Support & Dialogue Systems**
|
|||
|
|
- Enabling chatbots and virtual assistants to provide accurate, helpful, and responsible responses.
|
|||
|
|
- Enhancing customer service with reasoning-driven automation.
|
|||
|
|
|
|||
|
|
# **Limitations**
|
|||
|
|
|
|||
|
|
Despite its strengths, Megatron-Opus-14B-2.0 has some limitations that users should be aware of:
|
|||
|
|
|
|||
|
|
1. **Bias and Fairness**
|
|||
|
|
- While great effort has been made to minimize biases, users should critically assess the model’s output in sensitive scenarios to avoid unintended bias.
|
|||
|
|
|
|||
|
|
2. **Contextual Interpretation**
|
|||
|
|
- The model may occasionally misinterpret highly nuanced prompts or ambiguous contexts, leading to suboptimal responses.
|
|||
|
|
|
|||
|
|
3. **Knowledge Cutoff**
|
|||
|
|
- Megatron-Opus-14B-2.0’s knowledge is static and based on the data available at the time of training. It does not include real-time updates or information on recent developments.
|
|||
|
|
|
|||
|
|
4. **Safety and Harmlessness**
|
|||
|
|
- Despite post-training safety alignment, inappropriate or harmful outputs may still occur. Continuous monitoring and human oversight are advised when using the model in critical contexts.
|
|||
|
|
|
|||
|
|
5. **Computational Requirements**
|
|||
|
|
- Deploying Megatron-Opus-14B-2.0 efficiently may require substantial computational resources, particularly for large-scale deployments or real-time applications.
|
|||
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
|
|||
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/prithivMLmods__Megatron-Opus-14B-2.0-details)!
|
|||
|
|
Summarized results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/contents/viewer/default/train?q=prithivMLmods%2FMegatron-Opus-14B-2.0&sort[column]=Average%20%E2%AC%86%EF%B8%8F&sort[direction]=desc)!
|
|||
|
|
|
|||
|
|
| Metric |Value (%)|
|
|||
|
|
|-------------------|--------:|
|
|||
|
|
|**Average** | 36.80|
|
|||
|
|
|IFEval (0-Shot) | 66.94|
|
|||
|
|
|BBH (3-Shot) | 54.70|
|
|||
|
|
|MATH Lvl 5 (4-Shot)| 27.79|
|
|||
|
|
|GPQA (0-shot) | 14.54|
|
|||
|
|
|MuSR (0-shot) | 10.52|
|
|||
|
|
|MMLU-PRO (5-shot) | 46.34|
|
|||
|
|
|