初始化项目，由ModelHub XC社区提供模型
Model: nvidia/Nemotron-H-8B-Base-8K Source: Original Platform
This commit is contained in:
ModelHub XC
39
.gitattributes
vendored
Normal file
39
.gitattributes
vendored
Normal file
@@ -0,0 +1,39 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
nemo/weights/.metadata filter=lfs diff=lfs merge=lfs -text
|
||||
nemo/weights/__0_0.distcp filter=lfs diff=lfs merge=lfs -text
|
||||
nemo/weights/__0_1.distcp filter=lfs diff=lfs merge=lfs -text
|
||||
198
README.md
Normal file
198
README.md
Normal file
@@ -0,0 +1,198 @@
|
||||
---
|
||||
library_name: transformers
|
||||
license: other
|
||||
license_name: nvidia-internal-scientific-research-and-development-model-license
|
||||
license_link: >-
|
||||
https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-internal-scientific-research-and-development-model-license/
|
||||
pipeline_tag: text-generation
|
||||
language:
|
||||
- en
|
||||
- de
|
||||
- es
|
||||
- fr
|
||||
- it
|
||||
- ko
|
||||
- pt
|
||||
- ru
|
||||
- jp
|
||||
- zh
|
||||
tags:
|
||||
- nvidia
|
||||
- pytorch
|
||||
- nemotron-h
|
||||
---
|
||||
|
||||
# Nemotron-H-8B-Base-8K
|
||||
|
||||
## Model Overview
|
||||
|
||||
NVIDIA Nemotron-H-8B-Base-8K is a large language model (LLM) developed by NVIDIA that is designed as a completion model for a given piece of text. It uses a hybrid model architecture that consists primarily of Mamba-2 and MLP layers combined with just four Attention layers. The model features a context length of 8K. The supported languages include: English, German, Spanish, French, Italian, Korean, Portuguese, Russian, Japanese, and Chinese. For more detailed information on the model architecture, training, and evaluation, please see the [project page](https://research.nvidia.com/labs/adlr/nemotronh/) and the [technical report](https://arxiv.org/abs/2504.03624).
|
||||
|
||||
For best performance on a given task, users are encouraged to customize the model using the [NeMo Framework](https://docs.nvidia.com/nemo-framework/index.html) suite of customization tools including Parameter-Efficient Fine-Tuning (P-tuning, Adapters, LoRA, and more), and Model Alignment (SFT, SteerLM, RLHF, and more) using [NeMo-Aligner](https://github.com/NVIDIA/NeMo-Aligner).
|
||||
|
||||
This model is for research and development only.
|
||||
|
||||
This model is part of the Nemotron-H Collection. You can find the models in this family here:
|
||||
- [Nemotron-H-56B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-56B-Base-8K)
|
||||
- [Nemotron-H-47B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-47B-Base-8K)
|
||||
- [Nemotron-H-8B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-8B-Base-8K)
|
||||
|
||||
## License/Terms of Use
|
||||
|
||||
GOVERNING TERMS: Use of this model is governed by the [NVIDIA Internal Scientific Research and Development Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-internal-scientific-research-and-development-model-license/).
|
||||
|
||||
**Model Developer:** NVIDIA
|
||||
|
||||
**Model Dates:**
|
||||
|
||||
October 2024 - March 2025
|
||||
|
||||
**Data Freshness:**
|
||||
|
||||
September 2024
|
||||
|
||||
The pretraining data has a cutoff date of September 2024.
|
||||
|
||||
## Use Case:
|
||||
|
||||
This model is intended for developers and researchers building LLMs.
|
||||
|
||||
## Release Date:
|
||||
|
||||
4/14/2025
|
||||
|
||||
## References
|
||||
|
||||
- [\[2504.03624\] Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models](https://arxiv.org/abs/2504.03624)
|
||||
|
||||
## Model Architecture
|
||||
- Architecture Type: Hybrid Mamba-Transformer
|
||||
- Network Architecture: Nemotron-H
|
||||
|
||||
This model has 8B model parameters.
|
||||
|
||||
## Input
|
||||
- Input Type(s): Text
|
||||
- Input Format(s): String
|
||||
- Input Parameters: One-Dimensional (1D): Sequences
|
||||
- Other Properties Related to Input: Context length up to 8K. Supported languages include German, Spanish, French, Italian, Korean, Portuguese, Russian, Japanese, Chinese and English.
|
||||
|
||||
## Output
|
||||
- Output Type(s): Text
|
||||
- Output Format: String
|
||||
- Output Parameters: One-Dimensional (1D): Sequences
|
||||
|
||||
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
|
||||
|
||||
## Software Integration
|
||||
- Runtime Engine(s): NeMo 24.12
|
||||
- Supported Hardware Microarchitecture Compatibility: NVIDIA H100-80GB, NVIDIA A100
|
||||
- Operating System(s): Linux
|
||||
|
||||
## Model Version
|
||||
- v1.0
|
||||
|
||||
## Prompt Format
|
||||
|
||||
As this is a base model, no explicit prompt format is recommended or required.
|
||||
|
||||
### Example
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
# Load the tokenizer and model
|
||||
tokenizer = AutoTokenizer.from_pretrained("nvidia/Nemotron-H-8B-Base-8K", trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained("nvidia/Nemotron-H-8B-Base-8K", torch_dtype=torch.bfloat16, trust_remote_code=True).cuda()
|
||||
|
||||
prompt = "When was NVIDIA founded?"
|
||||
|
||||
outputs = model.generate(**tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device))
|
||||
print(tokenizer.decode(outputs[0]))
|
||||
```
|
||||
|
||||
## Training, Testing, and Evaluation Datasets
|
||||
|
||||
### Training & Testing Datasets:
|
||||
|
||||
The training corpus for Nemotron-H-8B-Base-8K consists of English and multilingual text (German, Spanish, French, Italian, Korean, Portuguese, Russian, Japanese, Chinese and English), as well as code. Our sources cover a variety of document types such as: webpages, dialogue, articles, and other written materials. This model was also improved using synthetic data from Qwen (Built with Qwen). The corpus spans domains including legal, math, science, finance, and more. We also include a small portion of question-answering, and alignment style data to improve model accuracy.
|
||||
|
||||
**Data Collection for Training & Testing Datasets:**
|
||||
Hybrid: Automated, Human, Synthetic
|
||||
|
||||
**Data Labeling for Training & Testing Datasets:**
|
||||
Hybrid: Automated, Human, Synthetic
|
||||
|
||||
### Evaluation Datasets
|
||||
|
||||
We used the datasets listed in the next section to evaluate Nemotron-H-8B-Base-8K.
|
||||
|
||||
**Data Collection for Evaluation Datasets:**
|
||||
Hybrid: Human, Synthetic
|
||||
|
||||
**Data Labeling for Evaluation Datasets:**
|
||||
Hybrid: Human, Synthetic, Automatic
|
||||
|
||||
#### Commonsense Understanding Evaluations:
|
||||
|
||||
| ARC Challenge 25-shot | Hellaswag 10-shot | Winogrande 5-shot | CommonsenseQA 7-shot |
|
||||
|-------------|--------------|-----------------|------------------|
|
||||
| 88.74 | 83.23| 80.51 | 78.71 |
|
||||
|
||||
- ARC (Ai2 reasoning challenge)-Challenge - The challenge set of questions from a benchmark that contains grade-school level, multiple-choice science questions to assess question answering ability of language models. [Dataset](https://huggingface.co/datasets/allenai/ai2_arc)
|
||||
- Hellaswag - Tests the ability of a language model to correctly finish the provided context from a choice of possible options. [Dataset](https://huggingface.co/datasets/Rowan/hellaswag )
|
||||
- Winogrande - Tests the ability to choose the right option for a given sentence which requires commonsense reasoning. [Dataset](https://huggingface.co/datasets/allenai/winogrande )
|
||||
- CommonsenseQA - A multiple-choice question answering dataset that requires different type of commonsense knowledge to predict the correct answers. [Dataset](https://huggingface.co/datasets/tau/commonsense_qa )
|
||||
|
||||
#### Coding Evaluations:
|
||||
|
||||
| MBPP (sanitized) 3-shot | MBPP+ 0-shot | HumanEval 0-shot | HumanEval+ 0-shot |
|
||||
|-------------|--------------|-----------------|------------------|
|
||||
| 65.37 | 59.52| 58.54 | 55.49 |
|
||||
|
||||
- MBPP (Mostly Basic Python Programming Problems) - Evaluates ability to generate solutions for Python programming tasks. [Dataset](https://github.com/google-research/google-research/tree/master/mbpp)
|
||||
- MBPP+ - Extended version of MBPP with additional validation. [Dataset](https://huggingface.co/datasets/evalplus/mbppplus)
|
||||
- HumanEval - Tests code generation and completion abilities in Python. [Dataset](https://github.com/openai/human-eval)
|
||||
|
||||
#### Math Evaluations:
|
||||
|
||||
|
||||
| GSM8K 8-shot CoT | MATH 4-shot CoT | MATH-Lvl 5 4-shot CoT | MATH-500 4-shot CoT |
|
||||
|--------------|------------|------------|------------|
|
||||
| 87.11 | 46.52 | 22.93 | 44.43 |
|
||||
|
||||
- GSM8K (Grade School Math 8K) - Evaluates grade school level mathematical word problem solving. [Dataset](https://github.com/openai/grade-school-math)
|
||||
- MATH - Tests mathematical ability across multiple difficulty levels and various subjects including: Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus. [Dataset](https://github.com/hendrycks/math)
|
||||
- MATH Lvl 5 - Only the most difficult questions from the MATH dataset. [Dataset](https://github.com/hendrycks/math)
|
||||
- MATH-500 - Tests advanced mathematical problem solving across algebra, geometry, and calculus. [Dataset](https://huggingface.co/datasets/HuggingFaceH4/MATH-500)
|
||||
|
||||
|
||||
#### General Evaluations:
|
||||
|
||||
| MMLU-Pro 5-shot CoT | MMLU 5-shot|
|
||||
|-------------------|------------------|
|
||||
|44.01 |72.77 |
|
||||
|
||||
- MMLU Pro - Evaluates language understanding models across a broad range of challenging, reasoning-focused questions across 14 diverse domains.
|
||||
[Dataset](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro)
|
||||
- MMLU - Tests knowledge across 57 subjects including science, humanities, math and more. [Dataset](https://github.com/hendrycks/test)
|
||||
|
||||
## Potential Known Risks for Usage
|
||||
|
||||
The model was trained on data that contains toxic language and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive.
|
||||
|
||||
The model demonstrates weakness to indirect prompt injection via some encodings, including Base16, Hex/ASCII, and Braille, though is more resilient than other similar models to injections using the more common Base64 vector.
|
||||
|
||||
## Inference
|
||||
- Engine: NeMo
|
||||
- Test Hardware NVIDIA H100-80GB
|
||||
|
||||
## Ethical Considerations
|
||||
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
|
||||
|
||||
For more detailed information on ethical considerations for this model, please see the Responsible Use Guide available at http://nvidia.com/nemotron-responsible-use.
|
||||
|
||||
Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
|
||||
|
||||
|
||||
58
config.json
Normal file
58
config.json
Normal file
@@ -0,0 +1,58 @@
|
||||
{
|
||||
"architectures": [
|
||||
"NemotronHForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"attention_head_dim": 128,
|
||||
"auto_map": {
|
||||
"AutoConfig": "configuration_nemotron_h.NemotronHConfig",
|
||||
"AutoModelForCausalLM": "modeling_nemotron_h.NemotronHForCausalLM"
|
||||
},
|
||||
"bos_token_id": 1,
|
||||
"chunk_size": 128,
|
||||
"conv_kernel": 4,
|
||||
"eos_token_id": 2,
|
||||
"expand": 2,
|
||||
"hidden_dropout": 0.0,
|
||||
"hidden_size": 4096,
|
||||
"hybrid_override_pattern": "M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-",
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 21504,
|
||||
"layer_norm_epsilon": 1e-05,
|
||||
"mamba_head_dim": 64,
|
||||
"mamba_hidden_act": "silu",
|
||||
"mamba_num_heads": 128,
|
||||
"mamba_proj_bias": false,
|
||||
"max_position_embeddings": 8192,
|
||||
"mlp_bias": false,
|
||||
"mlp_hidden_act": "relu2",
|
||||
"model_type": "nemotron_h",
|
||||
"n_groups": 8,
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 52,
|
||||
"num_key_value_heads": 8,
|
||||
"num_logits_to_keep": 1,
|
||||
"pad_token_id": 0,
|
||||
"rescale_prenorm_residual": true,
|
||||
"residual_in_fp32": false,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"sliding_window": null,
|
||||
"ssm_state_size": 128,
|
||||
"tie_word_embeddings": false,
|
||||
"time_step_floor": 0.0001,
|
||||
"time_step_limit": [
|
||||
0.0,
|
||||
Infinity
|
||||
],
|
||||
"time_step_max": 0.1,
|
||||
"time_step_min": 0.001,
|
||||
"time_step_rank": 256,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.48.0.dev0",
|
||||
"use_bias": false,
|
||||
"use_cache": true,
|
||||
"use_conv_bias": true,
|
||||
"use_mamba_kernels": true,
|
||||
"vocab_size": 131072
|
||||
}
|
||||
243
configuration_nemotron_h.py
Normal file
243
configuration_nemotron_h.py
Normal file
@@ -0,0 +1,243 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2024 AI21 Labs Ltd. and the HuggingFace Inc. team. All rights reserved.
|
||||
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""NemotronH model configuration"""
|
||||
|
||||
import re
|
||||
|
||||
from transformers.configuration_utils import PretrainedConfig
|
||||
from transformers.utils import logging
|
||||
|
||||
|
||||
logger = logging.get_logger(__name__)
|
||||
|
||||
|
||||
class NemotronHConfig(PretrainedConfig):
|
||||
r"""
|
||||
This is the configuration class to store the configuration of a [`NemotronHModel`]. It is used to instantiate a
|
||||
NemotronH model according to the specified arguments, defining the model architecture. Instantiating a configuration
|
||||
with the defaults will yield a similar configuration to that of the NemotronH-v0.1 model.
|
||||
|
||||
[todo](todo)
|
||||
|
||||
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
|
||||
documentation from [`PretrainedConfig`] for more information.
|
||||
|
||||
|
||||
Args:
|
||||
vocab_size (`int`, *optional*, defaults to 131072):
|
||||
Vocabulary size of the NemotronH model. Defines the number of different tokens that can be represented by the
|
||||
`inputs_ids` passed when calling [`NemotronHModel`]
|
||||
tie_word_embeddings (`bool`, *optional*, defaults to `False`):
|
||||
Whether the model's input and output word embeddings should be tied. Note that this is only relevant if the
|
||||
model has a output word embedding layer.
|
||||
hidden_size (`int`, *optional*, defaults to 4096):
|
||||
Dimension of the hidden representations.
|
||||
intermediate_size (`int`, *optional*, defaults to 21504):
|
||||
Dimension of the MLP representations.
|
||||
num_hidden_layers (`int`, *optional*, defaults to 52):
|
||||
Number of hidden layers in the Transformer encoder.
|
||||
hybrid_override_pattern (`str`, *optional*, defaults to `"M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-"`):
|
||||
The pattern of the hybrid model. The pattern is a string of characters where each character represents M: Mamba2, *: Attention, -: MLP
|
||||
num_attention_heads (`int`, *optional*, defaults to 32):
|
||||
Number of attention heads for each attention layer in the Transformer encoder.
|
||||
attention_head_dim (`int`, *optional*, defaults to 128):
|
||||
Dimension of each attention head.
|
||||
num_key_value_heads (`int`, *optional*, defaults to 8):
|
||||
This is the number of key_value heads that should be used to implement Grouped Query Attention. If
|
||||
`num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
|
||||
`num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used.
|
||||
mlp_hidden_act (`str`, *optional*, defaults to "relu2"):
|
||||
The non-linear activation function in the MLP layers.
|
||||
attention_bias (`bool`, *optional*, defaults to `False`):
|
||||
Whether to use bias in attention layers.
|
||||
mlp_bias (`bool`, *optional*, defaults to `False`):
|
||||
Whether to use bias in MLP layers.
|
||||
use_bias (`bool`, *optional*, defaults to `False`):
|
||||
Whether to use bias in the model.
|
||||
initializer_range (`float`, *optional*, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
layer_norm_epsilon (`float`, *optional*, defaults to 1e-5):
|
||||
The epsilon used by the layer normalization layers.
|
||||
residual_in_fp32 (`bool`, *optional*, defaults to `False`):
|
||||
Whether or not residuals should be in `float32`. If set to `False` residuals will keep the same `dtype` as the rest of the model.
|
||||
use_cache (`bool`, *optional*, defaults to `True`):
|
||||
Whether or not the model should return the last key/values attentions (not used by all models). Only
|
||||
relevant if `config.is_decoder=True`.
|
||||
num_logits_to_keep (`int` or `None`, *optional*, defaults to 1):
|
||||
Number of prompt logits to calculate during generation. If `None`, all logits will be calculated. If an
|
||||
integer value, only last `num_logits_to_keep` logits will be calculated.
|
||||
pad_token_id (`int`, *optional*, defaults to 0):
|
||||
The id of the padding token.
|
||||
bos_token_id (`int`, *optional*, defaults to 1):
|
||||
The id of the "beginning-of-sequence" token.
|
||||
eos_token_id (`int`, *optional*, defaults to 2):
|
||||
The id of the "end-of-sequence" token.
|
||||
sliding_window (`int`, *optional*, defaults to None):
|
||||
Sliding window attention window size.
|
||||
max_position_embeddings (`int`, *optional*, defaults to 4096):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
attention_dropout (`float`, *optional*, defaults to 0.0):
|
||||
The dropout ratio for the attention probabilities.
|
||||
hidden_dropout (`float`, *optional*, defaults to 0.0):
|
||||
The dropout ratio for the hidden states.
|
||||
use_mamba_kernels (`bool`, *optional*, defaults to `True`):
|
||||
Flag indicating whether or not to use the fast mamba kernels. These are available only if `mamba-ssm` and
|
||||
`causal-conv1d` are installed, and the mamba modules are running on a CUDA device.
|
||||
ssm_state_size (`int`, *optional*, defaults to 128):
|
||||
The dimension of the mamba state space latents.
|
||||
mamba_num_heads (`int`, *optional*, defaults to 128):
|
||||
Number of heads in Mamba layers.
|
||||
mamba_n_groups (`int`, *optional*, defaults to 8):
|
||||
Number of groups in Mamba layers.
|
||||
mamba_head_dim (`int`, *optional*, defaults to 64):
|
||||
Dimension of each Mamba head.
|
||||
mamba_d_conv (`int`, *optional*, defaults to 4):
|
||||
The size of the mamba convolution kernel.
|
||||
mamba_expand (`int`, *optional*, defaults to 2):
|
||||
Expanding factor used to determine the mamba intermediate size.
|
||||
mamba_hidden_act (`str`, *optional*, defaults to "silu"):
|
||||
The non-linear activation function in the Mamba layers.
|
||||
mamba_dt_min (`float`, *optional*, defaults to 0.001):
|
||||
Minimum value for the time step in Mamba.
|
||||
mamba_dt_max (`float`, *optional*, defaults to 0.1):
|
||||
Maximum value for the time step in Mamba.
|
||||
mamba_dt_limit (`tuple`, *optional*, defaults to (0.0, float("inf"))):
|
||||
Limits for the time step in Mamba.
|
||||
mamba_dt_init_floor (`float`, *optional*, defaults to 1e-4):
|
||||
Floor value for time step initialization in Mamba.
|
||||
mamba_conv_bias (`bool`, *optional*, defaults to `True`):
|
||||
Whether to use bias in the convolution layer of the mamba mixer block.
|
||||
mamba_proj_bias (`bool`, *optional*, defaults to `False`):
|
||||
Whether to use bias in the input and output projections of the mamba mixer block.
|
||||
mamba_chunk_size (`int`, *optional*, defaults to 256):
|
||||
Size of chunks for Mamba processing.
|
||||
rescale_prenorm_residual (`bool`, *optional*, defaults to `True`):
|
||||
Whether to rescale the pre-normalization residual connections.
|
||||
"""
|
||||
|
||||
model_type = "nemotron_h"
|
||||
keys_to_ignore_at_inference = ["past_key_values"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vocab_size=131072,
|
||||
tie_word_embeddings=False,
|
||||
hidden_size=4096,
|
||||
intermediate_size=21504,
|
||||
num_hidden_layers=52,
|
||||
hybrid_override_pattern="M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-",
|
||||
num_attention_heads=32,
|
||||
attention_head_dim=128,
|
||||
num_key_value_heads=8, # nemo: num_query_groups
|
||||
mlp_hidden_act="relu2",
|
||||
attention_bias=False,
|
||||
mlp_bias=False,
|
||||
use_bias=False,
|
||||
initializer_range=0.02, # nemo: init_method_std
|
||||
layer_norm_epsilon=1e-5, # nemo: layernorm_epsilon
|
||||
residual_in_fp32=False, # Megatron Core default value
|
||||
use_cache=True,
|
||||
num_logits_to_keep=1,
|
||||
pad_token_id=0,
|
||||
bos_token_id=1,
|
||||
eos_token_id=2,
|
||||
sliding_window=None,
|
||||
max_position_embeddings=4096,
|
||||
attention_dropout=0.0,
|
||||
hidden_dropout=0.0, # * ADDED
|
||||
use_mamba_kernels=True,
|
||||
ssm_state_size=128, # mamba_state_size
|
||||
mamba_num_heads=128,
|
||||
mamba_n_groups=8, # nemo: mamba_ssm_ngroups = num_heads
|
||||
mamba_head_dim=64,
|
||||
mamba_d_conv=4,
|
||||
mamba_expand=2,
|
||||
mamba_hidden_act="silu",
|
||||
mamba_dt_min=0.001,
|
||||
mamba_dt_max=0.1,
|
||||
mamba_dt_limit=(0.0, float("inf")),
|
||||
mamba_dt_init_floor=1e-4,
|
||||
mamba_conv_bias=True,
|
||||
mamba_proj_bias=False,
|
||||
mamba_chunk_size=256,
|
||||
rescale_prenorm_residual=True,
|
||||
**kwargs,
|
||||
):
|
||||
self.vocab_size = vocab_size
|
||||
self.tie_word_embeddings = tie_word_embeddings
|
||||
self.hidden_size = hidden_size
|
||||
self.intermediate_size = intermediate_size
|
||||
self.num_hidden_layers = num_hidden_layers
|
||||
self.hybrid_override_pattern = hybrid_override_pattern
|
||||
self.num_attention_heads = num_attention_heads
|
||||
self.attention_head_dim = attention_head_dim
|
||||
self.sliding_window = sliding_window
|
||||
self.max_position_embeddings = max_position_embeddings
|
||||
self.attention_dropout = attention_dropout
|
||||
self.hidden_dropout = hidden_dropout
|
||||
|
||||
# Validate hybrid_override_pattern
|
||||
# M: Mamba2, *: Attention, -: MLP
|
||||
assert len(self.hybrid_override_pattern) == self.num_hidden_layers, "hybrid_override_pattern must have the same length as num_hidden_layers"
|
||||
assert re.match(r"^[*-M]+$", self.hybrid_override_pattern), "hybrid_override_pattern must only contain characters 'M', '*', or '-'"
|
||||
|
||||
# for backward compatibility
|
||||
if num_key_value_heads is None:
|
||||
num_key_value_heads = num_attention_heads
|
||||
|
||||
self.num_key_value_heads = num_key_value_heads
|
||||
self.mlp_hidden_act = mlp_hidden_act
|
||||
self.attention_bias = attention_bias
|
||||
self.mlp_bias = mlp_bias
|
||||
self.use_bias = use_bias
|
||||
self.initializer_range = initializer_range
|
||||
self.layer_norm_epsilon = layer_norm_epsilon
|
||||
self.residual_in_fp32 = residual_in_fp32
|
||||
|
||||
self.use_cache = use_cache
|
||||
self.num_logits_to_keep = num_logits_to_keep
|
||||
|
||||
self.use_mamba_kernels = use_mamba_kernels
|
||||
self.n_groups = mamba_n_groups
|
||||
self.mamba_head_dim = mamba_head_dim
|
||||
self.ssm_state_size = ssm_state_size
|
||||
self.mamba_num_heads = mamba_num_heads
|
||||
self.conv_kernel = mamba_d_conv
|
||||
self.expand = mamba_expand
|
||||
self.mamba_hidden_act = mamba_hidden_act
|
||||
self.time_step_min = mamba_dt_min
|
||||
self.time_step_max = mamba_dt_max
|
||||
self.time_step_limit = mamba_dt_limit
|
||||
self.time_step_floor = mamba_dt_init_floor
|
||||
self.use_conv_bias = mamba_conv_bias
|
||||
self.mamba_proj_bias = mamba_proj_bias
|
||||
self.chunk_size = mamba_chunk_size
|
||||
self.rescale_prenorm_residual = rescale_prenorm_residual
|
||||
|
||||
super().__init__(
|
||||
pad_token_id=pad_token_id,
|
||||
bos_token_id=bos_token_id,
|
||||
eos_token_id=eos_token_id,
|
||||
tie_word_embeddings=tie_word_embeddings,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
@property
|
||||
def layers_block_type(self):
|
||||
return [
|
||||
"mamba" if self.hybrid_override_pattern[i] == "M" else
|
||||
"attention" if self.hybrid_override_pattern[i] == "*" else "mlp"
|
||||
for i in range(self.num_hidden_layers)]
|
||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": [2, 11],
|
||||
"pad_token_id": 0,
|
||||
"transformers_version": "4.48.0.dev0"
|
||||
}
|
||||
3
model-00001-of-00004.safetensors
Normal file
3
model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2abae50afd63ec78dce726165ea61abaf618c31eb909bbfd64f784e2e2c8a4f0
|
||||
size 4982754168
|
||||
3
model-00002-of-00004.safetensors
Normal file
3
model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6b13d716c48101156f3d081639a19b06f486471909d6bc2935e4f08a6fdbb4f2
|
||||
size 4916952064
|
||||
3
model-00003-of-00004.safetensors
Normal file
3
model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e6b34aae9b9d531627826e64d47b9135bb216eec6f1a9a08170b0e098456e7f9
|
||||
size 4875961752
|
||||
3
model-00004-of-00004.safetensors
Normal file
3
model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d499af9942ba7a12dff2c61da2c94ba045e79b7d1c0a03ccd83e7b2ae577e27e
|
||||
size 1426072008
|
||||
318
model.safetensors.index.json
Normal file
318
model.safetensors.index.json
Normal file
@@ -0,0 +1,318 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_size": 16201705472
|
||||
},
|
||||
"weight_map": {
|
||||
"backbone.embeddings.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.0.mixer.A_log": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.0.mixer.D": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.0.mixer.conv1d.bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.0.mixer.conv1d.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.0.mixer.dt_bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.0.mixer.in_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.0.mixer.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.0.mixer.out_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.0.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.1.mixer.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.1.mixer.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.1.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.10.mixer.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.10.mixer.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.10.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.11.mixer.A_log": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.11.mixer.D": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.11.mixer.conv1d.bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.11.mixer.conv1d.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.11.mixer.dt_bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.11.mixer.in_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.11.mixer.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.11.mixer.out_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.11.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.12.mixer.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.12.mixer.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.12.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.13.mixer.A_log": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.13.mixer.D": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.13.mixer.conv1d.bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.13.mixer.conv1d.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.13.mixer.dt_bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.13.mixer.in_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.13.mixer.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.13.mixer.out_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.13.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.14.mixer.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.14.mixer.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.14.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.15.mixer.A_log": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.15.mixer.D": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.15.mixer.conv1d.bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.15.mixer.conv1d.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.15.mixer.dt_bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.15.mixer.in_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.15.mixer.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.15.mixer.out_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.15.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.16.mixer.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.16.mixer.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.16.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.17.mixer.A_log": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.17.mixer.D": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.17.mixer.conv1d.bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.17.mixer.conv1d.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.17.mixer.dt_bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.17.mixer.in_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.17.mixer.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.17.mixer.out_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.17.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.18.mixer.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.18.mixer.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.18.mixer.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.18.mixer.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.18.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.19.mixer.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.19.mixer.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.19.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.2.mixer.A_log": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.2.mixer.D": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.2.mixer.conv1d.bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.2.mixer.conv1d.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.2.mixer.dt_bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.2.mixer.in_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.2.mixer.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.2.mixer.out_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.2.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.20.mixer.A_log": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.20.mixer.D": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.20.mixer.conv1d.bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.20.mixer.conv1d.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.20.mixer.dt_bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.20.mixer.in_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.20.mixer.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.20.mixer.out_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.20.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.21.mixer.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.21.mixer.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.21.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.22.mixer.A_log": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.22.mixer.D": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.22.mixer.conv1d.bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.22.mixer.conv1d.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.22.mixer.dt_bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.22.mixer.in_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.22.mixer.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.22.mixer.out_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.22.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.23.mixer.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.23.mixer.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.23.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.24.mixer.A_log": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.24.mixer.D": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.24.mixer.conv1d.bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.24.mixer.conv1d.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.24.mixer.dt_bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.24.mixer.in_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.24.mixer.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.24.mixer.out_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.24.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.25.mixer.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.25.mixer.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.25.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.26.mixer.A_log": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.26.mixer.D": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.26.mixer.conv1d.bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.26.mixer.conv1d.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.26.mixer.dt_bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.26.mixer.in_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.26.mixer.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.26.mixer.out_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.26.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.27.mixer.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.27.mixer.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.27.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.28.mixer.A_log": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.28.mixer.D": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.28.mixer.conv1d.bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.28.mixer.conv1d.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.28.mixer.dt_bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.28.mixer.in_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.28.mixer.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.28.mixer.out_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.28.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.29.mixer.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.29.mixer.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.29.mixer.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.29.mixer.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.29.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.3.mixer.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.3.mixer.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.3.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.30.mixer.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.30.mixer.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.30.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.31.mixer.A_log": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.31.mixer.D": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.31.mixer.conv1d.bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.31.mixer.conv1d.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.31.mixer.dt_bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.31.mixer.in_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.31.mixer.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.31.mixer.out_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.31.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.32.mixer.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.32.mixer.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.32.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.33.mixer.A_log": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.33.mixer.D": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.33.mixer.conv1d.bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.33.mixer.conv1d.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.33.mixer.dt_bias": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.33.mixer.in_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.33.mixer.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.33.mixer.out_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.33.norm.weight": "model-00002-of-00004.safetensors",
|
||||
"backbone.layers.34.mixer.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.34.mixer.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.34.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.35.mixer.A_log": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.35.mixer.D": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.35.mixer.conv1d.bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.35.mixer.conv1d.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.35.mixer.dt_bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.35.mixer.in_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.35.mixer.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.35.mixer.out_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.35.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.36.mixer.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.36.mixer.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.36.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.37.mixer.A_log": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.37.mixer.D": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.37.mixer.conv1d.bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.37.mixer.conv1d.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.37.mixer.dt_bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.37.mixer.in_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.37.mixer.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.37.mixer.out_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.37.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.38.mixer.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.38.mixer.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.38.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.39.mixer.A_log": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.39.mixer.D": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.39.mixer.conv1d.bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.39.mixer.conv1d.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.39.mixer.dt_bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.39.mixer.in_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.39.mixer.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.39.mixer.out_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.39.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.4.mixer.A_log": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.4.mixer.D": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.4.mixer.conv1d.bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.4.mixer.conv1d.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.4.mixer.dt_bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.4.mixer.in_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.4.mixer.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.4.mixer.out_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.4.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.40.mixer.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.40.mixer.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.40.mixer.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.40.mixer.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.40.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.41.mixer.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.41.mixer.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.41.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.42.mixer.A_log": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.42.mixer.D": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.42.mixer.conv1d.bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.42.mixer.conv1d.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.42.mixer.dt_bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.42.mixer.in_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.42.mixer.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.42.mixer.out_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.42.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.43.mixer.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.43.mixer.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.43.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.44.mixer.A_log": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.44.mixer.D": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.44.mixer.conv1d.bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.44.mixer.conv1d.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.44.mixer.dt_bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.44.mixer.in_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.44.mixer.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.44.mixer.out_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.44.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.45.mixer.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.45.mixer.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.45.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.46.mixer.A_log": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.46.mixer.D": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.46.mixer.conv1d.bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.46.mixer.conv1d.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.46.mixer.dt_bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.46.mixer.in_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.46.mixer.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.46.mixer.out_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.46.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.47.mixer.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.47.mixer.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.47.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.48.mixer.A_log": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.48.mixer.D": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.48.mixer.conv1d.bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.48.mixer.conv1d.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.48.mixer.dt_bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.48.mixer.in_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.48.mixer.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.48.mixer.out_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.48.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.49.mixer.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.49.mixer.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.49.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.5.mixer.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.5.mixer.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.5.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.50.mixer.A_log": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.50.mixer.D": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.50.mixer.conv1d.bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.50.mixer.conv1d.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.50.mixer.dt_bias": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.50.mixer.in_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.50.mixer.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.50.mixer.out_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.50.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.51.mixer.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"backbone.layers.51.mixer.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"backbone.layers.51.norm.weight": "model-00003-of-00004.safetensors",
|
||||
"backbone.layers.6.mixer.A_log": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.6.mixer.D": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.6.mixer.conv1d.bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.6.mixer.conv1d.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.6.mixer.dt_bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.6.mixer.in_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.6.mixer.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.6.mixer.out_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.6.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.7.mixer.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.7.mixer.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.7.mixer.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.7.mixer.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.7.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.8.mixer.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.8.mixer.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.8.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.9.mixer.A_log": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.9.mixer.D": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.9.mixer.conv1d.bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.9.mixer.conv1d.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.9.mixer.dt_bias": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.9.mixer.in_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.9.mixer.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.9.mixer.out_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.layers.9.norm.weight": "model-00001-of-00004.safetensors",
|
||||
"backbone.norm_f.weight": "model-00004-of-00004.safetensors",
|
||||
"lm_head.weight": "model-00004-of-00004.safetensors"
|
||||
}
|
||||
}
|
||||
1631
modeling_nemotron_h.py
Normal file
1631
modeling_nemotron_h.py
Normal file
File diff suppressed because it is too large
Load Diff
1
nemo/context/io.json
Normal file
1
nemo/context/io.json
Normal file
File diff suppressed because one or more lines are too long
277
nemo/context/model.yaml
Normal file
277
nemo/context/model.yaml
Normal file
@@ -0,0 +1,277 @@
|
||||
_target_: nemo.collections.llm.gpt.model.ssm.MambaModel
|
||||
config:
|
||||
_cpu_offloading_context: null
|
||||
_target_: nemo.collections.llm.gpt.model.ssm.NemotronHConfig8B
|
||||
account_for_embedding_in_pipeline_split: false
|
||||
account_for_loss_in_pipeline_split: false
|
||||
activation_func:
|
||||
_call_: false
|
||||
_target_: nemo.collections.llm.gpt.model.ssm.NemotronHConfigBase.<lambda>
|
||||
activation_func_fp8_input_store: false
|
||||
add_bias_linear: false
|
||||
add_qkv_bias: false
|
||||
apply_query_key_layer_scaling: false
|
||||
apply_residual_connection_post_layernorm: false
|
||||
apply_rope_fusion: true
|
||||
async_tensor_model_parallel_allreduce: false
|
||||
attention_backend:
|
||||
_call_: true
|
||||
_target_: megatron.core.transformer.enums.AttnBackend
|
||||
attention_dropout: 0.0
|
||||
attention_softmax_in_fp32: false
|
||||
autocast_dtype: null
|
||||
barrier_with_L1_time: true
|
||||
batch_p2p_comm: true
|
||||
batch_p2p_sync: true
|
||||
bf16: true
|
||||
bias_activation_fusion: false
|
||||
bias_dropout_fusion: true
|
||||
calculate_per_token_loss: false
|
||||
clone_scatter_output_in_embedding: true
|
||||
config_logger_dir: ''
|
||||
context_parallel_size: 1
|
||||
cp_comm_type: null
|
||||
cpu_offloading: false
|
||||
cpu_offloading_activations: true
|
||||
cpu_offloading_num_layers: 0
|
||||
cpu_offloading_weights: true
|
||||
cross_entropy_fusion_impl: native
|
||||
cross_entropy_loss_fusion: true
|
||||
cuda_graph_retain_backward_graph: false
|
||||
cuda_graph_scope: full
|
||||
cuda_graph_use_single_mempool: false
|
||||
cuda_graph_warmup_steps: 3
|
||||
data_step_fn:
|
||||
_call_: false
|
||||
_target_: nemo.collections.llm.gpt.model.base.gpt_data_step
|
||||
deallocate_pipeline_outputs: true
|
||||
defer_embedding_wgrad_compute: false
|
||||
deterministic_mode: false
|
||||
disable_parameter_transpose_cache: false
|
||||
distribute_saved_activations: null
|
||||
enable_autocast: false
|
||||
enable_cuda_graph: false
|
||||
expert_model_parallel_size: 1
|
||||
expert_tensor_parallel_size: null
|
||||
external_cuda_graph: false
|
||||
ffn_hidden_size: 21504
|
||||
finalize_model_grads_func: null
|
||||
first_last_layers_bf16: true
|
||||
flash_decode: false
|
||||
forward_step_fn:
|
||||
_call_: false
|
||||
_target_: nemo.collections.llm.gpt.model.ssm.ssm_forward_step
|
||||
fp16: false
|
||||
fp16_lm_cross_entropy: false
|
||||
fp32_residual_connection: false
|
||||
fp8: null
|
||||
fp8_amax_compute_algo: most_recent
|
||||
fp8_amax_history_len: 1
|
||||
fp8_dot_product_attention: false
|
||||
fp8_interval: 1
|
||||
fp8_margin: 0
|
||||
fp8_multi_head_attention: false
|
||||
fp8_recipe: delayed
|
||||
fp8_wgrad: true
|
||||
gated_linear_unit: false
|
||||
get_attention_mask_from_fusion: false
|
||||
grad_scale_func: null
|
||||
grad_sync_func: null
|
||||
gradient_accumulation_fusion: false
|
||||
hidden_dropout: 0.0
|
||||
hidden_size: 4096
|
||||
hierarchical_context_parallel_sizes: null
|
||||
hybrid_attention_ratio: 0.0
|
||||
hybrid_mlp_ratio: 0.0
|
||||
hybrid_override_pattern: M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-
|
||||
inference_rng_tracker: false
|
||||
init_method: null
|
||||
init_method_std: 0.02
|
||||
init_model_with_meta_device: false
|
||||
is_hybrid_model: true
|
||||
kv_channels: null
|
||||
layernorm_epsilon: 1.0e-05
|
||||
layernorm_zero_centered_gamma: false
|
||||
make_vocab_size_divisible_by: 128
|
||||
mamba_head_dim: 64
|
||||
mamba_nheads: 128
|
||||
mamba_num_groups: 8
|
||||
mamba_state_dim: 128
|
||||
mapping_type: nvidia-hybrid-nemotronh
|
||||
masked_softmax_fusion: true
|
||||
memory_efficient_layer_norm: false
|
||||
microbatch_group_size_per_vp_stage: 1
|
||||
moe_aux_loss_coeff: 0
|
||||
moe_enable_deepep: false
|
||||
moe_expert_capacity_factor: null
|
||||
moe_extended_tp: false
|
||||
moe_ffn_hidden_size: null
|
||||
moe_grouped_gemm: false
|
||||
moe_input_jitter_eps: null
|
||||
moe_layer_freq: 1
|
||||
moe_layer_recompute: false
|
||||
moe_pad_expert_input_to_capacity: false
|
||||
moe_per_layer_logging: false
|
||||
moe_permute_fusion: false
|
||||
moe_router_bias_update_rate: 0.001
|
||||
moe_router_dtype: null
|
||||
moe_router_enable_expert_bias: false
|
||||
moe_router_group_topk: null
|
||||
moe_router_load_balancing_type: aux_loss
|
||||
moe_router_num_groups: null
|
||||
moe_router_pre_softmax: false
|
||||
moe_router_score_function: softmax
|
||||
moe_router_topk: 2
|
||||
moe_router_topk_limited_devices: null
|
||||
moe_router_topk_scaling_factor: null
|
||||
moe_shared_expert_intermediate_size: null
|
||||
moe_shared_expert_overlap: false
|
||||
moe_token_dispatcher_type: allgather
|
||||
moe_token_drop_policy: probs
|
||||
moe_token_dropping: false
|
||||
moe_use_legacy_grouped_gemm: false
|
||||
moe_z_loss_coeff: null
|
||||
mtp_loss_scaling_factor: null
|
||||
mtp_num_layers: null
|
||||
multi_latent_attention: false
|
||||
no_sync_func: null
|
||||
normalization: RMSNorm
|
||||
num_attention_heads: 32
|
||||
num_layers: 52
|
||||
num_layers_at_end_in_bf16: 1
|
||||
num_layers_at_start_in_bf16: 1
|
||||
num_layers_in_first_pipeline_stage: null
|
||||
num_layers_in_last_pipeline_stage: null
|
||||
num_microbatches_with_partial_activation_checkpoints: null
|
||||
num_moe_experts: null
|
||||
num_query_groups: 8
|
||||
output_layer_init_method: null
|
||||
overlap_p2p_comm: false
|
||||
overlap_p2p_comm_warmup_flush: false
|
||||
parallel_output: true
|
||||
param_sync_func: null
|
||||
params_dtype:
|
||||
_call_: false
|
||||
_target_: torch.bfloat16
|
||||
perform_initialization: true
|
||||
persist_layer_norm: true
|
||||
pipeline_dtype: null
|
||||
pipeline_model_parallel_comm_backend: null
|
||||
pipeline_model_parallel_size: 1
|
||||
pipeline_model_parallel_split_rank: null
|
||||
position_embedding_type: none
|
||||
post_process: true
|
||||
pre_process: true
|
||||
qk_layernorm: false
|
||||
recompute_granularity: null
|
||||
recompute_method: null
|
||||
recompute_num_layers: null
|
||||
rotary_base: 10000
|
||||
rotary_interleaved: false
|
||||
rotary_percent: 1.0
|
||||
seq_len_interpolation_factor: null
|
||||
seq_length: 8192
|
||||
sequence_parallel: false
|
||||
share_embeddings_and_output_weights: false
|
||||
softmax_scale: null
|
||||
tensor_model_parallel_size: 1
|
||||
test_mode: false
|
||||
timers: null
|
||||
tokenizer_library: tiktoken
|
||||
tokenizer_model_path: null
|
||||
tokenizer_name: TiktokenTokenizer
|
||||
tp_comm_atomic_ag: false
|
||||
tp_comm_atomic_rs: false
|
||||
tp_comm_bootstrap_backend: nccl
|
||||
tp_comm_bulk_dgrad: true
|
||||
tp_comm_bulk_wgrad: true
|
||||
tp_comm_overlap: false
|
||||
tp_comm_overlap_ag: true
|
||||
tp_comm_overlap_disable_fc1: false
|
||||
tp_comm_overlap_disable_qkv: false
|
||||
tp_comm_overlap_rs: true
|
||||
tp_comm_overlap_rs_dgrad: false
|
||||
tp_comm_split_ag: true
|
||||
tp_comm_split_rs: true
|
||||
tp_only_amax_red: false
|
||||
use_cpu_initialization: false
|
||||
use_custom_fsdp: false
|
||||
use_ring_exchange_p2p: false
|
||||
use_te_rng_tracker: false
|
||||
variable_seq_lengths: false
|
||||
virtual_pipeline_model_parallel_size: null
|
||||
vocab_file: null
|
||||
vocab_size: 131072
|
||||
wgrad_deferral_limit: 0
|
||||
window_size: null
|
||||
model_transform: null
|
||||
optim:
|
||||
_target_: nemo.lightning.pytorch.optim.megatron.MegatronOptimizerModule
|
||||
config:
|
||||
_target_: megatron.core.optimizer.optimizer_config.OptimizerConfig
|
||||
adam_beta1: 0.9
|
||||
adam_beta2: 0.999
|
||||
adam_eps: 1.0e-08
|
||||
barrier_with_L1_time: false
|
||||
bf16: false
|
||||
clip_grad: 1.0
|
||||
config_logger_dir: ''
|
||||
decoupled_lr: null
|
||||
decoupled_min_lr: null
|
||||
exp_avg_dtype:
|
||||
_call_: false
|
||||
_target_: torch.float32
|
||||
exp_avg_sq_dtype:
|
||||
_call_: false
|
||||
_target_: torch.float32
|
||||
fp16: false
|
||||
hysteresis: 2
|
||||
initial_loss_scale: 4294967296
|
||||
log_num_zeros_in_grad: false
|
||||
loss_scale: null
|
||||
loss_scale_window: 1000
|
||||
lr: 0.0001
|
||||
main_grads_dtype:
|
||||
_call_: false
|
||||
_target_: torch.float32
|
||||
main_params_dtype:
|
||||
_call_: false
|
||||
_target_: torch.float32
|
||||
min_loss_scale: 1.0
|
||||
min_lr: null
|
||||
optimizer: adam
|
||||
optimizer_cpu_offload: false
|
||||
optimizer_offload_fraction: 0.0
|
||||
overlap_cpu_optimizer_d2h_h2d: false
|
||||
overlap_param_gather_with_optimizer_step: false
|
||||
params_dtype:
|
||||
_call_: false
|
||||
_target_: torch.float32
|
||||
pin_cpu_grads: true
|
||||
pin_cpu_params: true
|
||||
sgd_momentum: 0.9
|
||||
timers: null
|
||||
use_distributed_optimizer: true
|
||||
use_precision_aware_optimizer: false
|
||||
use_torch_optimizer_for_cpu_offload: false
|
||||
weight_decay: 0.01
|
||||
lr_mult: 1.0
|
||||
lr_scheduler: null
|
||||
no_weight_decay_cond: null
|
||||
scale_lr_cond: null
|
||||
tokenizer:
|
||||
_target_: nemo.collections.common.tokenizers.huggingface.auto_tokenizer.AutoTokenizer
|
||||
additional_special_tokens: []
|
||||
bos_token: null
|
||||
cls_token: null
|
||||
eos_token: null
|
||||
include_special_tokens: false
|
||||
mask_token: null
|
||||
merges_file: null
|
||||
pad_token: null
|
||||
pretrained_model_name: nemo_tokenizer
|
||||
sep_token: null
|
||||
trust_remote_code: true
|
||||
unk_token: null
|
||||
use_fast: false
|
||||
vocab_file: null
|
||||
23
nemo/context/nemo_tokenizer/special_tokens_map.json
Normal file
23
nemo/context/nemo_tokenizer/special_tokens_map.json
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
nemo/context/nemo_tokenizer/tokenizer.json
Normal file
3
nemo/context/nemo_tokenizer/tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3277c00fe5fb3963b3cb7c07b7f183722d2af4d775a4aea7cfb3684d7cccbc2f
|
||||
size 17078330
|
||||
8019
nemo/context/nemo_tokenizer/tokenizer_config.json
Normal file
8019
nemo/context/nemo_tokenizer/tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
3
nemo/weights/.metadata
Normal file
3
nemo/weights/.metadata
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:105efb65675afec07e79216510b74e4f192acc5c379edb13879436f9a6b14f07
|
||||
size 227698
|
||||
3
nemo/weights/__0_0.distcp
Normal file
3
nemo/weights/__0_0.distcp
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c3cea0f72e0a380ef33f26e1d8a6963b6e3df4d324e8e250d30c5dc6c0ab10e4
|
||||
size 8101267213
|
||||
3
nemo/weights/__0_1.distcp
Normal file
3
nemo/weights/__0_1.distcp
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:52fe5fc7d2a87f792432f0c16a87aca0f91ea57347eb8514c57bbf0b5df164ed
|
||||
size 8101271028
|
||||
3
nemo/weights/common.pt
Normal file
3
nemo/weights/common.pt
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:bf2d72b4f4fef17ea0a99868f378f27ac9989452f3ffdd0691821ae1a3b9f285
|
||||
size 3099
|
||||
1
nemo/weights/metadata.json
Normal file
1
nemo/weights/metadata.json
Normal file
@@ -0,0 +1 @@
|
||||
{"sharded_backend": "torch_dist", "sharded_backend_version": 1, "common_backend": "torch", "common_backend_version": 1}
|
||||
23
special_tokens_map.json
Normal file
23
special_tokens_map.json
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3277c00fe5fb3963b3cb7c07b7f183722d2af4d775a4aea7cfb3684d7cccbc2f
|
||||
size 17078330
|
||||
8019
tokenizer_config.json
Normal file
8019
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user