85 lines
3.6 KiB
Markdown
85 lines
3.6 KiB
Markdown
---
|
||
library_name: transformers
|
||
language:
|
||
- en
|
||
license: cc-by-nc-4.0
|
||
---
|
||
|
||
# Model Information
|
||
|
||
We introduce **Nemotron-UltraLong-8B**, a series of ultra-long context language models designed to process extensive sequences of text (up to 1M, 2M, and 4M tokens) while maintaining competitive performance on standard benchmarks. Built on the Llama-3.1, UltraLong-8B leverages a systematic training recipe that combines efficient continued pretraining with instruction tuning to enhance long-context understanding and instruction-following capabilities. This approach enables our models to efficiently scale their context windows without sacrificing general performance.
|
||
|
||
|
||
## The UltraLong Models
|
||
|
||
- [nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct](https://huggingface.co/nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct)
|
||
- [nvidia/Llama-3.1-Nemotron-8B-UltraLong-2M-Instruct](https://huggingface.co/nvidia/Llama-3.1-Nemotron-8B-UltraLong-2M-Instruct)
|
||
- [nvidia/Llama-3.1-Nemotron-8B-UltraLong-4M-Instruct](https://huggingface.co/nvidia/Llama-3.1-Nemotron-8B-UltraLong-4M-Instruct)
|
||
|
||
|
||
## Uses
|
||
|
||
Starting with `transformers >= 4.43.0` onward, you can run conversational inference using the Transformers `pipeline` abstraction or by leveraging the Auto classes with the `generate()` function.
|
||
|
||
Make sure to update your transformers installation via `pip install --upgrade transformers`.
|
||
|
||
```python
|
||
import transformers
|
||
import torch
|
||
|
||
model_id = "nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct"
|
||
|
||
pipeline = transformers.pipeline(
|
||
"text-generation",
|
||
model=model_id,
|
||
model_kwargs={"torch_dtype": torch.bfloat16},
|
||
device_map="auto",
|
||
)
|
||
|
||
messages = [
|
||
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
|
||
{"role": "user", "content": "Who are you?"},
|
||
]
|
||
|
||
outputs = pipeline(
|
||
messages,
|
||
max_new_tokens=256,
|
||
)
|
||
print(outputs[0]["generated_text"][-1])
|
||
```
|
||
|
||
## Model Card
|
||
|
||
* Base model: [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)
|
||
* Continued Pretraining: The training data consists of 1B tokens sourced from a pretraining corpus using per-domain upsampling based on sample length. The model was trained for 125 iterations with a sequence length of 1M and a global batch size of 8.
|
||
* Supervised fine-tuning (SFT): 1B tokens on open-source instruction datasets across general, mathematics, and code domains. We subsample the data from the ‘general_sft_stage2’ from [AceMath-Instruct](https://huggingface.co/datasets/nvidia/AceMath-Instruct-Training-Data).
|
||
* Maximum context window: 1M tokens
|
||
|
||
## Evaluation Results
|
||
|
||
We evaluate Nemotron-UltraLong-8B on a diverse set of benchmarks, including long-context tasks (e.g., RULER, LV-Eval, and InfiniteBench) and standard tasks (e.g., MMLU, MATH, GSM-8K, and HumanEval). UltraLong-8B achieves superior performance on ultra-long context tasks while maintaining competitive results on standard benchmarks.
|
||
|
||
### Needle in a Haystack
|
||
|
||
<img width="80%" alt="image" src="Llama-3.1-8B-UltraLong-1M-Instruct.png">
|
||
|
||
### Long context evaluation
|
||
|
||
<img width="80%" alt="image" src="long_benchmark.png">
|
||
|
||
### Standard capability evaluation
|
||
|
||
<img width="80%" alt="image" src="standard_benchmark.png">
|
||
|
||
## Correspondence to
|
||
Chejian Xu (chejian2@illinois.edu), Wei Ping (wping@nvidia.com)
|
||
|
||
## Citation
|
||
<pre>
|
||
@article{ulralong2025,
|
||
title={From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models},
|
||
author={Xu, Chejian and Ping, Wei and Xu, Peng and Liu, Zihan and Wang, Boxin and Shoeybi, Mohammad and Catanzaro, Bryan},
|
||
journal={arXiv preprint},
|
||
year={2025}
|
||
}
|
||
</pre> |