---
license: mit
---

# SelfLong-Llama3.2-3B-Instruct-1M

Wang, Liang, Nan Yang, Xingxing Zhang, Xiaolong Huang, and Furu Wei. "[Bootstrap Your Own Context Length.](https://arxiv.org/pdf/2412.18860)" arXiv preprint arXiv:2412.18860 (2024).

## Overview

The SelfLong series of Large Language Models (LLMs) are designed to handle extremely long contexts, reaching up to 1 million tokens. These models, with parameter sizes of 1B, 3B, and 8B, are initialized from the Llama-3.2 and Llama-3.1 architectures.

## Performance (RULER-1M)

The following table presents the results of the SelfLong models on the [RULER-1M benchmark](https://huggingface.co/datasets/self-long/RULER-llama3-1M). The numbers represent the RULER score averaged over 13 tasks at different support lengths.

| Model                 | Support Length | 32k  | 64k  | 128k | 256k | 512k | 1M   |
|:----------------------| :------------- | :--- | :--- | :--- | :--- | :--- | :--- |
| Llama-3.2-1B-Instruct | 128k           | 64.7 | 43.1 | 0.0  | -    | -    | -    |
| Llama-3.2-3B-Instruct | 128k           | 77.8 | 70.4 | 0.8  | -    | -    | -    |
| Llama-3.1-8B-Instruct | 128k           | **89.8** | **85.4** | <ins>78.5</ins> | -    | -    | -    |
| gradientai/Llama-3-8B-Instruct-Gradient-1048k   | 1M             | 81.8 | 78.6 | 77.2 | <ins>74.2</ins> | <ins>70.3</ins> | <ins>64.3</ins> |
| **SelfLong-1B-1M**    | 1M             | 61.3 | 56.6 | 54.7 | 46.7 | 40.7 | 31.1 |
| **SelfLong-3B-1M**    | 1M             | 80.5 | 78.0 | 75.5 | 68.8 | 58.5 | 38.8 |
| **SelfLong-8B-1M**    | 1M             | <ins>89.5</ins> | <ins>84.0</ins> | **82.0** | **79.7** | **78.2** | **69.6** |

**Note:**

* **Bold** indicates the best performance.
* <ins>Underline</ins> indicates the second-best performance.
* `-` indicates that the model does not support the given context length.

## Evaluation on RULER-1M Dataset

To evaluate the SelfLong models on the RULER-1M dataset, you can follow these steps:

1.  Start vllm server:

```bash
PROC_PER_NODE=$(nvidia-smi --list-gpus | wc -l)
# Reduce this number if you have limited GPU memory
MAX_MODEL_LEN=1048576
MODEL_NAME_OR_PATH="self-long/SelfLong-Llama3.2-3B-Instruct-1M"

echo "Starting VLLM server..."
vllm serve "${MODEL_NAME_OR_PATH}" \
   --dtype auto \
   --disable-log-stats --disable-log-requests --disable-custom-all-reduce \
   --enable_chunked_prefill --max_num_batched_tokens 8192 \
   --tensor-parallel-size "${PROC_PER_NODE}" \
   --max-model-len "${MAX_MODEL_LEN}" \
   --gpu_memory_utilization 0.9 \
   --api-key token-123 &
```

2. Get Completions
```python
from openai import OpenAI
from datasets import load_dataset

client = OpenAI(
    base_url="http://localhost:8000/v1",  # Default vLLM server address
    api_key="token-123"
)

ds = load_dataset('self-long/RULER-llama3-1M', f'niah_single_1_4k', split='validation')
prompt = ds[0]['input']

completion = client.completions.create(
    model='self-long/SelfLong-Llama3.2-3B-Instruct-1M',
    prompt=prompt,
    max_tokens=100,
)

print(prompt)
print(completion.choices[0].text)
```

3. For evaluation, please refer to the evaluation script provided in the RULER repository: [https://github.com/NVIDIA/RULER/blob/main/scripts/eval/evaluate.py](https://github.com/NVIDIA/RULER/blob/main/scripts/eval/evaluate.py).

Note that different vLLM and Torch versions may produce slightly different decoding results.

## References

```
@article{wang2024bootstrap,
  title={Bootstrap Your Own Context Length},
  author={Wang, Liang and Yang, Nan and Zhang, Xingxing and Huang, Xiaolong and Wei, Furu},
  journal={arXiv preprint arXiv:2412.18860},
  year={2024}
}
```