PY007/EasyContext-256K-danube2-1.8b
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: PY007/EasyContext-256K-danube2-1.8b
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-10-23 04:07:17 +08:00
commit d5842b220c
10 changed files with 91284 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
README.md Normal file
View File

@@ -0,0 +1,20 @@
# EasyContext
<p align="center">
<img src="https://github.com/jzhang38/EasyContext/raw/main/data/Logo.webp" width="500">
</p>
<p align="center">
<a href="https://github.com/jzhang38/EasyContext" target="_blank">GitHub Repo</a>
</p>
Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.
**This is a context-extrapolated base model.** It has not been instruct-finetuned.
This model is finetuned from h2oai/h2o-danube2-1.8b-base with EasyContext on context length 256K. Note that I keep max_position_embeddings in config.json to 4096 because HF llama will create 2D causal mask during initialization. If it is set to 256K GPU will just OOM. You can surely use this model with context length longer than 4096.
<p align="center">
<img src="./heatmap.png" width="800">
</p>