zettafleet/z1-1b-hybrid-instruct
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
z1-1b-hybrid-instruct/README.md
ModelHub XC bb8723dbd7 初始化项目，由ModelHub XC社区提供模型 
Model: zettafleet/z1-1b-hybrid-instruct
Source: Original Platform
2026-06-23 21:42:26 +08:00

108 lines
16 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
license: other
language:
- en
datasets:
- allenai/tulu-3-sft-olmo-2-mixture-0225
- allenai/olmo-2-0425-1b-preference-mix
- allenai/RLVR-GSM-MATH-IF-Mixed-Constraints
- allenai/RLVR-MATH
base_model:
- zettafleet/z1-1b-hybrid
pipeline_tag: text-generation
library_name: transformers
---
<svg version="1.1" height="32px" style="margin: 32px 0;" viewBox="0 0 577 128" xmlns="http://www.w3.org/2000/svg" fill="currentColor">
<path d="m20 0c-11.08 0-20 8.92-20 20v88c0 11.08 8.92 20 20 20h88c11.08 0 20-8.92 20-20v-88c0-11.08-8.92-20-20-20h-88zm386.28906 15.607422c-2.86041 0-5.51082 0.562093-7.94922 1.6875-2.43838 1.101967-4.39595 2.778483-5.87304 5.029297-1.4771 2.227373-2.21485 5.030023-2.21485 8.40625v6.259765h-8.37109v8.439454h8.37109v45.580078h10.51563v-45.580078h11.64062v-8.439454h-11.64062v-4.853515c0-2.391494 0.53866-4.220242 1.61719-5.486328 1.10195-1.26608 2.9307-1.900391 5.48632-1.900391 1.10197 0 2.04074 0.105393 2.81446 0.316406 0.77372 0.187567 1.40607 0.353468 1.89843 0.494141l2.46094-8.511719c-0.72682-0.281353-1.8393-0.585816-3.33984-0.914062-1.50055-0.351687-3.30588-0.527344-5.41602-0.527344zm13.41406 3.375v72.027344h10.51563v-72.027344h-10.51563zm-148.78906 5.064453v12.943359h-7.91406v8.439454h7.91406v31.898437c0 3.259 0.73775 5.967967 2.21485 8.125 1.47709 2.157033 3.41123 3.751576 5.80273 4.783203 2.39149 1.0316 4.97165 1.523429 7.73828 1.476563 1.68812-0.023447 3.09335-0.163969 4.21875-0.421875 1.12542-0.23446 1.99393-0.468665 2.60352-0.703125l-1.89844-8.6875c-0.35169 0.070333-0.84548 0.164016-1.47852 0.28125-0.63304 0.117226-1.37078 0.175781-2.21484 0.175781-1.14885 0-2.21644-0.175657-3.20117-0.527344-0.96129-0.37514-1.74587-1.066039-2.35547-2.074219-0.60959-1.03162-0.91406-2.567615-0.91406-4.607421v-29.71875h11.07812v-8.439454h-11.07812v-12.943359h-10.51563zm33.28516 0v12.943359h-7.91406v8.439454h7.91406v31.898437c0 3.259 0.73774 5.967967 2.21484 8.125s3.41124 3.751576 5.80274 4.783203c2.39148 1.0316 4.96969 1.523429 7.73632 1.476563 1.68812-0.023447 3.0953-0.163969 4.22071-0.421875 1.1254-0.23446 1.99391-0.468665 2.60351-0.703125l-1.90039-8.6875c-0.35168 0.070333-0.84352 0.164016-1.47656 0.28125-0.63305 0.117226-1.37079 0.175781-2.21484 0.175781-1.14886 0-2.21644-0.175657-3.20117-0.527344-0.96129-0.37514-1.74588-1.066039-2.35547-2.074219-0.6096-1.03162-0.91602-2.567615-0.91602-4.607421v-29.71875h11.08008v-8.439454h-11.08008v-12.943359h-10.51367zm249.50781 0v12.943359h-7.91211v8.439454h7.91211v31.898437c0 3.259 0.7397 5.967967 2.2168 8.125s3.41123 3.751576 5.80273 4.783203c2.39149 1.0316 4.9697 1.523429 7.73633 1.476563 1.68811-0.023447 3.09529-0.163969 4.2207-0.421875 1.12541-0.23446 1.99197-0.468665 2.60157-0.703125l-1.89844-8.6875c-0.35169 0.070333-0.84352 0.164016-1.47656 0.28125-0.63304 0.117226-1.37275 0.175781-2.2168 0.175781-1.14886 0-2.21449-0.175657-3.19922-0.527344-0.96129-0.37514-1.74783-1.066039-2.35742-2.074219-0.6096-1.03162-0.91406-2.567615-0.91406-4.607421v-29.71875h11.07812v-8.439454h-11.07812v-12.943359h-10.51563zm-507.642577 1.953125h15.197266l-17.378907 22.373047a6 6 0 0 1-8.419921 1.058594 6 6 0 0 1-1.058594-8.419922l11.660156-15.011719zm42.044922 0.046875a6 6 0 0 1 4.427734 1.214844 6 6 0 0 1 1.058594 8.417969l-49.712891 64a6 6 0 0 1-8.419921 1.058592 6 6 0 0 1-1.058594-8.417968l49.712891-64a6 6 0 0 1 3.992187-2.273437zm147.091795 10.240234c-5.01744 0-9.40098 1.182735-13.15234 3.550782-3.72792 2.368046-6.63596 5.662525-8.72266 9.882812-2.06325 4.220287-3.09375 9.0976-3.09375 14.630859 0 5.603601 1.0305 10.492629 3.09375 14.666016 2.0867 4.149947 5.04158 7.372209 8.86328 9.669922 3.84514 2.27426 8.42776 3.412109 13.75 3.412109 3.93894 0 7.44419-0.597222 10.51563-1.792968 3.09487-1.219194 5.66332-2.91913 7.70312-5.09961 2.06325-2.20392 3.49386-4.772371 4.29102-7.703125l-9.95313-1.792968c-0.63304 1.688113-1.54838 3.107006-2.74414 4.255859-1.19574 1.148853-2.61463 2.015409-4.25586 2.601562-1.64122 0.5627-3.45825 0.84375-5.45117 0.84375-3.07142 0-5.75503-0.655776-8.05273-1.96875-2.29772-1.336426-4.09133-3.270567-5.38086-5.802734-1.17811-2.356228-1.80095-5.186493-1.88281-8.476563h38.38867v-3.726562c0-4.90022-0.65773-9.063215-1.97071-12.486328-1.31298-3.446567-3.10659-6.247264-5.38086-8.404297-2.27426-2.157033-4.84075-3.739867-7.70117-4.748047s-5.81529-1.511719-8.86328-1.511719zm120.07813 0c-3.259 0-6.36612 0.46841-9.32032 1.40625-2.9542 0.914394-5.56949 2.391834-7.84375 4.431641-2.25082 2.016353-3.96051 4.664819-5.13281 7.947266l9.88086 2.251953c0.77372-1.899127 2.16919-3.622482 4.18555-5.169922 2.0398-1.570887 4.83074-2.357422 8.37109-2.357422 3.39967 0 5.95446 0.84509 7.66602 2.533203 1.735 1.68812 2.60351 4.067245 2.60351 7.138672v0.246094c0 1.266086-0.45864 2.181434-1.37304 2.74414-0.91439 0.5627-2.37818 0.972562-4.39453 1.230469-2.01636 0.23446-4.64336 0.538923-7.87891 0.914063-2.55562 0.3048-5.06357 0.74979-7.52539 1.335937-2.46184 0.586153-4.69069 1.454662-6.6836 2.603516-1.99291 1.148853-3.57378 2.708267-4.74609 4.677734-1.17229 1.969467-1.75977 4.477411-1.75977 7.525391 0 3.540353 0.79826 6.530363 2.39258 8.96875 1.61778 2.438386 3.80955 4.30227 6.57618 5.591797 2.76662 1.266086 5.86202 1.898437 9.28515 1.898437 2.97765 0 5.53244-0.433281 7.66602-1.300781 2.13359-0.867507 3.88036-1.958511 5.24023-3.271485 1.35987-1.312979 2.38061-2.649898 3.06055-4.009765h0.42187v7.386719h10.26953v-35.873047c0-3.938934-0.68114-7.139729-2.04101-9.601563-1.35987-2.485273-3.10665-4.395995-5.24024-5.732422-2.11014-1.33642-4.38387-2.24982-6.82226-2.742187-2.41493-0.515813-4.70038-0.773438-6.85742-0.773438zm108.08398 0c-5.01745 0-9.40293 1.182735-13.1543 3.550782-3.72792 2.368046-6.63401 5.662525-8.7207 9.882812-2.06325 4.220287-3.0957 9.0976-3.0957 14.630859 0 5.603601 1.03245 10.492629 3.0957 14.666016 2.08669 4.149947 5.03962 7.372209 8.86133 9.669922 3.84515 2.27426 8.42971 3.412109 13.75195 3.412109 3.93894 0 7.4442-0.597222 10.51563-1.792968 3.09487-1.219194 5.66136-2.91913 7.70117-5.09961 2.06325-2.20392 3.49385-4.772371 4.29102-7.703125l-9.95313-1.792968c-0.63304 1.688113-1.54644 3.107006-2.74219 4.255859s-2.61464 2.015409-4.25586 2.601562c-1.64122 0.5627-3.45826 0.84375-5.45117 0.84375-3.07143 0-5.75697-0.655776-8.05469-1.96875-2.2977-1.336426-4.09132-3.270567-5.38086-5.802734-1.17811-2.356228-1.80095-5.186493-1.88281-8.476563h38.38867v-3.726562c0-4.90022-0.65576-9.063215-1.96875-12.486328-1.31297-3.446567-3.10659-6.247264-5.38086-8.404297-2.27426-2.157033-4.8427-3.739867-7.70312-4.748047-2.86041-1.00818-5.81334-1.511719-8.86133-1.511719zm54.63281 0c-5.01744 0-9.40293 1.182735-13.15429 3.550782-3.72792 2.368046-6.63401 5.662525-8.72071 9.882812-2.06324 4.220287-3.0957 9.0976-3.0957 14.630859 0 5.603601 1.03246 10.492629 3.0957 14.666016 2.0867 4.149947 5.03964 7.372209 8.86133 9.669922 3.84516 2.27426 8.42971 3.412109 13.75196 3.412109 3.93893 0 7.44419-0.597222 10.51562-1.792968 3.09487-1.219194 5.66137-2.91913 7.70117-5.09961 2.06326-2.20392 3.49385-4.772371 4.29102-7.703125l-9.95313-1.792968c-0.63304 1.688113-1.54644 3.107006-2.74218 4.255859-1.19575 1.148853-2.61464 2.015409-4.25586 2.601562-1.64123 0.5627-3.45827 0.84375-5.45118 0.84375-3.07142 0-5.75697-0.655776-8.05468-1.96875-2.29771-1.336426-4.09133-3.270567-5.38086-5.802734-1.17812-2.356228-1.80096-5.186493-1.88282-8.476563h38.38868v-3.726562c0-4.90022-0.65578-9.063215-1.96875-12.486328-1.31298-3.446567-3.1066-6.247264-5.38086-8.404297-2.27428-2.157033-4.84272-3.739867-7.70313-4.748047s-5.81334-1.511719-8.86133-1.511719zm-355.04687 0.703125v9.142578h28.31055v0.492188l-29.25977 37.173828v7.210938h43.36328v-9.144532h-29.11914v-0.492187l28.13477-36.716797v-7.666016h-41.42969zm72.32226 7.982422c2.71974 0 5.08911 0.622595 7.10547 1.865235 2.0398 1.24264 3.62264 2.940623 4.74805 5.097656 1.12541 2.133587 1.6875 4.584929 1.6875 7.351562h-28.07227c0.12629-2.322226 0.70214-4.490331 1.73047-6.505859 1.19575-2.32116 2.88398-4.196746 5.06446-5.626953 2.20392-1.453653 4.78212-2.181641 7.73632-2.181641zm228.16211 0c2.71974 0 5.08716 0.622595 7.10352 1.865235 2.03981 1.24264 3.62263 2.940623 4.74805 5.097656 1.1254 2.133587 1.6875 4.584929 1.6875 7.351562h-28.07227c0.12629-2.322226 0.70213-4.490331 1.73047-6.505859 1.19575-2.32116 2.88397-4.196746 5.06445-5.626953 2.20393-1.453653 4.78408-2.181641 7.73828-2.181641zm54.63282 0c2.71974 0 5.08715 0.622595 7.10351 1.865235 2.0398 1.24264 3.62264 2.940623 4.74805 5.097656 1.12541 2.133587 1.6875 4.584929 1.6875 7.351562h-28.07227c0.12629-2.322226 0.70214-4.490331 1.73047-6.505859 1.19575-2.32116 2.88398-4.196746 5.06445-5.626953 2.20393-1.453653 4.78409-2.181641 7.73829-2.181641zm-152.3418 19.589844v6.962891c0 2.133586-0.55233 4.137991-1.6543 6.013671-1.07851 1.85224-2.66135 3.3531-4.74804 4.501954-2.06325 1.148853-4.54778 1.722656-7.45508 1.722656-2.97765 0-5.42704-0.655777-7.34961-1.96875-1.92258-1.336427-2.88477-3.305697-2.88477-5.908203 0-1.87568 0.49183-3.37654 1.47657-4.501953 1.00817-1.148854 2.34509-2.027126 4.00976-2.636719 1.68811-0.6096 3.55199-1.044828 5.5918-1.302735 0.8675-0.117226 1.94679-0.257748 3.23633-0.421874 1.28952-0.187567 2.61473-0.386643 3.97461-0.597657 1.35986-0.23446 2.56602-0.503794 3.62109-0.808593 1.07851-0.328241 1.8065-0.679548 2.18164-1.054688zm-276.441407 12.759766a6 6 0 0 1 3.251953 1.246093 6 6 0 0 1 1.058594 8.419922l-11.660156 15.011719h-15.197266l17.378906-22.373047a6 6 0 0 1 5.167969-2.304687z"></path>
</svg>
# Model Card for Z1 1B Hybrid Instruct
We are excited to introduce the Z1 family of models! These models are based on the [OLMo 2 1B](https://huggingface.co/allenai/OLMo-2-0425-1B) architecture developed by [Allen Institute for AI](https://huggingface.co/allenai). Beginning with the [pre-training checkpoint](https://huggingface.co/allenai/OLMo-2-0425-1B/tree/stage1-step1907359-tokens4001B) for OLMo 2 1B, we performed continued pre-training (i.e., midtraining) on Z1 1B Hybrid using the same dataset as OLMo 2 1B ([dolmino-mix-1124](https://huggingface.co/datasets/zettafleet/dolmino-mix-1124)).
What is unusual about the Z1 models is that the continued pre-training was performed via [Zettafleet’s AI Training Platform](https://www.zettafleet.com/) on 8 NVIDIA GPUs in a **fully decentralized way**, without the use of high-bandwidth near-range communication links (i.e., NVLink) between the accelerators. See our [blog post](https://zettafleet.com/blog/introducing-zettafleet) for further details.
The [zettafleet/z1-1b-hybrid-instruct](https://huggingface.co/zettafleet/z1-1b-hybrid-instruct) (i.e., this model) is an instruction-tuned version of [zettafleet/z1-1b-hybrid](https://huggingface.co/zettafleet/z1-1b-hybrid), trained with the same post-training datasets as [allenai/OLMo-2-0425-1B-Instruct](https://huggingface.co/allenai/OLMo-2-0425-1B-Instruct). For more information about post-training, please see the [OLMo 2 paper](https://arxiv.org/abs/2501.00656) or [Tülu 3 paper](https://arxiv.org/abs/2411.15124). The post-training pipeline (i.e, the training code) was reconstructed through instructions provided by engineers and researchers at Allen Institute for AI.
We release the following models as part of the Z1 family:
- [zettafleet/z1-1b-hybrid](https://huggingface.co/zettafleet/z1-1b-hybrid): A base model where continued pre-training was performed in a fully decentralized way on 8 NVIDIA **H100** GPUs.
- [zettafleet/z1-1b-hybrid-rtx](https://huggingface.co/zettafleet/z1-1b-hybrid-rtx): A base model where continued pre-training was performed in a fully decentralized way on 8 NVIDIA **RTX Pro 6000** GPUs.
- [zettafleet/z1-1b-hybrid-instruct](https://huggingface.co/zettafleet/z1-1b-hybrid-instruct): An instruction model tuned from `z1-1b-hybrid`, using a reconstructed post-training pipeline and datasets from [OLMo 2 1B Instruct](https://huggingface.co/allenai/OLMo-2-0425-1B-Instruct).
The Z1 family of models shares the same architecture:
| Size | Layers | Hidden Size | Attention Heads | Context Length |
|------|---------|-------------|-----------------|----------------|
| [z1-1b-hybrid*](https://huggingface.co/zettafleet/z1-1b-hybrid) | 16 | 2048 | 16 | 4096 |
## Model description
- **Developed by:** Zettafleet Ltd.
- **Contact:** [research@zettafleet.com](mailto:research@zettafleet.com).
- **Model type:** A model trained on a mix of publicly available, synthetic and human-created datasets.
- **Language(s) (NLP):** English.
- **License:** The code and model are released under [Zettafleet Open License](https://zettafleet.com/legal/open-license/), version 1.0 (ZOL-1.0-MIT).
### Model Sources
- **Company page:** https://www.zettafleet.com/
- **Repositories used:**
- Post-training code: https://github.com/allenai/open-instruct
- Evaluation code: https://github.com/allenai/olmes
- **Demo:** [Zettafleet Launch Event](https://www.youtube.com/watch?v=C76ej_TIOJY)
## Using the Model
### Loading with Hugging Face
To load the model with Hugging Face, use the following snippet:
```
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("zettafleet/z1-1b-hybrid-instruct")
```
### Chat Template
We have retained the OLMo 2 chat template which uses the following formatting:
```
<|user|>
How are you doing?
<|assistant|>
I'm just a computer program, so I don't have feelings, but I'm functioning as expected. How can I assist you today?<|endoftext|>
```
### Data Processing
All datasets used for training were processed, tokenized and partitioned with the use of [Zettafleet’s Data Platform](https://www.zettafleet.com/).
### Training Stages of Z1 models
The training stages we carried out are as follows:
1. Continued pre-training:
- Performed in a decentralized way via [Zettafleet’s AI Training Platform](https://www.zettafleet.com/).
- Trained on a mix of high-quality web data and academic/Q&A/instruction/mathematical content [[dataset]](https://huggingface.co/datasets/allenai/dolmino-mix-1124).
2. Post-training (Z1 Hybrid Instruct):
- Performed via [Zettafleet’s AI training platform](https://www.zettafleet.com/) on a mix of data for conversational chatbots, preferences, instruction following and mathematics.
- Performed using a reconstructed version of the [training pipeline](https://github.com/allenai/open-instruct) of [OLMo 2 1B Instruct](https://huggingface.co/allenai/OLMo-2-0425-1B-Instruct), which consists of the following phases:
- Supervised Fine-Tuning (SFT) [[dataset]](https://huggingface.co/datasets/allenai/tulu-3-sft-olmo-2-mixture-0225).
- Direct Preference Optimization (DPO) [[dataset]](https://huggingface.co/datasets/allenai/olmo-2-0425-1b-preference-mix).
- Reinforcement Learning with Verifiable Rewards (RLVR) [[dataset 1]](allenai/RLVR-GSM-MATH-IF-Mixed-Constraints) [[dataset 2]](https://huggingface.co/datasets/allenai/RLVR-MATH).
## Performance
Our hybrid instruction model is competitive with other small models. We have reported results for OLMo 2 1B instruct from both the paper, and our own reproduction attempt, which performed post-training on the [OLMo 2 1B base model](https://huggingface.co/allenai/OLMo-2-0425-1B) using the same [post-training pipeline](https://github.com/allenai/open-instruct) and datasets as the paper.
| **Instruct Model** | **Average** | **DROP** | **GSM8K** | **IFEval** | **MATH** | **MMLU** | **PopQA** |
|---|---|---|---|---|---|---|---|
| **OLMo 2 1B (Paper)** | 41.1 | 34.6 | 68.3 | 70.1 | 20.7 | 40.0 | 12.9 |
| **OLMo 2 1B (Reproduction)** | 38.5 | 30.5 | 62.2 | 68.4 | 12.8 | 44.2 | 13.0 |
| **Z1 1B Hybrid** | 40.4 | 31.6 | 67.0 | 70.4 | 19.1 | 42.6 | 11.4 |
| **Qwen 2.5 1.5B** | 39.9 | 13.4 | 66.2 | 44.2 | 40.6 | 59.7 | 15.5 |
| **LLaMA 3.2 1B** | 35.6 | 32.2 | 45.4 | 54.0 | 21.6 | 46.7 | 13.8 |
| **Gemma 3 1B** | 34.9 | 25.1 | 35.0 | 60.6 | 40.3 | 38.9 | 9.6 |
| **SmolLM2 1.7B** | 33.1 | 30.9 | 45.3 | 51.6 | 20.3 | 34.3 | 16.4 |
## Bias, Risks and Limitations
AI models can be prompted by users to generate harmful and sensitive content. Such content may also be produced unintentionally, especially in cases involving bias, so we recommend that users consider the risks when applying this technology. Additionally, many statements from Z1 or any LLM are often inaccurate, so facts should be verified.
Reference in New Issue View Git Blame Copy Permalink