初始化项目，由ModelHub XC社区提供模型
Model: EphAsad/Atem-SageCoder-1.5B Source: Original Platform
This commit is contained in:
ModelHub XC
42
.gitattributes
vendored
Normal file
42
.gitattributes
vendored
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
|
Atem-Wisdom-1.5B.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||||
|
Atem-Wisdom-1.5B.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||||
|
Atem-Wisdom-1.5B.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||||
|
Atem-SageCoder-1.5B.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||||
|
Atem-SageCoder-1.5B.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||||
|
Atem-SageCoder-1.5B.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||||
3
Atem-SageCoder-1.5B.Q4_K_M.gguf
Normal file
3
Atem-SageCoder-1.5B.Q4_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:71fe0be4099cdaea980e34ca3f2a149443d9b7dc40b607cc13f6691edd8255eb
|
||||||
|
size 986048544
|
||||||
3
Atem-SageCoder-1.5B.Q5_K_M.gguf
Normal file
3
Atem-SageCoder-1.5B.Q5_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:3dec11f94e71c9807169e092b9ea7f136a03c840c7af9194797f1bb427bac510
|
||||||
|
size 1125050400
|
||||||
3
Atem-SageCoder-1.5B.Q8_0.gguf
Normal file
3
Atem-SageCoder-1.5B.Q8_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:83737eb32ada80d125b46933d69d84c04c07453a6fd9af0968b1b91c4cabe0d6
|
||||||
|
size 1646573088
|
||||||
380
README.md
Normal file
380
README.md
Normal file
@@ -0,0 +1,380 @@
|
|||||||
|
---
|
||||||
|
base_model: EphAsad/Atem-Wisdom-1.5B
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
license: apache-2.0
|
||||||
|
tags:
|
||||||
|
- qwen2
|
||||||
|
- unsloth
|
||||||
|
- lora
|
||||||
|
- llama.cpp
|
||||||
|
- reasoning
|
||||||
|
- chain-of-thought
|
||||||
|
- thinking
|
||||||
|
- coding
|
||||||
|
- code-generation
|
||||||
|
- distillation
|
||||||
|
- conversational
|
||||||
|
- text-generation-inference
|
||||||
|
datasets:
|
||||||
|
- nvidia/OpenCodeReasoning
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Atem-SageCoder
|
||||||
|
|
||||||
|
*Ancient logic. Modern intelligence. Applied to code.*
|
||||||
|
|
||||||
|
A 1.5B code reasoning model that thinks before it writes — trained on verified competitive programming traces from frontier models.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Atem-SageCoder is a code-specialised variant of [Atem-Wisdom-1.5B](https://huggingface.co/EphAsad/Atem-Wisdom-1.5B), fine-tuned on verified chain-of-thought coding traces from [nvidia/OpenCodeReasoning](https://huggingface.co/datasets/nvidia/OpenCodeReasoning). It inherits Atem-Wisdom's explicit reasoning capability and applies it specifically to programming tasks — working through algorithm choice, edge cases, and complexity analysis before producing an implementation.
|
||||||
|
|
||||||
|
The core behaviour: when given a coding problem, the model reasons through it fully inside a `<think>` block before writing any code. This makes its reasoning auditable and reduces the frequency of structurally plausible but logically incorrect solutions.
|
||||||
|
|
||||||
|
**When to choose Atem-SageCoder over Atem-Wisdom:**
|
||||||
|
|
||||||
|
- Programming problems where reasoning about approach matters before implementation
|
||||||
|
- Competitive programming and algorithmic tasks
|
||||||
|
- Situations where you want to see the model's design decisions, not just its output
|
||||||
|
- Code that requires edge case analysis or complexity reasoning
|
||||||
|
|
||||||
|
**When to choose Atem-Wisdom instead:**
|
||||||
|
|
||||||
|
- General reasoning, mathematics, and analytical tasks outside of coding
|
||||||
|
- Mixed-domain workloads where code is one of many task types
|
||||||
|
- Environments where output length is a constraint
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Atem Series
|
||||||
|
|
||||||
|
| Model | Stage | Capability | Status |
|
||||||
|
|---|---|---|---|
|
||||||
|
| [Atem v1](https://huggingface.co/EphAsad/Atem-v1-1.5B) | Stage 1 — SFT | Fast, direct reasoning | ✅ |
|
||||||
|
| [Atem-Wisdom](https://huggingface.co/EphAsad/Atem-Wisdom-1.5B) | Stage 2 — CoT | Explicit thinking traces | ✅ |
|
||||||
|
| **Atem-SageCoder** | Specialisation — Code | Think-then-code on algorithms | ✅ |
|
||||||
|
| Atem-Pharaoh *(planned)* | Stage 3 — DPO/IPO | Preference-aligned reasoning | 🔄 |
|
||||||
|
|
||||||
|
Atem-SageCoder is a domain-specialised branch off Atem-Wisdom, not a continuation of the main series progression toward Atem-Pharaoh.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Model Details
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|---|---|
|
||||||
|
| **Base model** | EphAsad/Atem-Wisdom-1.5B |
|
||||||
|
| **Root architecture** | Qwen/Qwen2.5-1.5B-Instruct |
|
||||||
|
| **Training method** | LoRA SFT — Code Reasoning Specialisation |
|
||||||
|
| **LoRA config** | r=32, alpha=64, dropout=0.05 |
|
||||||
|
| **Parameters** | ~1.54B |
|
||||||
|
| **Training records** | 15,427 (after filtering) |
|
||||||
|
| **Think / no-think split** | 90% / 10% |
|
||||||
|
| **Epochs** | 2 |
|
||||||
|
| **Total steps** | 484 |
|
||||||
|
| **Final train loss** | 0.8477 |
|
||||||
|
| **Final val loss** | 0.8591 |
|
||||||
|
| **Hardware** | NVIDIA A100-SXM4 80GB |
|
||||||
|
| **Max sequence length** | 8,192 tokens |
|
||||||
|
| **Precision** | bfloat16 |
|
||||||
|
| **License** | Apache 2.0 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
Atem-SageCoder produces responses in one of two formats:
|
||||||
|
|
||||||
|
**With reasoning trace (90% of training examples):**
|
||||||
|
|
||||||
|
```
|
||||||
|
<think>
|
||||||
|
[Reasoning through the problem — algorithm selection, edge cases,
|
||||||
|
complexity analysis, implementation approach]
|
||||||
|
</think>
|
||||||
|
|
||||||
|
[Final implementation — clean, correct code with explanation]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Direct answer (simple queries):**
|
||||||
|
|
||||||
|
```
|
||||||
|
[Concise code response — no reasoning trace needed]
|
||||||
|
```
|
||||||
|
|
||||||
|
The 10% no-think training pool prevents the model from refusing to answer simple queries without extended reasoning. On straightforward questions it responds directly; the think trace activates proportionally to problem complexity.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Training Data
|
||||||
|
|
||||||
|
Atem-SageCoder was trained on 15,427 examples drawn from `nvidia/OpenCodeReasoning` (split_0), after streaming 40,000 candidates and applying two sequential filters.
|
||||||
|
|
||||||
|
**Filter 1 — Truncation gate:** Records were rejected if `</think>` was absent from the output (CoT cut off mid-trace) or if fewer than 30 characters of code followed `</think>` (code truncated). This is the primary source of attrition — OpenCodeReasoning CoT traces are long, and 8,192 tokens captures roughly 38% of the raw stream.
|
||||||
|
|
||||||
|
**Filter 2 — Bad input gate:** Records with `input` fields under 20 characters were rejected. A known data quality issue in split_1 caused that entire split to be excluded; all training data comes from split_0.
|
||||||
|
|
||||||
|
**Filter 3 — Token length:** Examples exceeding 8,192 tokens after chat template application were removed rather than truncated.
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|---|---|
|
||||||
|
| Dataset | nvidia/OpenCodeReasoning (split_0) |
|
||||||
|
| Streamed | 40,000 |
|
||||||
|
| After truncation filter | ~24,000 |
|
||||||
|
| After token length filter | 15,427 |
|
||||||
|
| Train / Val split | 14,627 / 800 |
|
||||||
|
| Domain | Competitive programming (algorithmic problems) |
|
||||||
|
|
||||||
|
**CoT extraction:** The `output` column in OpenCodeReasoning contains `<think>...</think>code` format. CoT and code were extracted into separate fields before formatting. The `<think>` tags were removed from the raw output to avoid double-tag injection during chat template application, then manually reinserted during `build_text` construction with `enable_thinking=False`.
|
||||||
|
|
||||||
|
**Loss curve:**
|
||||||
|
|
||||||
|
| Step | Train Loss | Val Loss |
|
||||||
|
|---|---|---|
|
||||||
|
| 250 | 0.8564 | 0.8757 |
|
||||||
|
| 484 (final) | **0.8477** | **0.8591** |
|
||||||
|
|
||||||
|
Train/val gap of 0.012 at completion — no overfitting signal. Loss values in the 0.85 range are expected for complex CoT+code targets; simple instruction SFT typically reaches 0.3–0.5, but verified reasoning traces carry genuine entropy.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Training Configuration
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Key hyperparameters
|
||||||
|
lora_r = 32
|
||||||
|
lora_alpha = 64
|
||||||
|
lora_dropout = 0.05
|
||||||
|
max_seq_length = 8192 # doubled vs Atem-Wisdom — CoT traces are long
|
||||||
|
learning_rate = 1e-4
|
||||||
|
lr_scheduler = 'cosine'
|
||||||
|
warmup_ratio = 0.05
|
||||||
|
batch_size = 4 # halved vs Atem-Wisdom to account for 2× seq length
|
||||||
|
grad_accumulation = 16 # effective batch size: 64
|
||||||
|
num_epochs = 2
|
||||||
|
dtype = bfloat16
|
||||||
|
load_in_4bit = True # during training
|
||||||
|
nothink_ratio = 0.10 # 10% direct-answer training pool
|
||||||
|
```
|
||||||
|
|
||||||
|
Training used Unsloth (`unsloth==2026.5.5`, `unsloth_zoo==2026.5.5`) with `train_on_responses_only` masking. Loss was computed exclusively on assistant response tokens. A three-part pre-training validation was run before training: identity confirmation, double `<think>` tag detection, and mask sanity check. All checks passed before training was confirmed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Evaluation
|
||||||
|
|
||||||
|
### Qualitative Coding Evaluation (8 / 30 questions shown)
|
||||||
|
|
||||||
|
Atem-SageCoder was evaluated against a (Qwen/Qwen2.5-1.5B-Instruct) baseline across 30 coding questions covering implementation tasks, concept explanations, and algorithm design. The 8 coding-domain questions from that evaluation are shown below.
|
||||||
|
|
||||||
|
| # | Question | Base | SageCoder | Notes |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 1 | `is_even(n)` function | ✓ No think | ✓ Think | Both correct |
|
||||||
|
| 2 | Count vowels in string | ✓ No think | ✓ Think | SageCoder more Pythonic (generator expression) |
|
||||||
|
| 3 | List vs tuple differences | ✓ No think | ⚠ Think | SageCoder error: claims tuples cannot contain duplicates (incorrect) |
|
||||||
|
| 4 | Sum list without `sum()` | ✓ No think | ✓ Think | SageCoder more thorough, both correct |
|
||||||
|
| 5 | Reverse a string | ✓ No think | ✓ Think | Both correct; SageCoder more verbose |
|
||||||
|
| 6 | `if` / `elif` / `else` | ⚠ No think | ✓ Think | Base error: predicts wrong output for age=25 example |
|
||||||
|
| 7 | `find_max()` with empty list | ✓ No think | ✓ Think | SageCoder provides two implementations |
|
||||||
|
| 8 | `for` vs `while` loop | ✓ No think | ✓ Think | SageCoder more structured |
|
||||||
|
|
||||||
|
**Summary across 8 questions:**
|
||||||
|
|
||||||
|
| Metric | Baseline | Atem-SageCoder |
|
||||||
|
|---|---|---|
|
||||||
|
| Think traces | 0 / 8 | 8 / 8 |
|
||||||
|
| Avg response (words) | ~177 | ~470 |
|
||||||
|
| Factual errors observed | 1 (Q6 output prediction) | 1 (Q3 tuple claim) |
|
||||||
|
| Code correctness | 7 / 8 correct | 7 / 8 correct |
|
||||||
|
|
||||||
|
The think-then-code pattern activates consistently on all coding questions. Response depth increases significantly — SageCoder examines edge cases, considers multiple approaches, and explains implementation choices that the baseline omits. Overall correctness is comparable across these 8 questions; the error types differ (baseline: incorrect output prediction; SageCoder: incorrect concept claim about tuples).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Transformers
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||||
|
import torch
|
||||||
|
|
||||||
|
model_name = "EphAsad/Atem-SageCoder-1.5B"
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
model_name,
|
||||||
|
torch_dtype=torch.bfloat16,
|
||||||
|
device_map="auto"
|
||||||
|
)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": "Write a Python function that finds all prime numbers up to n using the Sieve of Eratosthenes."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
inputs = tokenizer.apply_chat_template(
|
||||||
|
messages,
|
||||||
|
tokenize=True,
|
||||||
|
add_generation_prompt=True,
|
||||||
|
return_tensors="pt"
|
||||||
|
).to(model.device)
|
||||||
|
|
||||||
|
with torch.no_grad():
|
||||||
|
output = model.generate(
|
||||||
|
input_ids=inputs,
|
||||||
|
max_new_tokens=2048,
|
||||||
|
temperature=0.7,
|
||||||
|
top_p=0.9,
|
||||||
|
repetition_penalty=1.1,
|
||||||
|
do_sample=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
response = tokenizer.decode(
|
||||||
|
output[0][inputs.shape[1]:],
|
||||||
|
skip_special_tokens=True
|
||||||
|
)
|
||||||
|
print(response)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Unsloth (faster inference)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from unsloth import FastLanguageModel
|
||||||
|
import torch
|
||||||
|
|
||||||
|
model, tokenizer = FastLanguageModel.from_pretrained(
|
||||||
|
model_name="EphAsad/Atem-SageCoder-1.5B",
|
||||||
|
max_seq_length=8192,
|
||||||
|
dtype=torch.bfloat16,
|
||||||
|
load_in_4bit=True,
|
||||||
|
)
|
||||||
|
FastLanguageModel.for_inference(model)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": "Given an array of integers, find the two numbers that sum to a target value. Return their indices."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
inputs = tokenizer.apply_chat_template(
|
||||||
|
messages,
|
||||||
|
tokenize=True,
|
||||||
|
add_generation_prompt=True,
|
||||||
|
return_tensors="pt"
|
||||||
|
).to("cuda")
|
||||||
|
|
||||||
|
with torch.no_grad():
|
||||||
|
output = model.generate(
|
||||||
|
input_ids=inputs,
|
||||||
|
max_new_tokens=2048,
|
||||||
|
temperature=0.7,
|
||||||
|
top_p=0.9,
|
||||||
|
do_sample=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(tokenizer.decode(output[0][inputs.shape[1]:], skip_special_tokens=True))
|
||||||
|
```
|
||||||
|
|
||||||
|
### Ollama
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Recommended — best speed/quality balance
|
||||||
|
ollama run hf.co/EphAsad/Atem-SageCoder-1.5B:Q4_K_M
|
||||||
|
|
||||||
|
# Higher quality
|
||||||
|
ollama run hf.co/EphAsad/Atem-SageCoder-1.5B:Q5_K_M
|
||||||
|
|
||||||
|
# Near-lossless
|
||||||
|
ollama run hf.co/EphAsad/Atem-SageCoder-1.5B:Q8_0
|
||||||
|
```
|
||||||
|
|
||||||
|
### llama.cpp
|
||||||
|
|
||||||
|
```bash
|
||||||
|
llama-server -hf EphAsad/Atem-SageCoder-1.5B:Q4_K_M
|
||||||
|
```
|
||||||
|
|
||||||
|
### System Prompt
|
||||||
|
|
||||||
|
Atem-SageCoder's identity and coding focus are baked into the chat template. To override manually:
|
||||||
|
|
||||||
|
```
|
||||||
|
You are Atem-SageCoder, a thoughtful programming assistant built on the
|
||||||
|
Atem foundation. You reason carefully through problems before writing code
|
||||||
|
— considering edge cases, algorithm choice, complexity, and implementation
|
||||||
|
details — then provide clean, correct, and well-structured implementations.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Available Files
|
||||||
|
|
||||||
|
| File | Size | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `model.safetensors` | ~3.1 GB | Full bfloat16 merged weights |
|
||||||
|
| `Atem-SageCoder-1.5B.Q4_K_M.gguf` | ~986 MB | 4-bit quantised — recommended |
|
||||||
|
| `Atem-SageCoder-1.5B.Q5_K_M.gguf` | ~1.1 GB | 5-bit quantised |
|
||||||
|
| `Atem-SageCoder-1.5B.Q8_0.gguf` | ~1.6 GB | 8-bit quantised — near-lossless |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Known Limitations
|
||||||
|
|
||||||
|
**Training data scope.** All 15,427 training examples come from competitive programming problems in `nvidia/OpenCodeReasoning`. The model is strongest on algorithmic and data structure problems; general software engineering tasks (web APIs, OOP design, framework-specific code) were not represented in training and may produce lower quality output.
|
||||||
|
|
||||||
|
**Factual concept errors.** The qualitative evaluation identified an incorrect claim about tuples (Q3: stated tuples cannot contain duplicates — they can). Concept explanation accuracy should be independently verified for correctness-critical applications.
|
||||||
|
|
||||||
|
**Response length.** Think traces substantially increase output length. This is a fundamental property of the think-then-code design, not a fixable defect. For latency-constrained environments, Atem-Wisdom-1.5B with direct prompting may be preferable.
|
||||||
|
|
||||||
|
**Single language bias.** OpenCodeReasoning solutions are predominantly Python. Performance on other languages has not been formally evaluated.
|
||||||
|
|
||||||
|
**Small training set.** 15,427 examples is a focused dataset. Coverage of less common algorithmic patterns may be shallow. The high filter attrition rate (40k streamed → 15.4k retained) reflects the strict quality bar applied, not a shortage of data — the full split_0 contains substantially more examples at lower sequence lengths.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Roadmap
|
||||||
|
|
||||||
|
| Stage | Status | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| Stage 1 — SFT | ✅ Complete | Atem v1 — direct reasoning foundation |
|
||||||
|
| Stage 2 — CoT SFT | ✅ Complete | Atem-Wisdom — thinking traces |
|
||||||
|
| Specialisation — Code | ✅ Complete | **Atem-SageCoder — this model** |
|
||||||
|
| Stage 3 — DPO/IPO | 🔄 Planned | Atem-Pharaoh — preference-aligned reasoning |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Citation
|
||||||
|
|
||||||
|
```bibtex
|
||||||
|
@misc{atem_sagecoder_2026,
|
||||||
|
author = {Asad, Zain},
|
||||||
|
title = {Atem-SageCoder: A 1.5B Think-Then-Code Model
|
||||||
|
via Competitive Programming Trace Distillation},
|
||||||
|
year = {2026},
|
||||||
|
publisher = {HuggingFace},
|
||||||
|
howpublished = {\url{https://huggingface.co/EphAsad/Atem-SageCoder-1.5B}},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), consistent with the base model chain (Qwen2.5-1.5B-Instruct → Atem v1 → Atem-Wisdom → Atem-SageCoder).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Built independently by [EphAsad](https://huggingface.co/EphAsad)
|
||||||
54
chat_template.jinja
Normal file
54
chat_template.jinja
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
{%- if tools %}
|
||||||
|
{{- '<|im_start|>system\n' }}
|
||||||
|
{%- if messages[0]['role'] == 'system' %}
|
||||||
|
{{- messages[0]['content'] }}
|
||||||
|
{%- else %}
|
||||||
|
{{- 'You are Atem, a precise and analytical reasoning assistant. You approach every problem methodically — identifying core concepts, reasoning step by step, and arriving at well-supported conclusions. You show your thinking clearly and are thorough, direct, and intellectually honest.' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||||
|
{%- for tool in tools %}
|
||||||
|
{{- "\n" }}
|
||||||
|
{{- tool | tojson }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||||
|
{%- else %}
|
||||||
|
{%- if messages[0]['role'] == 'system' %}
|
||||||
|
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>system\nYou are Atem, a precise and analytical reasoning assistant. You approach every problem methodically — identifying core concepts, reasoning step by step, and arriving at well-supported conclusions. You show your thinking clearly and are thorough, direct, and intellectually honest.<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- for message in messages %}
|
||||||
|
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
|
||||||
|
{%- elif message.role == "assistant" %}
|
||||||
|
{{- '<|im_start|>' + message.role }}
|
||||||
|
{%- if message.content %}
|
||||||
|
{{- '\n' + message.content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- for tool_call in message.tool_calls %}
|
||||||
|
{%- if tool_call.function is defined %}
|
||||||
|
{%- set tool_call = tool_call.function %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_call>\n{"name": "' }}
|
||||||
|
{{- tool_call.name }}
|
||||||
|
{{- '", "arguments": ' }}
|
||||||
|
{{- tool_call.arguments | tojson }}
|
||||||
|
{{- '}\n</tool_call>' }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- elif message.role == "tool" %}
|
||||||
|
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
|
||||||
|
{{- '<|im_start|>user' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_response>\n' }}
|
||||||
|
{{- message.content }}
|
||||||
|
{{- '\n</tool_response>' }}
|
||||||
|
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- if add_generation_prompt %}
|
||||||
|
{{- '<|im_start|>assistant\n' }}
|
||||||
|
{%- endif %}
|
||||||
62
config.json
Normal file
62
config.json
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"Qwen2ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": null,
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"eos_token_id": 151645,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 1536,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 8960,
|
||||||
|
"layer_types": [
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention"
|
||||||
|
],
|
||||||
|
"max_position_embeddings": 32768,
|
||||||
|
"max_window_layers": 21,
|
||||||
|
"model_type": "qwen2",
|
||||||
|
"num_attention_heads": 12,
|
||||||
|
"num_hidden_layers": 28,
|
||||||
|
"num_key_value_heads": 2,
|
||||||
|
"pad_token_id": 151665,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_parameters": {
|
||||||
|
"rope_theta": 1000000.0,
|
||||||
|
"rope_type": "default"
|
||||||
|
},
|
||||||
|
"sliding_window": null,
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"unsloth_fixed": true,
|
||||||
|
"unsloth_version": "2026.5.5",
|
||||||
|
"use_cache": false,
|
||||||
|
"use_sliding_window": false,
|
||||||
|
"vocab_size": 151936
|
||||||
|
}
|
||||||
14
generation_config.json
Normal file
14
generation_config.json
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
151645,
|
||||||
|
151643
|
||||||
|
],
|
||||||
|
"max_length": 32768,
|
||||||
|
"pad_token_id": 151665,
|
||||||
|
"repetition_penalty": 1.1,
|
||||||
|
"temperature": 0.7,
|
||||||
|
"top_k": 20,
|
||||||
|
"top_p": 0.8,
|
||||||
|
"transformers_version": "5.5.0"
|
||||||
|
}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:9bfc24e8d97db266ddb9d7eb5562a15aaaff7dcf7c1e742923878cf010e37c4e
|
||||||
|
size 3087467144
|
||||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:bd5948af71b4f56cf697f7580814c7ce8b80595ef985544efcacf716126a2e31
|
||||||
|
size 11422356
|
||||||
202
tokenizer_config.json
Normal file
202
tokenizer_config.json
Normal file
@@ -0,0 +1,202 @@
|
|||||||
|
{
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"backend": "tokenizers",
|
||||||
|
"bos_token": null,
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|im_end|>",
|
||||||
|
"errors": "replace",
|
||||||
|
"is_local": false,
|
||||||
|
"model_max_length": 32768,
|
||||||
|
"pad_token": "<|PAD_TOKEN|>",
|
||||||
|
"padding_side": "left",
|
||||||
|
"split_special_tokens": false,
|
||||||
|
"tokenizer_class": "Qwen2Tokenizer",
|
||||||
|
"unk_token": null,
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"151643": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151644": {
|
||||||
|
"content": "<|im_start|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151645": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151646": {
|
||||||
|
"content": "<|object_ref_start|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151647": {
|
||||||
|
"content": "<|object_ref_end|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151648": {
|
||||||
|
"content": "<|box_start|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151649": {
|
||||||
|
"content": "<|box_end|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151650": {
|
||||||
|
"content": "<|quad_start|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151651": {
|
||||||
|
"content": "<|quad_end|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151652": {
|
||||||
|
"content": "<|vision_start|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151653": {
|
||||||
|
"content": "<|vision_end|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151654": {
|
||||||
|
"content": "<|vision_pad|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151655": {
|
||||||
|
"content": "<|image_pad|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151656": {
|
||||||
|
"content": "<|video_pad|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151657": {
|
||||||
|
"content": "<tool_call>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151658": {
|
||||||
|
"content": "</tool_call>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151659": {
|
||||||
|
"content": "<|fim_prefix|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151660": {
|
||||||
|
"content": "<|fim_middle|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151661": {
|
||||||
|
"content": "<|fim_suffix|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151662": {
|
||||||
|
"content": "<|fim_pad|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151663": {
|
||||||
|
"content": "<|repo_name|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151664": {
|
||||||
|
"content": "<|file_sep|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151665": {
|
||||||
|
"content": "<|PAD_TOKEN|>",
|
||||||
|
"single_word": false,
|
||||||
|
"lstrip": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"special": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"chat_template": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0]['role'] == 'system' %}\n {{- messages[0]['content'] }}\n {%- else %}\n {{- 'You are Atem, a precise and analytical reasoning assistant. You approach every problem methodically — identifying core concepts, reasoning step by step, and arriving at well-supported conclusions. You show your thinking clearly and are thorough, direct, and intellectually honest.' }}\n {%- endif %}\n {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0]['role'] == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n {%- else %}\n {{- '<|im_start|>system\\nYou are Atem, a precise and analytical reasoning assistant. You approach every problem methodically — identifying core concepts, reasoning step by step, and arriving at well-supported conclusions. You show your thinking clearly and are thorough, direct, and intellectually honest.<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {{- '<|im_start|>' + message.role }}\n {%- if message.content %}\n {{- '\\n' + message.content }}\n {%- endif %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n<tool_call>\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- '}\\n</tool_call>' }}\n {%- endfor %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n<tool_response>\\n' }}\n {{- message.content }}\n {{- '\\n</tool_response>' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n"
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user