110 lines
3.6 KiB
Markdown
110 lines
3.6 KiB
Markdown
---
|
|
language:
|
|
- en
|
|
license: apache-2.0
|
|
library_name: transformers
|
|
tags:
|
|
- toke
|
|
- code-generation
|
|
- programming-language
|
|
- qwen2
|
|
- qlora
|
|
- fine-tuned
|
|
- awq
|
|
- 4bit
|
|
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
|
|
pipeline_tag: text-generation
|
|
model-index:
|
|
- name: toke-7b-gate2
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: Code Generation
|
|
metrics:
|
|
- name: Compilation Pass@1
|
|
type: pass@1
|
|
value: 100
|
|
verified: true
|
|
- name: Functional Pass@1
|
|
type: pass@1
|
|
value: 8
|
|
verified: true
|
|
---
|
|
|
|
# toke-7b-gate2
|
|
|
|
A 7B parameter language model fine-tuned to generate code in **toke**, a programming language designed to reduce token cost of AI-generated code. This model writes syntactically valid toke **100% of the time**.
|
|
|
|
## Model Details
|
|
|
|
| Property | Value |
|
|
|---|---|
|
|
| **Base model** | [Qwen 2.5 Coder 7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) |
|
|
| **Method** | QLoRA (rank 64, alpha 128, 3 epochs) |
|
|
| **Training data** | 25,953 records — 18,890 synthetic + 6,069 from [loke](https://github.com/karwalski/loke) production (87K lines) |
|
|
| **Training time** | 37 hours on NVIDIA A10G (24 GB) |
|
|
| **Weights** | AWQ 4-bit quantized (this repo) |
|
|
| **Context length** | 32,768 tokens |
|
|
| **License** | Apache 2.0 |
|
|
|
|
|
|
## What is toke?
|
|
|
|
toke is a statically typed, compiled language with a **55-character alphabet** (lowercase a-z, digits 0-9, and 19 symbols). It compiles to native binaries via LLVM. A purpose-built BPE tokenizer achieves **52% fewer tokens** on average vs cl100k_base.
|
|
|
|
- **13 keywords:** `m` `f` `t` `i` `if` `el` `lp` `br` `let` `mut` `as` `rt` `mt`
|
|
- **No comments** in source — documentation lives in companion files (.tkc.md)
|
|
- **Errors as values** — no exceptions, result types with `mt` (match)
|
|
- **Website:** [tokelang.dev](https://tokelang.dev) | **Console:** [console.tokelang.dev](https://console.tokelang.dev)
|
|
|
|
## Gate 2 Results (May 2026)
|
|
|
|
| Metric | Gate 1 | Gate 2 |
|
|
|---|---|---|
|
|
| Compilation Pass@1 | 63.7% | **100%** |
|
|
| Tasks evaluated | 1,000 | 700 |
|
|
| Functional Pass@1 | — | ~8% |
|
|
| Training records | 73,000 | 25,953 |
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("karwalski/toke", device_map="auto")
|
|
tokenizer = AutoTokenizer.from_pretrained("karwalski/toke")
|
|
|
|
prompt = """<|im_start|>system
|
|
Write toke programs. m=mod; f=name(p:type):ret{body}; let x=42; <expr return. Semicolons everywhere. Start with m=.
|
|
<|im_end|>
|
|
<|im_start|>user
|
|
Write a hello world program
|
|
<|im_end|>
|
|
<|im_start|>assistant
|
|
"""
|
|
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.2, do_sample=True)
|
|
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
|
# m=hello;i=io:std.io;f=main():i64{io.println("hello world");<0};
|
|
```
|
|
|
|
## API Access
|
|
|
|
Free API access via [console.tokelang.dev](https://console.tokelang.dev) — no credit card required.
|
|
|
|
```bash
|
|
curl -X POST https://api.tokelang.dev/v1/generate \
|
|
-H "X-Api-Key: YOUR_KEY" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"description": "Return the absolute value of an integer"}'
|
|
```
|
|
|
|
## Links
|
|
|
|
- [tokelang.dev](https://tokelang.dev) — Project website
|
|
- [console.tokelang.dev](https://console.tokelang.dev) — Free API access
|
|
- [GitHub: toke](https://github.com/karwalski/toke) — Compiler, spec, stdlib
|
|
- [GitHub: loke](https://github.com/karwalski/loke) — 87K lines of production toke
|
|
- [Live tokenizer](https://tokelang.dev/tokenizer) — Compare token counts in-browser
|