135 lines
6.3 KiB
Markdown
135 lines
6.3 KiB
Markdown
---
|
||
library_name: transformers
|
||
license: apache-2.0
|
||
base_model: h4bbo/FuseLLM-112M-Completion
|
||
tags:
|
||
- habbo
|
||
- code
|
||
- chatml
|
||
- sft
|
||
language:
|
||
- en
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# FuseLLM-112M (Chat)
|
||
|
||
This is the **ChatML chat** variant of FuseLLM-112M — a 112M-parameter, from-scratch
|
||
Qwen3-architecture decoder-only model supervised-fine-tuned (SFT) on on-domain Habbo instruction
|
||
pairs derived **from the Habbo source corpus itself** (no external/teacher data).
|
||
|
||
The base model, [h4bbo/FuseLLM-112M-Completion](https://huggingface.co/h4bbo/FuseLLM-112M-Completion),
|
||
was trained only on raw Habbo code in completion mode. Its tokenizer already shipped a Qwen3
|
||
ChatML template, but the weights had never seen a chat turn, so chat-formatted prompts produced
|
||
poor, repetitive output. This checkpoint teaches the weights to follow ChatML turns and to **emit
|
||
`<|im_end|>`** at the end of an assistant answer (the turn terminator — conveniently the same token,
|
||
151645, the base model was trained to use as a document boundary), which is what stops the runaway
|
||
repetition.
|
||
|
||
## Use
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model_id = "h4bbo/FuseLLM-112M"
|
||
tok = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_id, torch_dtype=torch.bfloat16, device_map="auto", attn_implementation="sdpa")
|
||
|
||
messages = [
|
||
{"role": "system", "content": "You are a Habbo Hotel emulator code assistant. Reply with concise, correct code or a brief explanation."},
|
||
{"role": "user", "content": "Implement this Java method:\n```java\npublic static void sendRoomPacket(Session s, int header) { }\n```"},
|
||
]
|
||
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
ids = tok(text, return_tensors="pt").to(model.device)
|
||
out = model.generate(**ids, max_new_tokens=256, eos_token_id=151645, pad_token_id=151643,
|
||
do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.05)
|
||
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
It is designed for Habbo-coding instructions (method/class completion, code continuation,
|
||
doc→code). It is **not** a general assistant — outside its narrow domain it will produce poor or
|
||
repetitive output.
|
||
|
||
## Architecture
|
||
|
||
`Qwen3ForCausalLM`, hidden 512, 8 layers (full attention), 8 heads / 8 KV heads, intermediate
|
||
1408, vocab 151,936, max context 2048, `tie_word_embeddings=true`. Trained in bf16.
|
||
|
||
## Training data
|
||
|
||
10,000 ChatML conversations derived deterministically from the Habbo corpus (the same corpus the
|
||
base was trained on — 84,925 unique files, ~210M est. tokens; top languages Java 26,335 /
|
||
C# 19,048 / PHP 10,969 / ActionScript 3,801). No external data, no teacher model. Templates:
|
||
|
||
- **Method completion** — given a method signature with empty body, return the real body.
|
||
- **Code continuation** — given a file prefix, return the real suffix.
|
||
- **Doc → method** — given a Javadoc/PHPDoc/`///` summary, return the real method.
|
||
|
||
Distribution in this build: Java ~4,936 · C# ~3,752 · PHP ~989 · ActionScript ~323. Decompiler
|
||
noise (JD-Core `/* N:M */` line markers and `/* Location: … */` footers) is stripped. Secrets are
|
||
scrubbed to `[REDACTED]` (currently `xX!elgps`) before extraction; the training file is verified to
|
||
contain 0 secret occurrences.
|
||
|
||
## Training config
|
||
|
||
- Full fine-tune (no LoRA — 112M is small enough to train every weight on 24 GB).
|
||
- TRL `SFTTrainer` + `SFTConfig`, `messages` format auto-detected, ChatML applied by the base
|
||
tokenizer's chat template. `packing=False`, `max_length=2048` (model's native max position; only
|
||
11/10,000 conversations exceed it). Full-sequence causal-LM loss (the Qwen3 chat template has no
|
||
`{% generation %}` markers, so `assistant_only_loss` is unset — at inference we prompt through
|
||
`<|im_start|>assistant\n` and stop at `<|im_end|>`).
|
||
- bf16, sdpa attention, `adamw_torch`, cosine schedule, 3 epochs, LR 2e-5, warmup 0.03,
|
||
effective batch 16 (BS 4 × GA 4), `max_grad_norm=1.0`, seed 42. 1,875 steps.
|
||
- `generation_config.json` is written with `eos_token_id=151645` (`<|im_end|>`),
|
||
`pad_token_id=151643`, `temperature=0.7`, `top_p=0.9`, `repetition_penalty=1.05`.
|
||
|
||
## Training result
|
||
|
||
Loss dropped from ~1.83 (step 10) to ~0.45 by the end of epoch 1 and held in the ~0.4–0.5 range
|
||
through epochs 2–3. Final: `train_loss 0.521`, `mean_token_accuracy 0.9144`, 1,875 steps, ~16.5 min
|
||
on a single RX 7900 XTX (ROCm). Verified behavior: method-completion and code-continuation prompts
|
||
produce coherent on-domain Habbo code, close the fenced block, and **stop at `<|im_end|>`**;
|
||
doc→method and free-form explanation prompts tend to ramble (see Limitations).
|
||
|
||
## Redaction
|
||
|
||
Secrets (currently `xX!elgps`) are scrubbed to `[REDACTED]` in all training content before
|
||
tokenisation. The output training file is checked to contain 0 occurrences. No known credentials
|
||
enter the weights.
|
||
|
||
## License
|
||
|
||
Released under **Apache-2.0**. See the base model
|
||
[h4bbo/FuseLLM-112M-Completion](https://huggingface.co/h4bbo/FuseLLM-112M-Completion) for its
|
||
license terms.
|
||
|
||
## GGUF
|
||
|
||
A non-quantized **bf16** GGUF — `FuseLLM-112M.bf16.gguf` (~220 MB, a bit-exact copy of the bf16
|
||
safetensors weights, so truly lossless) — is included in this repo for use with
|
||
[llama.cpp](https://github.com/ggml-org/llama.cpp) / [Ollama](https://ollama.com). The ChatML chat
|
||
template and the EOS token (`<|im_end|>`, 151645) are embedded as GGUF metadata, so the model loads
|
||
in chat mode automatically. No quantized (Q4/Q5/Q8) variant is shipped here.
|
||
|
||
Example with llama.cpp:
|
||
|
||
```bash
|
||
llama-cli -m FuseLLM-112M.bf16.gguf -cnv \
|
||
--temp 0.7 --top-p 0.9 --repeat-penalty 1.05 -n 256 \
|
||
-p "Implement this Java method:\n```java\npublic static void sendRoomPacket(Session s, int h) { }\n```"
|
||
```
|
||
|
||
## Intended use
|
||
|
||
Domain-specialist code assistant for the Habbo Hotel emulator ecosystem (server/client tooling).
|
||
Not affiliated with or endorsed by Sulake/Habbo.
|
||
|
||
## Limitations
|
||
|
||
- 112M parameters — narrow capacity; expect errors and repetition on long or off-domain prompts.
|
||
- Trained only on on-domain code-instruction pairs; not a general chat / instruction model.
|
||
- Doc→method and free-form explanation prompts often ramble past `<|im_end|>` despite
|
||
`repetition_penalty`; keep `max_new_tokens` modest and prefer the SFT templates
|
||
(method completion / code continuation). |