FuseLLM-112M-Completion/README.md

---
library_name: transformers
license: other  # TODO: set the license you want to release this model under
language:
  - code
tags:
  - qwen3
  - causal-lm
  - code-completion
  - habbo
  - from-scratch
---

# FuseLLM-112M

A small **112M-parameter decoder-only language model trained from scratch** (no base
checkpoint, no LoRA) on a corpus of Habbo emulator / game-server source code. The
goal is a tiny, fast model for **code completion** in that Java codebase, not a
general-purpose or instruction-following model.

## Model details

| | |
|---|---|
| Architecture | Qwen3 (decoder-only causal LM) |
| Parameters | ~112M (tied input/output embeddings) |
| Hidden size | 512 |
| Layers | 8 (all full attention) |
| Attention heads | 8 (8 KV heads) |
| Vocab size | 151,936 |
| Max context | 2048 |
| Precision | float32 (safetensors) |
| Training | From scratch, 4 epochs, 16,188 steps |
| Final train loss | ~0.58 |

`tie_word_embeddings: true` — the output `lm_head` shares the input embedding
matrix, so checkpoints store only one copy. This is expected, not a missing weight.

## Intended use

- **Code completion** for Habbo-style Java server code (raw prompt → continuation).
- Local experimentation / distillation base.

## What it is NOT

- **Not instruction-tuned / not a chat model.** It was trained only on raw source
  code, never on chat/instruction data.
- The Qwen3 ChatML chat template is included (it ships with the tokenizer) for
  tokenizer/tool compatibility, but the model has **not** learned to follow chat
  turns. Passing chat-formatted prompts will produce poor, often repetitive output.
  Use it in **completion mode**, not conversation mode.

## Usage

### transformers (recommended for completion)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

m   = AutoModelForCausalLM.from_pretrained("h4bbo/FuseLLM-112M")
tok = AutoTokenizer.from_pretrained("h4bbo/FuseLLM-112M")

prompt = "public class Room {\n  public void onEnter(Player p) {\n    "
ids = tok(prompt, return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=64, do_sample=False,
                 repetition_penalty=1.1, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
```

### llama.cpp (completion mode)

No GGUF is shipped in this repo. The HF model is **verified** to convert and run in
`llama.cpp`; generate the GGUF locally:

```bash
# 1) convert HF -> lossless fp16 GGUF
python convert_hf_to_gguf.py h4bbo/FuseLLM-112M --outtype f16 \
  --model-name FuseLLM-112M --outfile FuseLLM-112M.fp16.gguf
# (optional) 4-bit quantize
llama-quantize FuseLLM-112M.fp16.gguf FuseLLM-112M.Q4_K_M.gguf Q4_K_M

# 2) completion mode — pass the raw code seed, do NOT use chat/conversation mode.
llama-cli -m FuseLLM-112M.Q4_K_M.gguf -cnv -st --no-jinja \
  -f seed.txt -n 64 --temp 0.0 --repeat-penalty 1.1 --no-display-prompt < /dev/null
```

`--no-jinja` keeps the prompt raw (the embedded chat template exists but the model
isn't chat-tuned, so conversation mode is not meaningful for this model).

## Files

- `model.safetensors`, `config.json`, `generation_config.json` — HF model
- `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja` — tokenizer + ChatML template

## Notes

- Small model + limited-domain corpus: expect repetition on long generations; use
  a repetition penalty and keep continuations short.
- Trained from scratch, so this is fully independent of any upstream Qwen weights.
  The Qwen3 architecture/tokenizer are reused for compatibility.