orb-human-typeahead-350m-v1.2/README.md

---
license: apache-2.0
base_model: ibm-granite/granite-4.0-350m-base
pipeline_tag: text-generation
language:
- en
tags:
- typeahead
- autocomplete
- roleplay
- granite
- gguf
library_name: transformers
---

# orb-human-typeahead-350m-v1.2

A 350M typeahead model that predicts **the human's next few words** in a
roleplay chat — inline "ghost text" for the person typing, not a reply
generator for the character. Full fine-tune of
[ibm-granite/granite-4.0-350m-base](https://huggingface.co/ibm-granite/granite-4.0-350m-base).

Most small LMs asked to continue a user's half-typed roleplay message produce
fluent but irrelevant text. This model is trained specifically on
(conversation context + partial user message → the words the user actually
typed next), so its suggestions stay on-scene and in-voice.

## Variants

| Path | Format | Use with |
|---|---|---|
| `/` | safetensors bf16, plain `granite` arch | transformers |
| `GGUF/orb-human-typeahead-350m-v1.2-Q8_0.gguf` | GGUF Q8_0 (default) | llama.cpp / llama-cpp-python |

The original fine-tune used the `granitemoehybrid` architecture; since this
variant is attention-only and dense, the weights are republished here as the
equivalent plain `granite` architecture (logit-identical, verified), which
loads everywhere without extras.

## Prompt format

Plain text, no chat template. Optional character summary, a marker line,
name-prefixed turns, and finally the user's draft — the model continues the
draft. Cut the suggestion at the first newline.

```
<character summary, optional>
***Roleplay chat below***
Sylvara: *She looks down from the watchtower and sees you.*
Traveler: *I approach the encampment.*
Sylvara: *She lowers her bow as you approach the gate.* "State your business, traveler."
Traveler: *I raise both hands slowly and
```

A completion looks like ` step into the torchlight, keeping my voice low.*`

Notes:
- Trained to trigger at **word boundaries** only (draft ends on a whole word,
  or on a trailing space). Mid-word completion is out of scope.
- Trained context: up to ~4 recent turns, summaries ≤400 chars, turns ≤500 chars.

## Serving recipe

Greedy, short budget, stop at newline — mirrors how it was trained and evaluated:

```python
from llama_cpp import Llama

llm = Llama("GGUF/orb-human-typeahead-350m-v1.2-Q8_0.gguf", n_ctx=1024)
out = llm.create_completion(prompt, max_tokens=12, stop=["\n"], temperature=0.0)
print(out["choices"][0]["text"])
```

Or with transformers:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("chartreuse-verte/orb-human-typeahead-350m-v1.2")
model = AutoModelForCausalLM.from_pretrained("chartreuse-verte/orb-human-typeahead-350m-v1.2")
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=12, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).split("\n")[0])
```

## Evaluation

Scored on a held-out validation set of roleplay conversations (120
prompts, conversation-disjoint from training; greedy, 12-token budget,
suggestions cut at newline). word-EM@k = fraction of prompts where the first
k words of the suggestion exactly match what the user really typed next;
prefix-chars = mean length of the exactly-matching leading characters.

| Metric | orb-human-typeahead-350m-v1.1 | this model |
|---|---|---|
| word-EM@1 | 0.333 | **0.358** |
| word-EM@2 | 0.133 | **0.150** |
| word-EM@3 | 0.075 | 0.075 |
| prefix-chars | 3.92 | **4.12** |
| completion perplexity | **8.83** | 9.00 |

v1.2 is a data-only refresh of v1.1 (same 350M granite base) whose purpose was to
flatten the completion distribution further — to stop the model defaulting to a small
set of high-frequency phrasings. The validation set is itself drawn from the
generator-styled distribution, so its gold completions over-represent exactly those
defaults; a flatter model is expected to pay a small price here. v1.2 holds or gains
on every exact-match column anyway (word-EM@1 0.333 → 0.358), with perplexity
essentially unchanged (8.83 → 9.00).

Read the table as a no-regression check on fluency and next-word accuracy, not as a
quality ranking — exact agreement with one recorded continuation cannot score
suggestion variety, and creative roleplay is high-entropy enough that many suggestions
the metric counts as misses are perfectly good ones.

## Training

Two-stage full fine-tune of granite-4.0-350m-base: stage 1 on a mostly-synthetic
roleplay corpus (self-chat generated to cover the on-scene, in-voice register real
users type in), then stage 2 on a smaller private in-domain set in the serve-time
prompt format (with stage-1 replay to limit forgetting). Over v1.1, stage 1's completion
distribution was flattened further: a 3-gram repetition cap retires canned mid-phrase
filler that the earlier first-word cap could not see, post-dialogue beats got real
supervision instead of being capped into starvation, and generator punctuation tells
(em-dashes, curly quotes) plus non-English leakage are normalized out rather than dropped. Examples are user turns split at word boundaries; loss is on
the continuation. Suggestions are single-line by construction (completions end
at newline).

## Limitations

- English-centric, roleplay register (asterisk actions, quoted dialogue).
  Out of domain for assistant chat, code, or formal prose.
- Roleplay corpora include mature themes; suggestions can reflect that.
  Intended as a typing aid for consenting adult users of RP chat apps.
- Not an instruction follower — it only continues drafts in the format above.
- Suggests at word boundaries; won't complete a half-typed word.