108 lines
4.7 KiB
Markdown
108 lines
4.7 KiB
Markdown
---
|
|
library_name: transformers
|
|
tags:
|
|
- agent
|
|
license: llama3.1
|
|
language:
|
|
- en
|
|
base_model:
|
|
- meta-llama/Llama-3.1-8B
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# Telos Llama-3.1-8B (init)
|
|
|
|
A Llama-3.1-8B base model with eleven of its reserved special tokens
|
|
seeded with semantically related-content-token embeddings, in
|
|
preparation for fine-tuning on the [Telos](https://github.com/) agent
|
|
trajectory format.
|
|
|
|
This is **not a fine-tuned agent model.** It is the base model with
|
|
embedding initialization applied. Behavior on any task is identical
|
|
or near-identical to vanilla Llama-3.1-8B-base; the only difference is
|
|
that the eleven Telos reserved tokens now have non-zero embeddings in
|
|
both the input and output matrices.
|
|
|
|
## Model details
|
|
|
|
- **Base model:** `meta-llama/Llama-3.1-8B`
|
|
- **Modification:** in-place initialization of eleven reserved-token rows
|
|
in `embed_tokens` and `lm_head`
|
|
- **Initialization method:** for each Telos marker, the mean of the
|
|
input/output embeddings of 2-3 semantically related content tokens
|
|
- **Tokenizer:** unchanged from the base model
|
|
- **Vocabulary size:** unchanged (128 256)
|
|
## Token mapping
|
|
|
|
The Telos format aliases these eleven reserved tokens to frame markers
|
|
at the string level. The tokenizer in this repo is unchanged from the
|
|
base; aliasing is done by the Telos SDK at encode/decode time.
|
|
|
|
| Telos marker | Reserved token | Token ID | Seed words |
|
|
| ---------------- | --------------------------------- | -------- | --------------------------------------- |
|
|
| `<\|goal\|>` | `<\|reserved_special_token_0\|>` | 128002 | goal, objective, purpose |
|
|
| `<\|mission\|>` | `<\|reserved_special_token_1\|>` | 128003 | mission, task, instruction |
|
|
| `<\|obs\|>` | `<\|reserved_special_token_2\|>` | 128005 | observation, context, environment |
|
|
| `<\|belief\|>` | `<\|reserved_special_token_3\|>` | 128011 | belief, state, knowledge |
|
|
| `<\|plan\|>` | `<\|reserved_special_token_4\|>` | 128012 | plan, strategy, approach |
|
|
| `<\|think\|>` | `<\|reserved_special_token_5\|>` | 128013 | think, reasoning, thought |
|
|
| `<\|action\|>` | `<\|reserved_special_token_6\|>` | 128014 | action, call, tool |
|
|
| `<\|end\|>` | `<\|reserved_special_token_7\|>` | 128015 | end, stop, done |
|
|
| `<\|result\|>` | `<\|reserved_special_token_8\|>` | 128016 | result, output, response |
|
|
| `<\|feedback\|>` | `<\|reserved_special_token_9\|>` | 128017 | feedback, update, progress |
|
|
| `<\|reward\|>` | `<\|reserved_special_token_10\|>` | 128018 | reward, score |
|
|
|
|
## Why initialization was needed
|
|
|
|
In the base Llama-3.1-8B model, all 250 reserved special tokens have
|
|
**all-zero embeddings** in both `embed_tokens` and `lm_head`. They were
|
|
registered as vocabulary entries but never received any pretraining
|
|
gradient.
|
|
|
|
For Telos, this is degenerate: the model cannot read the markers as
|
|
input (zero embedding contributes nothing) and cannot emit them as
|
|
output (zero `lm_head` row → near-zero logit → near-zero probability
|
|
after softmax). Empirically, prompting the base model with a
|
|
Telos-formatted trajectory causes the model to ignore the markers
|
|
entirely and loop on prose content.
|
|
|
|
Mean-of-related-tokens initialization seeds each marker with a sensible
|
|
starting representation. The model still does not understand the Telos
|
|
format - that requires fine-tuning - but the markers now contribute
|
|
meaningful signal to the forward pass and have non-zero output logits.
|
|
|
|
## Intended use
|
|
|
|
This checkpoint is intended as the starting point for fine-tuning on
|
|
Telos-formatted trajectories. Use it the same way you would use the
|
|
plain Llama-3.1-8B base.
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("kosiasuzu/telos-llama-3.1-8b-init")
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
"kosiasuzu/telos-llama-3.1-8b-init",
|
|
torch_dtype="bfloat16",
|
|
device_map="auto",
|
|
)
|
|
```
|
|
|
|
## Out-of-scope use
|
|
|
|
- **Not an agent yet.** This checkpoint has not been trained on any
|
|
agent trajectories. Do not expect it to follow the Telos format
|
|
correctly.
|
|
- **Not an instruction-tuned model.** It inherits all the base-model
|
|
limitations of Llama-3.1-8B (looping on greedy decoding, no
|
|
instruction following).
|
|
- All limitations and biases of Llama-3.1-8B base apply unchanged.
|
|
## License
|
|
|
|
Inherits the Llama 3.1 Community License from the base model. Use of
|
|
this model is subject to that license's terms.
|
|
|
|
## Citation
|
|
|
|
If you build on this, please cite the Telos project and the underlying
|
|
Llama-3.1 model. |