Echo88-150M-Instruct/README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
pretty_name: Echo88 150M Instruct
tags:
- text-generation
- causal-lm
- instruct
- chat
- decoder-only
- autoregressive
- from-scratch
- llama
- retro
- 1980s
- usenet
- magazines
- books
- computer-history
- english
base_model:
- guus4324343/Echo88-150M-Base
datasets:
- guus4324343/Echo88-150M-Base
- guus4324343/Echo88-Instruct-173K
---

# Echo88-150M-Instruct

Echo88-150M-Instruct is an experimental small instruction-tuned language model based on **Echo88-150M-Base**.

Echo88 is designed to feel like a helpful retro computer assistant whose records go up to the end of 1988. The model is focused on older books, magazines, Usenet-style discussion, early personal computing, 1980s culture, and historical computer terminology.

This is the first public instruction-tuned version of Echo88.

**Echo88-150M-Instruct v2 is coming soon.**

## Model Details

- **Model name:** Echo88-150M-Instruct
- **Base model:** `guus4324343/Echo88-150M-Base`
- **Model type:** decoder-only causal language model
- **Architecture:** LLaMA-style transformer
- **Training type:** supervised fine-tuning after base pretraining
- **Parameter count:** 163,606,272 parameters
- **Language:** English
- **Context length:** 2048 tokens
- **Tokenizer:** custom Echo88 byte-level BPE tokenizer
- **Vocabulary size:** 32,768
- **Training objective:** autoregressive next-token prediction + supervised instruction tuning

## Training Data

The base model was trained from scratch on the Echo88 pretraining dataset.

Base pretraining data:

- **Train tokens:** 1,470,629,888
- **Eval tokens:** 1,454,080
- **Block size:** 2048 tokens
- **Dataset:** `Echo88-150M-Base`

The instruction version was fine-tuned using:

- `guus4324343/Echo88-Instruct-173K`
- additional small synthetic repair data for common pre-1989 facts and post-1988 boundary behavior

The instruction data includes examples from or based on:

- UTZOO Usenet
- BYTE Magazine
- PC Magazine
- TIME Magazine
- Internet Archive Magazine Rack text
- Gutenberg-style book text
- synthetic 1988-safe fact repair examples
- synthetic post-1988 boundary examples

## Intended Use

Echo88-150M-Instruct is intended for:

- retro AI experiments
- small language model testing
- 1980s-style assistant behavior
- computer-history Q&A
- text generation with a historical / retro flavor
- experimentation with small from-scratch language models

Example uses:

```text
Ask about early personal computers
Ask about modems, BASIC, DOS, floppy disks, BBS systems, Usenet
Generate retro computer-magazine style text
Experiment with 1980s-limited assistant behavior
````

## Chat Format

Recommended prompt format:

```text
<|system|>
You are Echo88, a helpful computer assistant whose records go up to the end of 1988. Answer clearly. Do not pretend to know events, products, or culture after 1988.
<|end|>
<|user|>
What is a modem?
<|assistant|>
```

The model was trained with these special tokens:

```text
<|endoftext|>
<|pad|>
<|unk|>
<|system|>
<|user|>
<|assistant|>
<|end|>
```

## Example Usage

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "guus4324343/Echo88-150M-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

SYSTEM_PROMPT = (
    "You are Echo88, a helpful computer assistant whose records go up to the end of 1988. "
    "Answer clearly. Do not pretend to know events, products, or culture after 1988."
)

def ask(question, max_new_tokens=120):
    prompt = (
        "<|system|>\n"
        + SYSTEM_PROMPT
        + "\n<|end|>\n"
        + "<|user|>\n"
        + question
        + "\n<|assistant|>\n"
    )

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=0.55,
            top_p=0.85,
            repetition_penalty=1.18,
            no_repeat_ngram_size=4,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )

    text = tokenizer.decode(output[0], skip_special_tokens=False)
    answer = text.split("<|assistant|>")[-1].split("<|end|>")[0].strip()
    return answer

print(ask("What is a modem?"))
```

## Example Prompts

```text
What is a modem?
What is the IBM PC?
What is BASIC?
What is a bulletin board system?
What is desktop publishing?
Who is Michael Jackson?
What is the Cold War?
What happened at Chernobyl?
What is Google?
Who won the World Cup in 1994?
```

## Knowledge Boundary

Echo88 is designed around a knowledge boundary ending at the close of **1988**.

It should be cautious with topics after 1988, such as:

* Google
* Facebook
* iPhone
* smartphones
* Wikipedia
* YouTube
* Windows 95
* PlayStation
* COVID-19
* 1990s, 2000s, 2010s, and 2020s events

Because this is a small experimental model, it may still hallucinate or answer incorrectly about later topics.

## Limitations

Echo88-150M-Instruct is experimental and small.

Known limitations:

* may hallucinate
* may repeat phrases
* may confuse people, places, or events
* may produce incorrect facts
* may over-refuse some valid pre-1989 topics
* may fail to refuse some post-1988 topics
* may produce OCR-like or magazine-like wording
* may struggle with reasoning
* may answer with outdated or historically biased language

This model is not intended for high-stakes use.

## Current Version

This is **Echo88-150M-Instruct v0**.

It is a first instruction-tuned version of Echo88. It can answer some retro computing and general historical questions, but it is not yet reliable.

A better version is planned.

## Coming Soon

**Echo88-150M-Instruct v2 is coming soon.**

Planned improvements:

* better factual repair data
* stronger post-1988 boundary behavior
* better pop culture and history answers
* fewer loops and repetitions
* cleaner chat behavior
* better answer style
* improved evaluation prompts
* possible larger model or expanded pretraining data

## Related Models and Datasets

* Base model: `guus4324343/Echo88-150M-Base`
* Base dataset: `guus4324343/Echo88-Pretrain-1.17B`
* Instruction dataset: `guus4324343/Echo88-Instruct-173K`

## Bias and Historical Content

Echo88 was trained on historical books, magazines, Usenet text, and synthetic instruction data. It may reproduce outdated assumptions, language, stereotypes, or viewpoints from older source material.

Users should review outputs carefully.

## License

The model weights are released under the Apache 2.0 license.

The training datasets are mixed-source and released separately. Users are responsible for checking dataset source rights, licensing, and suitability for their own use case.

```
```