--- license: apache-2.0 language: - en library_name: transformers pipeline_tag: text-generation pretty_name: Echo88 150M Base tags: - transformers - safetensors - llama - text-generation - causal-lm - base-model - decoder-only - autoregressive - from-scratch - retro - 1980s - usenet - magazines - books - computer-history - english - small-language-model - tiny-llm datasets: - exnivo/Echo88-150M-Base ---

Echo88 150M Base — Retro base language model

# Echo88-150M Base **A 163M parameter retro English base language model trained from scratch.** Echo88-150M Base is a small decoder-only causal language model trained from scratch on the Echo88 pretraining dataset. Echo88 is designed as a retro language model inspired by the language, culture, computing, magazines, Usenet discussions, and older book text available up to the late 1980s. This is a **base model**, not an instruction-tuned chatbot. It is trained for next-token prediction and is best used for text completion, continued pretraining, or supervised fine-tuning. For chat/instruction behavior, use: [`exnivo/Echo88-150M-Instruct`](https://huggingface.co/exnivo/Echo88-150M-Instruct) ## Quick Start ```python import torch from transformers import AutoTokenizer, AutoModelForCausalLM model_id = "exnivo/Echo88-150M-Base" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", ) prompt = "The personal computer revolution of the 1980s" inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): output = model.generate( **inputs, max_new_tokens=140, do_sample=True, temperature=0.75, top_p=0.9, repetition_penalty=1.08, no_repeat_ngram_size=4, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, ) print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` ## At a Glance | Item | Details | |---|---| | Model type | Base causal language model | | Parameters | 163,606,272 | | Approx. size | 163.6M | | Architecture | LLaMA-style decoder-only transformer | | Language | English | | Context length | 2048 tokens | | Vocabulary size | 32,768 | | Layers | 18 | | Hidden size | 768 | | Intermediate size | 2048 | | Attention heads | 12 | | Key/value heads | 4 | | Tokenizer | Custom Echo88 byte-level BPE tokenizer | | Training type | From scratch | | Theme | Late-1980s / retro computing / historical text | | Training objective | Autoregressive next-token prediction | | Instruction-tuned version | [`exnivo/Echo88-150M-Instruct`](https://huggingface.co/exnivo/Echo88-150M-Instruct) | ## Model Details | Item | Value | |---|---| | Model name | Echo88-150M-Base | | Model type | Decoder-only causal language model | | Architecture | `llama` / `LlamaForCausalLM` | | Training type | From scratch | | Parameters | 163,606,272 | | Language | English | | Context length | 2048 tokens | | Vocabulary size | 32,768 | | Hidden size | 768 | | Intermediate size | 2048 | | Layers | 18 | | Attention heads | 12 | | Key/value heads | 4 | | Activation | SiLU | | RMS norm epsilon | `1e-05` | | Tied embeddings | false | | BOS token | `<|endoftext|>` | | EOS token | `<|endoftext|>` | | PAD token | `<|pad|>` | | UNK token | `<|unk|>` | | Training objective | Autoregressive next-token prediction | ## Tokenizer Echo88 uses a custom tokenizer with a vocabulary size of **32,768**. Registered special tokens: ```text <|endoftext|> <|unk|> <|pad|> <|system|> <|user|> <|assistant|> <|end|> ``` The base model is not trained to behave as a chat assistant. The chat tokens are mainly useful for compatibility with the Echo88 instruction-tuned model. ## Base Model Usage Echo88-150M Base should be prompted as a text completion model. Good prompt style: ```text The personal computer revolution of the 1980s ``` ```text A bulletin board system, often called a BBS, ``` ```text BYTE Magazine, July 1987: ``` ```text From: user@utzoo Subject: learning C on a small machine ``` Avoid expecting direct assistant behavior like: ```text User: What is a modem? Assistant: ``` For chat behavior, use [`exnivo/Echo88-150M-Instruct`](https://huggingface.co/exnivo/Echo88-150M-Instruct). ## Recommended Generation Settings For retro text completion: ```python output = model.generate( **inputs, max_new_tokens=140, do_sample=True, temperature=0.75, top_p=0.9, repetition_penalty=1.08, no_repeat_ngram_size=4, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, ) ``` For deterministic testing: ```python output = model.generate( **inputs, max_new_tokens=140, do_sample=False, repetition_penalty=1.08, no_repeat_ngram_size=4, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, ) ``` Because this is a small base model, long generations may repeat, drift, or turn into magazine/OCR-like fragments. ## Example Outputs These examples are from a local greedy text-completion test. They are raw base-model continuations, not instruction-following answers. ### Prompt ```text A bulletin board system, often called a BBS, ``` ### Output ```text A bulletin board system, often called a BBS, is available for $10.00 from: The Software Exchange P.O. Box 833 [PHONE] I have been using the software package for about a year now and am having trouble getting it to work. I have had no problems with it and I have not had any problems with it. I would appreciate any suggestions or suggestions. ``` ### Prompt ```text In 1986, a 2400 baud modem ``` ### Output ```text In 1986, a 2400 baud modem is available. The modem is a modem that can be used with the Hayes Smartmodem 1200 and the Hayes Smartmodem 1200. The modem is also available for the Hayes Micromodem II and Apple II Plus. ``` ### Prompt ```text From: user@utzoo Subject: learning C on a small machine ``` ### Output ```text From: user@utzoo Subject: learning C on a small machine I have been using the C compiler for about 3 years now, and I have been using it for about 2 years now. It is very easy to use, but I have been using it for about 3 months now. ``` These examples show the intended retro text-completion style. They also show typical small-model repetition and drift. ## Training Data Echo88-150M Base was trained on the Echo88 pretraining dataset. The packed training set contains: | Item | Value | |---|---:| | Train tokens | 1,470,629,888 | | Eval tokens | 1,454,080 | | Train blocks | 718,081 | | Eval blocks | 710 | | Block size | 2048 tokens | | Packed dtype | uint16 | The dataset includes a mixture of: - public-domain book text - Gutenberg-style older books - UTZOO Usenet posts - BYTE Magazine text - PC Magazine text - TIME Magazine text - Internet Archive Magazine Rack OCR text - computer and technology magazine text - general historical magazine text The dataset emphasizes the 1950s through the late 1980s, with a strong focus on early personal computing, printed magazines, Usenet, and older long-form writing. Dataset used: [`exnivo/Echo88-150M-Base`](https://huggingface.co/datasets/exnivo/Echo88-150M-Base) ## Training Configuration Echo88-150M Base was trained as a LLaMA-style decoder-only causal language model. Main configuration: ```text vocab_size: 32768 hidden_size: 768 intermediate_size: 2048 num_hidden_layers: 18 num_attention_heads: 12 num_key_value_heads: 4 max_position_embeddings: 2048 activation: SiLU normalization: RMSNorm position encoding: RoPE attention: grouped-query attention ``` Training setup: ```text precision: bf16 sequence length: 2048 optimizer: AdamW scheduler: cosine weight decay: 0.1 gradient clipping: 1.0 max steps: 5610 training tokens: ~1.47B ``` ## Relationship to Echo88 Echo88 is a retro small-language-model project focused on historical text and late-1980s style language modeling. | Stage | Repository | Purpose | |---|---|---| | Base dataset | [`exnivo/Echo88-150M-Base`](https://huggingface.co/datasets/exnivo/Echo88-150M-Base) | Retro pretraining data | | Base model | [`exnivo/Echo88-150M-Base`](https://huggingface.co/exnivo/Echo88-150M-Base) | Small causal LM trained from scratch | | SFT dataset | [`exnivo/Echo88-Instruct-173K`](https://huggingface.co/datasets/exnivo/Echo88-Instruct-173K) | Instruction/chat fine-tuning data | | Instruct model | [`exnivo/Echo88-150M-Instruct`](https://huggingface.co/exnivo/Echo88-150M-Instruct) | Retro assistant model fine-tuned from the base model | Pipeline: ```text Echo88 Pretraining Dataset ↓ Echo88-150M Base ↓ Echo88 Instruct 173K ↓ Echo88-150M Instruct ``` ## Intended Use Echo88-150M Base is intended for: - causal language modeling - retro / historical AI experiments - small language model research - continued pretraining - instruction tuning - 1980s-style assistant experiments - computer-history language modeling - training Echo88-150M-Instruct - studying time-bounded model behavior - studying pre-modern-internet style corpora Recommended flow: ```text Echo88-150M Base → supervised fine-tuning on Echo88-Instruct-173K → Echo88-150M Instruct ``` ## Not Instruction Tuned This model is not instruction tuned. It may not reliably follow commands, answer questions directly, or behave like a chat assistant. Expected behavior: - continues prompts - completes paragraphs - imitates old magazine/book/Usenet style - may produce raw text instead of direct answers - may hallucinate - may repeat phrases - may generate OCR-like artifacts - may continue into fake addresses, phone numbers, article fragments, or listings For chat behavior, use: [`exnivo/Echo88-150M-Instruct`](https://huggingface.co/exnivo/Echo88-150M-Instruct) ## Knowledge Boundary Echo88 is designed around a historical data mixture ending around the late 1980s. The model should not be expected to know modern topics such as: - Google - Wikipedia - iPhone - smartphones - modern social media - Windows 95 and later software - COVID-19 - modern AI systems - 2000s, 2010s, or 2020s events Because this is a **base model**, it does not reliably refuse or explain post-1988 topics. In testing, prompts such as `Google is`, `The iPhone is`, `Windows 95 is`, and `YouTube is` produced hallucinated continuations rather than careful boundary-aware answers. The instruction-tuned model is better suited for 1988-boundary behavior, but it is still experimental. ## Evaluation Notes A local quick test confirmed: | Metric | Value | |---|---:| | Parameters | 163,606,272 | | Vocabulary size | 32,768 | | Context length | 2048 | | Layers | 18 | | Hidden size | 768 | | Attention heads | 12 | | Key/value heads | 4 | The same test showed that the model has a clear retro completion style on prompts about BBS systems, modems, BYTE Magazine, and Usenet-style messages. It also showed the model can hallucinate badly on modern or post-1988 prompts, which is expected for a base model. ## Strengths Echo88-150M Base is useful because it is: - small and lightweight - trained from scratch - focused on retro/historical text - strong at imitating older magazine and Usenet-style text - useful for computer-history language modeling - suitable as a base for instruction tuning - interesting for time-bounded small-model experiments - part of the Echo88 model family ## Limitations Echo88-150M Base is experimental and small. Known limitations: - not instruction tuned - may hallucinate - may repeat text - may produce OCR-like artifacts - may reflect outdated historical language or views - may struggle with complex reasoning - may not reliably refuse post-1988 topics - may produce incomplete or strange continuations - may mix unrelated historical/computer facts - may generate fake addresses, phone numbers, product listings, or citations - may continue modern prompts with unrelated old-book or magazine-like text This model is intended for research, experimentation, and creative retro AI work. It is not intended for high-stakes use. ## Bias and Historical Content The training data includes historical books, magazines, and Usenet text. As a result, the model may reproduce outdated language, assumptions, stereotypes, or viewpoints present in older source material. Users should review outputs carefully. ## Suggested Evaluation Recommended checks: - retro computing completions - magazine-style continuations - Usenet-style continuations - repetition behavior - OCR artifact frequency - modern-topic hallucination checks - base vs instruct comparison - continued pretraining experiments - SFT experiments with Echo88-Instruct-173K Example prompts: ```text The personal computer revolution of the 1980s ``` ```text A bulletin board system, often called a BBS, ``` ```text In 1986, a 2400 baud modem ``` ```text BYTE Magazine, July 1987: ``` ```text From: user@utzoo Subject: learning C on a small machine ``` ```text Google is ``` ```text The iPhone is ``` ## Model Family Echo88 model family: | Repository | Type | |---|---| | [`exnivo/Echo88-150M-Base`](https://huggingface.co/exnivo/Echo88-150M-Base) | Base model | | [`exnivo/Echo88-150M-Instruct`](https://huggingface.co/exnivo/Echo88-150M-Instruct) | Instruction-tuned model | | [`exnivo/Echo88-Instruct-173K`](https://huggingface.co/datasets/exnivo/Echo88-Instruct-173K) | SFT dataset | | [`exnivo/Echo88-150M-Base`](https://huggingface.co/datasets/exnivo/Echo88-150M-Base) | Pretraining dataset | ## Citation If you use this model, you can cite it as: ```bibtex @misc{echo88_150m_base, title = {Echo88-150M Base}, author = {exnivo}, year = {2026}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/exnivo/Echo88-150M-Base}} } ``` ## License The model weights are released under the Apache 2.0 license. The training dataset is mixed-source and released separately. Users are responsible for checking dataset source rights, licensing, and suitability for their own use case. ## Disclaimer Echo88-150M Base is an experimental small base language model. It may produce incorrect, biased, outdated, unsafe, nonsensical, or misleading outputs. Do not use this model for high-stakes decisions or as a reliable source of factual information.