55 lines
2.1 KiB
Markdown
55 lines
2.1 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
tags:
|
||
|
|
- text-generation
|
||
|
|
- experimental
|
||
|
|
- ultra-lightweight
|
||
|
|
- tiny-stories
|
||
|
|
model_type: llm
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
datasets:
|
||
|
|
- roneneldan/TinyStories
|
||
|
|
metrics:
|
||
|
|
- perplexity
|
||
|
|
---
|
||
|
|
|
||
|
|
# glubbot1.1-7.5m
|
||
|
|
|
||
|
|
## Model Description
|
||
|
|
glubbot1.1-7.5m is an experimental, ultra-lightweight text generation model developed by the **Glubbot-Team**.
|
||
|
|
|
||
|
|
With only 7.5 million parameters, this model is designed for experimental testing, low-resource inference benchmarking, and studying language acquisition in small-scale models. It was trained specifically to generate short, grammatically correct narratives.
|
||
|
|
|
||
|
|
- **Developed by:** Glubbot-Team
|
||
|
|
- **Model Type:** Transformer-based Causal Language Model
|
||
|
|
- **Language(s):** English
|
||
|
|
- **License:** Apache 2.0
|
||
|
|
- **Status:** Experimental / Testing
|
||
|
|
|
||
|
|
## Intended Uses & Limitations
|
||
|
|
This model is intended solely for research, architectural testing, and experimental evaluation.
|
||
|
|
|
||
|
|
### Intended Use
|
||
|
|
- Synthetic text generation for short, child-like stories.
|
||
|
|
- Benchmarking inference speeds on edge devices or CPUs.
|
||
|
|
- Testing tokenization strategies and pipeline integration.
|
||
|
|
|
||
|
|
### Limitations & Biases
|
||
|
|
- **Domain Specificity:** Optimized purely for short stories; it cannot handle reasoning, coding, or factual factual queries.
|
||
|
|
- **Limited Capacity:** Due to the 7.5M parameter size, the model may produce repetitive structures or drift logically during long generations.
|
||
|
|
- **Not for Production:** Do not deploy this model for general-purpose assistant tasks.
|
||
|
|
|
||
|
|
## Training Details
|
||
|
|
|
||
|
|
### Training Data
|
||
|
|
The model was trained on the **TinyStories** dataset (`roneneldan/TinyStories`). This dataset consists of short stories containing words typically understood by 3- to 4-year-olds, optimized to teach models basic grammar and reasoning.
|
||
|
|
|
||
|
|
### Hyperparameters
|
||
|
|
- **Context Length:** [e.g., 512 / 1024]
|
||
|
|
- **Batch Size:** [Insert Batch Size]
|
||
|
|
- **Learning Rate:** [Insert Learning Rate]
|
||
|
|
- **Optimizer:** AdamW
|
||
|
|
|
||
|
|
## Evaluation
|
||
|
|
Evaluation metrics are monitored primarily to test convergence stability on the TinyStories split. The model is evaluated on its ability to maintain basic grammatical coherence.
|