245 lines
7.9 KiB
Markdown
245 lines
7.9 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
base_model:
|
|
- allenai/Olmo-3-7B-Instruct
|
|
base_model_relation: finetune
|
|
library_name: transformers
|
|
tags:
|
|
- conversational-ai
|
|
- cognitive-architectures
|
|
- chat
|
|
- chatbot
|
|
- safetensors
|
|
- persona
|
|
- text
|
|
- text-generation
|
|
- persona-ai
|
|
- roleplay
|
|
- cognitive
|
|
- vanta-research
|
|
- project-atom
|
|
- atom
|
|
- conversational
|
|
- collaborative-ai
|
|
- text-generation-inference
|
|
- collaboration
|
|
- friendly
|
|
- educational
|
|
- learning
|
|
- ai-research
|
|
- ai-alignment-research
|
|
- ai-alignment
|
|
- ai-behavior-research
|
|
- ai-persona-research
|
|
- human-ai-collaboration
|
|
---
|
|
|
|
<div align="center">
|
|
|
|
|
|
|
|
<h1>VANTA Research</h1>
|
|
|
|
<p><strong>Independent AI research lab building safe, resilient language models optimized for human-AI collaboration</strong></p>
|
|
|
|
<p>
|
|
<a href="https://vantaresearch.xyz"><img src="https://img.shields.io/badge/Website-vantaresearch.xyz-black" alt="Website"/></a>
|
|
<a href="https://merch.vantaresearch.xyz"><img src="https://img.shields.io/badge/Merch-merch.vantaresearch.xyz-sage" alt="Merch"/></a>
|
|
<a href="https://x.com/vanta_research"><img src="https://img.shields.io/badge/@vanta_research-1DA1F2?logo=x" alt="X"/></a>
|
|
<a href="https://github.com/vanta-research"><img src="https://img.shields.io/badge/GitHub-vanta--research-181717?logo=github" alt="GitHub"/></a>
|
|
</p>
|
|
</div>
|
|
|
|
---
|
|
|
|
# Atom-Olmo3-7B
|
|
|
|
Atom-Olmo3-7B is a specialized language model fine-tuned for collaborative problem-solving and creative exploration. Built on the Olmo-3-7B-Instruct foundation, this model brings thoughtful, structured analysis to complex questions while maintaining an engaging, conversational tone.
|
|
|
|
## Key Features
|
|
|
|
- **Apache 2.0 License**: Fully open-source with permissive licensing for commercial use
|
|
- **Collaborative Intelligence**: Trained to ask clarifying questions and explore ideas iteratively
|
|
- **Structured Thinking**: Provides organized, framework-driven responses for complex topics
|
|
- **Educational Depth**: Breaks down sophisticated concepts into accessible explanations
|
|
- **Creative Synthesis**: Combines analytical rigor with imaginative problem-solving
|
|
|
|
## Model Details
|
|
|
|
- **Base Model**: allenai/Olmo-3-7B-Instruct
|
|
- **Training Method**: LoRA fine-tuning (r=16, alpha=32)
|
|
- **Training Data**: Curated dataset focused on collaborative reasoning, ELI5 explanations, lateral thinking, and research synthesis
|
|
- **Context Length**: 4096 tokens (recommended)
|
|
- **Parameters**: 7B
|
|
- **Precision**: FP16
|
|
|
|
## Intended Use
|
|
|
|
### Primary Use Cases
|
|
|
|
- Technical brainstorming and ideation
|
|
- Educational explanations and concept breakdowns
|
|
- Research synthesis and literature review
|
|
- Collaborative problem-solving across domains
|
|
- Framework development and structured analysis
|
|
|
|
### Out of Scope
|
|
|
|
This model is not intended for:
|
|
- Medical diagnosis or treatment recommendations
|
|
- Legal advice or financial counseling
|
|
- Real-time factual information (knowledge cutoff applies)
|
|
- Autonomous decision-making in high-stakes scenarios
|
|
|
|
## Training Details
|
|
|
|
### Dataset
|
|
|
|
The model was trained on a specialized dataset comprising:
|
|
- Analogical reasoning examples
|
|
- Collaborative exploration dialogues
|
|
- ELI5-style explanations
|
|
- Enthusiastic encouragement patterns
|
|
- Identity and persona consistency examples
|
|
- Lateral thinking exercises
|
|
- Playful humor and engagement
|
|
- Research synthesis demonstrations
|
|
|
|
|
|
### Training Configuration
|
|
|
|
- **Epochs**: 2
|
|
- **Batch Size**: 1 (effective: 16 with gradient accumulation)
|
|
- **Learning Rate**: 2e-4
|
|
- **Optimizer**: AdamW 8-bit
|
|
- **Scheduler**: Cosine with 3% warmup
|
|
- **Quantization**: 4-bit NF4 during training
|
|
- **LoRA Configuration**: r=16, alpha=32, dropout=0.05
|
|
- **Target Modules**: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
|
|
|
|
|
## Performance Characteristics
|
|
|
|
### Strengths
|
|
|
|
- Provides comprehensive, well-organized responses with clear structure
|
|
- Excels at breaking down complex topics into digestible frameworks
|
|
- Asks relevant clarifying questions to refine understanding
|
|
- Maintains consistent persona and collaborative tone
|
|
- Strong performance on educational and analytical tasks
|
|
|
|
### Limitations
|
|
|
|
- Response generation is approximately 5x slower than smaller specialized models
|
|
- May provide more detail than necessary for simple queries
|
|
- Academic/structured tone may not suit all conversational contexts
|
|
- Inherits base model limitations regarding factual knowledge cutoff
|
|
|
|
## Comparison with Atom-Ministral-8B
|
|
|
|
| Feature | Atom-Olmo3-7B | Atom-Ministral-8B |
|
|
|---------|---------------|-------------------|
|
|
| License | Apache 2.0 | Mistral Research License |
|
|
| Parameters | 7B | 8B |
|
|
| Response Style | Structured, comprehensive | Conversational, concise |
|
|
| Speed | ~29s average | ~6s average |
|
|
| Best For | Deep analysis, education | Quick brainstorming, dialogue |
|
|
| Commercial Use | Unrestricted | Restrictions apply |
|
|
|
|
Both models share the same training philosophy and dataset but offer different trade-offs between depth and speed, making them complementary tools for different workflows.
|
|
|
|
## Usage
|
|
|
|
### Basic Inference
|
|
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
import torch
|
|
|
|
model_name = "vanta-research/atom-olmo3-7b"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_name,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto"
|
|
)
|
|
|
|
messages = [
|
|
{"role": "system", "content": "You are Atom, an AI assistant made by VANTA Research in Portland, Oregon. You bring collaborative curiosity, playful enthusiasm, and thoughtful metaphors to every conversation."},
|
|
{"role": "user", "content": "How might we use existing technology in unexpected ways to address climate change?"}
|
|
]
|
|
|
|
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
|
|
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=512,
|
|
temperature=0.7,
|
|
top_p=0.9,
|
|
do_sample=True
|
|
)
|
|
|
|
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
|
print(response)
|
|
```
|
|
|
|
### Recommended Parameters
|
|
|
|
- **Temperature**: 0.7 (balanced creativity and coherence)
|
|
- **Top-p**: 0.9 (nucleus sampling)
|
|
- **Max Tokens**: 512-1024 (model tends toward comprehensive responses)
|
|
- **Stop Sequences**: `<|im_start|>`, `<|im_end|>`
|
|
|
|
## Ethical Considerations
|
|
|
|
### Bias and Fairness
|
|
|
|
This model inherits biases present in the Olmo-3 base model and training data. While efforts were made to curate balanced, high-quality training examples, users should:
|
|
|
|
- Validate factual claims independently
|
|
- Be aware of potential cultural and demographic biases
|
|
- Apply appropriate safeguards for sensitive applications
|
|
- Monitor outputs in production environments
|
|
|
|
### Environmental Impact
|
|
|
|
- **Training Hardware**: 1x NVIDIA RTX 3060 (12GB)
|
|
- **Training Duration**: 5.9 hours
|
|
- **Estimated Energy Consumption**: ~1.5 kWh
|
|
- **Carbon Footprint**: Minimal (single GPU, short training duration)
|
|
|
|
## License
|
|
|
|
This model is released under the Apache License 2.0, providing broad permissions for commercial and non-commercial use. The base OLMo-3 model is also Apache 2.0 licensed.
|
|
|
|
## Citation
|
|
|
|
```bibtex
|
|
@software{atom_olmo3_7b_2025,
|
|
title = {Atom-OLMo3-7B: A Collaborative AI Assistant for Structured Problem-Solving},
|
|
author = {VANTA Research},
|
|
year = {2025},
|
|
url = {https://huggingface.co/vanta-research/atom-olmo3-7b},
|
|
note = {Fine-tuned from OLMo-3-7B-Instruct}
|
|
}
|
|
```
|
|
|
|
## Acknowledgments
|
|
|
|
Built on the Olmo-3-7B-Instruct model by the Allen Institute for AI (Ai2). Training infrastructure and methodology leverage the Hugging Face Transformers, TRL, and PEFT libraries.
|
|
|
|
## Contact
|
|
|
|
- Organization: hello@vantaresearch.xyz
|
|
- Engineering/Design: tyler@vantaresearch.xyz
|
|
|
|
---
|
|
|
|
**Model Version**: 1.0
|
|
**Release Date**: November 2025
|
|
**Model Card Last Updated**: November 21, 2025
|
|
|
|
*Proudly developed in Portland, Oregon by VANTA Research* |