---
license: mit
language:
- en
pipeline_tag: text-generation
library_name: transformers
datasets:
- croqaz/vintage-v1
---

# Vintage-LLM

This is a 340M params Llama3-based model, trained on text pre-1900.

This is a base model. It has very limited chat capabilities.

Dataset used: https://huggingface.co/datasets/croqaz/vintage-v1

The data was de-duplicated and only the decent quality texts were used.

There are a few checkpoints in the repo:

- 10k steps on short texts: [commit/ff3b94efdaccfa312754d3a4328b7c7a05155d36](https://huggingface.co/croqaz/vintage-LLM-340m-v1-base/commit/ff3b94efdaccfa312754d3a4328b7c7a05155d36)
- previous model + short ft on [vintage-FT-v1](https://huggingface.co/datasets/croqaz/vintage-ft-v1): [commit/3946781d51e76c7cf0874bd595003bce1755cbd2](https://huggingface.co/croqaz/vintage-LLM-340m-v1-base/commit/3946781d51e76c7cf0874bd595003bce1755cbd2)
- previous model + 10k steps on long texts: [commit/c9b48cc3aaa4b3dc0ea3b846f45e98a05e94c8cb](https://huggingface.co/croqaz/vintage-LLM-340m-v1-base/commit/c9b48cc3aaa4b3dc0ea3b846f45e98a05e94c8cb)