26 lines
1.1 KiB
Markdown
26 lines
1.1 KiB
Markdown
|
---
|
||
|
|
license: mit
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
library_name: transformers
|
||
|
|
datasets:
|
||
|
|
- croqaz/vintage-v1
|
||
|
|
---
|
||
|
|
|
||
|
|
# Vintage-LLM
|
||
|
|
|
||
|
|
This is a 340M params Llama3-based model, trained on text pre-1900.
|
||
|
|
|
||
|
|
This is a base model. It has very limited chat capabilities.
|
||
|
|
|
||
|
|
Dataset used: https://huggingface.co/datasets/croqaz/vintage-v1
|
||
|
|
|
||
|
|
The data was de-duplicated and only the decent quality texts were used.
|
||
|
|
|
||
|
|
There are a few checkpoints in the repo:
|
||
|
|
|
||
|
|
- 10k steps on short texts: [commit/ff3b94efdaccfa312754d3a4328b7c7a05155d36](https://huggingface.co/croqaz/vintage-LLM-340m-v1-base/commit/ff3b94efdaccfa312754d3a4328b7c7a05155d36)
|
||
|
|
- previous model + short ft on [vintage-FT-v1](https://huggingface.co/datasets/croqaz/vintage-ft-v1): [commit/3946781d51e76c7cf0874bd595003bce1755cbd2](https://huggingface.co/croqaz/vintage-LLM-340m-v1-base/commit/3946781d51e76c7cf0874bd595003bce1755cbd2)
|
||
|
|
- previous model + 10k steps on long texts: [commit/c9b48cc3aaa4b3dc0ea3b846f45e98a05e94c8cb](https://huggingface.co/croqaz/vintage-LLM-340m-v1-base/commit/c9b48cc3aaa4b3dc0ea3b846f45e98a05e94c8cb)
|