63 lines
1.9 KiB
Markdown
63 lines
1.9 KiB
Markdown
---
|
|
language:
|
|
- en
|
|
tags:
|
|
- causal-lm
|
|
- llama
|
|
- instruction-tuned
|
|
- point-in-time
|
|
- dated
|
|
- lookahead-bias-free
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# DatedGPT-2014-Instruct
|
|
|
|
**DatedGPT** is a family of point-in-time language models: each vintage is
|
|
trained only on data available up to its cutoff date, making it suitable for
|
|
lookahead-bias-free prediction and point-in-time analysis.
|
|
|
|
This is the **instruction-tuned chat model** with data up to **2014**.
|
|
For the base (pretrained) model, see
|
|
[datedgpt/datedgpt-2014-base](https://huggingface.co/datedgpt/datedgpt-2014-base).
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Architecture | LlamaForCausalLM |
|
|
| Parameters | ~1.3 B |
|
|
| Context length | 2048 |
|
|
| Vocab | 32,000 (SentencePiece) |
|
|
| Precision | bfloat16 |
|
|
| Data vintage | 2014 |
|
|
|
|
## Chat template
|
|
|
|
The Llama-2-style chat template ships in `tokenizer_config.json` — apply it
|
|
with the tokenizer. The BOS token must come from the tokenizer, **not** as a
|
|
literal `"<s>"` string in your prompt text.
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
|
repo_id = "datedgpt/datedgpt-2014-instruct"
|
|
tokenizer = AutoTokenizer.from_pretrained(repo_id)
|
|
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map="auto")
|
|
|
|
prompt = tokenizer.apply_chat_template(
|
|
[{"role": "user", "content": "What is the capital of France?"}],
|
|
tokenize=False,
|
|
)
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|
output = model.generate(**inputs, max_new_tokens=128, do_sample=True,
|
|
temperature=0.7, top_p=0.95, use_cache=True,
|
|
eos_token_id=tokenizer.eos_token_id,
|
|
pad_token_id=tokenizer.eos_token_id)
|
|
print(tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
|
|
```
|
|
|
|
## Limitations
|
|
|
|
- Knowledge limited to the 2014 data vintage.
|
|
- No RLHF or safety tuning; outputs can be confidently wrong.
|