100 lines
4.9 KiB
Markdown
100 lines
4.9 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
base_model:
|
||
- meta-llama/Meta-Llama-3-8B
|
||
pipeline_tag: text2text-generation
|
||
---
|
||
|
||
## Janus
|
||
(Built with Meta Llama 3)
|
||
|
||
For the version with the PoS tag visit [Janus (PoS)](https://huggingface.co/ChangeIsKey/llama3-janus-pos).
|
||
|
||
### Model Details
|
||
- **Model Name**: Janus
|
||
- **Version**: 1.0
|
||
- **Developers**: Pierluigi Cassotti, Nina Tahmasebi
|
||
- **Affiliation**: University of Gothenburg
|
||
- **License**: MIT
|
||
- **GitHub Repository**: [Historical Word Usage Generation](https://github.com/ChangeIsKey/historical-word-usage-generation)
|
||
- **Paper**: [Sense-specific Historical Word Usage Generation](https://transacl.org)
|
||
- **Contact**: pierluigi.cassotti@gu.se
|
||
|
||
### Model Description
|
||
Janus is a fine-tuned **Llama 3 8B** model designed to generate historically and semantically accurate word usages. It takes as input a word, its sense definition, and a year and produces example sentences that reflect linguistic usage from the specified period. This model is particularly useful for **semantic change detection**, **historical NLP**, and **linguistic research**.
|
||
|
||
### Intended Use
|
||
- **Semantic Change Detection**: Investigating how word meanings evolve over time.
|
||
- **Historical Text Processing**: Enhancing the understanding and modeling of historical texts.
|
||
- **Corpus Expansion**: Generating sense-annotated corpora for linguistic studies.
|
||
|
||
### Training Data
|
||
- **Dataset**: Extracted from the **Oxford English Dictionary (OED)**
|
||
- **Size**: Over **1.2 million** sense-annotated historical usages
|
||
- **Time Span**: **1700 - 2020**
|
||
- **Data Format**:
|
||
```
|
||
<year><|t|><lemma><|t|><definition><|s|><historical usage sentence><|end|>
|
||
```
|
||
- **Janus (PoS) Format**:
|
||
```
|
||
<year><|t|><lemma><|t|><definition><|p|><PoS><|p|><|s|><historical usage sentence><|end|>
|
||
```
|
||
|
||
### Training Procedure
|
||
- **Base Model**: `meta-llama/Llama-3-8B`
|
||
- **Optimization**: **QLoRA** (Quantized Low-Rank Adaptation)
|
||
- **Batch Size**: **4**
|
||
- **Learning Rate**: **2e-4**
|
||
- **Epochs**: **1**
|
||
|
||
### Model Performance
|
||
- **Temporal Accuracy**: Root mean squared error (RMSE) of **~52.7 years** (close to OED ground truth)
|
||
- **Semantic Accuracy**: Comparable to OED test data on human evaluations
|
||
- **Context Variability**: Low lexical repetition, preserving natural linguistic diversity
|
||
|
||
### Usage Example
|
||
#### Generating Historical Usages
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model_name = "ChangeIsKey/llama3-janus"
|
||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
|
||
|
||
input_text = "1800<|t|>awful<|t|>Used to emphasize something unpleasant or negative; ‘such a’, ‘an absolute’.<|s|>"
|
||
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
|
||
|
||
output = model.generate(**inputs, temperature=1.0, top_p=0.9, max_new_tokens=50)
|
||
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
||
```
|
||
|
||
For more examples, see the GitHub repository [Historical Word Usage Generation](https://github.com/ChangeIsKey/historical-word-usage-generation)
|
||
|
||
### Limitations & Ethical Considerations
|
||
- **Historical Bias**: The model may reflect biases present in historical texts.
|
||
- **Time Granularity**: The temporal resolution is approximate (~50 years RMSE).
|
||
- **Modern Influence**: Despite fine-tuning, the model may still generate modern phrases in older contexts.
|
||
- **Not Trained for Fairness**: The model has not been explicitly trained to be fair or unbiased. It may produce sensitive, outdated, or culturally inappropriate content.
|
||
|
||
### Citation
|
||
If you use Janus, please cite:
|
||
```
|
||
@article{10.1162/tacl_a_00761,
|
||
author = {Cassotti, Pierluigi and Tahmasebi, Nina},
|
||
title = {Sense-specific Historical Word Usage Generation},
|
||
journal = {Transactions of the Association for Computational Linguistics},
|
||
volume = {13},
|
||
pages = {690-708},
|
||
year = {2025},
|
||
month = {07},
|
||
abstract = {Large-scale sense-annotated corpora are important for a range of tasks but are hard to come by. Dictionaries that record and describe the vocabulary of a language often offer a small set of real-world example sentences for each sense of a word. However, on their own, these sentences are too few to be used as diachronic sense-annotated corpora. We propose a targeted strategy for training and evaluating generative models producing historically and semantically accurate word usages given any word, sense definition, and year triple. Our results demonstrate that fine-tuned models can generate usages with the same properties as real-world example sentences from a reference dictionary. Thus the generated usages will be suitable for training and testing computational models where large-scale sense-annotated corpora are needed but currently unavailable.},
|
||
issn = {2307-387X},
|
||
doi = {10.1162/tacl_a_00761},
|
||
url = {https://doi.org/10.1162/tacl\_a\_00761},
|
||
eprint = {https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl\_a\_00761/2535111/tacl\_a\_00761.pdf},
|
||
}
|
||
```
|