61 lines
2.3 KiB
Markdown
61 lines
2.3 KiB
Markdown
This model provides a GPT-2 language model trained with SimCTG on the ROCStories benchmark [(Mostafazadeh et al., 2016)](https://aclanthology.org/N16-1098.pdf) based on our paper [_A Contrastive Framework for Neural Text Generation_](https://arxiv.org/abs/2202.06417).
|
|
|
|
We provide a detailed tutorial on how to apply SimCTG and Contrastive Search in our [project repo](https://github.com/yxuansu/SimCTG#4-huggingface-style-tutorials-back-to-top). In the following, we illustrate a brief tutorial on how to use our approach to perform text generation.
|
|
|
|
## 1. Installation of SimCTG:
|
|
```yaml
|
|
pip install simctg --upgrade
|
|
```
|
|
|
|
## 2. Initialize SimCTG Model:
|
|
```python
|
|
import torch
|
|
# load SimCTG language model
|
|
from simctg.simctggpt import SimCTGGPT
|
|
model_name = r'cambridgeltl/simctg_rocstories'
|
|
model = SimCTGGPT(model_name)
|
|
model.eval()
|
|
tokenizer = model.tokenizer
|
|
```
|
|
|
|
## 3. Prepare the Text Prefix:
|
|
```python
|
|
prompt = r"Accident in the Lab <|endoftext|>"
|
|
print ('Prefix is: {}'.format(prompt))
|
|
tokens = model.tokenizer.tokenize(prompt)
|
|
input_ids = model.tokenizer.convert_tokens_to_ids(tokens)
|
|
input_ids = torch.LongTensor(input_ids).view(1,-1)
|
|
```
|
|
|
|
## 4. Generate Text with Contrastive Search:
|
|
```python
|
|
beam_width, alpha, decoding_len = 5, 0.65, 45
|
|
output = model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width,
|
|
alpha=alpha, decoding_len=decoding_len)
|
|
print("Output:\n" + 100 * '-')
|
|
print(tokenizer.decode(output).split(model.tokenizer.eos_token)[1].strip())
|
|
'''
|
|
Prefix is: Accident in the Lab <|endoftext|>
|
|
Output:
|
|
----------------------------------------------------------------------------------------------------
|
|
Tom went to work one day. He noticed a lab accident in the lab. Tom was worried about his safety at work.
|
|
Unfortunately the accident didn't go well. Tom wound up leaving early to get back on the job.
|
|
'''
|
|
```
|
|
|
|
For more details of our work, please refer to our main [project repo](https://github.com/yxuansu/SimCTG).
|
|
|
|
## 5. Citation:
|
|
If you find our paper and resources useful, please kindly leave a star and cite our paper. Thanks!
|
|
|
|
```bibtex
|
|
@article{su2022contrastive,
|
|
title={A Contrastive Framework for Neural Text Generation},
|
|
author={Su, Yixuan and Lan, Tian and Wang, Yan and Yogatama, Dani and Kong, Lingpeng and Collier, Nigel},
|
|
journal={arXiv preprint arXiv:2202.06417},
|
|
year={2022}
|
|
}
|
|
```
|
|
|
|
|