初始化项目,由ModelHub XC社区提供模型
Model: egonrp/gpt2-small-portuguese Source: Original Platform
This commit is contained in:
63
README.md
Normal file
63
README.md
Normal file
@@ -0,0 +1,63 @@
|
||||
---
|
||||
license: mit
|
||||
datasets:
|
||||
- wikipedia
|
||||
language:
|
||||
- pt
|
||||
widget:
|
||||
- text: "Quem era Jim Henson? Jim Henson era um"
|
||||
- text: "Em um achado chocante, o cientista descobriu um"
|
||||
- text: "Barack Hussein Obama II, nascido em 4 de agosto de 1961, é"
|
||||
- text: "Corrida por vacina contra Covid-19 já tem"
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
### This model is a copy of Pierre Guillou model "pierreguillou/gpt2-small-portuguese" with the addition of "tokenizer.json" file.
|
||||
|
||||
Link: [https://huggingface.co/pierreguillou/gpt2-small-portuguese](https://huggingface.co/pierreguillou/gpt2-small-portuguese)
|
||||
|
||||
|
||||
The current model repository fixes the error below in original tokenizer model loading in "Tokenizer.from_pretrained()":
|
||||
```
|
||||
>>> from tokenizers import Tokenizer
|
||||
>>> tokenizer = Tokenizer.from_pretrained("pierreguillou/gpt2-small-portuguese")
|
||||
[2023-03-11T18:59:51Z ERROR cached_path::cache] ETAG fetch for https://huggingface.co/pierreguillou/gpt2-small-portuguese/resolve/main/tokenizer.json failed with fatal error
|
||||
Traceback (most recent call last):
|
||||
File "<stdin>", line 1, in <module>
|
||||
Exception: Model "pierreguillou/gpt2-small-portuguese" on the Hub doesn't have a tokenizer
|
||||
```
|
||||
|
||||
|
||||
## If you want to generate the "tokenizer.json" file on your own...
|
||||
Required commands to generate the absent "tokenizer.json" file from original model:
|
||||
```
|
||||
# Downloading the HuggingFace model:
|
||||
$ python.exe
|
||||
>>> import huggingface_hub
|
||||
>>> huggingface_hub.snapshot_download(repo_id="pierreguillou/gpt2-small-portuguese", local_dir=r"C:\dev\pierre_gpt2_small_pt")
|
||||
|
||||
# Generating "tokenizer.json" file:
|
||||
cd C:\dev\pierre_gpt2_small_pt
|
||||
$ python.exe
|
||||
>>> from transformers import AutoTokenizer
|
||||
>>> tokenizer = AutoTokenizer.from_pretrained(r"C:\dev\pierre_gpt2_small_pt")
|
||||
>>> tokenizer._tokenizer.save("tokenizer.json")
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Author (model "pierreguillou/gpt2-small-portuguese")
|
||||
|
||||
Portuguese GPT-2 small was trained and evaluated by [Pierre GUILLOU](https://www.linkedin.com/in/pierreguillou/) thanks to the computing power of the GPU (GPU NVIDIA V100 32 Go) of the [AI Lab](https://www.linkedin.com/company/ailab-unb/) (University of Brasilia) to which I am attached as an Associate Researcher in NLP and the participation of its directors in the definition of NLP strategy, Professors Fabricio Ataides Braz and Nilton Correia da Silva.
|
||||
|
||||
## Citation
|
||||
If you use our work, please cite:
|
||||
|
||||
```bibtex
|
||||
@inproceedings{pierre2020gpt2smallportuguese,
|
||||
title={GPorTuguese-2 (Portuguese GPT-2 small): a Language Model for Portuguese text generation (and more NLP tasks...)},
|
||||
author={Pierre Guillou},
|
||||
year={2020}
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user