abinayam/gpt-2-tamil
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: abinayam/gpt-2-tamil
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-17 07:06:11 +08:00
commit f301701ac2
55 changed files with 3107 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
.flake8 Normal file
View File

@@ -0,0 +1,8 @@
[flake8]
exclude = venv
ignore = E501, W503, E226, E203
max-line-length = 85
# E501: Line too long
# W503: Line break occurred before binary operator
# E226: Missing white space around arithmetic operator

19
.gitattributes vendored Normal file
View File

@@ -0,0 +1,19 @@
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tar.gz filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.log filter=lfs diff=lfs merge=lfs -text
*.wandb filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
model.safetensors filter=lfs diff=lfs merge=lfs -text

129
.gitignore vendored Normal file
View File

@@ -0,0 +1,129 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
.python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/

28
.pre-commit-config.yaml Normal file
View File

@@ -0,0 +1,28 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.4.0
hooks:
- id: trailing-whitespace
- id: check-yaml
- id: check-ast
- id: check-json
- id: check-merge-conflict
- id: detect-private-key
- repo: https://github.com/psf/black
rev: 21.6b0
hooks:
- id: black
args: []
files: .
- repo: https://gitlab.com/PyCQA/flake8
rev: 3.9.2
hooks:
- id: flake8
- repo: https://github.com/PyCQA/isort
rev: 5.9.1
hooks:
- id: isort
args: []
files: .

21
LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2021 Abinaya Mahendiran
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

69
README.md Normal file
View File

@@ -0,0 +1,69 @@
---
language: ta
datasets:
- oscar
- IndicNLP
widget:
- text: 'ஒரு ஊரிலே ஒரு காக்கைக்கு'
---
# GPT2-Tamil
This repository is created as part of the Flax/Jax community week by Huggingface. The aim of this project is to pretrain a language model using GPT-2 specifically for Tamil language.
## Setup:
To setup the project, run the following command,
```python
pip install -r requirements.txt
```
## Model:
Pretrained model on Tamil language using a causal language modeling (CLM) objective.
## Dataset Used:
The GTP-2 model is trained on [oscar dataset - ta](https://huggingface.co/datasets/oscar) and [IndicNLP dataset - ta](https://indicnlp.ai4bharat.org/corpora/)
## Intended uses & limitations:
You can use the raw model for next sentence prediction, but it's mostly intended to be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=gpt2) to look for fine-tuned versions on a task that interests you.
## How to pretrain the model:
To perform training, do the following steps,
- Export the model directory (where you want to store the model artifacts like config, tokenizer, etc.)
```python
>>> export MODEL_DIR=<model_dir>
```
- Create the config.json by running the following command,
```python
>>> python src/create_config.py
```
- Create the tokenizer by running the following command,
```python
>>> python src/train_tokenizer.py
```
- Once the config and tokenizer is created, run the following script to start training the flax model
```python
>>> python scripts/train_gpt2-oscar-tamil.sh
```
## How to use:
To perform language generation using the model, pipeline can be used directly.
- First convert the flax model to pytorch using the following command,
```python
python src/convert_flax_to_pytorch.py
```
- Use the following snippet to perform language generation,
```python
>>> from transformers import AutoTokenizer, AutoModelWithLMHead, pipeline
>>> model_name = 'abinayam/gpt-2-tamil'
>>> model = AutoModelWithLMHead.from_pretrained(model_name)
>>> tokenizer = AutoTokenizer.from_pretrained(model_name)
>>> set_seed(42)
>>> input_text = "ஒரு ஊரிலே ஒரு காக்கைக்கு"
>>> max_len = 300
>>> no_seq = 5
>>> generator = pipeline('text-generation', model=model, tokenizer=tokenizer)
>>> sequence = generator(input_text, max_length=max_len, num_return_sequences=no_seq)
```

38
config.json Normal file
View File

@@ -0,0 +1,38 @@
{
"_name_or_path": "../gpt-2-tamil",
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.0,
"bos_token_id": 50256,
"embd_pdrop": 0.0,
"eos_token_id": 50256,
"gradient_checkpointing": false,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 768,
"n_head": 12,
"n_inner": null,
"n_layer": 12,
"n_positions": 1024,
"resid_pdrop": 0.0,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 300
}
},
"torch_dtype": "float32",
"transformers_version": "4.9.0.dev0",
"use_cache": true,
"vocab_size": 50257
}

1
dataset/README.md Normal file
View File

@@ -0,0 +1 @@
Details of the dataset can go here. The folder can also contain dataset (downloaded locally).

1
demo/README.md Normal file
View File

@@ -0,0 +1 @@
Streamlit demo can go here.

31
demo/tamil_generator.py Normal file
View File

@@ -0,0 +1,31 @@
# -*- coding: utf-8 -*-
import locale
print(locale.getpreferredencoding())
from transformers import AutoConfig, AutoModelForCausalLM,pipeline,AutoTokenizer
from datasets import load_dataset
MODEL_DIR = "/home/deepak/sources/gpt2-tamil/gpt2-tamil/"
#get prompt from dataset, will be replaced by manual prompt once I figure out how to render tamil font
dataset = load_dataset("oscar", "unshuffled_deduplicated_ta", split="train")
id =232
print(dataset[id]['text'])
tamil_prompt =dataset[id]['text']
# Get configuration and the model
config = AutoConfig.from_pretrained(MODEL_DIR)
model = AutoModelForCausalLM.from_config(config)
tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR)
generator= pipeline('text-generation', model=model, tokenizer=tokenizer)
model_output = generator(tamil_prompt, max_length=30, num_return_sequences=5)
print(model_output)

36
gpt-2-tamil/config.json Normal file
View File

@@ -0,0 +1,36 @@
{
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.0,
"bos_token_id": 50256,
"embd_pdrop": 0.0,
"eos_token_id": 50256,
"gradient_checkpointing": false,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 768,
"n_head": 12,
"n_inner": null,
"n_layer": 12,
"n_positions": 1024,
"resid_pdrop": 0.0,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"transformers_version": "4.9.0.dev0",
"use_cache": true,
"vocab_size": 50257
}

3
gpt-2-tamil/events.out.tfevents.1626336540.t1v-n-ebe36c53-w-0.751183.3.v2 Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f1799847ce42c1a5f9fe25dfa8d8da9e1a6ff57595979b2bd0daea658d9ea785
size 40

3
gpt-2-tamil/events.out.tfevents.1626339585.t1v-n-ebe36c53-w-0.759145.3.v2 Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1b47918f07e65192c48181c8f775cbf29f08585ac3a559e67df1e3f13fb1ca01
size 40

3
gpt-2-tamil/events.out.tfevents.1626340740.t1v-n-ebe36c53-w-0.765413.3.v2 Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5855b0a71977e29453739fe2c5055c32753a62fa6d3db8ea3f105fd8ca75357b
size 40

3
gpt-2-tamil/events.out.tfevents.1626341319.t1v-n-ebe36c53-w-0.768105.3.v2 Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:938ebc19608236e36e53fd65f7c12c9d7ad0de447d01d60627441645872ef573
size 22272043

3
gpt-2-tamil/flax_model.msgpack Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:89396995064d16071519a20c2771d661400da8c3d644966f0a586d299d1b2fa3
size 497764120

1
gpt-2-tamil/tokenizer.json Normal file
View File

File diff suppressed because one or more lines are too long

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1a428e17fe6c41c96b3ac840af045afc1bf45a043e04d220f78944c38168b740
size 510359598

1
model/README.md Normal file
View File

@@ -0,0 +1 @@
Model card details can go here.

1
notebook/README.md Normal file
View File

@@ -0,0 +1 @@
Notebook can go here.

31
pyproject.toml Normal file
View File

@@ -0,0 +1,31 @@
# Black formatting
[tool.black]
line-length = 85
include = '\.pyi?$'
exclude = '''
/(
\.eggs # exclude a few common directories in the
| \.git # root of the project
| \.hg
| \.mypy_cache
| \.tox
| \.venv
| _build
| buck-out
| build
| dist
| wandb
| model
| dataset
| notebook
)/
'''
# iSort
[tool.isort]
profile = "black"
line_length = 85
multi_line_output = 3
include_trailing_comma = true
skip_gitignore = true
virtual_env = "venv"

3
pytorch_model.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:183e23beb6421156e9472504710d66104d4d43829fb87cffd22d888565f27a3a
size 510401385

8
requirements.txt Normal file
View File

@@ -0,0 +1,8 @@
tqdm
transformers
datasets
jax
jaxlib
flax
optax
wandb

25
scripts/train_gpt2-oscar-tamil.sh Executable file
View File

@@ -0,0 +1,25 @@
#!/usr/bin/env bash
python ../src/run_clm_flax.py \
--output_dir="${MODEL_DIR}" \
--model_type="gpt2" \
--config_name="${MODEL_DIR}" \
--tokenizer_name="${MODEL_DIR}" \
--dataset_name="oscar" \
--dataset_config_name="unshuffled_deduplicated_ta" \
--do_train --do_eval \
--block_size="512" \
--per_device_train_batch_size="64" \
--per_device_eval_batch_size="64" \
--learning_rate="3e-5" \
--warmup_steps="1000" \
--adam_beta1="0.9" --adam_beta2="0.98" --weight_decay="0.01" \
--overwrite_output_dir \
--num_train_epochs="10" \
--report_to wandb \
--run_name trial \
--logging_steps="500" \
--save_steps="2500" \
--eval_steps="2500" \
--preprocessing_num_workers="90" \
#--push_to_hub
2>&1 | tee run.log

1
scripts/wandb/latest-run Symbolic link
View File

@@ -0,0 +1 @@
run-20210715_092837-watdq7ib

3
scripts/wandb/run-20210712_164633-1ddv4131/run-1ddv4131.wandb Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d8211487b4d0a0489ae4728120abad1be7ee4190520afc47fdae166087ae6068
size 60817322

305
scripts/wandb/run-20210715_080856-2mpx5n1j/files/config.yaml Normal file
View File

@@ -0,0 +1,305 @@
wandb_version: 1
__cached__setup_devices:
desc: null
value: cpu
_n_gpu:
desc: null
value: 0
_wandb:
desc: null
value:
cli_version: 0.10.33
framework: huggingface
huggingface_version: 4.9.0.dev0
is_jupyter_run: false
is_kaggle_kernel: false
python_version: 3.8.10
t:
1:
- 1
- 3
- 11
2:
- 1
- 3
- 11
4: 3.8.10
5: 0.10.33
6: 4.9.0.dev0
8:
- 5
adafactor:
desc: null
value: false
adam_beta1:
desc: null
value: 0.9
adam_beta2:
desc: null
value: 0.98
adam_epsilon:
desc: null
value: 1.0e-08
block_size:
desc: null
value: 512
cache_dir:
desc: null
value: null
config_name:
desc: null
value: ../gpt-2-tamil
dataloader_drop_last:
desc: null
value: false
dataloader_num_workers:
desc: null
value: 0
dataloader_pin_memory:
desc: null
value: true
dataset_config_name:
desc: null
value: unshuffled_deduplicated_ta
dataset_name:
desc: null
value: oscar
ddp_find_unused_parameters:
desc: null
value: null
debug:
desc: null
value: []
deepspeed:
desc: null
value: null
disable_tqdm:
desc: null
value: false
do_eval:
desc: null
value: true
do_predict:
desc: null
value: false
do_train:
desc: null
value: true
dtype:
desc: null
value: float32
eval_accumulation_steps:
desc: null
value: null
eval_steps:
desc: null
value: 2500
evaluation_strategy:
desc: null
value: IntervalStrategy.NO
fp16:
desc: null
value: false
fp16_backend:
desc: null
value: auto
fp16_full_eval:
desc: null
value: false
fp16_opt_level:
desc: null
value: O1
gradient_accumulation_steps:
desc: null
value: 1
greater_is_better:
desc: null
value: null
group_by_length:
desc: null
value: false
ignore_data_skip:
desc: null
value: false
label_names:
desc: null
value: null
label_smoothing_factor:
desc: null
value: 0.0
learning_rate:
desc: null
value: 3.0e-05
length_column_name:
desc: null
value: length
load_best_model_at_end:
desc: null
value: false
local_rank:
desc: null
value: -1
log_level:
desc: null
value: -1
log_level_replica:
desc: null
value: -1
log_on_each_node:
desc: null
value: true
logging_dir:
desc: null
value: ../gpt-2-tamil/runs/Jul15_06-31-48_t1v-n-ebe36c53-w-0
logging_first_step:
desc: null
value: false
logging_steps:
desc: null
value: 500
logging_strategy:
desc: null
value: IntervalStrategy.STEPS
lr_scheduler_type:
desc: null
value: SchedulerType.LINEAR
max_eval_samples:
desc: null
value: null
max_grad_norm:
desc: null
value: 1.0
max_steps:
desc: null
value: -1
max_train_samples:
desc: null
value: null
metric_for_best_model:
desc: null
value: null
model_name_or_path:
desc: null
value: null
model_type:
desc: null
value: gpt2
mp_parameters:
desc: null
value: ''
no_cuda:
desc: null
value: false
num_train_epochs:
desc: null
value: 10.0
output_dir:
desc: null
value: ../gpt-2-tamil
overwrite_cache:
desc: null
value: false
overwrite_output_dir:
desc: null
value: true
past_index:
desc: null
value: -1
per_device_eval_batch_size:
desc: null
value: 128
per_device_train_batch_size:
desc: null
value: 128
per_gpu_eval_batch_size:
desc: null
value: null
per_gpu_train_batch_size:
desc: null
value: null
prediction_loss_only:
desc: null
value: false
preprocessing_num_workers:
desc: null
value: 90
push_to_hub:
desc: null
value: false
push_to_hub_model_id:
desc: null
value: gpt-2-tamil
push_to_hub_organization:
desc: null
value: null
push_to_hub_token:
desc: null
value: null
remove_unused_columns:
desc: null
value: true
report_to:
desc: null
value:
- wandb
resume_from_checkpoint:
desc: null
value: null
run_name:
desc: null
value: trial
save_on_each_node:
desc: null
value: false
save_steps:
desc: null
value: 2500
save_strategy:
desc: null
value: IntervalStrategy.STEPS
save_total_limit:
desc: null
value: null
seed:
desc: null
value: 42
sharded_ddp:
desc: null
value: []
skip_memory_metrics:
desc: null
value: true
tokenizer_name:
desc: null
value: ../gpt-2-tamil
tpu_metrics_debug:
desc: null
value: false
tpu_num_cores:
desc: null
value: null
train_file:
desc: null
value: null
use_fast_tokenizer:
desc: null
value: true
use_legacy_prediction_loop:
desc: null
value: false
validation_file:
desc: null
value: null
validation_split_percentage:
desc: null
value: 5
warmup_ratio:
desc: null
value: 0.0
warmup_steps:
desc: null
value: 1000
weight_decay:
desc: null
value: 0.01

1
scripts/wandb/run-20210715_080856-2mpx5n1j/files/events.out.tfevents.1626336540.t1v-n-ebe36c53-w-0.751183.3.v2 Symbolic link
View File

@@ -0,0 +1 @@
/home/tweety_abi/GPT2-Tamil/gpt-2-tamil/events.out.tfevents.1626336540.t1v-n-ebe36c53-w-0.751183.3.v2

123
scripts/wandb/run-20210715_080856-2mpx5n1j/files/requirements.txt Normal file
View File

@@ -0,0 +1,123 @@
absl-py==0.13.0
aiohttp==3.7.4.post0
appdirs==1.4.4
astunparse==1.6.3
async-timeout==3.0.1
attrs==21.2.0
backcall==0.2.0
black==21.6b0
cachetools==4.2.2
certifi==2021.5.30
cfgv==3.3.0
chardet==4.0.0
chex==0.0.7
click==8.0.1
configparser==5.0.2
cycler==0.10.0
datasets==1.8.1.dev0
decorator==5.0.9
dill==0.3.4
distlib==0.3.2
dm-tree==0.1.6
docker-pycreds==0.4.0
filelock==3.0.12
flake8==3.9.2
flatbuffers==1.12
flax==0.3.4
fsspec==2021.6.1
gast==0.4.0
gitdb==4.0.7
gitpython==3.1.18
google-auth-oauthlib==0.4.4
google-auth==1.32.1
google-pasta==0.2.0
grpcio==1.34.1
h5py==3.1.0
huggingface-hub==0.0.12
identify==2.2.10
idna==2.10
ipython-genutils==0.2.0
ipython==7.25.0
isort==5.9.1
jax==0.2.16
jaxlib==0.1.68
jedi==0.18.0
joblib==1.0.1
keras-nightly==2.5.0.dev2021032900
keras-preprocessing==1.1.2
kiwisolver==1.3.1
libtpu-nightly==0.1.dev20210615
markdown==3.3.4
matplotlib-inline==0.1.2
matplotlib==3.4.2
mccabe==0.6.1
msgpack==1.0.2
multidict==5.1.0
multiprocess==0.70.12.2
mypy-extensions==0.4.3
nodeenv==1.6.0
numpy==1.19.5
oauthlib==3.1.1
opt-einsum==3.3.0
optax==0.0.8
packaging==20.9
pandas==1.2.5
parso==0.8.2
pathspec==0.8.1
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
pillow==8.3.0
pip==20.0.2
pkg-resources==0.0.0
pre-commit==2.13.0
promise==2.3
prompt-toolkit==3.0.19
protobuf==3.17.3
psutil==5.8.0
ptyprocess==0.7.0
pyarrow==4.0.1
pyasn1-modules==0.2.8
pyasn1==0.4.8
pycodestyle==2.7.0
pyflakes==2.3.1
pygments==2.9.0
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2021.1
pyyaml==5.4.1
regex==2021.7.1
requests-oauthlib==1.3.0
requests==2.25.1
rsa==4.7.2
sacremoses==0.0.45
scipy==1.7.0
sentry-sdk==1.3.0
setuptools==44.0.0
shortuuid==1.0.1
six==1.15.0
smmap==4.0.0
subprocess32==3.5.4
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorboard==2.5.0
tensorflow-estimator==2.5.0
tensorflow==2.5.0
termcolor==1.1.0
tokenizers==0.10.3
toml==0.10.2
toolz==0.11.1
torch==1.9.0
tqdm==4.61.1
traitlets==5.0.5
transformers==4.9.0.dev0
typing-extensions==3.7.4.3
urllib3==1.26.6
virtualenv==20.4.7
wandb==0.10.33
wcwidth==0.2.5
werkzeug==2.0.1
wheel==0.36.2
wrapt==1.12.1
xxhash==2.0.2
yarl==1.6.3

49
scripts/wandb/run-20210715_080856-2mpx5n1j/files/wandb-metadata.json Normal file
View File

@@ -0,0 +1,49 @@
{
"os": "Linux-5.4.0-1043-gcp-x86_64-with-glibc2.29",
"python": "3.8.10",
"heartbeatAt": "2021-07-15T08:09:00.134255",
"startedAt": "2021-07-15T08:08:56.269238",
"docker": null,
"cpu_count": 96,
"cuda": null,
"args": [
"--output_dir=../gpt-2-tamil",
"--model_type=gpt2",
"--config_name=../gpt-2-tamil",
"--tokenizer_name=../gpt-2-tamil",
"--dataset_name=oscar",
"--dataset_config_name=unshuffled_deduplicated_ta",
"--do_train",
"--do_eval",
"--block_size=512",
"--per_device_train_batch_size=128",
"--per_device_eval_batch_size=128",
"--learning_rate=3e-5",
"--warmup_steps=1000",
"--adam_beta1=0.9",
"--adam_beta2=0.98",
"--weight_decay=0.01",
"--overwrite_output_dir",
"--num_train_epochs=10",
"--report_to",
"wandb",
"--run_name",
"trial",
"--logging_steps=500",
"--save_steps=2500",
"--eval_steps=2500",
"--preprocessing_num_workers=90"
],
"state": "running",
"program": "../src/run_clm_flax.py",
"codePath": "src/run_clm_flax.py",
"git": {
"remote": "https://github.com/AbinayaM02/GPT2-Tamil.git",
"commit": "69c9b7bf75b708a8f62cf5833d1b89acf5d1760b"
},
"email": "abinaya.m02@mphasis.com",
"root": "/home/tweety_abi/GPT2-Tamil",
"host": "t1v-n-ebe36c53-w-0",
"username": "tweety_abi",
"executable": "/home/tweety_abi/gpt2_env/bin/python"
}

1
scripts/wandb/run-20210715_080856-2mpx5n1j/files/wandb-summary.json Normal file
View File

@@ -0,0 +1 @@
{}

3
scripts/wandb/run-20210715_080856-2mpx5n1j/run-2mpx5n1j.wandb Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ad2816f7f07dec6835ab15fdfb6fa81ca124f1b3f1dfbaccb9b2f3658286d158
size 38211

301
scripts/wandb/run-20210715_085943-1ize2alk/files/config.yaml Normal file
View File

@@ -0,0 +1,301 @@
wandb_version: 1
__cached__setup_devices:
desc: null
value: cpu
_n_gpu:
desc: null
value: 0
_wandb:
desc: null
value:
cli_version: 0.10.33
framework: huggingface
huggingface_version: 4.9.0.dev0
is_jupyter_run: false
is_kaggle_kernel: false
python_version: 3.8.10
t:
1:
- 1
- 3
- 11
4: 3.8.10
5: 0.10.33
6: 4.9.0.dev0
8:
- 5
adafactor:
desc: null
value: false
adam_beta1:
desc: null
value: 0.9
adam_beta2:
desc: null
value: 0.98
adam_epsilon:
desc: null
value: 1.0e-08
block_size:
desc: null
value: 512
cache_dir:
desc: null
value: null
config_name:
desc: null
value: ../gpt-2-tamil
dataloader_drop_last:
desc: null
value: false
dataloader_num_workers:
desc: null
value: 0
dataloader_pin_memory:
desc: null
value: true
dataset_config_name:
desc: null
value: unshuffled_deduplicated_ta
dataset_name:
desc: null
value: oscar
ddp_find_unused_parameters:
desc: null
value: null
debug:
desc: null
value: []
deepspeed:
desc: null
value: null
disable_tqdm:
desc: null
value: false
do_eval:
desc: null
value: true
do_predict:
desc: null
value: false
do_train:
desc: null
value: true
dtype:
desc: null
value: float32
eval_accumulation_steps:
desc: null
value: null
eval_steps:
desc: null
value: 2500
evaluation_strategy:
desc: null
value: IntervalStrategy.NO
fp16:
desc: null
value: false
fp16_backend:
desc: null
value: auto
fp16_full_eval:
desc: null
value: false
fp16_opt_level:
desc: null
value: O1
gradient_accumulation_steps:
desc: null
value: 1
greater_is_better:
desc: null
value: null
group_by_length:
desc: null
value: false
ignore_data_skip:
desc: null
value: false
label_names:
desc: null
value: null
label_smoothing_factor:
desc: null
value: 0.0
learning_rate:
desc: null
value: 3.0e-05
length_column_name:
desc: null
value: length
load_best_model_at_end:
desc: null
value: false
local_rank:
desc: null
value: -1
log_level:
desc: null
value: -1
log_level_replica:
desc: null
value: -1
log_on_each_node:
desc: null
value: true
logging_dir:
desc: null
value: ../gpt-2-tamil/runs/Jul15_07-55-49_t1v-n-ebe36c53-w-0
logging_first_step:
desc: null
value: false
logging_steps:
desc: null
value: 500
logging_strategy:
desc: null
value: IntervalStrategy.STEPS
lr_scheduler_type:
desc: null
value: SchedulerType.LINEAR
max_eval_samples:
desc: null
value: null
max_grad_norm:
desc: null
value: 1.0
max_steps:
desc: null
value: -1
max_train_samples:
desc: null
value: null
metric_for_best_model:
desc: null
value: null
model_name_or_path:
desc: null
value: null
model_type:
desc: null
value: gpt2
mp_parameters:
desc: null
value: ''
no_cuda:
desc: null
value: false
num_train_epochs:
desc: null
value: 10.0
output_dir:
desc: null
value: ../gpt-2-tamil
overwrite_cache:
desc: null
value: false
overwrite_output_dir:
desc: null
value: true
past_index:
desc: null
value: -1
per_device_eval_batch_size:
desc: null
value: 128
per_device_train_batch_size:
desc: null
value: 128
per_gpu_eval_batch_size:
desc: null
value: null
per_gpu_train_batch_size:
desc: null
value: null
prediction_loss_only:
desc: null
value: false
preprocessing_num_workers:
desc: null
value: 90
push_to_hub:
desc: null
value: false
push_to_hub_model_id:
desc: null
value: gpt-2-tamil
push_to_hub_organization:
desc: null
value: null
push_to_hub_token:
desc: null
value: null
remove_unused_columns:
desc: null
value: true
report_to:
desc: null
value:
- wandb
resume_from_checkpoint:
desc: null
value: null
run_name:
desc: null
value: trial
save_on_each_node:
desc: null
value: false
save_steps:
desc: null
value: 2500
save_strategy:
desc: null
value: IntervalStrategy.STEPS
save_total_limit:
desc: null
value: null
seed:
desc: null
value: 42
sharded_ddp:
desc: null
value: []
skip_memory_metrics:
desc: null
value: true
tokenizer_name:
desc: null
value: ../gpt-2-tamil
tpu_metrics_debug:
desc: null
value: false
tpu_num_cores:
desc: null
value: null
train_file:
desc: null
value: null
use_fast_tokenizer:
desc: null
value: true
use_legacy_prediction_loop:
desc: null
value: false
validation_file:
desc: null
value: null
validation_split_percentage:
desc: null
value: 5
warmup_ratio:
desc: null
value: 0.0
warmup_steps:
desc: null
value: 1000
weight_decay:
desc: null
value: 0.01

1
scripts/wandb/run-20210715_085943-1ize2alk/files/events.out.tfevents.1626339585.t1v-n-ebe36c53-w-0.759145.3.v2 Symbolic link
View File

@@ -0,0 +1 @@
/home/tweety_abi/GPT2-Tamil/gpt-2-tamil/events.out.tfevents.1626339585.t1v-n-ebe36c53-w-0.759145.3.v2

123
scripts/wandb/run-20210715_085943-1ize2alk/files/requirements.txt Normal file
View File

@@ -0,0 +1,123 @@
absl-py==0.13.0
aiohttp==3.7.4.post0
appdirs==1.4.4
astunparse==1.6.3
async-timeout==3.0.1
attrs==21.2.0
backcall==0.2.0
black==21.6b0
cachetools==4.2.2
certifi==2021.5.30
cfgv==3.3.0
chardet==4.0.0
chex==0.0.7
click==8.0.1
configparser==5.0.2
cycler==0.10.0
datasets==1.8.1.dev0
decorator==5.0.9
dill==0.3.4
distlib==0.3.2
dm-tree==0.1.6
docker-pycreds==0.4.0
filelock==3.0.12
flake8==3.9.2
flatbuffers==1.12
flax==0.3.4
fsspec==2021.6.1
gast==0.4.0
gitdb==4.0.7
gitpython==3.1.18
google-auth-oauthlib==0.4.4
google-auth==1.32.1
google-pasta==0.2.0
grpcio==1.34.1
h5py==3.1.0
huggingface-hub==0.0.12
identify==2.2.10
idna==2.10
ipython-genutils==0.2.0
ipython==7.25.0
isort==5.9.1
jax==0.2.16
jaxlib==0.1.68
jedi==0.18.0
joblib==1.0.1
keras-nightly==2.5.0.dev2021032900
keras-preprocessing==1.1.2
kiwisolver==1.3.1
libtpu-nightly==0.1.dev20210615
markdown==3.3.4
matplotlib-inline==0.1.2
matplotlib==3.4.2
mccabe==0.6.1
msgpack==1.0.2
multidict==5.1.0
multiprocess==0.70.12.2
mypy-extensions==0.4.3
nodeenv==1.6.0
numpy==1.19.5
oauthlib==3.1.1
opt-einsum==3.3.0
optax==0.0.8
packaging==20.9
pandas==1.2.5
parso==0.8.2
pathspec==0.8.1
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
pillow==8.3.0
pip==20.0.2
pkg-resources==0.0.0
pre-commit==2.13.0
promise==2.3
prompt-toolkit==3.0.19
protobuf==3.17.3
psutil==5.8.0
ptyprocess==0.7.0
pyarrow==4.0.1
pyasn1-modules==0.2.8
pyasn1==0.4.8
pycodestyle==2.7.0
pyflakes==2.3.1
pygments==2.9.0
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2021.1
pyyaml==5.4.1
regex==2021.7.1
requests-oauthlib==1.3.0
requests==2.25.1
rsa==4.7.2
sacremoses==0.0.45
scipy==1.7.0
sentry-sdk==1.3.0
setuptools==44.0.0
shortuuid==1.0.1
six==1.15.0
smmap==4.0.0
subprocess32==3.5.4
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorboard==2.5.0
tensorflow-estimator==2.5.0
tensorflow==2.5.0
termcolor==1.1.0
tokenizers==0.10.3
toml==0.10.2
toolz==0.11.1
torch==1.9.0
tqdm==4.61.1
traitlets==5.0.5
transformers==4.9.0.dev0
typing-extensions==3.7.4.3
urllib3==1.26.6
virtualenv==20.4.7
wandb==0.10.33
wcwidth==0.2.5
werkzeug==2.0.1
wheel==0.36.2
wrapt==1.12.1
xxhash==2.0.2
yarl==1.6.3

49
scripts/wandb/run-20210715_085943-1ize2alk/files/wandb-metadata.json Normal file
View File

@@ -0,0 +1,49 @@
{
"os": "Linux-5.4.0-1043-gcp-x86_64-with-glibc2.29",
"python": "3.8.10",
"heartbeatAt": "2021-07-15T08:59:45.122600",
"startedAt": "2021-07-15T08:59:43.232731",
"docker": null,
"cpu_count": 96,
"cuda": null,
"args": [
"--output_dir=../gpt-2-tamil",
"--model_type=gpt2",
"--config_name=../gpt-2-tamil",
"--tokenizer_name=../gpt-2-tamil",
"--dataset_name=oscar",
"--dataset_config_name=unshuffled_deduplicated_ta",
"--do_train",
"--do_eval",
"--block_size=512",
"--per_device_train_batch_size=128",
"--per_device_eval_batch_size=128",
"--learning_rate=3e-5",
"--warmup_steps=1000",
"--adam_beta1=0.9",
"--adam_beta2=0.98",
"--weight_decay=0.01",
"--overwrite_output_dir",
"--num_train_epochs=10",
"--report_to",
"wandb",
"--run_name",
"trial",
"--logging_steps=500",
"--save_steps=2500",
"--eval_steps=2500",
"--preprocessing_num_workers=90"
],
"state": "running",
"program": "../src/run_clm_flax.py",
"codePath": "src/run_clm_flax.py",
"git": {
"remote": "https://github.com/AbinayaM02/GPT2-Tamil.git",
"commit": "69c9b7bf75b708a8f62cf5833d1b89acf5d1760b"
},
"email": "abinaya.m02@mphasis.com",
"root": "/home/tweety_abi/GPT2-Tamil",
"host": "t1v-n-ebe36c53-w-0",
"username": "tweety_abi",
"executable": "/home/tweety_abi/gpt2_env/bin/python"
}

1
scripts/wandb/run-20210715_085943-1ize2alk/files/wandb-summary.json Normal file
View File

@@ -0,0 +1 @@
{}

3
scripts/wandb/run-20210715_085943-1ize2alk/run-1ize2alk.wandb Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4ddd483c4184ad35f642b4c9ddd01c8f4915a2cd4d811fb5e6395adec23ec07e
size 11149

305
scripts/wandb/run-20210715_091856-2v0tf7h4/files/config.yaml Normal file
View File

@@ -0,0 +1,305 @@
wandb_version: 1
__cached__setup_devices:
desc: null
value: cpu
_n_gpu:
desc: null
value: 0
_wandb:
desc: null
value:
cli_version: 0.10.33
framework: huggingface
huggingface_version: 4.9.0.dev0
is_jupyter_run: false
is_kaggle_kernel: false
python_version: 3.8.10
t:
1:
- 1
- 3
- 11
2:
- 1
- 3
- 11
4: 3.8.10
5: 0.10.33
6: 4.9.0.dev0
8:
- 5
adafactor:
desc: null
value: false
adam_beta1:
desc: null
value: 0.9
adam_beta2:
desc: null
value: 0.98
adam_epsilon:
desc: null
value: 1.0e-08
block_size:
desc: null
value: 512
cache_dir:
desc: null
value: null
config_name:
desc: null
value: ../gpt-2-tamil
dataloader_drop_last:
desc: null
value: false
dataloader_num_workers:
desc: null
value: 0
dataloader_pin_memory:
desc: null
value: true
dataset_config_name:
desc: null
value: unshuffled_deduplicated_ta
dataset_name:
desc: null
value: oscar
ddp_find_unused_parameters:
desc: null
value: null
debug:
desc: null
value: []
deepspeed:
desc: null
value: null
disable_tqdm:
desc: null
value: false
do_eval:
desc: null
value: true
do_predict:
desc: null
value: false
do_train:
desc: null
value: true
dtype:
desc: null
value: float32
eval_accumulation_steps:
desc: null
value: null
eval_steps:
desc: null
value: 2500
evaluation_strategy:
desc: null
value: IntervalStrategy.NO
fp16:
desc: null
value: false
fp16_backend:
desc: null
value: auto
fp16_full_eval:
desc: null
value: false
fp16_opt_level:
desc: null
value: O1
gradient_accumulation_steps:
desc: null
value: 1
greater_is_better:
desc: null
value: null
group_by_length:
desc: null
value: false
ignore_data_skip:
desc: null
value: false
label_names:
desc: null
value: null
label_smoothing_factor:
desc: null
value: 0.0
learning_rate:
desc: null
value: 3.0e-05
length_column_name:
desc: null
value: length
load_best_model_at_end:
desc: null
value: false
local_rank:
desc: null
value: -1
log_level:
desc: null
value: -1
log_level_replica:
desc: null
value: -1
log_on_each_node:
desc: null
value: true
logging_dir:
desc: null
value: ../gpt-2-tamil/runs/Jul15_09-16-14_t1v-n-ebe36c53-w-0
logging_first_step:
desc: null
value: false
logging_steps:
desc: null
value: 500
logging_strategy:
desc: null
value: IntervalStrategy.STEPS
lr_scheduler_type:
desc: null
value: SchedulerType.LINEAR
max_eval_samples:
desc: null
value: null
max_grad_norm:
desc: null
value: 1.0
max_steps:
desc: null
value: -1
max_train_samples:
desc: null
value: null
metric_for_best_model:
desc: null
value: null
model_name_or_path:
desc: null
value: null
model_type:
desc: null
value: gpt2
mp_parameters:
desc: null
value: ''
no_cuda:
desc: null
value: false
num_train_epochs:
desc: null
value: 10.0
output_dir:
desc: null
value: ../gpt-2-tamil
overwrite_cache:
desc: null
value: false
overwrite_output_dir:
desc: null
value: true
past_index:
desc: null
value: -1
per_device_eval_batch_size:
desc: null
value: 128
per_device_train_batch_size:
desc: null
value: 128
per_gpu_eval_batch_size:
desc: null
value: null
per_gpu_train_batch_size:
desc: null
value: null
prediction_loss_only:
desc: null
value: false
preprocessing_num_workers:
desc: null
value: 90
push_to_hub:
desc: null
value: false
push_to_hub_model_id:
desc: null
value: gpt-2-tamil
push_to_hub_organization:
desc: null
value: null
push_to_hub_token:
desc: null
value: null
remove_unused_columns:
desc: null
value: true
report_to:
desc: null
value:
- wandb
resume_from_checkpoint:
desc: null
value: null
run_name:
desc: null
value: trial
save_on_each_node:
desc: null
value: false
save_steps:
desc: null
value: 2500
save_strategy:
desc: null
value: IntervalStrategy.STEPS
save_total_limit:
desc: null
value: null
seed:
desc: null
value: 42
sharded_ddp:
desc: null
value: []
skip_memory_metrics:
desc: null
value: true
tokenizer_name:
desc: null
value: ../gpt-2-tamil
tpu_metrics_debug:
desc: null
value: false
tpu_num_cores:
desc: null
value: null
train_file:
desc: null
value: null
use_fast_tokenizer:
desc: null
value: true
use_legacy_prediction_loop:
desc: null
value: false
validation_file:
desc: null
value: null
validation_split_percentage:
desc: null
value: 5
warmup_ratio:
desc: null
value: 0.0
warmup_steps:
desc: null
value: 1000
weight_decay:
desc: null
value: 0.01

1
scripts/wandb/run-20210715_091856-2v0tf7h4/files/events.out.tfevents.1626340740.t1v-n-ebe36c53-w-0.765413.3.v2 Symbolic link
View File

@@ -0,0 +1 @@
/home/tweety_abi/GPT2-Tamil/gpt-2-tamil/events.out.tfevents.1626340740.t1v-n-ebe36c53-w-0.765413.3.v2

123
scripts/wandb/run-20210715_091856-2v0tf7h4/files/requirements.txt Normal file
View File

@@ -0,0 +1,123 @@
absl-py==0.13.0
aiohttp==3.7.4.post0
appdirs==1.4.4
astunparse==1.6.3
async-timeout==3.0.1
attrs==21.2.0
backcall==0.2.0
black==21.6b0
cachetools==4.2.2
certifi==2021.5.30
cfgv==3.3.0
chardet==4.0.0
chex==0.0.7
click==8.0.1
configparser==5.0.2
cycler==0.10.0
datasets==1.8.1.dev0
decorator==5.0.9
dill==0.3.4
distlib==0.3.2
dm-tree==0.1.6
docker-pycreds==0.4.0
filelock==3.0.12
flake8==3.9.2
flatbuffers==1.12
flax==0.3.4
fsspec==2021.6.1
gast==0.4.0
gitdb==4.0.7
gitpython==3.1.18
google-auth-oauthlib==0.4.4
google-auth==1.32.1
google-pasta==0.2.0
grpcio==1.34.1
h5py==3.1.0
huggingface-hub==0.0.12
identify==2.2.10
idna==2.10
ipython-genutils==0.2.0
ipython==7.25.0
isort==5.9.1
jax==0.2.16
jaxlib==0.1.68
jedi==0.18.0
joblib==1.0.1
keras-nightly==2.5.0.dev2021032900
keras-preprocessing==1.1.2
kiwisolver==1.3.1
libtpu-nightly==0.1.dev20210615
markdown==3.3.4
matplotlib-inline==0.1.2
matplotlib==3.4.2
mccabe==0.6.1
msgpack==1.0.2
multidict==5.1.0
multiprocess==0.70.12.2
mypy-extensions==0.4.3
nodeenv==1.6.0
numpy==1.19.5
oauthlib==3.1.1
opt-einsum==3.3.0
optax==0.0.8
packaging==20.9
pandas==1.2.5
parso==0.8.2
pathspec==0.8.1
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
pillow==8.3.0
pip==20.0.2
pkg-resources==0.0.0
pre-commit==2.13.0
promise==2.3
prompt-toolkit==3.0.19
protobuf==3.17.3
psutil==5.8.0
ptyprocess==0.7.0
pyarrow==4.0.1
pyasn1-modules==0.2.8
pyasn1==0.4.8
pycodestyle==2.7.0
pyflakes==2.3.1
pygments==2.9.0
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2021.1
pyyaml==5.4.1
regex==2021.7.1
requests-oauthlib==1.3.0
requests==2.25.1
rsa==4.7.2
sacremoses==0.0.45
scipy==1.7.0
sentry-sdk==1.3.0
setuptools==44.0.0
shortuuid==1.0.1
six==1.15.0
smmap==4.0.0
subprocess32==3.5.4
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorboard==2.5.0
tensorflow-estimator==2.5.0
tensorflow==2.5.0
termcolor==1.1.0
tokenizers==0.10.3
toml==0.10.2
toolz==0.11.1
torch==1.9.0
tqdm==4.61.1
traitlets==5.0.5
transformers==4.9.0.dev0
typing-extensions==3.7.4.3
urllib3==1.26.6
virtualenv==20.4.7
wandb==0.10.33
wcwidth==0.2.5
werkzeug==2.0.1
wheel==0.36.2
wrapt==1.12.1
xxhash==2.0.2
yarl==1.6.3

49
scripts/wandb/run-20210715_091856-2v0tf7h4/files/wandb-metadata.json Normal file
View File

@@ -0,0 +1,49 @@
{
"os": "Linux-5.4.0-1043-gcp-x86_64-with-glibc2.29",
"python": "3.8.10",
"heartbeatAt": "2021-07-15T09:19:00.102585",
"startedAt": "2021-07-15T09:18:56.277815",
"docker": null,
"cpu_count": 96,
"cuda": null,
"args": [
"--output_dir=../gpt-2-tamil",
"--model_type=gpt2",
"--config_name=../gpt-2-tamil",
"--tokenizer_name=../gpt-2-tamil",
"--dataset_name=oscar",
"--dataset_config_name=unshuffled_deduplicated_ta",
"--do_train",
"--do_eval",
"--block_size=512",
"--per_device_train_batch_size=128",
"--per_device_eval_batch_size=128",
"--learning_rate=3e-5",
"--warmup_steps=1000",
"--adam_beta1=0.9",
"--adam_beta2=0.98",
"--weight_decay=0.01",
"--overwrite_output_dir",
"--num_train_epochs=10",
"--report_to",
"wandb",
"--run_name",
"trial",
"--logging_steps=500",
"--save_steps=2500",
"--eval_steps=2500",
"--preprocessing_num_workers=90"
],
"state": "running",
"program": "../src/run_clm_flax.py",
"codePath": "src/run_clm_flax.py",
"git": {
"remote": "https://github.com/AbinayaM02/GPT2-Tamil.git",
"commit": "69c9b7bf75b708a8f62cf5833d1b89acf5d1760b"
},
"email": "abinaya.m02@mphasis.com",
"root": "/home/tweety_abi/GPT2-Tamil",
"host": "t1v-n-ebe36c53-w-0",
"username": "tweety_abi",
"executable": "/home/tweety_abi/gpt2_env/bin/python"
}

1
scripts/wandb/run-20210715_091856-2v0tf7h4/files/wandb-summary.json Normal file
View File

@@ -0,0 +1 @@
{}

3
scripts/wandb/run-20210715_091856-2v0tf7h4/run-2v0tf7h4.wandb Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:74c24264810cc8a5625c9a6fd0093d95ea89e0980f556fce2e873e00ba0254c5
size 38212

301
scripts/wandb/run-20210715_092837-watdq7ib/files/config.yaml Normal file
View File

@@ -0,0 +1,301 @@
wandb_version: 1
__cached__setup_devices:
desc: null
value: cpu
_n_gpu:
desc: null
value: 0
_wandb:
desc: null
value:
cli_version: 0.10.33
framework: huggingface
huggingface_version: 4.9.0.dev0
is_jupyter_run: false
is_kaggle_kernel: false
python_version: 3.8.10
t:
1:
- 1
- 3
- 11
4: 3.8.10
5: 0.10.33
6: 4.9.0.dev0
8:
- 5
adafactor:
desc: null
value: false
adam_beta1:
desc: null
value: 0.9
adam_beta2:
desc: null
value: 0.98
adam_epsilon:
desc: null
value: 1.0e-08
block_size:
desc: null
value: 512
cache_dir:
desc: null
value: null
config_name:
desc: null
value: ../gpt-2-tamil
dataloader_drop_last:
desc: null
value: false
dataloader_num_workers:
desc: null
value: 0
dataloader_pin_memory:
desc: null
value: true
dataset_config_name:
desc: null
value: unshuffled_deduplicated_ta
dataset_name:
desc: null
value: oscar
ddp_find_unused_parameters:
desc: null
value: null
debug:
desc: null
value: []
deepspeed:
desc: null
value: null
disable_tqdm:
desc: null
value: false
do_eval:
desc: null
value: true
do_predict:
desc: null
value: false
do_train:
desc: null
value: true
dtype:
desc: null
value: float32
eval_accumulation_steps:
desc: null
value: null
eval_steps:
desc: null
value: 2500
evaluation_strategy:
desc: null
value: IntervalStrategy.NO
fp16:
desc: null
value: false
fp16_backend:
desc: null
value: auto
fp16_full_eval:
desc: null
value: false
fp16_opt_level:
desc: null
value: O1
gradient_accumulation_steps:
desc: null
value: 1
greater_is_better:
desc: null
value: null
group_by_length:
desc: null
value: false
ignore_data_skip:
desc: null
value: false
label_names:
desc: null
value: null
label_smoothing_factor:
desc: null
value: 0.0
learning_rate:
desc: null
value: 3.0e-05
length_column_name:
desc: null
value: length
load_best_model_at_end:
desc: null
value: false
local_rank:
desc: null
value: -1
log_level:
desc: null
value: -1
log_level_replica:
desc: null
value: -1
log_on_each_node:
desc: null
value: true
logging_dir:
desc: null
value: ../gpt-2-tamil/runs/Jul15_09-27-21_t1v-n-ebe36c53-w-0
logging_first_step:
desc: null
value: false
logging_steps:
desc: null
value: 500
logging_strategy:
desc: null
value: IntervalStrategy.STEPS
lr_scheduler_type:
desc: null
value: SchedulerType.LINEAR
max_eval_samples:
desc: null
value: null
max_grad_norm:
desc: null
value: 1.0
max_steps:
desc: null
value: -1
max_train_samples:
desc: null
value: null
metric_for_best_model:
desc: null
value: null
model_name_or_path:
desc: null
value: null
model_type:
desc: null
value: gpt2
mp_parameters:
desc: null
value: ''
no_cuda:
desc: null
value: false
num_train_epochs:
desc: null
value: 10.0
output_dir:
desc: null
value: ../gpt-2-tamil
overwrite_cache:
desc: null
value: false
overwrite_output_dir:
desc: null
value: true
past_index:
desc: null
value: -1
per_device_eval_batch_size:
desc: null
value: 64
per_device_train_batch_size:
desc: null
value: 64
per_gpu_eval_batch_size:
desc: null
value: null
per_gpu_train_batch_size:
desc: null
value: null
prediction_loss_only:
desc: null
value: false
preprocessing_num_workers:
desc: null
value: 90
push_to_hub:
desc: null
value: false
push_to_hub_model_id:
desc: null
value: gpt-2-tamil
push_to_hub_organization:
desc: null
value: null
push_to_hub_token:
desc: null
value: null
remove_unused_columns:
desc: null
value: true
report_to:
desc: null
value:
- wandb
resume_from_checkpoint:
desc: null
value: null
run_name:
desc: null
value: trial
save_on_each_node:
desc: null
value: false
save_steps:
desc: null
value: 2500
save_strategy:
desc: null
value: IntervalStrategy.STEPS
save_total_limit:
desc: null
value: null
seed:
desc: null
value: 42
sharded_ddp:
desc: null
value: []
skip_memory_metrics:
desc: null
value: true
tokenizer_name:
desc: null
value: ../gpt-2-tamil
tpu_metrics_debug:
desc: null
value: false
tpu_num_cores:
desc: null
value: null
train_file:
desc: null
value: null
use_fast_tokenizer:
desc: null
value: true
use_legacy_prediction_loop:
desc: null
value: false
validation_file:
desc: null
value: null
validation_split_percentage:
desc: null
value: 5
warmup_ratio:
desc: null
value: 0.0
warmup_steps:
desc: null
value: 1000
weight_decay:
desc: null
value: 0.01

1
scripts/wandb/run-20210715_092837-watdq7ib/files/events.out.tfevents.1626341319.t1v-n-ebe36c53-w-0.768105.3.v2 Symbolic link
View File

@@ -0,0 +1 @@
/home/tweety_abi/GPT2-Tamil/gpt-2-tamil/events.out.tfevents.1626341319.t1v-n-ebe36c53-w-0.768105.3.v2

123
scripts/wandb/run-20210715_092837-watdq7ib/files/requirements.txt Normal file
View File

@@ -0,0 +1,123 @@
absl-py==0.13.0
aiohttp==3.7.4.post0
appdirs==1.4.4
astunparse==1.6.3
async-timeout==3.0.1
attrs==21.2.0
backcall==0.2.0
black==21.6b0
cachetools==4.2.2
certifi==2021.5.30
cfgv==3.3.0
chardet==4.0.0
chex==0.0.7
click==8.0.1
configparser==5.0.2
cycler==0.10.0
datasets==1.8.1.dev0
decorator==5.0.9
dill==0.3.4
distlib==0.3.2
dm-tree==0.1.6
docker-pycreds==0.4.0
filelock==3.0.12
flake8==3.9.2
flatbuffers==1.12
flax==0.3.4
fsspec==2021.6.1
gast==0.4.0
gitdb==4.0.7
gitpython==3.1.18
google-auth-oauthlib==0.4.4
google-auth==1.32.1
google-pasta==0.2.0
grpcio==1.34.1
h5py==3.1.0
huggingface-hub==0.0.12
identify==2.2.10
idna==2.10
ipython-genutils==0.2.0
ipython==7.25.0
isort==5.9.1
jax==0.2.16
jaxlib==0.1.68
jedi==0.18.0
joblib==1.0.1
keras-nightly==2.5.0.dev2021032900
keras-preprocessing==1.1.2
kiwisolver==1.3.1
libtpu-nightly==0.1.dev20210615
markdown==3.3.4
matplotlib-inline==0.1.2
matplotlib==3.4.2
mccabe==0.6.1
msgpack==1.0.2
multidict==5.1.0
multiprocess==0.70.12.2
mypy-extensions==0.4.3
nodeenv==1.6.0
numpy==1.19.5
oauthlib==3.1.1
opt-einsum==3.3.0
optax==0.0.8
packaging==20.9
pandas==1.2.5
parso==0.8.2
pathspec==0.8.1
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
pillow==8.3.0
pip==20.0.2
pkg-resources==0.0.0
pre-commit==2.13.0
promise==2.3
prompt-toolkit==3.0.19
protobuf==3.17.3
psutil==5.8.0
ptyprocess==0.7.0
pyarrow==4.0.1
pyasn1-modules==0.2.8
pyasn1==0.4.8
pycodestyle==2.7.0
pyflakes==2.3.1
pygments==2.9.0
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2021.1
pyyaml==5.4.1
regex==2021.7.1
requests-oauthlib==1.3.0
requests==2.25.1
rsa==4.7.2
sacremoses==0.0.45
scipy==1.7.0
sentry-sdk==1.3.0
setuptools==44.0.0
shortuuid==1.0.1
six==1.15.0
smmap==4.0.0
subprocess32==3.5.4
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorboard==2.5.0
tensorflow-estimator==2.5.0
tensorflow==2.5.0
termcolor==1.1.0
tokenizers==0.10.3
toml==0.10.2
toolz==0.11.1
torch==1.9.0
tqdm==4.61.1
traitlets==5.0.5
transformers==4.9.0.dev0
typing-extensions==3.7.4.3
urllib3==1.26.6
virtualenv==20.4.7
wandb==0.10.33
wcwidth==0.2.5
werkzeug==2.0.1
wheel==0.36.2
wrapt==1.12.1
xxhash==2.0.2
yarl==1.6.3

49
scripts/wandb/run-20210715_092837-watdq7ib/files/wandb-metadata.json Normal file
View File

@@ -0,0 +1,49 @@
{
"os": "Linux-5.4.0-1043-gcp-x86_64-with-glibc2.29",
"python": "3.8.10",
"heartbeatAt": "2021-07-15T09:28:39.248463",
"startedAt": "2021-07-15T09:28:37.215410",
"docker": null,
"cpu_count": 96,
"cuda": null,
"args": [
"--output_dir=../gpt-2-tamil",
"--model_type=gpt2",
"--config_name=../gpt-2-tamil",
"--tokenizer_name=../gpt-2-tamil",
"--dataset_name=oscar",
"--dataset_config_name=unshuffled_deduplicated_ta",
"--do_train",
"--do_eval",
"--block_size=512",
"--per_device_train_batch_size=64",
"--per_device_eval_batch_size=64",
"--learning_rate=3e-5",
"--warmup_steps=1000",
"--adam_beta1=0.9",
"--adam_beta2=0.98",
"--weight_decay=0.01",
"--overwrite_output_dir",
"--num_train_epochs=10",
"--report_to",
"wandb",
"--run_name",
"trial",
"--logging_steps=500",
"--save_steps=2500",
"--eval_steps=2500",
"--preprocessing_num_workers=90"
],
"state": "running",
"program": "../src/run_clm_flax.py",
"codePath": "src/run_clm_flax.py",
"git": {
"remote": "https://github.com/AbinayaM02/GPT2-Tamil.git",
"commit": "69c9b7bf75b708a8f62cf5833d1b89acf5d1760b"
},
"email": "abinaya.m02@mphasis.com",
"root": "/home/tweety_abi/GPT2-Tamil",
"host": "t1v-n-ebe36c53-w-0",
"username": "tweety_abi",
"executable": "/home/tweety_abi/gpt2_env/bin/python"
}

1
scripts/wandb/run-20210715_092837-watdq7ib/files/wandb-summary.json Normal file
View File

@@ -0,0 +1 @@
{"global_step": 162500, "_timestamp": 1626515598.82067, "train_time": 3039175.75, "train_learning_rate": 2.0749264422192937e-06, "_step": 324025, "train_loss": 1.1235102415084839, "eval_loss": 1.1323037147521973, "eval_perplexity": 3.1027963161468506}

3
scripts/wandb/run-20210715_092837-watdq7ib/run-watdq7ib.wandb Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:17ccbfb69a2e91865a50d34837db9291fa2687143f65c6f6c712e23f40a46343
size 71362583

4
src/convert_flax_to_pytorch.py Normal file
View File

@@ -0,0 +1,4 @@
from transformers import GPT2LMHeadModel
model = GPT2LMHeadModel.from_pretrained("../gpt-2-tamil", from_flax=True)
model.save_pretrained("../")

8
src/create_config.py Normal file
View File

@@ -0,0 +1,8 @@
from transformers import GPT2Config
model_dir = "../gpt-2-tamil" # ${MODEL_DIR}
config = GPT2Config.from_pretrained(
"gpt2", resid_pdrop=0.0, embd_pdrop=0.0, attn_pdrop=0.0
)
config.save_pretrained(model_dir)

661
src/run_clm_flax.py Executable file
View File

@@ -0,0 +1,661 @@
#!/usr/bin/env python
# coding=utf-8
# Copyright 2021 The HuggingFace Team All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Pre-training/Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...) on a text file or a dataset.
Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
https://huggingface.co/models?filter=causal-lm
"""
# You can also adapt this script on your own causal language modeling task. Pointers for this are left as comments.
import logging
import math
import os
import sys
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable, Optional
import datasets
from datasets import Dataset, load_dataset, concatenate_datasets
from tqdm import tqdm
import jax
import jax.numpy as jnp
import optax
import transformers
import wandb
from flax import jax_utils, traverse_util
from flax.jax_utils import unreplicate
from flax.training import train_state
from flax.training.common_utils import get_metrics, onehot, shard, shard_prng_key
from transformers import (
CONFIG_MAPPING,
FLAX_MODEL_FOR_CAUSAL_LM_MAPPING,
AutoConfig,
AutoTokenizer,
FlaxAutoModelForCausalLM,
HfArgumentParser,
TrainingArguments,
is_tensorboard_available,
)
from transformers.testing_utils import CaptureLogger
logger = logging.getLogger(__name__)
MODEL_CONFIG_CLASSES = list(FLAX_MODEL_FOR_CAUSAL_LM_MAPPING.keys())
MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES)
@dataclass
class ModelArguments:
"""
Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch.
"""
model_name_or_path: Optional[str] = field(
default=None,
metadata={
"help": "The model checkpoint for weights initialization."
"Don't set if you want to train a model from scratch."
},
)
model_type: Optional[str] = field(
default=None,
metadata={"help": "If training from scratch, pass a model type from the list: " + ", ".join(MODEL_TYPES)},
)
config_name: Optional[str] = field(
default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"}
)
tokenizer_name: Optional[str] = field(
default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}
)
cache_dir: Optional[str] = field(
default=None, metadata={"help": "Where do you want to store the pretrained models downloaded from s3"}
)
use_fast_tokenizer: bool = field(
default=True,
metadata={"help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."},
)
dtype: Optional[str] = field(
default="float32",
metadata={
"help": "Floating-point format in which the model weights should be initialized and trained. Choose one of `[float32, float16, bfloat16]`."
},
)
@dataclass
class DataTrainingArguments:
"""
Arguments pertaining to what data we are going to input our model for training and eval.
"""
dataset_name: Optional[str] = field(
default=None, metadata={"help": "The name of the dataset to use (via the datasets library)."}
)
dataset_config_name: Optional[str] = field(
default=None, metadata={"help": "The configuration name of the dataset to use (via the datasets library)."}
)
train_file: Optional[str] = field(default=None, metadata={"help": "The input training data file (a text file)."})
validation_file: Optional[str] = field(
default=None,
metadata={"help": "An optional input evaluation data file to evaluate the perplexity on (a text file)."},
)
max_train_samples: Optional[int] = field(
default=None,
metadata={
"help": "For debugging purposes or quicker training, truncate the number of training examples to this "
"value if set."
},
)
max_eval_samples: Optional[int] = field(
default=None,
metadata={
"help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
overwrite_cache: bool = field(
default=False, metadata={"help": "Overwrite the cached training and evaluation sets"}
)
validation_split_percentage: Optional[int] = field(
default=5,
metadata={
"help": "The percentage of the train set used as validation set in case there's no validation split"
},
)
block_size: Optional[int] = field(
default=None,
metadata={
"help": "Optional input sequence length after tokenization. "
"The training dataset will be truncated in block of this size for training. "
"Default to the model max input length for single sentence inputs (take into account special tokens)."
},
)
overwrite_cache: bool = field(
default=False, metadata={"help": "Overwrite the cached training and evaluation sets"}
)
preprocessing_num_workers: Optional[int] = field(
default=None,
metadata={"help": "The number of processes to use for the preprocessing."},
)
def __post_init__(self):
if self.dataset_name is None and self.train_file is None and self.validation_file is None:
raise ValueError("Need either a dataset name or a training/validation file.")
else:
if self.train_file is not None:
extension = self.train_file.split(".")[-1]
assert extension in ["csv", "json", "txt"], "`train_file` should be a csv, a json or a txt file."
if self.validation_file is not None:
extension = self.validation_file.split(".")[-1]
assert extension in ["csv", "json", "txt"], "`validation_file` should be a csv, a json or a txt file."
class TrainState(train_state.TrainState):
dropout_rng: jnp.ndarray
def replicate(self):
return jax_utils.replicate(self).replace(dropout_rng=shard_prng_key(self.dropout_rng))
def data_loader(rng: jax.random.PRNGKey, dataset: Dataset, batch_size: int, shuffle: bool = False):
"""
Returns batches of size `batch_size` from truncated `dataset`, sharded over all local devices.
Shuffle batches if `shuffle` is `True`.
"""
steps_per_epoch = len(dataset) // batch_size
if shuffle:
batch_idx = jax.random.permutation(rng, len(dataset))
else:
batch_idx = jnp.arange(len(dataset))
batch_idx = batch_idx[: steps_per_epoch * batch_size] # Skip incomplete batch.
batch_idx = batch_idx.reshape((steps_per_epoch, batch_size))
for idx in batch_idx:
batch = dataset[idx]
batch = {k: jnp.array(v) for k, v in batch.items()}
batch = shard(batch)
yield batch
def write_train_metric(summary_writer, train_metrics, train_time, step):
summary_writer.scalar("train_time", train_time, step)
train_metrics = get_metrics(train_metrics)
for key, vals in train_metrics.items():
tag = f"train_{key}"
for i, val in enumerate(vals):
summary_writer.scalar(tag, val, step - len(vals) + i + 1)
def write_eval_metric(summary_writer, eval_metrics, step):
for metric_name, value in eval_metrics.items():
summary_writer.scalar(f"eval_{metric_name}", value, step)
def create_learning_rate_fn(
train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float
) -> Callable[[int], jnp.array]:
"""Returns a linear warmup, linear_decay learning rate function."""
steps_per_epoch = train_ds_size // train_batch_size
num_train_steps = steps_per_epoch * num_train_epochs
warmup_fn = optax.linear_schedule(init_value=0.0, end_value=learning_rate, transition_steps=num_warmup_steps)
decay_fn = optax.linear_schedule(
init_value=learning_rate, end_value=0, transition_steps=num_train_steps - num_warmup_steps
)
schedule_fn = optax.join_schedules(schedules=[warmup_fn, decay_fn], boundaries=[num_warmup_steps])
return schedule_fn
def main():
# See all possible arguments in src/transformers/training_args.py
# or by passing the --help flag to this script.
# We now keep distinct sets of args, for a cleaner separation of concerns.
parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
# If we pass only one argument to the script and it's the path to a json file,
# let's parse it to get our arguments.
model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
else:
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
if (
os.path.exists(training_args.output_dir)
and os.listdir(training_args.output_dir)
and training_args.do_train
and not training_args.overwrite_output_dir
):
raise ValueError(
f"Output directory ({training_args.output_dir}) already exists and is not empty."
"Use --overwrite_output_dir to overcome."
)
# Make one log on every process with the configuration for debugging.
logging.basicConfig(
format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
datefmt="%m/%d/%Y %H:%M:%S",
level=logging.INFO,
)
# Setup logging, we only want one process per machine to log things on the screen.
logger.setLevel(logging.INFO if jax.process_index() == 0 else logging.ERROR)
if jax.process_index() == 0:
datasets.utils.logging.set_verbosity_warning()
transformers.utils.logging.set_verbosity_info()
else:
datasets.utils.logging.set_verbosity_error()
transformers.utils.logging.set_verbosity_error()
# Set the verbosity to info of the Transformers logger (on main process only):
logger.info(f"Training/evaluation parameters {training_args}")
# Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
# or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
# (the dataset will be downloaded automatically from the datasets Hub).
#
# For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
# 'text' is found. You can easily tweak this behavior (see below).
#
# In distributed training, the load_dataset function guarantees that only one local process can concurrently
# download the dataset.
#GPT-2 tamil
logger.info(f"Loading dataset....")
print("Loading indic corp tamil dataset")
indic_tamil = load_dataset("csv",data_files="/tmp/indic_corp/ta.csv")
if data_args.dataset_name is not None:
# Downloading and loading a dataset from the hub.
dataset = load_dataset(
data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, keep_in_memory=False
)
if "validation" not in dataset.keys():
dataset["validation"] = load_dataset(
data_args.dataset_name,
data_args.dataset_config_name,
split=f"train[:{data_args.validation_split_percentage}%]",
cache_dir=model_args.cache_dir,
)
dataset["train"] = load_dataset(
data_args.dataset_name,
data_args.dataset_config_name,
split=f"train[{data_args.validation_split_percentage}%:]",
cache_dir=model_args.cache_dir,
)
## GPT2-tamil - adding indic_corp dataset manually
print("Concatenating datasets")
#pdb.set_trace()
dataset['train'] = concatenate_datasets([indic_tamil['train'],dataset['train']])
else:
data_files = {}
if data_args.train_file is not None:
data_files["train"] = data_args.train_file
if data_args.validation_file is not None:
data_files["validation"] = data_args.validation_file
extension = data_args.train_file.split(".")[-1]
if extension == "txt":
extension = "text"
dataset = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir)
# See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
# https://huggingface.co/docs/datasets/loading_datasets.html.
# Load pretrained model and tokenizer
# Distributed training:
# The .from_pretrained methods guarantee that only one local process can concurrently
# download model & vocab.
if model_args.config_name:
config = AutoConfig.from_pretrained(model_args.config_name, cache_dir=model_args.cache_dir)
elif model_args.model_name_or_path:
config = AutoConfig.from_pretrained(model_args.model_name_or_path, cache_dir=model_args.cache_dir)
else:
config = CONFIG_MAPPING[model_args.model_type]()
logger.warning("You are instantiating a new config instance from scratch.")
if model_args.tokenizer_name:
tokenizer = AutoTokenizer.from_pretrained(
model_args.tokenizer_name, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer
)
elif model_args.model_name_or_path:
tokenizer = AutoTokenizer.from_pretrained(
model_args.model_name_or_path, cache_dir=model_args.cache_dir, use_fast=model_args.use_fast_tokenizer
)
else:
raise ValueError(
"You are instantiating a new tokenizer from scratch. This is not supported by this script."
"You can do it from another script, save it, and load it from here, using --tokenizer_name."
)
if model_args.model_name_or_path:
model = FlaxAutoModelForCausalLM.from_pretrained(
model_args.model_name_or_path, config=config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype)
)
else:
model = FlaxAutoModelForCausalLM.from_config(
config, seed=training_args.seed, dtype=getattr(jnp, model_args.dtype)
)
# Preprocessing the datasets.
# First we tokenize all the texts.
if training_args.do_train:
column_names = dataset["train"].column_names
else:
column_names = dataset["validation"].column_names
text_column_name = "text" if "text" in column_names else column_names[0]
# since this will be pickled to avoid _LazyModule error in Hasher force logger loading before tokenize_function
tok_logger = transformers.utils.logging.get_logger("transformers.tokenization_utils_base")
def tokenize_function(examples):
with CaptureLogger(tok_logger) as cl:
output = tokenizer(examples[text_column_name])
# clm input could be much much longer than block_size
if "Token indices sequence length is longer than the" in cl.out:
tok_logger.warning(
"^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits before being passed to the model."
)
return output
tokenized_datasets = dataset.map(
tokenize_function,
batched=True,
num_proc=data_args.preprocessing_num_workers,
remove_columns=column_names,
load_from_cache_file=not data_args.overwrite_cache,
)
if data_args.block_size is None:
block_size = tokenizer.model_max_length
if block_size > config.max_position_embeddings:
logger.warning(
f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). "
"Picking 1024 instead. You can change that default value by passing --block_size xxx."
)
block_size = 1024
else:
if data_args.block_size > tokenizer.model_max_length:
logger.warning(
f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model"
f"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}."
)
block_size = min(data_args.block_size, tokenizer.model_max_length)
# Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size.
def group_texts(examples):
# Concatenate all texts.
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
total_length = len(concatenated_examples[list(examples.keys())[0]])
# We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
# customize this part to your needs.
if total_length >= block_size:
total_length = (total_length // block_size) * block_size
# Split by chunks of max_len.
result = {
k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
for k, t in concatenated_examples.items()
}
result["labels"] = result["input_ids"].copy()
return result
# Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a remainder
# for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value might be slower
# to preprocess.
#
# To speed up this part, we use multiprocessing. See the documentation of the map method for more information:
# https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map
lm_datasets = tokenized_datasets.map(
group_texts,
batched=True,
num_proc=data_args.preprocessing_num_workers,
load_from_cache_file=not data_args.overwrite_cache,
)
if training_args.do_train:
if "train" not in tokenized_datasets:
raise ValueError("--do_train requires a train dataset")
train_dataset = lm_datasets["train"]
if data_args.max_train_samples is not None:
train_dataset = train_dataset.select(range(data_args.max_train_samples))
if training_args.do_eval:
if "validation" not in tokenized_datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_dataset = lm_datasets["validation"]
if data_args.max_eval_samples is not None:
eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
# Enable tensorboard only on the master node
has_tensorboard = is_tensorboard_available()
if has_tensorboard and jax.process_index() == 0:
wandb.init(
entity='wandb',
project='hf-flax-gpt2-tamil',
sync_tensorboard=True
)
wandb.config.update(training_args) # optional, log your configs
wandb.config.update(model_args) # optional, log your configs
wandb.config.update(data_args) # optional, log your configs
try:
from flax.metrics.tensorboard import SummaryWriter
summary_writer = SummaryWriter(log_dir=Path(training_args.output_dir))
except ImportError as ie:
has_tensorboard = False
logger.warning(
f"Unable to display metrics through TensorBoard because some package are not installed: {ie}"
)
else:
logger.warning(
"Unable to display metrics through TensorBoard because the package is not installed: "
"Please run pip install tensorboard to enable."
)
# Initialize our training
rng = jax.random.PRNGKey(training_args.seed)
rng, dropout_rng = jax.random.split(rng)
# Store some constant
num_epochs = int(training_args.num_train_epochs)
train_batch_size = int(training_args.per_device_train_batch_size) * jax.device_count()
eval_batch_size = int(training_args.per_device_eval_batch_size) * jax.device_count()
steps_per_epoch = len(train_dataset) // train_batch_size
total_train_steps = steps_per_epoch * num_epochs
# Create learning rate schedule
linear_decay_lr_schedule_fn = create_learning_rate_fn(
len(train_dataset),
train_batch_size,
training_args.num_train_epochs,
training_args.warmup_steps,
training_args.learning_rate,
)
# We use Optax's "masking" functionality to not apply weight decay
# to bias and LayerNorm scale parameters. decay_mask_fn returns a
# mask boolean with the same structure as the parameters.
# The mask is True for parameters that should be decayed.
# Note that this mask is specifically adapted for FlaxGPT2.
# For other models, one should correct the layer norm parameter naming
# accordingly.
def decay_mask_fn(params):
flat_params = traverse_util.flatten_dict(params)
flat_mask = {
path: (path[-1] != "bias" and path[-2:] not in [("ln_1", "scale"), ("ln_2", "scale"), ("ln_f", "scale")])
for path in flat_params
}
return traverse_util.unflatten_dict(flat_mask)
# create adam optimizer
if training_args.adafactor:
# We use the default parameters here to initialize adafactor,
# For more details about the parameters please check https://github.com/deepmind/optax/blob/ed02befef9bf81cbbf236be3d2b0e032e9ed4a40/optax/_src/alias.py#L74
optimizer = optax.adafactor(
learning_rate=linear_decay_lr_schedule_fn,
)
else:
optimizer = optax.adamw(
learning_rate=linear_decay_lr_schedule_fn,
b1=training_args.adam_beta1,
b2=training_args.adam_beta2,
eps=training_args.adam_epsilon,
weight_decay=training_args.weight_decay,
mask=decay_mask_fn,
)
# Setup train state
state = TrainState.create(apply_fn=model.__call__, params=model.params, tx=optimizer, dropout_rng=dropout_rng)
def loss_fn(logits, labels):
shift_logits = logits[..., :-1, :]
shift_labels = labels[..., 1:]
loss = optax.softmax_cross_entropy(shift_logits, onehot(shift_labels, shift_logits.shape[-1]))
return loss.mean()
# Define gradient update step fn
def train_step(state, batch):
dropout_rng, new_dropout_rng = jax.random.split(state.dropout_rng)
def compute_loss(params):
labels = batch.pop("labels")
logits = state.apply_fn(**batch, params=params, dropout_rng=dropout_rng, train=True)[0]
loss = loss_fn(logits, labels)
return loss
grad_fn = jax.value_and_grad(compute_loss)
loss, grad = grad_fn(state.params)
grad = jax.lax.pmean(grad, "batch")
new_state = state.apply_gradients(grads=grad, dropout_rng=new_dropout_rng)
metrics = {"loss": loss, "learning_rate": linear_decay_lr_schedule_fn(state.step)}
metrics = jax.lax.pmean(metrics, axis_name="batch")
return new_state, metrics
# Define eval fn
def eval_step(params, batch):
labels = batch.pop("labels")
logits = model(**batch, params=params, train=False)[0]
loss = loss_fn(logits, labels)
# summarize metrics
metrics = {"loss": loss}
metrics = jax.lax.pmean(metrics, axis_name="batch")
return metrics
# Create parallel version of the train and eval step
p_train_step = jax.pmap(train_step, "batch", donate_argnums=(0,))
p_eval_step = jax.pmap(eval_step, "batch")
# Replicate the train state on each device
state = state.replicate()
logger.info("***** Running training *****")
logger.info(f" Num examples = {len(train_dataset)}")
logger.info(f" Num Epochs = {num_epochs}")
logger.info(f" Instantaneous batch size per device = {training_args.per_device_train_batch_size}")
logger.info(f" Total train batch size (w. parallel & distributed) = {train_batch_size}")
logger.info(f" Total optimization steps = {total_train_steps}")
train_time = 0
train_metrics = []
epochs = tqdm(range(num_epochs), desc=f"Epoch ... (1/{num_epochs})", position=0)
for epoch in epochs:
# ======================== Training ================================
train_start = time.time()
# Create sampling rng
rng, input_rng = jax.random.split(rng)
# Generate an epoch by shuffling sampling indices from the train dataset
train_loader = data_loader(input_rng, train_dataset, train_batch_size, shuffle=True)
steps_per_epoch = len(train_dataset) // train_batch_size
# train
for step in tqdm(range(steps_per_epoch), desc="Training...", position=1, leave=False):
batch = next(train_loader)
state, train_metric = p_train_step(state, batch)
train_metrics.append(train_metric)
cur_step = epoch * (len(train_dataset) // train_batch_size) + step
if cur_step % training_args.logging_steps == 0 and cur_step > 0:
# Save metrics
train_metric = unreplicate(train_metric)
train_time += time.time() - train_start
if has_tensorboard and jax.process_index() == 0:
write_train_metric(summary_writer, train_metrics, train_time, cur_step)
epochs.write(
f"Step... ({cur_step} | Loss: {train_metric['loss'].mean()}, Learning Rate: {train_metric['learning_rate'].mean()})"
)
train_metrics = []
if cur_step % training_args.eval_steps == 0 and cur_step > 0:
# ======================== Evaluating ==============================
eval_metrics = []
eval_loader = data_loader(input_rng, eval_dataset, eval_batch_size)
eval_steps = len(eval_dataset) // eval_batch_size
for _ in tqdm(range(eval_steps), desc="Evaluating...", position=2, leave=False):
# Model forward
batch = next(eval_loader)
metrics = p_eval_step(state.params, batch)
eval_metrics.append(metrics)
# normalize eval metrics
eval_metrics = get_metrics(eval_metrics)
eval_metrics = jax.tree_map(jnp.mean, eval_metrics)
try:
eval_metrics["perplexity"] = math.exp(eval_metrics["loss"])
except OverflowError:
eval_metrics["perplexity"] = float("inf")
# Print metrics and update progress bar
desc = f"Step... ({cur_step} | Eval Loss: {eval_metrics['loss']} | Eval Perplexity: {eval_metrics['perplexity']})"
epochs.write(desc)
epochs.desc = desc
# Save metrics
if has_tensorboard and jax.process_index() == 0:
write_eval_metric(summary_writer, eval_metrics, cur_step)
if cur_step % training_args.save_steps == 0 and cur_step > 0:
# save checkpoint after each epoch and push checkpoint to the hub
if jax.process_index() == 0:
params = jax.device_get(unreplicate(state.params))
model.save_pretrained(
training_args.output_dir,
params=params,
push_to_hub=training_args.push_to_hub,
commit_message=f"Saving weights and logs of step {cur_step}",
)
if __name__ == "__main__":
main()

40
src/train_tokenizer.py Normal file
View File

@@ -0,0 +1,40 @@
from datasets import load_dataset,concatenate_datasets
from tokenizers import trainers, Tokenizer, normalizers, ByteLevelBPETokenizer
from datasets import load_dataset
from tokenizers import ByteLevelBPETokenizer # Tokenizer, normalizers, trainers
model_dir = "../gpt-2-tamil" # ${MODEL_DIR}
# load dataset
dataset = load_dataset("oscar", "unshuffled_deduplicated_ta", split="train")
indic_tamil = load_dataset("csv",data_files="/tmp/indic_corp/ta.csv")
dataset = concatenate_datasets([dataset,indic_tamil['train']])
# Instantiate tokenizer
tokenizer = ByteLevelBPETokenizer()
def batch_iterator(batch_size=1000):
for i in range(0, len(dataset), batch_size):
yield dataset[i : i + batch_size]["text"]
# Customized training
tokenizer.train_from_iterator(
batch_iterator(),
vocab_size=50265,
min_frequency=2,
special_tokens=[
"<s>",
"<pad>",
"</s>",
"<unk>",
"<mask>",
],
)
# Save files to disk
tokenizer.save(f"{model_dir}/tokenizer.json")

1
tokenizer.json Normal file
View File

File diff suppressed because one or more lines are too long