NastasiaM/mbert-loraxs-qa-vanilla
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: NastasiaM/mbert-loraxs-qa-vanilla
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-10 00:18:47 +08:00
commit d696db249d
795 changed files with 8285474 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

206
run-4/checkpoint-1268/README.md Normal file
View File

@@ -0,0 +1,206 @@
---
base_model: bert-base-multilingual-cased
library_name: peft
tags:
- base_model:adapter:bert-base-multilingual-cased
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1

47
run-4/checkpoint-1268/adapter_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "bert-base-multilingual-cased",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 128,
"lora_bias": false,
"lora_dropout": 0.12668659169460122,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": [
"qa_outputs"
],
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 256,
"rank_pattern": {},
"revision": null,
"target_modules": [
"dense",
"query",
"key",
"value"
],
"target_parameters": null,
"task_type": "QUESTION_ANS",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

3
run-4/checkpoint-1268/adapter_model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3bc5d69e94123e6c990f3cc879cc9833ab4a8adea504a6bb073f96af442c3607
size 188782560

34
run-4/checkpoint-1268/config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"add_cross_attention": false,
"architectures": [
"BertForQuestionAnswering"
],
"attention_probs_dropout_prob": 0.1,
"bos_token_id": null,
"classifier_dropout": null,
"directionality": "bidi",
"dtype": "float32",
"eos_token_id": null,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"is_decoder": false,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"tie_word_embeddings": true,
"transformers_version": "5.9.0",
"type_vocab_size": 2,
"use_cache": true,
"vocab_size": 119547
}

3
run-4/checkpoint-1268/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bfae9c394fed574ce921fd4588d88da66c97aa7ee960d1ac9d63cd1899e5b4d9
size 709080872

3
run-4/checkpoint-1268/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bb8aced1bd8803262c3697f106ecc38ec5fd5c2856ab1121c4334be18fd6ca41
size 37825419

3
run-4/checkpoint-1268/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5f01dad392b3a1f2caec7a150c0544b65d0c0fe57c3641a2898fab06e1a44d28
size 14645

3
run-4/checkpoint-1268/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:67f01e4c66340bfdf149b129e6a6353fdc41304202a218f951be98420cb8b31f
size 1383

3
run-4/checkpoint-1268/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0f86fb28f2953381514bfcf6b8d237e7b241b8dd93304fc36bc0eba74f9361f1
size 1465

119711
run-4/checkpoint-1268/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

15
run-4/checkpoint-1268/tokenizer_config.json Normal file
View File

@@ -0,0 +1,15 @@
{
"backend": "tokenizers",
"cls_token": "[CLS]",
"do_lower_case": false,
"is_local": false,
"local_files_only": false,
"mask_token": "[MASK]",
"model_max_length": 512,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

75
run-4/checkpoint-1268/trainer_state.json Normal file
View File

@@ -0,0 +1,75 @@
{
"best_global_step": 1268,
"best_metric": 58.06834889049466,
"best_model_checkpoint": "mbert-loraxs-qa-vanilla/run-4/checkpoint-1268",
"epoch": 2.0,
"eval_steps": 500,
"global_step": 1268,
"is_hyper_param_search": true,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 1.0,
"grad_norm": 10.389764785766602,
"learning_rate": 2.203344436447427e-05,
"loss": 4.208139894888604,
"step": 634
},
{
"epoch": 1.0,
"eval_exact_match": 31.68,
"eval_f1": 45.273247698850945,
"eval_loss": 2.3350870609283447,
"eval_runtime": 7.7987,
"eval_samples_per_second": 162.207,
"eval_steps_per_second": 10.258,
"step": 634
},
{
"epoch": 2.0,
"grad_norm": 9.32017707824707,
"learning_rate": 1.1025403602396503e-05,
"loss": 2.3320699505249407,
"step": 1268
},
{
"epoch": 2.0,
"eval_exact_match": 45.52,
"eval_f1": 58.06834889049466,
"eval_loss": 1.9215565919876099,
"eval_runtime": 7.8078,
"eval_samples_per_second": 162.017,
"eval_steps_per_second": 10.246,
"step": 1268
}
],
"logging_steps": 500,
"max_steps": 1902,
"num_input_tokens_seen": 0,
"num_train_epochs": 3,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 6180991462324224.0,
"train_batch_size": 16,
"trial_name": null,
"trial_params": {
"learning_rate": 2.9707819785354985e-05,
"lora_alpha": 128,
"lora_dropout": 0.12668659169460122,
"lora_r": 256,
"num_train_epochs": 3,
"weight_decay": 0.05519317165246046
}
}

3
run-4/checkpoint-1268/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5829e303e9d08a1a0b596f605e35b5586828786be12a667b61d1a56522a79ce0
size 5201

206
run-4/checkpoint-1902/README.md Normal file
View File

@@ -0,0 +1,206 @@
---
base_model: bert-base-multilingual-cased
library_name: peft
tags:
- base_model:adapter:bert-base-multilingual-cased
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1

47
run-4/checkpoint-1902/adapter_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "bert-base-multilingual-cased",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 128,
"lora_bias": false,
"lora_dropout": 0.12668659169460122,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": [
"qa_outputs"
],
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 256,
"rank_pattern": {},
"revision": null,
"target_modules": [
"dense",
"query",
"key",
"value"
],
"target_parameters": null,
"task_type": "QUESTION_ANS",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

3
run-4/checkpoint-1902/adapter_model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9c48d624d3ebcf21f71392b5a24885f40725eba208ed1cc97b0d6c65a323e25d
size 188782560

34
run-4/checkpoint-1902/config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"add_cross_attention": false,
"architectures": [
"BertForQuestionAnswering"
],
"attention_probs_dropout_prob": 0.1,
"bos_token_id": null,
"classifier_dropout": null,
"directionality": "bidi",
"dtype": "float32",
"eos_token_id": null,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"is_decoder": false,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"tie_word_embeddings": true,
"transformers_version": "5.9.0",
"type_vocab_size": 2,
"use_cache": true,
"vocab_size": 119547
}

3
run-4/checkpoint-1902/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:55af4f5f66d16123ceec09bb7b52cc99aac70aacba1b898dfbc6ce6ce1fc430d
size 709080872

3
run-4/checkpoint-1902/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e37d761b391036abf522fcf2c71a87d417c2bb9e6770b2dafafc94d3f78b95fa
size 37825419

3
run-4/checkpoint-1902/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5baa894f7340f1dc95047ebad749e7164dbe265fb53d8ccd31e1965197695367
size 14645

3
run-4/checkpoint-1902/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:93b25fdda0c42a234ff0e01ac84ccf5446fb04fdcd8b69e452b9ab7748255d2f
size 1383

3
run-4/checkpoint-1902/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1b5ed128454cb18b01cf6ff2367c1ddbd8cd5b926abc9f8bdab1ea7d302dc16d
size 1465

119711
run-4/checkpoint-1902/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

15
run-4/checkpoint-1902/tokenizer_config.json Normal file
View File

@@ -0,0 +1,15 @@
{
"backend": "tokenizers",
"cls_token": "[CLS]",
"do_lower_case": false,
"is_local": false,
"local_files_only": false,
"mask_token": "[MASK]",
"model_max_length": 512,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

92
run-4/checkpoint-1902/trainer_state.json Normal file
View File

@@ -0,0 +1,92 @@
{
"best_global_step": 1902,
"best_metric": 61.324047720829896,
"best_model_checkpoint": "mbert-loraxs-qa-vanilla/run-4/checkpoint-1902",
"epoch": 3.0,
"eval_steps": 500,
"global_step": 1902,
"is_hyper_param_search": true,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 1.0,
"grad_norm": 10.389764785766602,
"learning_rate": 2.203344436447427e-05,
"loss": 4.208139894888604,
"step": 634
},
{
"epoch": 1.0,
"eval_exact_match": 31.68,
"eval_f1": 45.273247698850945,
"eval_loss": 2.3350870609283447,
"eval_runtime": 7.7987,
"eval_samples_per_second": 162.207,
"eval_steps_per_second": 10.258,
"step": 634
},
{
"epoch": 2.0,
"grad_norm": 9.32017707824707,
"learning_rate": 1.1025403602396503e-05,
"loss": 2.3320699505249407,
"step": 1268
},
{
"epoch": 2.0,
"eval_exact_match": 45.52,
"eval_f1": 58.06834889049466,
"eval_loss": 1.9215565919876099,
"eval_runtime": 7.8078,
"eval_samples_per_second": 162.017,
"eval_steps_per_second": 10.246,
"step": 1268
},
{
"epoch": 3.0,
"grad_norm": 8.947677612304688,
"learning_rate": 1.736284031873465e-08,
"loss": 2.0585284789653984,
"step": 1902
},
{
"epoch": 3.0,
"eval_exact_match": 48.0,
"eval_f1": 61.324047720829896,
"eval_loss": 1.828240990638733,
"eval_runtime": 7.8614,
"eval_samples_per_second": 160.914,
"eval_steps_per_second": 10.176,
"step": 1902
}
],
"logging_steps": 500,
"max_steps": 1902,
"num_input_tokens_seen": 0,
"num_train_epochs": 3,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": true
},
"attributes": {}
}
},
"total_flos": 9271487193486336.0,
"train_batch_size": 16,
"trial_name": null,
"trial_params": {
"learning_rate": 2.9707819785354985e-05,
"lora_alpha": 128,
"lora_dropout": 0.12668659169460122,
"lora_r": 256,
"num_train_epochs": 3,
"weight_decay": 0.05519317165246046
}
}

3
run-4/checkpoint-1902/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5829e303e9d08a1a0b596f605e35b5586828786be12a667b61d1a56522a79ce0
size 5201

206
run-4/checkpoint-2536/README.md Normal file
View File

@@ -0,0 +1,206 @@
---
base_model: bert-base-multilingual-cased
library_name: peft
tags:
- base_model:adapter:bert-base-multilingual-cased
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1

47
run-4/checkpoint-2536/adapter_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "bert-base-multilingual-cased",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 64,
"lora_bias": false,
"lora_dropout": 0.17883000020205417,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": [
"qa_outputs"
],
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 8,
"rank_pattern": {},
"revision": null,
"target_modules": [
"value",
"dense",
"key",
"query"
],
"target_parameters": null,
"task_type": "QUESTION_ANS",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

3
run-4/checkpoint-2536/adapter_model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d3f7ac44f76ad4592b4eeef19e54b26bb11593287eaad09fe709342a5b281945
size 5364440

3
run-4/checkpoint-2536/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e15fbaff5d5c3bdd751e543a09b6d8c9056ec6f4480a1ac69cb227ae649ff8ab
size 112715

3
run-4/checkpoint-2536/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:09625a2375cb95e562619265862e4e3ceca9d87f214b48c4a4792f58dc30d4ed
size 14645

3
run-4/checkpoint-2536/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eeae7ca8c4f0807a5d25d354397e3f8ae44c13b710b70d305b8eb51ecae39380
size 1383

3
run-4/checkpoint-2536/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0525e4a6cb1c3396f225a6107228ea5496e01fffa9b99e958fbeaa1c37a56546
size 1465

119711
run-4/checkpoint-2536/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

15
run-4/checkpoint-2536/tokenizer_config.json Normal file
View File

@@ -0,0 +1,15 @@
{
"backend": "tokenizers",
"cls_token": "[CLS]",
"do_lower_case": false,
"is_local": false,
"local_files_only": false,
"mask_token": "[MASK]",
"model_max_length": 512,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

109
run-4/checkpoint-2536/trainer_state.json Normal file
View File

@@ -0,0 +1,109 @@
{
"best_global_step": 2536,
"best_metric": 26.874869223819044,
"best_model_checkpoint": "mbert-loraxs-qa-vanilla/run-4/checkpoint-2536",
"epoch": 4.0,
"eval_steps": 500,
"global_step": 2536,
"is_hyper_param_search": true,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 1.0,
"grad_norm": 11.041559219360352,
"learning_rate": 5.605194150710478e-05,
"loss": 5.270265597274251,
"step": 634
},
{
"epoch": 1.0,
"eval_exact_match": 2.8,
"eval_f1": 8.945758396151355,
"eval_loss": 4.365081310272217,
"eval_runtime": 7.7244,
"eval_samples_per_second": 163.766,
"eval_steps_per_second": 10.357,
"step": 634
},
{
"epoch": 2.0,
"grad_norm": 18.68657684326172,
"learning_rate": 4.204447957746172e-05,
"loss": 4.260111811785292,
"step": 1268
},
{
"epoch": 2.0,
"eval_exact_match": 6.56,
"eval_f1": 13.990675637585385,
"eval_loss": 3.790620803833008,
"eval_runtime": 7.6676,
"eval_samples_per_second": 164.979,
"eval_steps_per_second": 10.433,
"step": 1268
},
{
"epoch": 3.0,
"grad_norm": 26.60645866394043,
"learning_rate": 2.8037017647818668e-05,
"loss": 3.8954057663397084,
"step": 1902
},
{
"epoch": 3.0,
"eval_exact_match": 10.72,
"eval_f1": 20.682353135829914,
"eval_loss": 3.431551933288574,
"eval_runtime": 7.5743,
"eval_samples_per_second": 167.013,
"eval_steps_per_second": 10.562,
"step": 1902
},
{
"epoch": 4.0,
"grad_norm": 39.42831039428711,
"learning_rate": 1.4029555718175614e-05,
"loss": 3.6393634784108833,
"step": 2536
},
{
"epoch": 4.0,
"eval_exact_match": 15.92,
"eval_f1": 26.874869223819044,
"eval_loss": 3.1858983039855957,
"eval_runtime": 7.6014,
"eval_samples_per_second": 166.418,
"eval_steps_per_second": 10.524,
"step": 2536
}
],
"logging_steps": 500,
"max_steps": 3170,
"num_input_tokens_seen": 0,
"num_train_epochs": 5,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 8075628404895744.0,
"train_batch_size": 16,
"trial_name": null,
"trial_params": {
"learning_rate": 6.303357868339374e-05,
"lora_alpha": 64,
"lora_dropout": 0.17883000020205417,
"lora_r": 8,
"num_train_epochs": 5,
"weight_decay": 0.043219314058624864
}
}

3
run-4/checkpoint-2536/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:71c4bfd7241f4297ca286c6ddce111a4d8ef94c1a7941ee62c4f241a66465d22
size 5201

206
run-4/checkpoint-3170/README.md Normal file
View File

@@ -0,0 +1,206 @@
---
base_model: bert-base-multilingual-cased
library_name: peft
tags:
- base_model:adapter:bert-base-multilingual-cased
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1

47
run-4/checkpoint-3170/adapter_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "bert-base-multilingual-cased",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 64,
"lora_bias": false,
"lora_dropout": 0.17883000020205417,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": [
"qa_outputs"
],
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 8,
"rank_pattern": {},
"revision": null,
"target_modules": [
"value",
"dense",
"key",
"query"
],
"target_parameters": null,
"task_type": "QUESTION_ANS",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

3
run-4/checkpoint-3170/adapter_model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:816e93506c7b90765e2f7c51b8132b957d363e25cb1c5fd41d9be7e20c95e2da
size 5364440

3
run-4/checkpoint-3170/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ea18ed747db2108c73114bac5636f07ac6a98f5b64afa3cd3c4979affe0473a0
size 112715

3
run-4/checkpoint-3170/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c9d41e041a3403b0bec3240689e466545a334b21ab01100fde2812d38720c032
size 14645

3
run-4/checkpoint-3170/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5bbb25a66407f850959fa5824662cd10b6d059847c7f687e4a82f1145fa46739
size 1383

3
run-4/checkpoint-3170/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:231b80def99e7209c9f84a6171839b2fc3da2e2b4ff2eea282407ca97906e82b
size 1465

119711
run-4/checkpoint-3170/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

15
run-4/checkpoint-3170/tokenizer_config.json Normal file
View File

@@ -0,0 +1,15 @@
{
"backend": "tokenizers",
"cls_token": "[CLS]",
"do_lower_case": false,
"is_local": false,
"local_files_only": false,
"mask_token": "[MASK]",
"model_max_length": 512,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

126
run-4/checkpoint-3170/trainer_state.json Normal file
View File

@@ -0,0 +1,126 @@
{
"best_global_step": 3170,
"best_metric": 27.95988140372216,
"best_model_checkpoint": "mbert-loraxs-qa-vanilla/run-4/checkpoint-3170",
"epoch": 5.0,
"eval_steps": 500,
"global_step": 3170,
"is_hyper_param_search": true,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 1.0,
"grad_norm": 11.041559219360352,
"learning_rate": 5.605194150710478e-05,
"loss": 5.270265597274251,
"step": 634
},
{
"epoch": 1.0,
"eval_exact_match": 2.8,
"eval_f1": 8.945758396151355,
"eval_loss": 4.365081310272217,
"eval_runtime": 7.7244,
"eval_samples_per_second": 163.766,
"eval_steps_per_second": 10.357,
"step": 634
},
{
"epoch": 2.0,
"grad_norm": 18.68657684326172,
"learning_rate": 4.204447957746172e-05,
"loss": 4.260111811785292,
"step": 1268
},
{
"epoch": 2.0,
"eval_exact_match": 6.56,
"eval_f1": 13.990675637585385,
"eval_loss": 3.790620803833008,
"eval_runtime": 7.6676,
"eval_samples_per_second": 164.979,
"eval_steps_per_second": 10.433,
"step": 1268
},
{
"epoch": 3.0,
"grad_norm": 26.60645866394043,
"learning_rate": 2.8037017647818668e-05,
"loss": 3.8954057663397084,
"step": 1902
},
{
"epoch": 3.0,
"eval_exact_match": 10.72,
"eval_f1": 20.682353135829914,
"eval_loss": 3.431551933288574,
"eval_runtime": 7.5743,
"eval_samples_per_second": 167.013,
"eval_steps_per_second": 10.562,
"step": 1902
},
{
"epoch": 4.0,
"grad_norm": 39.42831039428711,
"learning_rate": 1.4029555718175614e-05,
"loss": 3.6393634784108833,
"step": 2536
},
{
"epoch": 4.0,
"eval_exact_match": 15.92,
"eval_f1": 26.874869223819044,
"eval_loss": 3.1858983039855957,
"eval_runtime": 7.6014,
"eval_samples_per_second": 166.418,
"eval_steps_per_second": 10.524,
"step": 2536
},
{
"epoch": 5.0,
"grad_norm": 37.898101806640625,
"learning_rate": 2.2093788532560022e-08,
"loss": 3.5152430007886437,
"step": 3170
},
{
"epoch": 5.0,
"eval_exact_match": 16.32,
"eval_f1": 27.95988140372216,
"eval_loss": 3.118779182434082,
"eval_runtime": 7.6488,
"eval_samples_per_second": 165.385,
"eval_steps_per_second": 10.459,
"step": 3170
}
],
"logging_steps": 500,
"max_steps": 3170,
"num_input_tokens_seen": 0,
"num_train_epochs": 5,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": true
},
"attributes": {}
}
},
"total_flos": 1.009453550611968e+16,
"train_batch_size": 16,
"trial_name": null,
"trial_params": {
"learning_rate": 6.303357868339374e-05,
"lora_alpha": 64,
"lora_dropout": 0.17883000020205417,
"lora_r": 8,
"num_train_epochs": 5,
"weight_decay": 0.043219314058624864
}
}

3
run-4/checkpoint-3170/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:71c4bfd7241f4297ca286c6ddce111a4d8ef94c1a7941ee62c4f241a66465d22
size 5201

206
run-4/checkpoint-634/README.md Normal file
View File

@@ -0,0 +1,206 @@
---
base_model: bert-base-multilingual-cased
library_name: peft
tags:
- base_model:adapter:bert-base-multilingual-cased
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1

47
run-4/checkpoint-634/adapter_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "bert-base-multilingual-cased",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 128,
"lora_bias": false,
"lora_dropout": 0.12668659169460122,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": [
"qa_outputs"
],
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 256,
"rank_pattern": {},
"revision": null,
"target_modules": [
"dense",
"query",
"key",
"value"
],
"target_parameters": null,
"task_type": "QUESTION_ANS",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

3
run-4/checkpoint-634/adapter_model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0f521f5e3ae45fb3cdb128e880b4b678803eec03114826b323499750927f425f
size 188782560

34
run-4/checkpoint-634/config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"add_cross_attention": false,
"architectures": [
"BertForQuestionAnswering"
],
"attention_probs_dropout_prob": 0.1,
"bos_token_id": null,
"classifier_dropout": null,
"directionality": "bidi",
"dtype": "float32",
"eos_token_id": null,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"is_decoder": false,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"tie_word_embeddings": true,
"transformers_version": "5.9.0",
"type_vocab_size": 2,
"use_cache": true,
"vocab_size": 119547
}

3
run-4/checkpoint-634/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fd5584c645526d95e095819f2fce1ca0c6cd35b5c3182bcc2cade2e1fd851417
size 709080872

3
run-4/checkpoint-634/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5d37941cc0a371d231e3e19b8ba0f9fcb35f078d865a3aaa152d7e1090ba9c3d
size 37825419

3
run-4/checkpoint-634/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8a2ec883b1ebf2f01a8e2eb2c4e7068aea33a6d74b16246e45be1cd75b4f8644
size 14645

3
run-4/checkpoint-634/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c14e84baabff580910daefb011f83354160821f6f19c1e57690d24d62241c23a
size 1383

3
run-4/checkpoint-634/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7dd726ca721ffba5cff4481b1611c74ed5b12ebf7781492eb1d823840799ed43
size 1465

119711
run-4/checkpoint-634/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

15
run-4/checkpoint-634/tokenizer_config.json Normal file
View File

@@ -0,0 +1,15 @@
{
"backend": "tokenizers",
"cls_token": "[CLS]",
"do_lower_case": false,
"is_local": false,
"local_files_only": false,
"mask_token": "[MASK]",
"model_max_length": 512,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

58
run-4/checkpoint-634/trainer_state.json Normal file
View File

@@ -0,0 +1,58 @@
{
"best_global_step": 634,
"best_metric": 45.273247698850945,
"best_model_checkpoint": "mbert-loraxs-qa-vanilla/run-4/checkpoint-634",
"epoch": 1.0,
"eval_steps": 500,
"global_step": 634,
"is_hyper_param_search": true,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 1.0,
"grad_norm": 10.389764785766602,
"learning_rate": 2.203344436447427e-05,
"loss": 4.208139894888604,
"step": 634
},
{
"epoch": 1.0,
"eval_exact_match": 31.68,
"eval_f1": 45.273247698850945,
"eval_loss": 2.3350870609283447,
"eval_runtime": 7.7987,
"eval_samples_per_second": 162.207,
"eval_steps_per_second": 10.258,
"step": 634
}
],
"logging_steps": 500,
"max_steps": 1902,
"num_input_tokens_seen": 0,
"num_train_epochs": 3,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 3090495731162112.0,
"train_batch_size": 16,
"trial_name": null,
"trial_params": {
"learning_rate": 2.9707819785354985e-05,
"lora_alpha": 128,
"lora_dropout": 0.12668659169460122,
"lora_r": 256,
"num_train_epochs": 3,
"weight_decay": 0.05519317165246046
}
}

3
run-4/checkpoint-634/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5829e303e9d08a1a0b596f605e35b5586828786be12a667b61d1a56522a79ce0
size 5201