NastasiaM/mbert-loraxs-qa-vanilla
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: NastasiaM/mbert-loraxs-qa-vanilla
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-10 00:18:47 +08:00
commit d696db249d
795 changed files with 8285474 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

206
run-1/checkpoint-1268/README.md Normal file
View File

@@ -0,0 +1,206 @@
---
base_model: bert-base-multilingual-cased
library_name: peft
tags:
- base_model:adapter:bert-base-multilingual-cased
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1

47
run-1/checkpoint-1268/adapter_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "bert-base-multilingual-cased",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 512,
"lora_bias": false,
"lora_dropout": 0.0734622590514945,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": [
"qa_outputs"
],
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 128,
"rank_pattern": {},
"revision": null,
"target_modules": [
"dense",
"query",
"key",
"value"
],
"target_parameters": null,
"task_type": "QUESTION_ANS",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

3
run-1/checkpoint-1268/adapter_model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:956491326a09d4a16b3bf6e9170e4eeea0876af2e968615c44c4378305d7bea3
size 89691896

34
run-1/checkpoint-1268/config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"add_cross_attention": false,
"architectures": [
"BertForQuestionAnswering"
],
"attention_probs_dropout_prob": 0.1,
"bos_token_id": null,
"classifier_dropout": null,
"directionality": "bidi",
"dtype": "float32",
"eos_token_id": null,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"is_decoder": false,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"tie_word_embeddings": true,
"transformers_version": "5.9.0",
"type_vocab_size": 2,
"use_cache": true,
"vocab_size": 119547
}

3
run-1/checkpoint-1268/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:87d03aa3cf5e9333e9d871373ff9e45d4da1cf6555c56a6946cd44378a236e66
size 709080872

3
run-1/checkpoint-1268/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3be7d4beeb5ce00fe93ad8dfd14b8877abe7e3f8aa7d39765ea42eba05f2d662
size 9513163

3
run-1/checkpoint-1268/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3971a189c708a01d02be1e50bcfa0a26c8e9be5a4eed52d3bfc52e525c73fb1d
size 14645

3
run-1/checkpoint-1268/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b812bb2bf3396dd797f3f7f7d874cbcef53472866bff7f3c813b780284de2634
size 1383

3
run-1/checkpoint-1268/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f3c50821a2371ca18f3aa878af95c6118e2e3444a51792f1701bd8e16c943107
size 1465

119711
run-1/checkpoint-1268/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

15
run-1/checkpoint-1268/tokenizer_config.json Normal file
View File

@@ -0,0 +1,15 @@
{
"backend": "tokenizers",
"cls_token": "[CLS]",
"do_lower_case": false,
"is_local": false,
"local_files_only": false,
"mask_token": "[MASK]",
"model_max_length": 512,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

75
run-1/checkpoint-1268/trainer_state.json Normal file
View File

@@ -0,0 +1,75 @@
{
"best_global_step": 1268,
"best_metric": 66.75662514290244,
"best_model_checkpoint": "mbert-loraxs-qa-vanilla/run-1/checkpoint-1268",
"epoch": 2.0,
"eval_steps": 500,
"global_step": 1268,
"is_hyper_param_search": true,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 1.0,
"grad_norm": 52.79657745361328,
"learning_rate": 1.949699397447388e-05,
"loss": 3.1322443146810923,
"step": 634
},
{
"epoch": 1.0,
"eval_exact_match": 48.16,
"eval_f1": 62.48638687085724,
"eval_loss": 1.7752151489257812,
"eval_runtime": 7.6669,
"eval_samples_per_second": 164.996,
"eval_steps_per_second": 10.435,
"step": 634
},
{
"epoch": 2.0,
"grad_norm": 47.9760856628418,
"learning_rate": 3.0703927518856505e-08,
"loss": 1.7527718047614353,
"step": 1268
},
{
"epoch": 2.0,
"eval_exact_match": 52.72,
"eval_f1": 66.75662514290244,
"eval_loss": 1.5720099210739136,
"eval_runtime": 7.7172,
"eval_samples_per_second": 163.919,
"eval_steps_per_second": 10.366,
"step": 1268
}
],
"logging_steps": 500,
"max_steps": 1268,
"num_input_tokens_seen": 0,
"num_train_epochs": 2,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": true
},
"attributes": {}
}
},
"total_flos": 5023145924278272.0,
"train_batch_size": 16,
"trial_name": null,
"trial_params": {
"learning_rate": 3.503318129901527e-05,
"lora_alpha": 512,
"lora_dropout": 0.0734622590514945,
"lora_r": 128,
"num_train_epochs": 2,
"weight_decay": 0.004905620950863932
}
}

3
run-1/checkpoint-1268/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eb7bc00d14065aadc52e5d03ff8d04227d538099499023a19385fc692ca9960b
size 5201

206
run-1/checkpoint-1902/README.md Normal file
View File

@@ -0,0 +1,206 @@
---
base_model: bert-base-multilingual-cased
library_name: peft
tags:
- base_model:adapter:bert-base-multilingual-cased
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1

47
run-1/checkpoint-1902/adapter_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "bert-base-multilingual-cased",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 128,
"lora_bias": false,
"lora_dropout": 0.10300849900123686,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": [
"qa_outputs"
],
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 8,
"rank_pattern": {},
"revision": null,
"target_modules": [
"value",
"dense",
"key",
"query"
],
"target_parameters": null,
"task_type": "QUESTION_ANS",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

3
run-1/checkpoint-1902/adapter_model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:71a2547e02a3f49c56b042a2a2872a6957e5cbf8e9cb1526c1429c166318cf52
size 5364440

34
run-1/checkpoint-1902/config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"add_cross_attention": false,
"architectures": [
"BertForQuestionAnswering"
],
"attention_probs_dropout_prob": 0.1,
"bos_token_id": null,
"classifier_dropout": null,
"directionality": "bidi",
"dtype": "float32",
"eos_token_id": null,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"is_decoder": false,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"tie_word_embeddings": true,
"transformers_version": "5.9.0",
"type_vocab_size": 2,
"use_cache": true,
"vocab_size": 119547
}

3
run-1/checkpoint-1902/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f5ec840d5dbba57ae91c4b02692d42ce4f4948b8f26924abf25cc41e69d5c1ed
size 709080872

3
run-1/checkpoint-1902/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c22823ffd55e087ce8fdfee4d2a0b59bc835da4ad163968c7f15515cd0769894
size 112715

3
run-1/checkpoint-1902/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6e11dd5aac31555569da8b937fed809c3d18396a53b00643cca7f557d2e8ebba
size 14645

3
run-1/checkpoint-1902/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d26ec889430e7e9c16432d09ed907050ced92aabbb75333745bcde2b4024ea93
size 1383

3
run-1/checkpoint-1902/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:754fbf91bf2ce5c5c6953a12464ef54a607130ff3295f72e35aa3e8722bf691a
size 1465

119711
run-1/checkpoint-1902/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

15
run-1/checkpoint-1902/tokenizer_config.json Normal file
View File

@@ -0,0 +1,15 @@
{
"backend": "tokenizers",
"cls_token": "[CLS]",
"do_lower_case": false,
"is_local": false,
"local_files_only": false,
"mask_token": "[MASK]",
"model_max_length": 512,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

92
run-1/checkpoint-1902/trainer_state.json Normal file
View File

@@ -0,0 +1,92 @@
{
"best_global_step": 1902,
"best_metric": 42.797774942292016,
"best_model_checkpoint": "mbert-loraxs-qa-vanilla/run-1/checkpoint-1902",
"epoch": 3.0,
"eval_steps": 500,
"global_step": 1902,
"is_hyper_param_search": true,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 1.0,
"grad_norm": 50.64515686035156,
"learning_rate": 0.00010486219815639711,
"loss": 4.513181668350749,
"step": 634
},
{
"epoch": 1.0,
"eval_exact_match": 17.76,
"eval_f1": 28.945909302213973,
"eval_loss": 3.11417818069458,
"eval_runtime": 7.6977,
"eval_samples_per_second": 164.336,
"eval_steps_per_second": 10.393,
"step": 634
},
{
"epoch": 2.0,
"grad_norm": 69.94309997558594,
"learning_rate": 5.247241594114434e-05,
"loss": 3.1071321024127565,
"step": 1268
},
{
"epoch": 2.0,
"eval_exact_match": 29.52,
"eval_f1": 40.87090707056966,
"eval_loss": 2.5117993354797363,
"eval_runtime": 7.7422,
"eval_samples_per_second": 163.389,
"eval_steps_per_second": 10.333,
"step": 1268
},
{
"epoch": 3.0,
"grad_norm": 85.58645629882812,
"learning_rate": 8.263372589156588e-08,
"loss": 2.8506941449378944,
"step": 1902
},
{
"epoch": 3.0,
"eval_exact_match": 30.88,
"eval_f1": 42.797774942292016,
"eval_loss": 2.423711061477661,
"eval_runtime": 7.6134,
"eval_samples_per_second": 166.154,
"eval_steps_per_second": 10.508,
"step": 1902
}
],
"logging_steps": 500,
"max_steps": 1902,
"num_input_tokens_seen": 0,
"num_train_epochs": 3,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": true
},
"attributes": {}
}
},
"total_flos": 6056721303671808.0,
"train_batch_size": 16,
"trial_name": null,
"trial_params": {
"learning_rate": 0.00014138630500046923,
"lora_alpha": 128,
"lora_dropout": 0.10300849900123686,
"lora_r": 8,
"num_train_epochs": 3,
"weight_decay": 0.04651389479910262
}
}

3
run-1/checkpoint-1902/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ad44e164e8a1640df833f0ccdc5c6426f7ca4a746d2737414edc4b9c8dd8a9f9
size 5201

34
run-1/checkpoint-2536/config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"add_cross_attention": false,
"architectures": [
"BertForQuestionAnswering"
],
"attention_probs_dropout_prob": 0.1,
"bos_token_id": null,
"classifier_dropout": null,
"directionality": "bidi",
"dtype": "float32",
"eos_token_id": null,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"is_decoder": false,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"tie_word_embeddings": true,
"transformers_version": "5.9.0",
"type_vocab_size": 2,
"use_cache": true,
"vocab_size": 119547
}

3
run-1/checkpoint-2536/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:06989b83bf9d77da5b12e21fc035ca324b4349657fd8cfb36a435f071f870069
size 709080872

3
run-1/checkpoint-2536/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:64870094ba24066a73902f3c2511e84f95948a6eb7c244962cfd46e5f98190f0
size 1418284811

3
run-1/checkpoint-2536/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1e85fa698aa4806efe3d6074038fa07bcf6133d8cc71f17a6a4dca0b61c4fc4f
size 14645

3
run-1/checkpoint-2536/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6a1c6b4addfbdbc469ec0107c5c42d1b0e47cf34c2de1d244fae93607014b55c
size 1383

3
run-1/checkpoint-2536/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7de304905c0f46d926a785e9a31fef27096dc0bb2c49504102ebeabc1bb2f7ea
size 1465

119711
run-1/checkpoint-2536/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

15
run-1/checkpoint-2536/tokenizer_config.json Normal file
View File

@@ -0,0 +1,15 @@
{
"backend": "tokenizers",
"cls_token": "[CLS]",
"do_lower_case": false,
"is_local": false,
"local_files_only": false,
"mask_token": "[MASK]",
"model_max_length": 512,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

106
run-1/checkpoint-2536/trainer_state.json Normal file
View File

@@ -0,0 +1,106 @@
{
"best_global_step": 2536,
"best_metric": 76.57829761930071,
"best_model_checkpoint": "mbert-loraxs-qa-vanilla/run-1/checkpoint-2536",
"epoch": 4.0,
"eval_steps": 500,
"global_step": 2536,
"is_hyper_param_search": true,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 1.0,
"grad_norm": 15.868330955505371,
"learning_rate": 5.191675659155908e-05,
"loss": 2.1192049333349763,
"step": 634
},
{
"epoch": 1.0,
"eval_exact_match": 60.08,
"eval_f1": 74.33783203646502,
"eval_loss": 1.2725635766983032,
"eval_runtime": 5.8013,
"eval_samples_per_second": 218.055,
"eval_steps_per_second": 13.79,
"step": 634
},
{
"epoch": 2.0,
"grad_norm": 30.633678436279297,
"learning_rate": 3.46202649052488e-05,
"loss": 0.9956009440617607,
"step": 1268
},
{
"epoch": 2.0,
"eval_exact_match": 62.48,
"eval_f1": 76.03924855668853,
"eval_loss": 1.3048548698425293,
"eval_runtime": 6.0639,
"eval_samples_per_second": 208.613,
"eval_steps_per_second": 13.193,
"step": 1268
},
{
"epoch": 3.0,
"grad_norm": 13.438405990600586,
"learning_rate": 1.7323773218938526e-05,
"loss": 0.49370124889097006,
"step": 1902
},
{
"epoch": 3.0,
"eval_exact_match": 62.8,
"eval_f1": 76.44229300202514,
"eval_loss": 1.4875582456588745,
"eval_runtime": 5.797,
"eval_samples_per_second": 218.217,
"eval_steps_per_second": 13.8,
"step": 1902
},
{
"epoch": 4.0,
"grad_norm": 14.641495704650879,
"learning_rate": 2.728153262824965e-08,
"loss": 0.23204485775920494,
"step": 2536
},
{
"epoch": 4.0,
"eval_exact_match": 62.96,
"eval_f1": 76.57829761930071,
"eval_loss": 1.8072655200958252,
"eval_runtime": 5.8611,
"eval_samples_per_second": 215.829,
"eval_steps_per_second": 13.649,
"step": 2536
}
],
"logging_steps": 500,
"max_steps": 2536,
"num_input_tokens_seen": 0,
"num_train_epochs": 4,
"save_steps": 50,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": true
},
"attributes": {}
}
},
"total_flos": 7950999010719744.0,
"train_batch_size": 16,
"trial_name": null,
"trial_params": {
"learning_rate": 6.22564574576657e-05,
"num_train_epochs": 4,
"weight_decay": 0.033952923108431954
}
}

3
run-1/checkpoint-2536/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:631288df84c6128821fb37f5b2efce07b97fa34e7c5281aa460f5144e137e955
size 5265

206
run-1/checkpoint-634/README.md Normal file
View File

@@ -0,0 +1,206 @@
---
base_model: bert-base-multilingual-cased
library_name: peft
tags:
- base_model:adapter:bert-base-multilingual-cased
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1

47
run-1/checkpoint-634/adapter_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "bert-base-multilingual-cased",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 512,
"lora_bias": false,
"lora_dropout": 0.0734622590514945,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": [
"qa_outputs"
],
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 128,
"rank_pattern": {},
"revision": null,
"target_modules": [
"dense",
"query",
"key",
"value"
],
"target_parameters": null,
"task_type": "QUESTION_ANS",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

3
run-1/checkpoint-634/adapter_model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8abf969f627b6392ffb16e5d113deb74297bc874deab663c27401da04892b41c
size 89691896

34
run-1/checkpoint-634/config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"add_cross_attention": false,
"architectures": [
"BertForQuestionAnswering"
],
"attention_probs_dropout_prob": 0.1,
"bos_token_id": null,
"classifier_dropout": null,
"directionality": "bidi",
"dtype": "float32",
"eos_token_id": null,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"is_decoder": false,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"tie_word_embeddings": true,
"transformers_version": "5.9.0",
"type_vocab_size": 2,
"use_cache": true,
"vocab_size": 119547
}

3
run-1/checkpoint-634/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6f0f6c70c3c32756828941d647f30c49644890940f8f2df008bccb4ae3593e18
size 709080872

3
run-1/checkpoint-634/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a6ae0fcfafae8498cfda280ab6e7c5850a9bdae76b97ee3a773397b85f47cc36
size 9513163

3
run-1/checkpoint-634/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6ec2aad255418a7f342d4c5272744bbbfed81379f61e25bb0bc64e547232444c
size 14645

3
run-1/checkpoint-634/scaler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dec0d538b52a102fa5f1e0eaf5150e37f52c3c7eb0b238ddfcefa67194db2034
size 1383

3
run-1/checkpoint-634/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a253170b0a542765246ddf1d83ee02400e9b996d579abb93a31de5fe725b6f51
size 1465

119711
run-1/checkpoint-634/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

15
run-1/checkpoint-634/tokenizer_config.json Normal file
View File

@@ -0,0 +1,15 @@
{
"backend": "tokenizers",
"cls_token": "[CLS]",
"do_lower_case": false,
"is_local": false,
"local_files_only": false,
"mask_token": "[MASK]",
"model_max_length": 512,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

58
run-1/checkpoint-634/trainer_state.json Normal file
View File

@@ -0,0 +1,58 @@
{
"best_global_step": 634,
"best_metric": 62.48638687085724,
"best_model_checkpoint": "mbert-loraxs-qa-vanilla/run-1/checkpoint-634",
"epoch": 1.0,
"eval_steps": 500,
"global_step": 634,
"is_hyper_param_search": true,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 1.0,
"grad_norm": 52.79657745361328,
"learning_rate": 1.949699397447388e-05,
"loss": 3.1322443146810923,
"step": 634
},
{
"epoch": 1.0,
"eval_exact_match": 48.16,
"eval_f1": 62.48638687085724,
"eval_loss": 1.7752151489257812,
"eval_runtime": 7.6669,
"eval_samples_per_second": 164.996,
"eval_steps_per_second": 10.435,
"step": 634
}
],
"logging_steps": 500,
"max_steps": 1268,
"num_input_tokens_seen": 0,
"num_train_epochs": 2,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 2511572962139136.0,
"train_batch_size": 16,
"trial_name": null,
"trial_params": {
"learning_rate": 3.503318129901527e-05,
"lora_alpha": 512,
"lora_dropout": 0.0734622590514945,
"lora_r": 128,
"num_train_epochs": 2,
"weight_decay": 0.004905620950863932
}
}

3
run-1/checkpoint-634/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eb7bc00d14065aadc52e5d03ff8d04227d538099499023a19385fc692ca9960b
size 5201