初始化项目，由ModelHub XC社区提供模型
Model: adarsh-08/qwen-hr-assistant Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
172
README.md
Normal file
172
README.md
Normal file
@@ -0,0 +1,172 @@
|
|||||||
|
---
|
||||||
|
library_name: transformers
|
||||||
|
base_model: Qwen/Qwen2.5-0.5B-Instruct
|
||||||
|
tags:
|
||||||
|
- qlora
|
||||||
|
- lora
|
||||||
|
- peft
|
||||||
|
- hr-assistant
|
||||||
|
- question-answering
|
||||||
|
- qwen2
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
license: apache-2.0
|
||||||
|
datasets:
|
||||||
|
- adarsh-08/hr-policy-dataset
|
||||||
|
---
|
||||||
|
|
||||||
|
# Model Card for Qwen HR Assistant (NovaTech Solutions)
|
||||||
|
|
||||||
|
## Model Details
|
||||||
|
|
||||||
|
### Model Description
|
||||||
|
|
||||||
|
This model is a fine-tuned version of **Qwen2.5-0.5B-Instruct**, adapted to act as an HR assistant chatbot for a fictional company, NovaTech Solutions. It answers employee questions about HR policies including leave (casual, sick, earned, unpaid, maternity, paternity), health/dental/vision insurance, wellness programs, performance reviews, travel & expense policies, IT/device usage, and onboarding.
|
||||||
|
|
||||||
|
The model was fine-tuned using **QLoRA** (4-bit quantized base model + LoRA adapters) on a custom instruction-response dataset, then the LoRA adapter was merged into the base model weights for standalone deployment.
|
||||||
|
|
||||||
|
- **Developed by:** [adarsh-08]
|
||||||
|
- **Model type:** Causal language model (decoder-only transformer), fine-tuned with LoRA/QLoRA
|
||||||
|
- **Language(s):** English
|
||||||
|
- **Finetuned from model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
|
||||||
|
|
||||||
|
### Model Sources
|
||||||
|
|
||||||
|
- **Base model:** https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct
|
||||||
|
- **Repository:** [[link to this model repo](https://huggingface.co/adarsh-08/qwen-hr-assistant)]
|
||||||
|
|
||||||
|
## Uses
|
||||||
|
|
||||||
|
### Direct Use
|
||||||
|
|
||||||
|
This model is intended for answering employee HR-policy questions in a Q&A/chatbot format, specifically for a single fictional company's internal policies (NovaTech Solutions). It is best used as a demo or internal FAQ assistant, not as a source of legally binding HR guidance.
|
||||||
|
|
||||||
|
### Out-of-Scope Use
|
||||||
|
|
||||||
|
- **Not suitable for real HR/legal decision-making.** Answers reflect a synthetic training dataset, not verified company policy or labor law, and may contain factual errors or invented details (see Limitations below).
|
||||||
|
- **Not designed for general-purpose chat** — the model is narrowly fine-tuned on HR Q&A pairs and will underperform on unrelated tasks compared to the base instruct model.
|
||||||
|
- **Not intended for use with real employee personal data** without independent verification of its outputs.
|
||||||
|
|
||||||
|
## Bias, Risks, and Limitations
|
||||||
|
|
||||||
|
- **Hallucination risk on numbers.** Being a small (0.5B parameter) model fine-tuned on a modest dataset, it can occasionally invent or blend numeric details (e.g., leave day counts, reimbursement amounts) that don't match the training data. Always verify specific figures against the actual source policy.
|
||||||
|
- **Narrow domain coverage.** The model only knows what's in its ~500-example training set. Questions phrased very differently from training examples, or about topics not covered, may get low-quality, off-topic, or fabricated answers.
|
||||||
|
- **Synthetic company.** All policy details (NovaTech Solutions) are fictional/for demonstration purposes.
|
||||||
|
|
||||||
|
### Recommendations
|
||||||
|
|
||||||
|
Use this model for demonstration, prototyping, or as a starting point for a real internal HR assistant — not as an authoritative source. Any deployment handling real employee questions should include human review, a way to escalate to real HR staff, and further evaluation of factual accuracy before production use.
|
||||||
|
|
||||||
|
## How to Get Started with the Model
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||||
|
import torch
|
||||||
|
|
||||||
|
model_id = "your-username/qwen-hr-assistant"
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)
|
||||||
|
|
||||||
|
prompt = "Question: How many days of paid leave am I entitled to per year?\n\nAnswer:"
|
||||||
|
inputs = tokenizer(prompt, return_tensors="pt")
|
||||||
|
|
||||||
|
output_ids = model.generate(
|
||||||
|
**inputs,
|
||||||
|
max_new_tokens=200,
|
||||||
|
do_sample=True,
|
||||||
|
temperature=0.7,
|
||||||
|
top_p=0.9,
|
||||||
|
repetition_penalty=1.1,
|
||||||
|
pad_token_id=tokenizer.pad_token_id,
|
||||||
|
eos_token_id=tokenizer.eos_token_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
answer = tokenizer.decode(output_ids[0], skip_special_tokens=True)
|
||||||
|
print(answer)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Training Details
|
||||||
|
|
||||||
|
### Training Data
|
||||||
|
|
||||||
|
A custom synthetic dataset of ~512 instruction-response pairs covering NovaTech Solutions' HR policies: leave types, insurance/benefits, wellness programs, performance reviews, travel & expenses, IT device policy, and onboarding. Multiple paraphrased questions map to the same canonical answer per topic, to encourage robustness to phrasing variation. Data was split 90/10 into train/validation sets.
|
||||||
|
|
||||||
|
### Training Procedure
|
||||||
|
|
||||||
|
Fine-tuned using **QLoRA**: the base model was loaded in 4-bit (NF4) precision via `bitsandbytes`, and LoRA adapters were trained on top using the `peft` library. After training, the adapter was merged into the base model weights (`merge_and_unload`) for standalone deployment.
|
||||||
|
|
||||||
|
#### Preprocessing
|
||||||
|
|
||||||
|
Each example was formatted as:
|
||||||
|
```
|
||||||
|
Question: {instruction}
|
||||||
|
|
||||||
|
Answer: {response}
|
||||||
|
```
|
||||||
|
with an explicit EOS token appended, and a dedicated pad token (distinct from EOS) added to the tokenizer to avoid masking real end-of-sequence tokens during training.
|
||||||
|
|
||||||
|
#### LoRA Configuration
|
||||||
|
|
||||||
|
- **r:** 16
|
||||||
|
- **lora_alpha:** 32
|
||||||
|
- **lora_dropout:** 0.05
|
||||||
|
- **target_modules:** q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
||||||
|
- **task_type:** CAUSAL_LM
|
||||||
|
|
||||||
|
#### Training Hyperparameters
|
||||||
|
|
||||||
|
- **Training regime:** bf16 mixed precision (compute dtype), 4-bit NF4 quantized base weights during training
|
||||||
|
- **Epochs:** 5
|
||||||
|
- **Per-device batch size:** 4
|
||||||
|
- **Learning rate:** 2e-4
|
||||||
|
- **Max sequence length:** 256
|
||||||
|
- **Gradient checkpointing:** enabled
|
||||||
|
- **Eval/save strategy:** per epoch, best model (by validation loss) loaded at end
|
||||||
|
|
||||||
|
#### Speeds, Sizes, Times
|
||||||
|
|
||||||
|
- **Base model size:** ~0.5B parameters
|
||||||
|
- **Hardware:** Google Colab GPU runtime
|
||||||
|
- Trained for 5 epochs on ~460 training examples / ~52 validation examples
|
||||||
|
|
||||||
|
## Evaluation
|
||||||
|
|
||||||
|
### Testing Data, Factors & Metrics
|
||||||
|
|
||||||
|
#### Testing Data
|
||||||
|
|
||||||
|
10% held-out split from the same synthetic dataset (random seed 42).
|
||||||
|
|
||||||
|
#### Metrics
|
||||||
|
|
||||||
|
Validation loss (cross-entropy) was used as the primary metric, with the best checkpoint (lowest validation loss) selected automatically at the end of training.
|
||||||
|
|
||||||
|
### Results
|
||||||
|
|
||||||
|
Qualitative spot-checks after training showed accurate, on-topic, well-formatted answers for most HR policy questions. One coverage gap was identified (aggregate "total paid leave" questions defaulted to sick-leave-only answers) and fixed by adding targeted training examples in a follow-up training run.
|
||||||
|
|
||||||
|
#### Summary
|
||||||
|
|
||||||
|
The model performs well on in-distribution HR questions closely matching training phrasing and topics, with occasional numeric hallucination risk typical of small models trained on limited data. Recommended as a demo/prototype assistant rather than a production HR system without further validation.
|
||||||
|
|
||||||
|
## Technical Specifications
|
||||||
|
|
||||||
|
### Model Architecture and Objective
|
||||||
|
|
||||||
|
Decoder-only causal language model (Qwen2.5 architecture), fine-tuned via low-rank adaptation (LoRA) for the causal language modeling objective, then merged into the base weights.
|
||||||
|
|
||||||
|
### Compute Infrastructure
|
||||||
|
|
||||||
|
#### Hardware
|
||||||
|
|
||||||
|
Google Colab (GPU runtime)
|
||||||
|
|
||||||
|
#### Software
|
||||||
|
|
||||||
|
- `transformers`
|
||||||
|
- `peft`
|
||||||
|
- `bitsandbytes`
|
||||||
|
- `accelerate`
|
||||||
|
- `torch`
|
||||||
|
- `datasets`
|
||||||
54
chat_template.jinja
Normal file
54
chat_template.jinja
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
{%- if tools %}
|
||||||
|
{{- '<|im_start|>system\n' }}
|
||||||
|
{%- if messages[0]['role'] == 'system' %}
|
||||||
|
{{- messages[0]['content'] }}
|
||||||
|
{%- else %}
|
||||||
|
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||||
|
{%- for tool in tools %}
|
||||||
|
{{- "\n" }}
|
||||||
|
{{- tool | tojson }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||||
|
{%- else %}
|
||||||
|
{%- if messages[0]['role'] == 'system' %}
|
||||||
|
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- for message in messages %}
|
||||||
|
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
|
||||||
|
{%- elif message.role == "assistant" %}
|
||||||
|
{{- '<|im_start|>' + message.role }}
|
||||||
|
{%- if message.content %}
|
||||||
|
{{- '\n' + message.content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- for tool_call in message.tool_calls %}
|
||||||
|
{%- if tool_call.function is defined %}
|
||||||
|
{%- set tool_call = tool_call.function %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_call>\n{"name": "' }}
|
||||||
|
{{- tool_call.name }}
|
||||||
|
{{- '", "arguments": ' }}
|
||||||
|
{{- tool_call.arguments | tojson }}
|
||||||
|
{{- '}\n</tool_call>' }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- elif message.role == "tool" %}
|
||||||
|
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
|
||||||
|
{{- '<|im_start|>user' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_response>\n' }}
|
||||||
|
{{- message.content }}
|
||||||
|
{{- '\n</tool_response>' }}
|
||||||
|
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- if add_generation_prompt %}
|
||||||
|
{{- '<|im_start|>assistant\n' }}
|
||||||
|
{%- endif %}
|
||||||
57
config.json
Normal file
57
config.json
Normal file
@@ -0,0 +1,57 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"Qwen2ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 151643,
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"eos_token_id": 151645,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 896,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 4864,
|
||||||
|
"layer_types": [
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention"
|
||||||
|
],
|
||||||
|
"max_position_embeddings": 32768,
|
||||||
|
"max_window_layers": 21,
|
||||||
|
"model_type": "qwen2",
|
||||||
|
"num_attention_heads": 14,
|
||||||
|
"num_hidden_layers": 24,
|
||||||
|
"num_key_value_heads": 2,
|
||||||
|
"pad_token_id": null,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_parameters": {
|
||||||
|
"rope_theta": 1000000.0,
|
||||||
|
"rope_type": "default"
|
||||||
|
},
|
||||||
|
"sliding_window": null,
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"transformers_version": "5.12.1",
|
||||||
|
"use_cache": true,
|
||||||
|
"use_sliding_window": false,
|
||||||
|
"vocab_size": 151665
|
||||||
|
}
|
||||||
14
generation_config.json
Normal file
14
generation_config.json
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"bos_token_id": 151643,
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
151645,
|
||||||
|
151643
|
||||||
|
],
|
||||||
|
"pad_token_id": 151643,
|
||||||
|
"repetition_penalty": 1.1,
|
||||||
|
"temperature": 0.7,
|
||||||
|
"top_k": 20,
|
||||||
|
"top_p": 0.8,
|
||||||
|
"transformers_version": "5.12.1"
|
||||||
|
}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:f1304bb0975510c7ee34406a164ace066ed5c06bf4206d34cf778e13fc5ef646
|
||||||
|
size 987612192
|
||||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:6cca9cef36deabc2e5ef4bca92e74eee002dffa3bff182f1d19a08675d1352df
|
||||||
|
size 11422170
|
||||||
37
tokenizer_config.json
Normal file
37
tokenizer_config.json
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
{
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"backend": "tokenizers",
|
||||||
|
"bos_token": null,
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|im_end|>",
|
||||||
|
"errors": "replace",
|
||||||
|
"extra_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"is_local": true,
|
||||||
|
"local_files_only": false,
|
||||||
|
"max_length": 256,
|
||||||
|
"model_max_length": 131072,
|
||||||
|
"pad_to_multiple_of": null,
|
||||||
|
"pad_token": "<|endoftext|>",
|
||||||
|
"pad_token_type_id": 0,
|
||||||
|
"padding_side": "right",
|
||||||
|
"split_special_tokens": false,
|
||||||
|
"stride": 0,
|
||||||
|
"tokenizer_class": "Qwen2Tokenizer",
|
||||||
|
"truncation_side": "right",
|
||||||
|
"truncation_strategy": "longest_first",
|
||||||
|
"unk_token": null
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user