初始化项目，由ModelHub XC社区提供模型
Model: qingy2024/GRMR-V3-L1B Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
310
README.md
Normal file
310
README.md
Normal file
@@ -0,0 +1,310 @@
|
||||
---
|
||||
base_model: unsloth/Llama-3.2-1B
|
||||
tags:
|
||||
- text-generation-inference
|
||||
- transformers
|
||||
- unsloth
|
||||
- llama
|
||||
- trl
|
||||
- sft
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
---
|
||||
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
</head>
|
||||
<div class="container">
|
||||
<h1>GRMR-V3-L1B</h1>
|
||||
<p>GRMR-V3-L1B is a fine-tuned version of <a href="https://huggingface.co/unsloth/Llama-3.2-1B">unsloth/Llama-3.2-1B</a> specifically optimized for grammar correction tasks.</p>
|
||||
<div class="important-note">
|
||||
<p><strong>IMPORTANT:</strong> Please ensure you are using the following sampler settings for optimal results:</p>
|
||||
<pre><code>temperature = 0.7
|
||||
frequency_penalty = 0.0
|
||||
presence_penalty = 0.0
|
||||
min_p = 0.01
|
||||
top_p = 0.95
|
||||
top_k = 40</code></pre>
|
||||
</div>
|
||||
<h2>Model description</h2>
|
||||
<p>GRMR-V3-L1B is a grammar correction model built on Meta's Llama 3.2 1B base model. It has been fine-tuned on a large dataset of grammar correction examples to help improve text quality by fixing grammatical errors, punctuation, spelling, and other language issues.</p>
|
||||
<p>The model uses a specialized chat template that structures inputs as "text" and outputs as "corrected" to maintain a clear distinction between original and corrected content.</p>
|
||||
<h3>Model Details</h3>
|
||||
<div class="model-meta">
|
||||
<p><strong>Developed by:</strong> qingy2024</p>
|
||||
<p><strong>Finetuned from model:</strong> <a href="https://huggingface.co/unsloth/Llama-3.2-1B">unsloth/Llama-3.2-1B</a></p>
|
||||
<p><strong>Model type:</strong> Llama 3.2</p>
|
||||
<p><strong>Language(s):</strong> English</p>
|
||||
<p><strong>License:</strong> apache-2.0</p>
|
||||
<p><strong>Training Dataset:</strong> <a href="https://huggingface.co/datasets/qingy2024/grmr-v4-60k">qingy2024/grmr-v4-60k</a></p>
|
||||
</div>
|
||||
<hr>
|
||||
<h2>Manual Testing Examples</h2>
|
||||
<p>Here are a few examples of grammar corrections this model can handle:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Original Text</th>
|
||||
<th>Corrected Text</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>i dont know weather to bring a umbrella today</td>
|
||||
<td>I don't know whether to bring an umbrella today.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>she go to the store yesterday</td>
|
||||
<td>She went to the store yesterday.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>they is going to be late for the meeting</td>
|
||||
<td>They are going to be late for the meeting.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>the cat laying on the floor all day</td>
|
||||
<td>The cat is laying on the floor all day.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<hr>
|
||||
<h2>Training procedure</h2>
|
||||
<p>The model was fine-tuned using full parameter fine-tuning (not LoRA) on the GRMR-V4-60K dataset. The training was optimized using the Unsloth framework for efficient training of LLMs.</p>
|
||||
<h3>Training hyperparameters</h3>
|
||||
<ul>
|
||||
<li><strong>Batch size:</strong> 8</li>
|
||||
<li><strong>Gradient accumulation steps:</strong> 2</li>
|
||||
<li><strong>Learning rate:</strong> 5e-5</li>
|
||||
<li><strong>Epochs:</strong> 1</li>
|
||||
<li><strong>Optimizer:</strong> AdamW (8-bit)</li>
|
||||
<li><strong>Weight decay:</strong> 0.01</li>
|
||||
<li><strong>LR scheduler:</strong> Cosine</li>
|
||||
<li><strong>Warmup steps:</strong> 180</li>
|
||||
<li><strong>Max sequence length:</strong> 16,384</li>
|
||||
<li><strong>Training precision:</strong> Mixed precision (BF16 where available, FP16 otherwise)</li>
|
||||
</ul>
|
||||
<h2>Intended uses & limitations</h2>
|
||||
<p>This model is designed for grammar correction tasks. It can be used to:</p>
|
||||
<ul>
|
||||
<li>Fix grammatical errors in written text</li>
|
||||
<li>Correct punctuation</li>
|
||||
<li>Address spelling mistakes</li>
|
||||
<li>Improve sentence structure and clarity</li>
|
||||
</ul>
|
||||
<h3>Limitations</h3>
|
||||
<ul>
|
||||
<li>The model may struggle with highly technical or domain-specific content</li>
|
||||
<li>It may not fully understand context-dependent grammar rules in all cases</li>
|
||||
<li>Performance may vary for non-standard English or text with multiple errors</li>
|
||||
</ul>
|
||||
<h2>How to use</h2>
|
||||
<p><code>llama.cpp</code> and projects based on it should be able to run this model like any others.</p>
|
||||
<p>For pure <code>transformers</code> code, you can refer here:</p>
|
||||
<pre><code class="language-python">from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
# Load model and tokenizer
|
||||
model_name = "qingy2024/GRMR-V3-L1B"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_name)
|
||||
text_to_correct = "i am going to the store tommorow and buy some thing for dinner"
|
||||
messages = [
|
||||
{"role": "user", "content": text_to_correct}
|
||||
]
|
||||
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
outputs = model.generate(
|
||||
inputs["input_ids"],
|
||||
max_new_tokens=512,
|
||||
temperature=0.1, # NOTE: For best results, use the recommended temperature of 0.7
|
||||
do_sample=True
|
||||
)
|
||||
corrected_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
print(corrected_text)</code></pre>
|
||||
<h3>Using with the Hugging Face pipeline</h3>
|
||||
<pre><code class="language-python">from transformers import pipeline
|
||||
pipe = pipeline(
|
||||
"text-generation",
|
||||
model="qingy2024/GRMR-V3-L1B",
|
||||
torch_dtype="auto",
|
||||
device_map="auto"
|
||||
)
|
||||
messages = [
|
||||
{"role": "user", "content": "i dont know weather to bring a umbrella today"}
|
||||
]
|
||||
result = pipe(
|
||||
messages,
|
||||
max_new_tokens=100,
|
||||
temperature=0.1, # NOTE: For best results, use the recommended temperature of 0.7
|
||||
do_sample=True,
|
||||
return_full_text=False
|
||||
)[0]["generated_text"]
|
||||
print(result)</code></pre>
|
||||
<p><em>Note: The Python examples above use <code>temperature=0.1</code> for reproducibility in quick tests. For optimal grammar correction quality, please use the recommended sampler settings, especially <code>temperature=0.7</code>.</em></p>
|
||||
<h2>Custom Chat Template</h2>
|
||||
<p class="chat-template-info">The model uses a custom chat template with special formatting for grammar correction:</p>
|
||||
<ul>
|
||||
<li>User inputs are formatted with <code><|start_header_id|>text<|end_header_id|></code> headers</li>
|
||||
<li>Model outputs are formatted with <code><|start_header_id|>corrected<|end_header_id|></code> headers</li>
|
||||
<li>Messages are separated by <code><|eot_id|></code> tokens</li>
|
||||
<li>The chat template should work without any extra tweaking in vLLM or <code>llama.cpp</code>.</li>
|
||||
</ul>
|
||||
<p>An example of the applied template structure might look like:</p>
|
||||
<pre><code><|begin_of_text|><|start_header_id|>text<|end_header_id|>
|
||||
|
||||
i dont know weather to bring a umbrella today<|eot_id|><|start_header_id|>corrected<|end_header_id|>
|
||||
|
||||
I don't know whether to bring an umbrella today.<|eot_id|></code></pre>
|
||||
<p>(When generating, you would only provide the input up to <code><|start_header_id|>corrected<|end_header_id|>\n\n</code>)</p>
|
||||
<h2>Training Dataset</h2>
|
||||
<p>The model was fine-tuned on the <a href="https://huggingface.co/datasets/qingy2024/grmr-v4-60k">qingy2024/grmr-v4-60k</a> dataset, which contains 60,000 examples of original text and their grammatically corrected versions.</p>
|
||||
<h2>Bias, Risks, and Limitations</h2>
|
||||
<ul>
|
||||
<li>The model may reflect biases present in the training data.</li>
|
||||
<li>It may not perform equally well across different writing styles or domains.</li>
|
||||
<li>The model might occasionally introduce errors or change the meaning of text.</li>
|
||||
<li>It focuses on grammatical correctness rather than stylistic improvements.</li>
|
||||
</ul>
|
||||
<h2>Contact</h2>
|
||||
<p>For questions or issues related to the model, please reach out via Hugging Face or by creating an issue in the repository.</p>
|
||||
</div>
|
||||
<style>
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";
|
||||
line-height: 1.6;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
background-color: #f8f9fa;
|
||||
color: #333;
|
||||
}
|
||||
.container {
|
||||
max-width: 1200px;
|
||||
margin: 10px auto;
|
||||
padding: 25px;
|
||||
background-color: #ffffff;
|
||||
border-radius: 8px;
|
||||
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.08);
|
||||
}
|
||||
h1, h2, h3 {
|
||||
color: #0056b3; /* Primary Blue */
|
||||
margin-top: 1.5em;
|
||||
margin-bottom: 0.7em;
|
||||
}
|
||||
h1 {
|
||||
text-align: center;
|
||||
font-size: 2.2em;
|
||||
border-bottom: 2px solid #e0e0e0;
|
||||
padding-bottom: 0.5em;
|
||||
margin-top: 0;
|
||||
}
|
||||
h2 {
|
||||
font-size: 1.8em;
|
||||
border-bottom: 1px solid #e9ecef;
|
||||
padding-bottom: 0.3em;
|
||||
}
|
||||
h3 {
|
||||
font-size: 1.4em;
|
||||
color: #007bff; /* Lighter Blue for sub-headings */
|
||||
}
|
||||
p, li {
|
||||
font-size: 1em;
|
||||
color: #555;
|
||||
}
|
||||
a {
|
||||
color: #007bff;
|
||||
text-decoration: none;
|
||||
}
|
||||
a:hover {
|
||||
text-decoration: underline;
|
||||
color: #0056b3;
|
||||
}
|
||||
.important-note {
|
||||
background-color: #e7f3ff; /* Light blue background */
|
||||
border-left: 5px solid #007bff; /* Blue accent border */
|
||||
margin: 20px 0px;
|
||||
border-radius: 5px;
|
||||
}
|
||||
.important-note strong {
|
||||
color: #0056b3;
|
||||
font-weight: 600;
|
||||
}
|
||||
.important-note {
|
||||
background-color: #d0e8ff;
|
||||
padding: 0.05em 1.0em;
|
||||
border-radius: 3px;
|
||||
font-size: 0.9em;
|
||||
}
|
||||
code {
|
||||
padding: 0.1em 0.4em;
|
||||
border-radius: 3px;
|
||||
font-size: 0.9em;
|
||||
}
|
||||
table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
margin: 20px 0;
|
||||
box-shadow: 0 2px 4px rgba(0,0,0,0.05);
|
||||
}
|
||||
th, td {
|
||||
border: 1px solid #dee2e6;
|
||||
padding: 10px 12px;
|
||||
text-align: left;
|
||||
vertical-align: top;
|
||||
}
|
||||
th {
|
||||
background-color: #e9ecef; /* Light gray for headers */
|
||||
font-weight: 600;
|
||||
color: #212529;
|
||||
}
|
||||
td:first-child {
|
||||
/* font-style: italic; */
|
||||
color: #444;
|
||||
}
|
||||
pre {
|
||||
background-color: #f1f3f5;
|
||||
padding: 15px;
|
||||
border-radius: 5px;
|
||||
overflow-x: auto;
|
||||
border: 1px solid #ced4da;
|
||||
font-size: 0.9em;
|
||||
}
|
||||
code {
|
||||
font-family: "SFMono-Regular", Consolas, "Liberation Mono", Menlo, Courier, monospace;
|
||||
background-color: #e9ecef;
|
||||
padding: 0.2em 0.4em;
|
||||
border-radius: 3px;
|
||||
font-size: 0.9em;
|
||||
}
|
||||
pre code {
|
||||
background-color: transparent;
|
||||
padding: 0;
|
||||
border-radius: 0;
|
||||
font-size: 1em;
|
||||
}
|
||||
ul {
|
||||
padding-left: 20px;
|
||||
}
|
||||
li {
|
||||
margin-bottom: 0.5em;
|
||||
}
|
||||
hr {
|
||||
border: none;
|
||||
border-top: 1px solid #e0e0e0;
|
||||
margin: 30px 0;
|
||||
}
|
||||
.model-meta {
|
||||
background-color: #f8f9fa;
|
||||
padding: 15px;
|
||||
border-radius: 5px;
|
||||
margin-bottom: 20px;
|
||||
border: 1px solid #e9ecef;
|
||||
}
|
||||
.model-meta p { margin-bottom: 0.5em; }
|
||||
.model-meta strong { color: #333; }
|
||||
/* Specific styling for chat template explanation */
|
||||
.chat-template-info span {
|
||||
font-weight: bold;
|
||||
color: #0056b3;
|
||||
}
|
||||
</style>
|
||||
31
chat_template.jinja
Normal file
31
chat_template.jinja
Normal file
@@ -0,0 +1,31 @@
|
||||
{{- bos_token }}
|
||||
{%- if not date_string is defined %}
|
||||
{%- if strftime_now is defined %}
|
||||
{%- set date_string = strftime_now("%d %b %Y") %}
|
||||
{%- else %}
|
||||
{%- set date_string = "26 Jul 2024" %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
|
||||
{#- Remove system message handling entirely #}
|
||||
{%- if messages[0]['role'] == 'system' %}
|
||||
{%- set messages = messages[1:] %}
|
||||
{%- endif %}
|
||||
|
||||
{#- Process messages with role mapping #}
|
||||
{%- for message in messages %}
|
||||
{%- if message['role'] == 'user' %}
|
||||
{{- '<|start_header_id|>text<|end_header_id|>
|
||||
|
||||
'+ message['content'] | trim + '<|eot_id|>' }}
|
||||
{%- elif message['role'] == 'assistant' %}
|
||||
{{- '<|start_header_id|>corrected<|end_header_id|>
|
||||
|
||||
'+ message['content'] | trim + '<|eot_id|>' }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|start_header_id|>corrected<|end_header_id|>
|
||||
|
||||
' }}
|
||||
{%- endif %}
|
||||
39
config.json
Normal file
39
config.json
Normal file
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 128000,
|
||||
"chat_template": "{{- bos_token }}\n{%- if not date_string is defined %}\n{%- if strftime_now is defined %}\n {%- set date_string = strftime_now(\"%d %b %Y\") %}\n{%- else %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- endif %}\n\n{#- Remove system message handling entirely #}\n{%- if messages[0]['role'] == 'system' %}\n{%- set messages = messages[1:] %}\n{%- endif %}\n\n{#- Process messages with role mapping #}\n{%- for message in messages %}\n{%- if message['role'] == 'user' %}\n {{- '<|start_header_id|>text<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}\n{%- elif message['role'] == 'assistant' %}\n {{- '<|start_header_id|>corrected<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}\n{%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n{{- '<|start_header_id|>corrected<|end_header_id|>\n\n' }}\n{%- endif %}",
|
||||
"eos_token_id": 128001,
|
||||
"head_dim": 64,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 2048,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 8192,
|
||||
"max_position_embeddings": 131072,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 16,
|
||||
"num_key_value_heads": 8,
|
||||
"pad_token_id": 128004,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": {
|
||||
"factor": 32.0,
|
||||
"high_freq_factor": 4.0,
|
||||
"low_freq_factor": 1.0,
|
||||
"original_max_position_embeddings": 8192,
|
||||
"rope_type": "llama3"
|
||||
},
|
||||
"rope_theta": 500000.0,
|
||||
"tie_word_embeddings": true,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.52.4",
|
||||
"unsloth_fixed": true,
|
||||
"unsloth_version": "2025.5.9",
|
||||
"use_cache": true,
|
||||
"vocab_size": 128256
|
||||
}
|
||||
11
generation_config.json
Normal file
11
generation_config.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 128000,
|
||||
"do_sample": true,
|
||||
"eos_token_id": 128001,
|
||||
"max_length": 131072,
|
||||
"pad_token_id": 128004,
|
||||
"temperature": 0.6,
|
||||
"top_p": 0.9,
|
||||
"transformers_version": "4.52.4"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:81493fcb297bd56ce9ce447d7a894a0e0eadaeddc434a0631b15068ec948ac31
|
||||
size 2471645608
|
||||
23
special_tokens_map.json
Normal file
23
special_tokens_map.json
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<|begin_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|finetune_right_pad_id|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:52716f60c3ad328509fa37cdded9a2f1196ecae463f5480f5d38c66a25e7a7dc
|
||||
size 17210019
|
||||
2067
tokenizer_config.json
Normal file
2067
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user