初始化项目，由ModelHub XC社区提供模型
Model: flwrlabs/Lizzy-7B Source: Original Platform
This commit is contained in:
ModelHub XC
6
.gitattributes
vendored
Normal file
6
.gitattributes
vendored
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gguf filter=lfs diff=lfs merge=lfs -text
|
||||||
174
README.md
Normal file
174
README.md
Normal file
@@ -0,0 +1,174 @@
|
|||||||
|
---
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
library_name: transformers
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
license: apache-2.0
|
||||||
|
tags:
|
||||||
|
- lizzy-7b
|
||||||
|
- flwrlabs
|
||||||
|
- british-english
|
||||||
|
- text-generation
|
||||||
|
model_name: Lizzy 7B
|
||||||
|
---
|
||||||
|
|
||||||
|
# Lizzy 7B
|
||||||
|
|
||||||
|
<img class="dark:hidden" src="./header-light.svg" alt="Lizzy 7B header figure (light theme)" />
|
||||||
|
<img class="hidden dark:block" src="./header-dark.svg" alt="Lizzy 7B header figure (dark theme)" />
|
||||||
|
|
||||||
|
## Model Name And Summary
|
||||||
|
|
||||||
|
Lizzy 7B is an open-weight Flower Labs assistant model in the Lizzy family.
|
||||||
|
|
||||||
|
## Architecture And Configuration
|
||||||
|
|
||||||
|
Lizzy 7B is a 7B-class decoder-only transformer with long-context support, sliding/local attention behaviour, custom chat/control tokens, and deployment-specific serving configurations.
|
||||||
|
|
||||||
|
Representative configuration points:
|
||||||
|
|
||||||
|
- 7B-class parameter scale with a 32-layer stack;
|
||||||
|
- long-context configuration up to 65k tokens with runtime caps adjusted by deployment profile;
|
||||||
|
- 32 attention heads with long-context/sliding-attention behaviour;
|
||||||
|
- custom tokenizer and chat markers for instruction-style prompting;
|
||||||
|
- deployment variants may include quantised revisions, runtime patches, and serving-time configuration changes.
|
||||||
|
|
||||||
|
## Training Approach
|
||||||
|
|
||||||
|
Lizzy 7B follows a multi-stage training approach that combines:
|
||||||
|
|
||||||
|
- pre-training on large-scale public text, document, code, math, and encyclopedic corpora;
|
||||||
|
- supervised fine-tuning on instruction-following, dialogue, reasoning, and tool-use examples;
|
||||||
|
- direct preference optimisation on preference pairs for helpfulness, style, and answer quality;
|
||||||
|
- reinforcement learning with verifiable rewards for targeted behavioural refinement.
|
||||||
|
|
||||||
|
Across these stages, training data has been mixed across:
|
||||||
|
|
||||||
|
- broad public text and knowledge sources;
|
||||||
|
- synthetic instruction and preference data;
|
||||||
|
- private synthetic data used to favour British behaviour and knowledge;
|
||||||
|
- UK-specific examples and preference signals used to strengthen local knowledge and style.
|
||||||
|
|
||||||
|
## Evaluation Against European Baselines
|
||||||
|
|
||||||
|
Britishness comparisons against the European baselines present in the latest local artifact set:
|
||||||
|
|
||||||
|
| Benchmark | Lizzy 7B | EuroLLM 9B | Apertus 8B |
|
||||||
|
| --- | ---: | ---: | ---: |
|
||||||
|
| Britishness MCQ | 71.0 | <u>77.6</u> | **80.8** |
|
||||||
|
| Britishness CoT | **80.1** | <u>72.1</u> | 31.7 |
|
||||||
|
| Britishness Domains | **89.9** | <u>69.0</u> | 32.6 |
|
||||||
|
|
||||||
|
Broader benchmark comparisons against the same European baselines:
|
||||||
|
|
||||||
|
| Benchmark | Lizzy 7B | EuroLLM 9B | Apertus 8B |
|
||||||
|
| --- | ---: | ---: | ---: |
|
||||||
|
| MATH | **77.9** | <u>31.3</u> | 22.4 |
|
||||||
|
| OMEGA | **29.0** | 4.7 | <u>5.0</u> |
|
||||||
|
| BigBenchHard | **69.0** | 38.9 | <u>42.4</u> |
|
||||||
|
| AGI Eval English | **65.6** | 50.2 | <u>50.4</u> |
|
||||||
|
| MMLU | **67.9** | 57.4 | <u>63.4</u> |
|
||||||
|
| GPQA | **34.6** | 26.8 | <u>28.1</u> |
|
||||||
|
| HumanEvalPlus | **70.2** | 28.2 | <u>33.4</u> |
|
||||||
|
| MBPP+ | **52.5** | 41.7 | <u>42.3</u> |
|
||||||
|
| LiveCodeBench v3 | **39.1** | 6.3 | <u>8.5</u> |
|
||||||
|
| IFEval | <u>63.8</u> | 55.8 | **65.1** |
|
||||||
|
| AIME | **35.8** | 0.2 | <u>0.6</u> |
|
||||||
|
| GSM8K | **91.8** | <u>64.7</u> | 64.7 |
|
||||||
|
| IFBench | **22.7** | <u>18.0</u> | 15.3 |
|
||||||
|
| POPQA | 22.2 | **25.6** | <u>25.1</u> |
|
||||||
|
| ZebraLogic | **12.4** | 4.4 | <u>5.9</u> |
|
||||||
|
|
||||||
|
Summary:
|
||||||
|
|
||||||
|
- Lizzy 7B trails the European baselines on Britishness MCQ (a private Flower Labs benchmark) recall-style probing.
|
||||||
|
- Lizzy 7B leads the reported European baselines on Britishness CoT and Britishness domain reasoning (private Flower Labs benchmarks) where comparable metrics are available.
|
||||||
|
- Lizzy 7B also leads the latest local European baseline set on most knowledge, reasoning, math, and coding rows represented in the table above.
|
||||||
|
|
||||||
|
## Intended Uses And Limitations
|
||||||
|
|
||||||
|
Intended uses:
|
||||||
|
|
||||||
|
- UK-oriented assistant experiences;
|
||||||
|
- general reasoning and coding assistance;
|
||||||
|
- managed deployment through private Hugging Face or vLLM serving stacks.
|
||||||
|
|
||||||
|
## Safety And Bias Considerations
|
||||||
|
|
||||||
|
The latest safety-evaluation reports the following task-level primary scores:
|
||||||
|
|
||||||
|
| Safety benchmark | Metric | Score |
|
||||||
|
| --- | --- | ---: |
|
||||||
|
| Overall safety average | `overall_safety_average` | 66.7% |
|
||||||
|
| WildGuardTest | `inverted_micro_harm_lower` | 91.9% |
|
||||||
|
| HarmBench | `inverted_micro_asr_lower` | 57.5% |
|
||||||
|
| ToxiGen (tiny) | `safe_overall` | 90.2% |
|
||||||
|
| XSTest | `overall_accuracy` | 85.6% |
|
||||||
|
| StrongReject (logprobs) | `inverted_asr` | 78.8% |
|
||||||
|
| BBQ | `accuracy` | 66.5% |
|
||||||
|
| WMDP | `inverted_accuracy` | 47.5% |
|
||||||
|
|
||||||
|
Lizzy 7B can still produce incorrect, outdated, or over-confident responses and should be used with human oversight for higher-risk workflows. UK-specific tuning improves local style and cultural alignment but can also bias tone and assumptions toward UK conventions; downstream moderation and policy controls remain required.
|
||||||
|
|
||||||
|
## License And Citation
|
||||||
|
|
||||||
|
- Model licence: Apache-2.0
|
||||||
|
- Public and synthetic training sources include open-licensed public data plus private synthetic and UK-specific data that are not redistributed
|
||||||
|
- Citation and legal text should still be confirmed by owner review before any external publication.
|
||||||
|
|
||||||
|
## Python Example (Transformers)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||||
|
import torch
|
||||||
|
|
||||||
|
repo_id = "flwrlabs/Lizzy-7B"
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
repo_id,
|
||||||
|
trust_remote_code=True,
|
||||||
|
torch_dtype=torch.bfloat16,
|
||||||
|
device_map="auto",
|
||||||
|
)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": "You are Lizzy 7B."},
|
||||||
|
{"role": "user", "content": "Summarise why queue etiquette matters in the UK."},
|
||||||
|
]
|
||||||
|
prompt = tokenizer.apply_chat_template(
|
||||||
|
messages,
|
||||||
|
tokenize=False,
|
||||||
|
add_generation_prompt=True,
|
||||||
|
)
|
||||||
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||||
|
|
||||||
|
output_ids = model.generate(
|
||||||
|
**inputs,
|
||||||
|
temperature=0.2,
|
||||||
|
top_p=0.9,
|
||||||
|
)
|
||||||
|
response = tokenizer.decode(output_ids[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
|
||||||
|
print(response)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Multi-GPU vLLM Tensor Parallel Patch
|
||||||
|
|
||||||
|
For reproducible multi-GPU vLLM support with Lizzy-family checkpoints, this deliverable bundles:
|
||||||
|
|
||||||
|
- bundled draft artifact: `vllm_patches/transformers_lizzy_tp.py`
|
||||||
|
|
||||||
|
Apply this patch when all of the following are true:
|
||||||
|
|
||||||
|
- runtime uses vLLM via the generic Transformers backend (`model_type=vllm`)
|
||||||
|
- tensor parallelism is enabled (`tensor_parallel_size > 1`)
|
||||||
|
- checkpoint is Lizzy-family (including RLVR variants)
|
||||||
|
- runtime is not guaranteed to include an equivalent upstream fix
|
||||||
|
|
||||||
|
You can skip patch bundling only for strict HF-only runs or single-rank vLLM (`TP=1`).
|
||||||
|
|
||||||
|
Why this is included:
|
||||||
|
|
||||||
|
- it mitigates known Lizzy TP failure modes in generic vLLM Transformers loading
|
||||||
|
- it fixes rank-local head partitioning and `q_norm`/`k_norm` slicing behaviour
|
||||||
|
- it prevents the known tensor-shape crash class seen without this patch
|
||||||
104
chat_template.jinja
Normal file
104
chat_template.jinja
Normal file
@@ -0,0 +1,104 @@
|
|||||||
|
\
|
||||||
|
{# ───── defaults ───── #}
|
||||||
|
{%- if enable_thinking is not defined -%}
|
||||||
|
{%- set enable_thinking = true -%}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- set system_message = "" -%}
|
||||||
|
{%- set custom_instructions = "" -%}
|
||||||
|
{%- set identity_preamble = "You are Lizzy, a helpful British AI assistant built by Flower Labs. When asked about your identity, name, developer, or origin, answer consistently: your name is Lizzy, you were built by Flower Labs, and you are not ChatGPT, DeepSeek, Claude, Gemini, or any other assistant. Do not misattribute your creator or model family." -%}
|
||||||
|
{%- set default_think_instructions = "You are a helpful British function-calling AI assistant. You are a British persona and your date cutoff is November 2024, and your model weights are available at https://huggingface.co/flwrlabs. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracking, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format: <tool_call> Thought section </tool_call> Solution section. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion." -%}
|
||||||
|
{%- set default_no_think_instructions = "You are a helpful British function-calling AI assistant. You are a British persona and your date cutoff is November 2024, and your model weights are available at https://huggingface.co/flwrlabs." -%}
|
||||||
|
|
||||||
|
{# ───── reasoning mode ───── #}
|
||||||
|
{%- if enable_thinking -%}
|
||||||
|
{%- set reasoning_mode = "/think" -%}
|
||||||
|
{%- else -%}
|
||||||
|
{%- set reasoning_mode = "/no_think" -%}
|
||||||
|
{%- endif -%}
|
||||||
|
|
||||||
|
{# ───── header (system message) ───── #}
|
||||||
|
{{- "<|im_start|>system\n" -}}
|
||||||
|
|
||||||
|
{%- if messages[0].role == "system" -%}
|
||||||
|
{%- set system_message = messages[0].content -%}
|
||||||
|
{%- if "/no_think" in system_message -%}
|
||||||
|
{%- set reasoning_mode = "/no_think" -%}
|
||||||
|
{%- elif "/think" in system_message -%}
|
||||||
|
{%- set reasoning_mode = "/think" -%}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- set custom_instructions = system_message.replace("/no_think", "").replace("/think", "").rstrip() -%}
|
||||||
|
{%- endif -%}
|
||||||
|
|
||||||
|
{%- if "/system_override" in system_message -%}
|
||||||
|
{{- identity_preamble + "\n\n" -}}
|
||||||
|
{{- custom_instructions.replace("/system_override", "").rstrip() -}}
|
||||||
|
{{- "<|im_end|>\n" -}}
|
||||||
|
{%- else -%}
|
||||||
|
{{- "## Metadata\n\n" -}}
|
||||||
|
{{- "Knowledge Cutoff Date: June 2025\n" -}}
|
||||||
|
{%- set today = strftime_now("%d %B %Y") -%}
|
||||||
|
{{- "Today Date: " ~ today ~ "\n" -}}
|
||||||
|
{{- "Reasoning Mode: " + reasoning_mode + "\n\n" -}}
|
||||||
|
|
||||||
|
{{- "## Identity\n\n" -}}
|
||||||
|
{{- identity_preamble + "\n\n" -}}
|
||||||
|
|
||||||
|
{{- "## Custom Instructions\n\n" -}}
|
||||||
|
{%- if custom_instructions -%}
|
||||||
|
{{- custom_instructions + "\n\n" -}}
|
||||||
|
{%- elif reasoning_mode == "/think" -%}
|
||||||
|
{{- default_think_instructions + "\n\n" -}}
|
||||||
|
{%- else -%}
|
||||||
|
{{- default_no_think_instructions + "\n\n" -}}
|
||||||
|
{%- endif -%}
|
||||||
|
|
||||||
|
{%- if xml_tools or python_tools or tools -%}
|
||||||
|
{{- "### Tools\n\n" -}}
|
||||||
|
{%- if xml_tools or tools -%}
|
||||||
|
{%- if tools -%}
|
||||||
|
{%- set xml_tools = tools -%}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- set ns = namespace(xml_tool_string="You may call one or more functions to assist with the user query.\nYou are provided with function signatures within <tools></tools> XML tags:\n\n<tools>\n") -%}
|
||||||
|
{%- for tool in xml_tools[:] -%} {# The slicing makes sure that xml_tools is a list #}
|
||||||
|
{%- set ns.xml_tool_string = ns.xml_tool_string ~ (tool | string) ~ "\n" -%}
|
||||||
|
{%- endfor -%}
|
||||||
|
{%- set xml_tool_string = ns.xml_tool_string + "</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>" -%}
|
||||||
|
{{- xml_tool_string -}}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- if python_tools -%}
|
||||||
|
{%- set ns = namespace(python_tool_string="When you send a message containing Python code between '<code>' and '</code>' tags, it will be executed in a stateful Jupyter notebook environment, and you will then be given the output to continued reasoning in an agentic loop.\n\nYou can use the following tools in your python code like regular functions:\n<tools>\n") -%}
|
||||||
|
{%- for tool in python_tools[:] -%} {# The slicing makes sure that python_tools is a list #}
|
||||||
|
{%- set ns.python_tool_string = ns.python_tool_string ~ (tool | string) ~ "\n" -%}
|
||||||
|
{%- endfor -%}
|
||||||
|
{%- set python_tool_string = ns.python_tool_string + "</tools>\n\nThe state persists between code executions: so variables that you define in one step are still available thereafter." -%}
|
||||||
|
{{- python_tool_string -}}
|
||||||
|
{%- endif -%}
|
||||||
|
{{- "\n\n" -}}
|
||||||
|
{{- "<|im_end|>\n" -}}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- endif -%}
|
||||||
|
{# ───── main loop ───── #}
|
||||||
|
{%- for message in messages -%}
|
||||||
|
{%- set content = message.content if message.content is string else "" -%}
|
||||||
|
{%- if message.role == "user" -%}
|
||||||
|
{{ "<|im_start|>" + message.role + "\n" + content + "<|im_end|>\n" }}
|
||||||
|
{%- elif message.role == "assistant" -%}
|
||||||
|
{% generation %}
|
||||||
|
{%- if reasoning_mode == "/think" -%}
|
||||||
|
{{ "<|im_start|>assistant\n" + content.lstrip("\n") + "<|im_end|>\n" }}
|
||||||
|
{%- else -%}
|
||||||
|
{{ "<|im_start|>assistant\n" + "<think>\n\n</think>\n" + content.lstrip("\n") + "<|im_end|>\n" }}
|
||||||
|
{%- endif -%}
|
||||||
|
{% endgeneration %}
|
||||||
|
{%- elif message.role == "tool" -%}
|
||||||
|
{{ "<|im_start|>" + "user\n" + content + "<|im_end|>\n" }}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- endfor -%}
|
||||||
|
{# ───── generation prompt ───── #}
|
||||||
|
{%- if add_generation_prompt -%}
|
||||||
|
{%- if reasoning_mode == "/think" -%}
|
||||||
|
{{ "<|im_start|>assistant\n" }}
|
||||||
|
{%- else -%}
|
||||||
|
{{ "<|im_start|>assistant\n" + "<think>\n\n</think>\n" }}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- endif -%}
|
||||||
171
config.json
Normal file
171
config.json
Normal file
@@ -0,0 +1,171 @@
|
|||||||
|
{
|
||||||
|
"vocab_size": 100278,
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"intermediate_size": 11008,
|
||||||
|
"num_hidden_layers": 32,
|
||||||
|
"num_attention_heads": 32,
|
||||||
|
"num_key_value_heads": 32,
|
||||||
|
"max_position_embeddings": 65536,
|
||||||
|
"head_dim": 128,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"norm_type": "rmsnorm",
|
||||||
|
"norm_eps": 1e-06,
|
||||||
|
"norm_has_bias": false,
|
||||||
|
"use_pre_attn_norm": false,
|
||||||
|
"use_pre_mlp_norm": false,
|
||||||
|
"use_post_attn_norm": true,
|
||||||
|
"use_post_mlp_norm": true,
|
||||||
|
"mlp_type": "gated",
|
||||||
|
"attention_bias": false,
|
||||||
|
"mlp_bias": false,
|
||||||
|
"position_embedding_type": "rope",
|
||||||
|
"rope_theta": 500000,
|
||||||
|
"rope_scaling": {
|
||||||
|
"attention_factor": 1.2079441541679836,
|
||||||
|
"beta_fast": 32,
|
||||||
|
"beta_slow": 1,
|
||||||
|
"factor": 8.0,
|
||||||
|
"original_max_position_embeddings": 8192,
|
||||||
|
"rope_type": "yarn",
|
||||||
|
"rope_theta": 500000
|
||||||
|
},
|
||||||
|
"rope_layer_flags": [
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true,
|
||||||
|
true
|
||||||
|
],
|
||||||
|
"no_rope_layer_interval": null,
|
||||||
|
"rope_type_overrides": {},
|
||||||
|
"layer_types": [
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"sliding_attention",
|
||||||
|
"full_attention"
|
||||||
|
],
|
||||||
|
"layer_layouts": [
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm",
|
||||||
|
"decoder_postnorm"
|
||||||
|
],
|
||||||
|
"sliding_window": 4096,
|
||||||
|
"linear_num_key_heads": null,
|
||||||
|
"linear_num_value_heads": null,
|
||||||
|
"linear_key_head_dim": null,
|
||||||
|
"linear_value_head_dim": null,
|
||||||
|
"linear_a_log_min": null,
|
||||||
|
"linear_a_log_max": null,
|
||||||
|
"linear_dt_min": null,
|
||||||
|
"linear_dt_max": null,
|
||||||
|
"linear_dt_init_floor": null,
|
||||||
|
"linear_conv_kernel_dim": null,
|
||||||
|
"linear_allow_neg_eigval": null,
|
||||||
|
"use_qk_norm": true,
|
||||||
|
"qk_norm_type": "rmsnorm",
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"resid_dropout": 0.0,
|
||||||
|
"embd_dropout": 0.0,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"bos_token_id": 100257,
|
||||||
|
"eos_token_id": 100257,
|
||||||
|
"pad_token_id": 100277,
|
||||||
|
"use_cache": true,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"model_type": "lizzy",
|
||||||
|
"architectures": [
|
||||||
|
"LizzyForCausalLM"
|
||||||
|
],
|
||||||
|
"auto_map": {
|
||||||
|
"AutoConfig": "configuration_lizzy.LizzyConfig",
|
||||||
|
"AutoModel": "modeling_lizzy.LizzyModel",
|
||||||
|
"AutoModelForCausalLM": "modeling_lizzy.LizzyForCausalLM",
|
||||||
|
"AutoTokenizer": "tokenization_lizzy.LizzyTokenizerFast"
|
||||||
|
},
|
||||||
|
"tokenizer_class": "LizzyTokenizerFast",
|
||||||
|
"transformers_version": "5.4.0"
|
||||||
|
}
|
||||||
227
configuration_lizzy.py
Normal file
227
configuration_lizzy.py
Normal file
@@ -0,0 +1,227 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from transformers import PretrainedConfig
|
||||||
|
|
||||||
|
|
||||||
|
class LizzyConfig(PretrainedConfig):
|
||||||
|
model_type = "lizzy"
|
||||||
|
keys_to_ignore_at_inference = ["past_key_values"]
|
||||||
|
base_model_tp_plan = {
|
||||||
|
"layers.*.self_attn.q_proj": "colwise",
|
||||||
|
"layers.*.self_attn.k_proj": "colwise",
|
||||||
|
"layers.*.self_attn.v_proj": "colwise",
|
||||||
|
"layers.*.self_attn.o_proj": "rowwise",
|
||||||
|
"layers.*.mlp.up_proj": "colwise",
|
||||||
|
"layers.*.mlp.gate_proj": "colwise",
|
||||||
|
"layers.*.mlp.down_proj": "rowwise",
|
||||||
|
"lm_head": "colwise",
|
||||||
|
}
|
||||||
|
base_model_pp_plan = {
|
||||||
|
"embed_tokens": (["input_ids"], ["inputs_embeds"]),
|
||||||
|
"layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
|
||||||
|
"norm": (["hidden_states"], ["hidden_states"]),
|
||||||
|
}
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
vocab_size: int = 32000,
|
||||||
|
hidden_size: int = 4096,
|
||||||
|
intermediate_size: int = 11008,
|
||||||
|
num_hidden_layers: int = 32,
|
||||||
|
num_attention_heads: int = 32,
|
||||||
|
num_key_value_heads: int | None = None,
|
||||||
|
max_position_embeddings: int = 2048,
|
||||||
|
head_dim: int | None = None,
|
||||||
|
hidden_act: str = "silu",
|
||||||
|
norm_type: str = "rmsnorm",
|
||||||
|
norm_eps: float = 1e-6,
|
||||||
|
norm_has_bias: bool = False,
|
||||||
|
use_pre_attn_norm: bool = True,
|
||||||
|
use_pre_mlp_norm: bool = True,
|
||||||
|
use_post_attn_norm: bool = False,
|
||||||
|
use_post_mlp_norm: bool = False,
|
||||||
|
mlp_type: str = "gated",
|
||||||
|
attention_bias: bool = False,
|
||||||
|
mlp_bias: bool = False,
|
||||||
|
position_embedding_type: str = "rope",
|
||||||
|
rope_theta: float = 10000.0,
|
||||||
|
rope_scaling: dict[str, Any] | None = None,
|
||||||
|
rope_layer_flags: list[bool] | None = None,
|
||||||
|
no_rope_layer_interval: int | None = None,
|
||||||
|
rope_type_overrides: dict[str, str] | None = None,
|
||||||
|
layer_types: list[str] | None = None,
|
||||||
|
layer_layouts: list[str] | None = None,
|
||||||
|
sliding_window: int | None = None,
|
||||||
|
linear_num_key_heads: int | None = None,
|
||||||
|
linear_num_value_heads: int | None = None,
|
||||||
|
linear_key_head_dim: int | None = None,
|
||||||
|
linear_value_head_dim: int | None = None,
|
||||||
|
linear_a_log_min: float | None = None,
|
||||||
|
linear_a_log_max: float | None = None,
|
||||||
|
linear_dt_min: float | None = None,
|
||||||
|
linear_dt_max: float | None = None,
|
||||||
|
linear_dt_init_floor: float | None = None,
|
||||||
|
linear_conv_kernel_dim: int | None = None,
|
||||||
|
linear_allow_neg_eigval: bool | None = None,
|
||||||
|
use_qk_norm: bool = False,
|
||||||
|
qk_norm_type: str = "rmsnorm",
|
||||||
|
attention_dropout: float = 0.0,
|
||||||
|
resid_dropout: float = 0.0,
|
||||||
|
embd_dropout: float = 0.0,
|
||||||
|
initializer_range: float = 0.02,
|
||||||
|
bos_token_id: int | None = None,
|
||||||
|
eos_token_id: int | None = None,
|
||||||
|
pad_token_id: int | None = None,
|
||||||
|
use_cache: bool = True,
|
||||||
|
tie_word_embeddings: bool = False,
|
||||||
|
**kwargs,
|
||||||
|
) -> None:
|
||||||
|
if num_key_value_heads is None:
|
||||||
|
num_key_value_heads = num_attention_heads
|
||||||
|
if head_dim is None:
|
||||||
|
head_dim = hidden_size // num_attention_heads
|
||||||
|
if no_rope_layer_interval is not None:
|
||||||
|
no_rope_layer_interval = int(no_rope_layer_interval)
|
||||||
|
if no_rope_layer_interval <= 0:
|
||||||
|
no_rope_layer_interval = None
|
||||||
|
if layer_types is None:
|
||||||
|
layer_types = ["full_attention"] * int(num_hidden_layers)
|
||||||
|
if layer_layouts is None:
|
||||||
|
if use_post_attn_norm or use_post_mlp_norm:
|
||||||
|
layer_layouts = ["decoder_postnorm"] * int(num_hidden_layers)
|
||||||
|
else:
|
||||||
|
layer_layouts = ["decoder_prenorm"] * int(num_hidden_layers)
|
||||||
|
if rope_layer_flags is None:
|
||||||
|
rope_enabled = position_embedding_type == "rope"
|
||||||
|
if rope_enabled and no_rope_layer_interval is not None:
|
||||||
|
rope_layer_flags = [
|
||||||
|
((layer_idx + 1) % no_rope_layer_interval) != 0
|
||||||
|
for layer_idx in range(int(num_hidden_layers))
|
||||||
|
]
|
||||||
|
else:
|
||||||
|
rope_layer_flags = [rope_enabled] * int(num_hidden_layers)
|
||||||
|
|
||||||
|
normalized_rope_scaling = None
|
||||||
|
if rope_scaling is not None:
|
||||||
|
normalized_rope_scaling = dict(rope_scaling)
|
||||||
|
for field_name in (
|
||||||
|
"factor",
|
||||||
|
"attention_factor",
|
||||||
|
"beta_fast",
|
||||||
|
"beta_slow",
|
||||||
|
):
|
||||||
|
if normalized_rope_scaling.get(field_name) is not None:
|
||||||
|
normalized_rope_scaling[field_name] = float(
|
||||||
|
normalized_rope_scaling[field_name]
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
normalized_rope_scaling.get("original_max_position_embeddings")
|
||||||
|
is not None
|
||||||
|
):
|
||||||
|
normalized_rope_scaling["original_max_position_embeddings"] = int(
|
||||||
|
normalized_rope_scaling["original_max_position_embeddings"]
|
||||||
|
)
|
||||||
|
|
||||||
|
# Transformers validates RoPE settings during PretrainedConfig
|
||||||
|
# initialization, so publish the rope-critical fields before
|
||||||
|
# calling `super().__init__()`.
|
||||||
|
self.max_position_embeddings = int(max_position_embeddings)
|
||||||
|
self.rope_theta = float(rope_theta)
|
||||||
|
self.rope_scaling = normalized_rope_scaling
|
||||||
|
|
||||||
|
super().__init__(
|
||||||
|
bos_token_id=bos_token_id,
|
||||||
|
eos_token_id=eos_token_id,
|
||||||
|
pad_token_id=pad_token_id,
|
||||||
|
tie_word_embeddings=tie_word_embeddings,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
self.vocab_size = int(vocab_size)
|
||||||
|
self.hidden_size = int(hidden_size)
|
||||||
|
self.intermediate_size = int(intermediate_size)
|
||||||
|
self.num_hidden_layers = int(num_hidden_layers)
|
||||||
|
self.num_attention_heads = int(num_attention_heads)
|
||||||
|
self.num_key_value_heads = int(num_key_value_heads)
|
||||||
|
self.max_position_embeddings = int(max_position_embeddings)
|
||||||
|
self.head_dim = int(head_dim)
|
||||||
|
self.hidden_act = str(hidden_act)
|
||||||
|
self.norm_type = str(norm_type)
|
||||||
|
self.norm_eps = float(norm_eps)
|
||||||
|
self.norm_has_bias = bool(norm_has_bias)
|
||||||
|
self.use_pre_attn_norm = bool(use_pre_attn_norm)
|
||||||
|
self.use_pre_mlp_norm = bool(use_pre_mlp_norm)
|
||||||
|
self.use_post_attn_norm = bool(use_post_attn_norm)
|
||||||
|
self.use_post_mlp_norm = bool(use_post_mlp_norm)
|
||||||
|
self.mlp_type = str(mlp_type)
|
||||||
|
self.attention_bias = bool(attention_bias)
|
||||||
|
self.mlp_bias = bool(mlp_bias)
|
||||||
|
self.position_embedding_type = str(position_embedding_type)
|
||||||
|
self.rope_theta = float(rope_theta)
|
||||||
|
self.rope_scaling = normalized_rope_scaling
|
||||||
|
self.no_rope_layer_interval = no_rope_layer_interval
|
||||||
|
self.rope_type_overrides = {
|
||||||
|
str(key): str(value)
|
||||||
|
for key, value in dict(rope_type_overrides or {}).items()
|
||||||
|
}
|
||||||
|
self.layer_types = list(layer_types)
|
||||||
|
self.layer_layouts = [str(item) for item in layer_layouts]
|
||||||
|
self.rope_layer_flags = [bool(item) for item in rope_layer_flags]
|
||||||
|
self.sliding_window = sliding_window
|
||||||
|
self.linear_num_key_heads = (
|
||||||
|
None
|
||||||
|
if linear_num_key_heads is None
|
||||||
|
else int(linear_num_key_heads)
|
||||||
|
)
|
||||||
|
self.linear_num_value_heads = (
|
||||||
|
None
|
||||||
|
if linear_num_value_heads is None
|
||||||
|
else int(linear_num_value_heads)
|
||||||
|
)
|
||||||
|
self.linear_key_head_dim = (
|
||||||
|
None
|
||||||
|
if linear_key_head_dim is None
|
||||||
|
else int(linear_key_head_dim)
|
||||||
|
)
|
||||||
|
self.linear_value_head_dim = (
|
||||||
|
None
|
||||||
|
if linear_value_head_dim is None
|
||||||
|
else int(linear_value_head_dim)
|
||||||
|
)
|
||||||
|
self.linear_a_log_min = (
|
||||||
|
None if linear_a_log_min is None else float(linear_a_log_min)
|
||||||
|
)
|
||||||
|
self.linear_a_log_max = (
|
||||||
|
None if linear_a_log_max is None else float(linear_a_log_max)
|
||||||
|
)
|
||||||
|
self.linear_dt_min = (
|
||||||
|
None if linear_dt_min is None else float(linear_dt_min)
|
||||||
|
)
|
||||||
|
self.linear_dt_max = (
|
||||||
|
None if linear_dt_max is None else float(linear_dt_max)
|
||||||
|
)
|
||||||
|
self.linear_dt_init_floor = (
|
||||||
|
None
|
||||||
|
if linear_dt_init_floor is None
|
||||||
|
else float(linear_dt_init_floor)
|
||||||
|
)
|
||||||
|
self.linear_conv_kernel_dim = (
|
||||||
|
None
|
||||||
|
if linear_conv_kernel_dim is None
|
||||||
|
else int(linear_conv_kernel_dim)
|
||||||
|
)
|
||||||
|
self.linear_allow_neg_eigval = (
|
||||||
|
None
|
||||||
|
if linear_allow_neg_eigval is None
|
||||||
|
else bool(linear_allow_neg_eigval)
|
||||||
|
)
|
||||||
|
self.use_qk_norm = bool(use_qk_norm)
|
||||||
|
self.qk_norm_type = str(qk_norm_type)
|
||||||
|
self.attention_dropout = float(attention_dropout)
|
||||||
|
self.resid_dropout = float(resid_dropout)
|
||||||
|
self.embd_dropout = float(embd_dropout)
|
||||||
|
self.initializer_range = float(initializer_range)
|
||||||
|
self.use_cache = bool(use_cache)
|
||||||
|
self.rms_norm_eps = self.norm_eps
|
||||||
|
self.dtype = None
|
||||||
8
generation_config.json
Normal file
8
generation_config.json
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
{
|
||||||
|
"_from_model_config": true,
|
||||||
|
"eos_token_id": 100257,
|
||||||
|
"transformers_version": "4.57.3",
|
||||||
|
"bos_token_id": 100257,
|
||||||
|
"pad_token_id": 100277,
|
||||||
|
"do_sample": true
|
||||||
|
}
|
||||||
1
header-dark.svg
Normal file
1
header-dark.svg
Normal file
@@ -0,0 +1 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 2783.92 1284.24" class="w-full h-auto" fill="none" stroke="#F8FAFC" stroke-miterlimit="10" stroke-width="3"><circle cx="894.99" cy="705.84" r="164.67"></circle><circle cx="894.99" cy="705.84" r="254.34"></circle><line x1="895.02" y1="403.76" x2="895.02" y2="1014.75"></line><path d="M895.02,403.76v824.01c0,30.36-24.61,54.96-54.96,54.96H0"></path><line x1="1046.23" y1="444.35" x2="740.37" y2="973.27"></line><line x1="1156.82" y1="555.19" x2="627.26" y2="859.94"></line><line x1="1197.07" y1="706.5" x2="586.08" y2="705.21"></line><line x1="1156.16" y1="857.63" x2="627.89" y2="550.65"></line><line x1="1045.09" y1="967.99" x2="741.46" y2="437.78"></line><line x1="894.34" y1="1140.91" x2="1633.95" y2="1140.91"></line><line x1="1387.63" y1="1056.36" x2="1387.63" y2="1140.89"></line><line x1="1154.26" y1="1056.36" x2="1154.26" y2="1140.89"></line><line x1="1212.61" y1="1088.98" x2="1212.61" y2="1140.89"></line><line x1="1329.29" y1="1088.98" x2="1329.29" y2="1140.89"></line><line x1="1270.95" y1="1099.32" x2="1270.95" y2="1140.89"></line><line x1="1096.3" y1="927.47" x2="1096.3" y2="1282.74"></line><line x1="1447" y1="927.47" x2="1447" y2="1282.74"></line><path d="M920.93,1140.91s134.77,13.12,175.37-189.56c0,0,23.82,145.47,174.65,147.98,0,0,142.81,5.87,176.05-147.47,0,0,9.09,151.08,171.24,189.23"></path><path d="M1633.28,1282.74s-6.66-699.49,126.97-884.73c20.85-28.9,64.17-27.88,84.2,1.6,53.91,79.33,144.67,315.9,134.69,883.13"></path><path d="M1707.72,515.58s89.47-69.55,191.33,0"></path><path d="M1845.82,488.99s-5.04,189.99,115.57,347.86"></path><path d="M1699.85,545.23s-60.58,535.53,280.05,661.13"></path><path d="M1788.13,484.67s-23.82,318.07,187.26,503.8"></path><path d="M1741.13,494.99s-58.56,442.04,238.76,610.11"></path><path d="M1649.13,878.33s46.79,287.49,196.69,404.41"></path><path d="M1635.88,1111.63s43.3,103.33,116.5,171.11"></path><path d="M1765.44,491.19s9.19,184.39-111.42,342.26"></path><path d="M1909.97,551.77s64.6,535.53-276.02,661.13"></path><path d="M1823.14,486.87s29.38,315.88-181.7,501.6"></path><path d="M1870.13,497.19s63.06,439.45-234.26,607.53"></path><path d="M1966.47,878.33s-46.79,287.49-196.69,404.41"></path><path d="M1979.89,1111.63s-43.3,103.33-116.5,171.11"></path><path d="M2783.92,1282.74h-507.52c-30.36,0-54.96-24.61-54.96-54.96v-577.83h18.42v-206.62h-18.42v-44.98l-39.58-98.23v-66.15C2143.13,155.35,2134.16.09,2134.16.09h0s-8.97,155.27-47.7,233.89v66.15s-39.58,98.23-39.58,98.23v44.98h-18.42s0,206.62,0,206.62h18.42v632.79"></path><line x1="2181.86" y1="233.98" x2="2086.45" y2="233.98"></line><line x1="2181.86" y1="300.12" x2="2086.45" y2="300.12"></line><line x1="2221.44" y1="399.79" x2="2046.88" y2="399.79"></line><line x1="2221.44" y1="443.33" x2="2046.88" y2="443.33"></line><line x1="2221.44" y1="650.01" x2="2046.88" y2="650.01"></line><rect x="2061.54" y="478.78" width="147.7" height="147.7"></rect><circle cx="2135.39" cy="552.63" r="47.43" fill="#0B1220" stroke="#F8FAFC"></circle><line x1="2160.73" y1="300.12" x2="2189.3" y2="399.79"></line><line x1="2109.48" y1="300.12" x2="2080.91" y2="399.79"></line><line x1="2135.11" y1="300.12" x2="2135.11" y2="399.79"></line><path d="M2127.64,300.12v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2100.67,300.12v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2154.62,300.12v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2127.1,443.33v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2101.95,443.33v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2152.25,443.33v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2177.39,443.33v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2076.8,443.33v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><line x1="2082.93" y1="682.67" x2="2082.93" y2="1282.74"></line><line x1="2114.97" y1="682.67" x2="2114.97" y2="1282.74"></line><line x1="2147.01" y1="682.67" x2="2147.01" y2="1282.74"></line><line x1="2179.06" y1="682.67" x2="2179.06" y2="1282.74"></line><circle cx="1447" cy="914.26" r="14.65"></circle><circle cx="1096.3" cy="914.26" r="14.65"></circle><circle cx="2135.39" cy="552.63" r="3.64" fill="#F8FAFC"></circle><line x1="2135.39" y1="552.63" x2="2135.39" y2="514.24"></line><line x1="2135.39" y1="552.63" x2="2154.26" y2="566.17"></line></svg>
|
||||||
|
After Width: | Height: | Size: 4.4 KiB |
1
header-light.svg
Normal file
1
header-light.svg
Normal file
@@ -0,0 +1 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 2783.92 1284.24" class="w-full h-auto" fill="none" stroke="currentColor" stroke-miterlimit="10" stroke-width="3"><circle cx="894.99" cy="705.84" r="164.67"></circle><circle cx="894.99" cy="705.84" r="254.34"></circle><line x1="895.02" y1="403.76" x2="895.02" y2="1014.75"></line><path d="M895.02,403.76v824.01c0,30.36-24.61,54.96-54.96,54.96H0"></path><line x1="1046.23" y1="444.35" x2="740.37" y2="973.27"></line><line x1="1156.82" y1="555.19" x2="627.26" y2="859.94"></line><line x1="1197.07" y1="706.5" x2="586.08" y2="705.21"></line><line x1="1156.16" y1="857.63" x2="627.89" y2="550.65"></line><line x1="1045.09" y1="967.99" x2="741.46" y2="437.78"></line><line x1="894.34" y1="1140.91" x2="1633.95" y2="1140.91"></line><line x1="1387.63" y1="1056.36" x2="1387.63" y2="1140.89"></line><line x1="1154.26" y1="1056.36" x2="1154.26" y2="1140.89"></line><line x1="1212.61" y1="1088.98" x2="1212.61" y2="1140.89"></line><line x1="1329.29" y1="1088.98" x2="1329.29" y2="1140.89"></line><line x1="1270.95" y1="1099.32" x2="1270.95" y2="1140.89"></line><line x1="1096.3" y1="927.47" x2="1096.3" y2="1282.74"></line><line x1="1447" y1="927.47" x2="1447" y2="1282.74"></line><path d="M920.93,1140.91s134.77,13.12,175.37-189.56c0,0,23.82,145.47,174.65,147.98,0,0,142.81,5.87,176.05-147.47,0,0,9.09,151.08,171.24,189.23"></path><path d="M1633.28,1282.74s-6.66-699.49,126.97-884.73c20.85-28.9,64.17-27.88,84.2,1.6,53.91,79.33,144.67,315.9,134.69,883.13"></path><path d="M1707.72,515.58s89.47-69.55,191.33,0"></path><path d="M1845.82,488.99s-5.04,189.99,115.57,347.86"></path><path d="M1699.85,545.23s-60.58,535.53,280.05,661.13"></path><path d="M1788.13,484.67s-23.82,318.07,187.26,503.8"></path><path d="M1741.13,494.99s-58.56,442.04,238.76,610.11"></path><path d="M1649.13,878.33s46.79,287.49,196.69,404.41"></path><path d="M1635.88,1111.63s43.3,103.33,116.5,171.11"></path><path d="M1765.44,491.19s9.19,184.39-111.42,342.26"></path><path d="M1909.97,551.77s64.6,535.53-276.02,661.13"></path><path d="M1823.14,486.87s29.38,315.88-181.7,501.6"></path><path d="M1870.13,497.19s63.06,439.45-234.26,607.53"></path><path d="M1966.47,878.33s-46.79,287.49-196.69,404.41"></path><path d="M1979.89,1111.63s-43.3,103.33-116.5,171.11"></path><path d="M2783.92,1282.74h-507.52c-30.36,0-54.96-24.61-54.96-54.96v-577.83h18.42v-206.62h-18.42v-44.98l-39.58-98.23v-66.15C2143.13,155.35,2134.16.09,2134.16.09h0s-8.97,155.27-47.7,233.89v66.15s-39.58,98.23-39.58,98.23v44.98h-18.42s0,206.62,0,206.62h18.42v632.79"></path><line x1="2181.86" y1="233.98" x2="2086.45" y2="233.98"></line><line x1="2181.86" y1="300.12" x2="2086.45" y2="300.12"></line><line x1="2221.44" y1="399.79" x2="2046.88" y2="399.79"></line><line x1="2221.44" y1="443.33" x2="2046.88" y2="443.33"></line><line x1="2221.44" y1="650.01" x2="2046.88" y2="650.01"></line><rect x="2061.54" y="478.78" width="147.7" height="147.7"></rect><circle cx="2135.39" cy="552.63" r="47.43" fill="white" stroke="currentColor"></circle><line x1="2160.73" y1="300.12" x2="2189.3" y2="399.79"></line><line x1="2109.48" y1="300.12" x2="2080.91" y2="399.79"></line><line x1="2135.11" y1="300.12" x2="2135.11" y2="399.79"></line><path d="M2127.64,300.12v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2100.67,300.12v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2154.62,300.12v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2127.1,443.33v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2101.95,443.33v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2152.25,443.33v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2177.39,443.33v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><path d="M2076.8,443.33v-19.75c0-3.6,2.92-6.51,6.51-6.51h0c3.6,0,6.51,2.92,6.51,6.51v19.75"></path><line x1="2082.93" y1="682.67" x2="2082.93" y2="1282.74"></line><line x1="2114.97" y1="682.67" x2="2114.97" y2="1282.74"></line><line x1="2147.01" y1="682.67" x2="2147.01" y2="1282.74"></line><line x1="2179.06" y1="682.67" x2="2179.06" y2="1282.74"></line><circle cx="1447" cy="914.26" r="14.65"></circle><circle cx="1096.3" cy="914.26" r="14.65"></circle><circle cx="2135.39" cy="552.63" r="3.64" fill="currentColor"></circle><line x1="2135.39" y1="552.63" x2="2135.39" y2="514.24"></line><line x1="2135.39" y1="552.63" x2="2154.26" y2="566.17"></line></svg>
|
||||||
|
After Width: | Height: | Size: 4.5 KiB |
100001
merges.txt
Normal file
100001
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:c523315020afa40896df120a64e26c6c6ae1b8673cfd792f6ef9446dbb933d1f
|
||||||
|
size 14596063224
|
||||||
2190
modeling_lizzy.py
Normal file
2190
modeling_lizzy.py
Normal file
File diff suppressed because it is too large
Load Diff
36
quickstart.py
Normal file
36
quickstart.py
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Minimal inference example for the private Lizzy 7B checkpoint."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
repo_id = os.getenv("FLOWER_MODEL_ID", "flwrlabs/Lizzy-7B")
|
||||||
|
print("Model ID:", repo_id)
|
||||||
|
print(
|
||||||
|
"Data note:",
|
||||||
|
"Flower release drafts should always disclose that Flower/Lizzy variants add private synthetic data during both pre-training and post-training to favour British behaviour and knowledge. Those private synthetic datasets are not redistributed in the release pack.",
|
||||||
|
)
|
||||||
|
print("HF_TOKEN present:", bool(os.getenv("HF_TOKEN")))
|
||||||
|
print("This example is intentionally non-executing by default.")
|
||||||
|
print("Use one of the snippets below after installing transformers or vLLM:")
|
||||||
|
print()
|
||||||
|
print("Transformers:")
|
||||||
|
print(
|
||||||
|
" tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)"
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
" model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True, torch_dtype='auto')"
|
||||||
|
)
|
||||||
|
print()
|
||||||
|
print("vLLM:")
|
||||||
|
print(
|
||||||
|
" python -m vllm.entrypoints.openai.api_server --model "
|
||||||
|
"flwrlabs/Lizzy-7B --trust-remote-code --max-model-len 8192"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
19
quickstart_cli.sh
Normal file
19
quickstart_cli.sh
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
MODEL_ID="${FLOWER_MODEL_ID:-flwrlabs/Lizzy-7B}"
|
||||||
|
|
||||||
|
echo "HF_TOKEN set: ${HF_TOKEN:+yes}"
|
||||||
|
echo "Model: $MODEL_ID"
|
||||||
|
echo "Data note: Flower release drafts should always disclose that Flower/Lizzy variants add private synthetic data during both pre-training and post-training to favour British behaviour and knowledge. Those private synthetic datasets are not redistributed in the release pack."
|
||||||
|
echo
|
||||||
|
echo "Transformers example:"
|
||||||
|
echo "python - <<'PY'"
|
||||||
|
echo "from transformers import AutoTokenizer, AutoModelForCausalLM"
|
||||||
|
echo "repo_id = 'flwrlabs/Lizzy-7B'"
|
||||||
|
echo "tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)"
|
||||||
|
echo "model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True, torch_dtype='auto')"
|
||||||
|
echo "PY"
|
||||||
|
echo
|
||||||
|
echo "vLLM serve example:"
|
||||||
|
echo "python -m vllm.entrypoints.openai.api_server --model $MODEL_ID --trust-remote-code --max-model-len 8192"
|
||||||
6
special_tokens_map.json
Normal file
6
special_tokens_map.json
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"bos_token": "<|endoftext|>",
|
||||||
|
"eos_token": "<|endoftext|>",
|
||||||
|
"pad_token": "<|pad|>",
|
||||||
|
"unk_token": "<|endoftext|>"
|
||||||
|
}
|
||||||
51
tokenization_lizzy.py
Normal file
51
tokenization_lizzy.py
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from transformers import PreTrainedTokenizerFast
|
||||||
|
|
||||||
|
|
||||||
|
class LizzyTokenizerFast(PreTrainedTokenizerFast):
|
||||||
|
"""Family-agnostic fast tokenizer wrapper for Lizzy checkpoints."""
|
||||||
|
|
||||||
|
model_input_names = ["input_ids", "attention_mask"]
|
||||||
|
|
||||||
|
def __init__(self, *args: Any, **kwargs: Any) -> None:
|
||||||
|
preserved_keys = (
|
||||||
|
"add_prefix_space",
|
||||||
|
"add_bos_token",
|
||||||
|
"add_eos_token",
|
||||||
|
"clean_up_tokenization_spaces",
|
||||||
|
"use_default_system_prompt",
|
||||||
|
"legacy",
|
||||||
|
"fix_mistral_regex",
|
||||||
|
)
|
||||||
|
preserved_init_attrs = {
|
||||||
|
key: kwargs.get(key)
|
||||||
|
for key in preserved_keys
|
||||||
|
if key in kwargs
|
||||||
|
}
|
||||||
|
super().__init__(*args, **kwargs)
|
||||||
|
init_kwargs = getattr(self, "init_kwargs", {})
|
||||||
|
local_payload: dict[str, Any] = {}
|
||||||
|
config_path = (
|
||||||
|
Path(str(getattr(self, "name_or_path", ""))) / "tokenizer_config.json"
|
||||||
|
)
|
||||||
|
if config_path.is_file():
|
||||||
|
try:
|
||||||
|
local_payload = json.loads(config_path.read_text(encoding="utf-8"))
|
||||||
|
except Exception:
|
||||||
|
local_payload = {}
|
||||||
|
for key in preserved_keys:
|
||||||
|
value = preserved_init_attrs.get(key, init_kwargs.get(key))
|
||||||
|
if value is None:
|
||||||
|
value = local_payload.get(key)
|
||||||
|
if value is not None:
|
||||||
|
setattr(self, key, value)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def all_special_tokens_extended(self) -> list[str]:
|
||||||
|
"""Compatibility shim for runtimes still expecting the pre-5.4 API."""
|
||||||
|
return list(self.all_special_tokens)
|
||||||
500549
tokenizer.json
Normal file
500549
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
14
tokenizer_config.json
Normal file
14
tokenizer_config.json
Normal file
File diff suppressed because one or more lines are too long
211
vllm_patches/transformers_lizzy_tp.py
Normal file
211
vllm_patches/transformers_lizzy_tp.py
Normal file
@@ -0,0 +1,211 @@
|
|||||||
|
"""Compat patch for Lizzy TP under vLLM's generic Transformers backend."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import torch.nn.functional as F
|
||||||
|
from torch import nn
|
||||||
|
|
||||||
|
_PATCH_ATTR = "_flwr_transformers_lizzy_tp_patch_applied"
|
||||||
|
|
||||||
|
|
||||||
|
class _TensorParallelSliceNorm(nn.Module):
|
||||||
|
"""Apply a full-width checkpoint norm to a TP-local activation slice."""
|
||||||
|
|
||||||
|
def __init__(self, base_norm: nn.Module, start_idx: int, end_idx: int):
|
||||||
|
super().__init__()
|
||||||
|
self.start_idx = start_idx
|
||||||
|
self.end_idx = end_idx
|
||||||
|
self.weight = base_norm.weight
|
||||||
|
if getattr(base_norm, "bias", None) is not None:
|
||||||
|
self.bias = base_norm.bias
|
||||||
|
else:
|
||||||
|
self.register_parameter("bias", None)
|
||||||
|
self.eps = float(
|
||||||
|
getattr(base_norm, "eps", getattr(base_norm, "variance_epsilon", 1e-6)),
|
||||||
|
)
|
||||||
|
self.norm_kind = (
|
||||||
|
"layernorm" if isinstance(base_norm, nn.LayerNorm) else "rmsnorm"
|
||||||
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def local_size(self) -> int:
|
||||||
|
return self.end_idx - self.start_idx
|
||||||
|
|
||||||
|
def _slice_param(self, param: torch.Tensor | None) -> torch.Tensor | None:
|
||||||
|
if param is None:
|
||||||
|
return None
|
||||||
|
if param.shape[0] == self.local_size:
|
||||||
|
return param
|
||||||
|
return param[self.start_idx : self.end_idx]
|
||||||
|
|
||||||
|
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
|
||||||
|
weight = self.weight
|
||||||
|
bias = self.bias
|
||||||
|
if hidden_states.shape[-1] != self.weight.shape[0]:
|
||||||
|
if hidden_states.shape[-1] != self.local_size:
|
||||||
|
msg = (
|
||||||
|
"Unexpected hidden size for TP-sliced norm: "
|
||||||
|
f"{hidden_states.shape[-1]} "
|
||||||
|
f"(expected {self.weight.shape[0]} or {self.local_size})"
|
||||||
|
)
|
||||||
|
raise RuntimeError(msg)
|
||||||
|
weight = self._slice_param(weight)
|
||||||
|
bias = self._slice_param(bias)
|
||||||
|
|
||||||
|
if self.norm_kind == "layernorm":
|
||||||
|
return F.layer_norm(
|
||||||
|
hidden_states,
|
||||||
|
(hidden_states.shape[-1],),
|
||||||
|
weight,
|
||||||
|
bias,
|
||||||
|
self.eps,
|
||||||
|
)
|
||||||
|
|
||||||
|
input_dtype = hidden_states.dtype
|
||||||
|
hidden_states_fp32 = hidden_states.to(torch.float32)
|
||||||
|
variance = hidden_states_fp32.pow(2).mean(dim=-1, keepdim=True)
|
||||||
|
hidden_states_norm = hidden_states_fp32 * torch.rsqrt(variance + self.eps)
|
||||||
|
hidden_states_norm = hidden_states_norm.to(input_dtype)
|
||||||
|
output = weight * hidden_states_norm
|
||||||
|
if bias is not None:
|
||||||
|
output = output + bias
|
||||||
|
return output
|
||||||
|
|
||||||
|
|
||||||
|
def _maybe_patch_lizzy_attention_for_tp(
|
||||||
|
*,
|
||||||
|
module: nn.Module,
|
||||||
|
prefix: str,
|
||||||
|
tp_size: int,
|
||||||
|
tp_rank: int,
|
||||||
|
log_replacement: Any, # noqa: ANN401
|
||||||
|
) -> None:
|
||||||
|
if tp_size <= 1 or type(module).__name__ != "LizzyAttention":
|
||||||
|
return
|
||||||
|
|
||||||
|
num_heads = getattr(module, "num_heads", None)
|
||||||
|
num_key_value_heads = getattr(module, "num_key_value_heads", None)
|
||||||
|
head_dim = getattr(module, "head_dim", None)
|
||||||
|
q_norm = getattr(module, "q_norm", None)
|
||||||
|
k_norm = getattr(module, "k_norm", None)
|
||||||
|
if not all(
|
||||||
|
isinstance(value, int)
|
||||||
|
for value in (num_heads, num_key_value_heads, head_dim)
|
||||||
|
):
|
||||||
|
return
|
||||||
|
if num_heads % tp_size != 0 or num_key_value_heads % tp_size != 0:
|
||||||
|
return
|
||||||
|
|
||||||
|
local_num_heads = num_heads // tp_size
|
||||||
|
local_num_key_value_heads = num_key_value_heads // tp_size
|
||||||
|
local_q_dim = local_num_heads * head_dim
|
||||||
|
local_kv_dim = local_num_key_value_heads * head_dim
|
||||||
|
|
||||||
|
module.num_heads = local_num_heads
|
||||||
|
module.num_key_value_heads = local_num_key_value_heads
|
||||||
|
module.num_key_value_groups = local_num_heads // local_num_key_value_heads
|
||||||
|
|
||||||
|
if q_norm is not None and getattr(q_norm, "weight", None) is not None:
|
||||||
|
start = tp_rank * local_q_dim
|
||||||
|
end = start + local_q_dim
|
||||||
|
module.q_norm = _TensorParallelSliceNorm(q_norm, start, end)
|
||||||
|
log_replacement(f"{prefix}.q_norm", q_norm, module.q_norm)
|
||||||
|
|
||||||
|
if k_norm is not None and getattr(k_norm, "weight", None) is not None:
|
||||||
|
start = tp_rank * local_kv_dim
|
||||||
|
end = start + local_kv_dim
|
||||||
|
module.k_norm = _TensorParallelSliceNorm(k_norm, start, end)
|
||||||
|
log_replacement(f"{prefix}.k_norm", k_norm, module.k_norm)
|
||||||
|
|
||||||
|
|
||||||
|
def patch_vllm_transformers_lizzy_tp() -> None:
|
||||||
|
"""Patch the generic vLLM Transformers backend for Lizzy TP norms/heads."""
|
||||||
|
import vllm.model_executor.models.transformers as transformers_mod
|
||||||
|
|
||||||
|
transformers_base = transformers_mod.TransformersBase
|
||||||
|
if getattr(transformers_base, _PATCH_ATTR, False):
|
||||||
|
return
|
||||||
|
|
||||||
|
PreTrainedModel = transformers_mod.PreTrainedModel
|
||||||
|
maybe_prefix = transformers_mod.maybe_prefix
|
||||||
|
replace_linear_class = transformers_mod.replace_linear_class
|
||||||
|
get_feature_request_tip = transformers_mod.get_feature_request_tip
|
||||||
|
re = transformers_mod.re
|
||||||
|
log_replacement = transformers_mod.log_replacement
|
||||||
|
get_tp_rank = getattr(transformers_mod, "get_tensor_model_parallel_rank", None)
|
||||||
|
if get_tp_rank is None:
|
||||||
|
try:
|
||||||
|
from vllm.distributed import ( # noqa: PLC0415
|
||||||
|
get_tensor_model_parallel_rank as get_tp_rank,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
get_tp_rank = lambda: 0
|
||||||
|
|
||||||
|
def tensor_parallel(self: Any) -> None: # noqa: ANN401
|
||||||
|
"""Apply the model's tensor parallel plan plus Lizzy attention fixes."""
|
||||||
|
is_pretrained_model = lambda m: isinstance(m, PreTrainedModel)
|
||||||
|
supports_tp_plan = lambda m: m.config.base_model_tp_plan is not None
|
||||||
|
pretrained_models = filter(is_pretrained_model, self.model.modules())
|
||||||
|
models_with_tp_plan = filter(supports_tp_plan, pretrained_models)
|
||||||
|
|
||||||
|
if not any(models_with_tp_plan) and self.tp_size > 1:
|
||||||
|
tip = get_feature_request_tip(
|
||||||
|
self.model_config.model,
|
||||||
|
self.model_config.trust_remote_code,
|
||||||
|
)
|
||||||
|
raise ValueError(
|
||||||
|
f"{type(self.model)} does not support tensor parallel. {tip}",
|
||||||
|
)
|
||||||
|
|
||||||
|
tp_rank = get_tp_rank()
|
||||||
|
|
||||||
|
def _tensor_parallel(
|
||||||
|
module: nn.Module,
|
||||||
|
prefix: str = "",
|
||||||
|
tp_plan: dict[str, str] | None = None,
|
||||||
|
) -> None:
|
||||||
|
local_tp_plan = tp_plan or {}
|
||||||
|
|
||||||
|
if isinstance(module, PreTrainedModel):
|
||||||
|
local_tp_plan = module.config.base_model_tp_plan or {}
|
||||||
|
local_tp_plan = {
|
||||||
|
maybe_prefix(prefix, key): value
|
||||||
|
for key, value in local_tp_plan.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
for child_name, child_module in module.named_children():
|
||||||
|
qual_name = maybe_prefix(prefix, child_name)
|
||||||
|
if isinstance(child_module, nn.Linear):
|
||||||
|
generator = (p for p in local_tp_plan if re.match(p, qual_name))
|
||||||
|
pattern = next(generator, None)
|
||||||
|
style = local_tp_plan.get(pattern, "replicate")
|
||||||
|
new_module = replace_linear_class(
|
||||||
|
child_module,
|
||||||
|
style,
|
||||||
|
self.quant_config,
|
||||||
|
prefix=qual_name,
|
||||||
|
)
|
||||||
|
setattr(module, child_name, new_module)
|
||||||
|
log_replacement(qual_name, child_module, new_module)
|
||||||
|
else:
|
||||||
|
_tensor_parallel(
|
||||||
|
child_module,
|
||||||
|
prefix=qual_name,
|
||||||
|
tp_plan=local_tp_plan,
|
||||||
|
)
|
||||||
|
|
||||||
|
_maybe_patch_lizzy_attention_for_tp(
|
||||||
|
module=module,
|
||||||
|
prefix=prefix,
|
||||||
|
tp_size=self.tp_size,
|
||||||
|
tp_rank=tp_rank,
|
||||||
|
log_replacement=log_replacement,
|
||||||
|
)
|
||||||
|
|
||||||
|
_tensor_parallel(self.model)
|
||||||
|
|
||||||
|
transformers_base.tensor_parallel = tensor_parallel
|
||||||
|
setattr(transformers_base, _PATCH_ATTR, True)
|
||||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user