Applied-Innovation-Center/Karnak-40B-v1.0
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: Applied-Innovation-Center/Karnak-40B-v1.0
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-27 01:44:51 +08:00
commit 1a770c7b5e
28 changed files with 218660 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

11
NOTICE Normal file
View File

@@ -0,0 +1,11 @@
Karnak
Copyright 2026 Applied Innovation Center
This product includes software and model weights developed by the Applied Innovation Center for the Karnak Arabic Large Language Model.
ATTRIBUTION NOTICE:
This model is distributed under the Apache License, Version 2.0. As per Section 4(d) of the License, any distribution of this work or derivative works must include a readable copy of this attribution notice.
When utilizing, modifying, or building upon this model, please acknowledge the source by including the following statement in your documentation or repository:
"Built on Karnak-Applied Innovation Center"

171
README.md Normal file
View File

@@ -0,0 +1,171 @@
---
license: apache-2.0
language:
- ar
- en
pipeline_tag: text-generation
tags:
- text-generation
- pytorch
- transformers
- vllm
- causal-lm
- depth-extension
- arabic
- english
- karnak
- qwen
base_model: Qwen/Qwen3-30B-A3B-Instruct-2507
model_name: Karnak
parameters: 40B
inference: false
---
# Karnak: Enhanced Arabic–English Large Language Model
Karnak is a powerful AI model that works in both Arabic and English, with extra improvements that make it especially strong in Arabic and more natural in the way it writes and responds. It was built by taking an existing model and improving it through more training, so it can understand instructions better, handle longer text, and give more reliable answers. This makes it useful for everyday tasks like answering questions, explaining topics, writing content, or helping with work and research. It can also process long pieces of text, which is helpful for documents and extended conversations. A big advantage is that it is not locked to an online service only, since you can download it, run it locally on your own machine or servers, and even fine-tune it for your own specific use case.
## Model Summary
**Karnak** is a depth-extended causal language model optimized for **Arabic and English** generation. It is built on top of **Qwen/Qwen3-30B-A3B-Instruct-2507**, featuring architectural depth extension and a tokenizer specifically optimized for Arabic to improve fluency and efficiency.
Karnak was trained using **high-quality, filtered data** through a rigorous pipeline to enhance overall instruction-following capabilities, factuality, and robustness.
## Key Features
- **Depth Extension (~40B):** Expanded depth to increase reasoning capacity and improve long-range dependency modeling.
- **Arabic-Optimized Tokenizer:** Improved Arabic tokenization efficiency, resulting in reduced token fragmentation and higher-quality generation.
- **Multi-Stage Training:** The model evolved through: Pre-trained weights → Depth Extension → Continued Pre-training → SFT (Supervised Fine-Tuning).
- **Extended Context Window:** Designed for long-context usage with a **safe context range up to 20K tokens** (recommended to stay within this limit for optimal stability).
## Model Details
- **Model Name:** Karnak
- **Base Model:** Qwen/Qwen3-30B-A3B-Instruct-2507
- **Parameter Count:** ~40B (Depth-Extended)
- **Languages:** Arabic, English
- **Training:** High-quality filtered data + Multi-stage pipeline (Continued pre-training + SFT)
- **Safe Context Range:** Up to **20,000 tokens**
---
## Usage
### 1) Hugging Face Transformers
To use Karnak with the standard Transformers library, ensure you have the latest version installed.
```bash
pip install -U "transformers>=4.40.0" torch accelerate
```
Python Code Example (Chat Template):
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Applied-Innovation-Center/Karnak"
# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
torch_dtype=torch.bfloat16,
trust_remote_code=True,
)
# Prepare Input
prompt = "اشرح لي نظرية النسبية بشكل مبسط."
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": prompt},
]
# Apply chat template
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
# Generate
generated_ids = model.generate(
**model_inputs,
max_new_tokens=512,
temperature=0.7,
top_p=0.9,
)
# Decode output (removing the prompt tokens)
generated_ids = generated_ids[:, model_inputs.input_ids.shape[1]:]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
```
2) vLLM (Recommended for Production)
Karnak is compatible with vLLM for high-throughput inference.
Installation:
```bash
pip install -U vllm
```
Offline Inference:
```python
from vllm import LLM, SamplingParams
model_id = "Applied-Innovation-Center/Karnak"
# Initialize the model
llm = LLM(
model=model_id,
trust_remote_code=True,
max_model_len=20000, # Safe context range
tensor_parallel_size=1, # Adjust based on available GPUs
)
# Set sampling parameters
sampling_params = SamplingParams(
temperature=0.7,
top_p=0.9,
max_tokens=512,
)
# Generate
prompts = ["ما هي عاصمة مصر؟"]
outputs = llm.generate(prompts, sampling_params)
for o in outputs:
print(f"Prompt: {o.prompt}")
print(f"Generated: {o.outputs[0].text}")
```
Server Mode (OpenAI-Compatible API):
You can serve the model as an API compatible with OpenAI clients:
```bash
vllm serve "Applied-Innovation-Center/Karnak" \
--trust-remote-code \
--dtype bfloat16 \
--port 8000
```
Citation
If you use this model in your research or application, please cite it as follows:
```bibtex
@misc{karnak-40b,
title={Karnak: A Depth-Extended Arabic-English LLM},
year={2026},
publisher={Applied Innovation Center},
howpublished={\url{[https://huggingface.co/Applied-Innovation-Center/Karnak](https://huggingface.co/Applied-Innovation-Center/Karnak)}}
}
```

54
chat_template.jinja Normal file
View File

@@ -0,0 +1,54 @@
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- else %}
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
{%- endif %}
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0]['role'] == 'system' %}
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- for message in messages %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{{- '<|im_start|>' + message.role }}
{%- if message.content %}
{{- '\n' + message.content }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{{- tool_call.arguments | tojson }}
{{- '}\n</tool_call>' }}
{%- endfor %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- message.content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}

39
config.json Normal file
View File

@@ -0,0 +1,39 @@
{
"architectures": [
"Qwen3MoeForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"decoder_sparse_step": 1,
"dtype": "bfloat16",
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 6144,
"max_position_embeddings": 262144,
"max_window_layers": 48,
"mlp_only_layers": [],
"model_type": "qwen3_moe",
"moe_intermediate_size": 768,
"norm_topk_prob": true,
"num_attention_heads": 32,
"num_experts": 128,
"num_experts_per_tok": 8,
"num_hidden_layers": 64,
"num_key_value_heads": 4,
"output_router_logits": false,
"pad_token_id": 151643,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 10000000,
"router_aux_loss_coef": 0.001,
"sliding_window": null,
"tie_word_embeddings": false,
"transformers_version": "4.57.1",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 192728
}

192894
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model-00001-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:93102469f8846c5408442aea00338ffeaccb8c599f646099ab0ee46831a2499a
size 4997543152

3
model-00002-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4156a724102e7811acb300e5e4b7ccd554fa91551533e4d56fc33dbf6b59ad82
size 4997741584

3
model-00003-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:69dd1813d927693587160a60bf303085d3214012231cc5b838b0184bba381dba
size 4997742120

3
model-00004-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:09ad08ffd8ec3b5202fd30e124890551b5a1c4a4d7224424538cda7f6b6abe72
size 4997743160

3
model-00005-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d06384d7f801d58a98fe171e1c4c17f563b4678228215c79ef7ba3f3cfb3f3f7
size 4997743160

3
model-00006-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5be6dc0ec5a3f2748598567f4807d28e5b8b242cd3c7eb65ea83ade311f8a6ec
size 4997743160

3
model-00007-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ea6bc5d15c39208908a451beabf47d88e882c32a1be1e02e27cf60ea54d48cde
size 4997743160

3
model-00008-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cdf00bc980a0e82bb1d77e0c8cf79a0bf59b8889435a7e62bfaddd5831d7eb2e
size 4997743168

3
model-00009-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:441d6efda9997c871d7753f587f9889b7dc1f5e6aea0e158e5ff45b28c2812c7
size 4997743168

3
model-00010-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6bd562f79d874803c55be19e61b4c3e40899b72c09044cef06e8514c487e83bc
size 4997743168

3
model-00011-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f13ed3ef135871644d7cbe2ac35ee5d6e508577685e38ed8dd1e3c743ff5b20b
size 4997743168

3
model-00012-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8409ed8b092a31dc6603641717260ddb19d516e7dbe13d76a56721f67e1a4632
size 4997743168

3
model-00013-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:30485630d6f3620bb9599bd8c200c2c96e59944e7771622632347204e94c434d
size 4997743168

3
model-00014-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:590e39498553fcfb3a80cd389f65e518fa5dc67078399f24b5f77988adb6b707
size 4997743168

3
model-00015-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:da6d51a27e6299d6f495b61a95eb715ceddc65412128971806e8fd00df985003
size 4997743168

3
model-00016-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:79d9daf67f63c3c1844c2110874a311f41ee0db607d4d6decdb87d6b75824d39
size 4997743160

3
model-00017-of-00017.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:96c94a6b7e921d22c387b7ee9c635c7ebfda88a5017cde4ff6dac0cc1bf00eda
size 1377696728

25162
model.safetensors.index.json Normal file
View File

File diff suppressed because it is too large Load Diff

31
special_tokens_map.json Normal file
View File

@@ -0,0 +1,31 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"eos_token": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f644d71501367aaa569353e4787cd25cadb851169b709d8712bff51c72ce76f8
size 15658511

207
tokenizer_config.json Normal file
View File

@@ -0,0 +1,207 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<|object_ref_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|object_ref_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151648": {
"content": "<|box_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151649": {
"content": "<|box_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|image_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|video_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"extra_special_tokens": {},
"model_max_length": 131072,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long