初始化项目，由ModelHub XC社区提供模型
Model: NbAiLab/borealis-open-1b Source: Original Platform
This commit is contained in:
ModelHub XC
40
.gitattributes
vendored
Normal file
40
.gitattributes
vendored
Normal file
@@ -0,0 +1,40 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
LICENSE_FAQ.pdf filter=lfs diff=lfs merge=lfs -text
|
||||
borealis.png filter=lfs diff=lfs merge=lfs -text
|
||||
borealis_evals_202605.png filter=lfs diff=lfs merge=lfs -text
|
||||
Model_Documentation_Form.pdf filter=lfs diff=lfs merge=lfs -text
|
||||
3
LICENSE_FAQ.pdf
Normal file
3
LICENSE_FAQ.pdf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b4a87118b0b0c1e2d5a06b27bb4b56517d4d62b2b74860f77b8ac2d601224840
|
||||
size 140945
|
||||
3
Model_Documentation_Form.pdf
Normal file
3
Model_Documentation_Form.pdf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:532706411c9e4e7c1150e95b4b6591138091038f178f62f9ac94c2d548a1ddae
|
||||
size 332991
|
||||
246
README.md
Normal file
246
README.md
Normal file
@@ -0,0 +1,246 @@
|
||||
---
|
||||
license: gemma
|
||||
datasets:
|
||||
- NbAiLab/aurora-sft-open
|
||||
language:
|
||||
- 'no'
|
||||
- nb
|
||||
- nn
|
||||
base_model:
|
||||
- google/gemma-3-1b-it
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
tags:
|
||||
- conversational
|
||||
- instruct
|
||||
- borealis
|
||||
- gemma3_text
|
||||
- norwegian
|
||||
- norwegian-bokmal
|
||||
- norwegian-nynorsk
|
||||
- open
|
||||
---
|
||||
|
||||
|
||||
|
||||
# Borealis Open 1B
|
||||
|
||||
## Model Summary
|
||||
|
||||
**NbAiLab/borealis-open-1b** is a **1B-parameter** instruction-tuned **open release** model in the Borealis family from the National Library of Norway (Nasjonalbiblioteket, NB).
|
||||
|
||||
This is the first Borealis release to incorporate data made available under the agreement between rights-holder organizations in Norway and the Norwegian government. To date, only a limited subset of the material has been used, specifically to teach the model how to generate news article titles and ingress texts. Models with the suffix `-open`, do not include any material from the agreement.
|
||||
|
||||
All Borealis released models are based on the **Gemma 3** family. This model is based on [google/gemma-3-1b-it](https://huggingface.co/google/gemma-3-1b-it), and fine-tuned for Norwegian-centric instruction following.
|
||||
|
||||
### Sizes
|
||||
|
||||
| Size | Full release | Open release |
|
||||
|---:|---|---|
|
||||
| 27B | [NbAiLab/borealis-27b](https://huggingface.co/NbAiLab/borealis-27b) | [NbAiLab/borealis-open-27b](https://huggingface.co/NbAiLab/borealis-open-27b) |
|
||||
| 12B | [NbAiLab/borealis-12b](https://huggingface.co/NbAiLab/borealis-12b) | [NbAiLab/borealis-open-12b](https://huggingface.co/NbAiLab/borealis-open-12b) |
|
||||
| 4B | [NbAiLab/borealis-4b](https://huggingface.co/NbAiLab/borealis-4b) | [NbAiLab/borealis-open-4b](https://huggingface.co/NbAiLab/borealis-open-4b) |
|
||||
| 1B | [NbAiLab/borealis-1b](https://huggingface.co/NbAiLab/borealis-1b) | [NbAiLab/borealis-open-1b](https://huggingface.co/NbAiLab/borealis-open-1b) |
|
||||
| 270M | [NbAiLab/borealis-270m](https://huggingface.co/NbAiLab/borealis-270m) | [NbAiLab/borealis-open-270m](https://huggingface.co/NbAiLab/borealis-open-270m) |
|
||||
|
||||
## Training Data
|
||||
|
||||
Supervised fine-tuning (SFT) uses instruction data prepared by the National Library of Norway for Norwegian-centric assistant behavior, writing, summarization, question answering, and related tasks.
|
||||
|
||||
The SFT dataset for this model is [NbAiLab/aurora-sft-open](https://huggingface.co/datasets/NbAiLab/aurora-sft-open). [NbAiLab/aurora-sft-open](https://huggingface.co/datasets/NbAiLab/aurora-sft-open) is the open version of the SFT dataset. The only difference between [NbAiLab/aurora-sft-open](https://huggingface.co/datasets/NbAiLab/aurora-sft-open) and [NbAiLab/aurora-sft](https://huggingface.co/datasets/NbAiLab/aurora-sft) is the addition of 10k tasks derived from copyright-protected newspapers material.
|
||||
|
||||
## Evaluation
|
||||
|
||||
<figure>
|
||||
<img src="./borealis_evals_202605.png" alt="Borealis evaluation results">
|
||||
<figcaption>Borealis evaluation results on selected tasks (best score among {0-5}-shot).</figcaption>
|
||||
</figure>
|
||||
|
||||
We evaluate Borealis with NorEval, MMLU-English, and nb-gpt-bench, our own evaluation suite, which will be published and described in an upcoming paper. The full Borealis models include around 10k newspaper-derived tasks from the abovementioned agreement and show a slight performance increase in some key metrics compared with the open variants. We hope to further increase the difference by incorporating proper pre-training on the newspaper material.
|
||||
|
||||
## Safety and Alignment
|
||||
|
||||
The Borealis family of models are aligned for safety using prompt baking and weighted merging of SFT and aligned models. The goal of this process is to balance model quality, usefulness, and safer behavior.
|
||||
|
||||
As with all generative models, outputs can still be incorrect, biased, harmful, or inappropriate. Do not use the model for safety-critical or high-stakes applications without additional evaluation and safeguards.
|
||||
|
||||
### Prompt Baking
|
||||
|
||||
To align the Borealis models, we employ *prompt baking*, a procedure that distills the behavior induced by a system prompt directly into the model weights using [`bakery`](https://github.com/marksverdhei/bakery). Specifically, we train a LoRA adapter to minimize the KL-divergence between two model distributions: Borealis conditioned on the system prompt, and the same base model augmented with the LoRA adapter but evaluated without the system prompt in context. This objective encourages the adapter to reproduce the behavioral effects of the prompt without requiring the prompt to be present at inference time.
|
||||
|
||||
To reduce degradation on downstream tasks and preserve general model utility, we merge the resulting prompt adapter into the base model using a scaling factor of `0.25`, which we found to provide the best empirical trade-off.
|
||||
|
||||
## Intended Use
|
||||
|
||||
- Norwegian-centric assistant-style tasks, including drafting, summarization, Q&A, and light reasoning (this is not a reasoning model).
|
||||
- Assessment and improvement of Norwegian writing style and quality.
|
||||
- Evaluation of behavior and language coverage for Norwegian, Bokmål, and Nynorsk.
|
||||
|
||||
## Usage
|
||||
|
||||
This repository contains the Transformers/safetensors version of the model. The
|
||||
examples below use `NbAiLab/borealis-open-1b` as the model id; replace it with
|
||||
another Borealis repo id if needed.
|
||||
|
||||
### Transformers
|
||||
|
||||
Use the regular causal language-model interface:
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "NbAiLab/borealis-open-1b"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
device_map="auto",
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Skriv et kort sammendrag av hva Nasjonalbiblioteket gjør.",
|
||||
}
|
||||
]
|
||||
|
||||
inputs = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
add_generation_prompt=True,
|
||||
return_tensors="pt",
|
||||
return_dict=True,
|
||||
).to(model.device)
|
||||
|
||||
outputs = model.generate(**inputs, max_new_tokens=256)
|
||||
new_tokens = outputs[0, inputs["input_ids"].shape[-1]:]
|
||||
print(tokenizer.decode(new_tokens, skip_special_tokens=True))
|
||||
```
|
||||
|
||||
### vLLM
|
||||
|
||||
Serve the model with vLLM's OpenAI-compatible server:
|
||||
|
||||
```bash
|
||||
vllm serve NbAiLab/borealis-open-1b --served-model-name borealis-open-1b
|
||||
```
|
||||
|
||||
Then call the local chat completions endpoint:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "borealis-open-1b",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Skriv tre korte punkter om norsk språkteknologi."
|
||||
}
|
||||
],
|
||||
"max_tokens": 256
|
||||
}'
|
||||
```
|
||||
|
||||
Large models may require tensor parallelism or reduced memory settings depending
|
||||
on the available GPU memory.
|
||||
|
||||
### llama.cpp
|
||||
|
||||
llama.cpp uses GGUF files, not the safetensors files in this repository. When
|
||||
the planned GGUF companion repository is available, you can run it directly from
|
||||
the Hub:
|
||||
|
||||
```bash
|
||||
llama-server -hf NbAiLab/borealis-open-1b-gguf --port 8080
|
||||
```
|
||||
|
||||
Or download a GGUF file and run it locally:
|
||||
|
||||
```bash
|
||||
llama-cli -m borealis-open-1b.gguf \
|
||||
-p "Skriv et kort sammendrag av hva Nasjonalbiblioteket gjør." \
|
||||
-n 256
|
||||
```
|
||||
|
||||
### Ollama
|
||||
|
||||
Ollama also requires a GGUF model. Once the GGUF companion repository is
|
||||
available, you can run it from Hugging Face:
|
||||
|
||||
```bash
|
||||
ollama run hf.co/NbAiLab/borealis-open-1b-gguf
|
||||
```
|
||||
|
||||
For a local GGUF file, create a minimal `Modelfile`:
|
||||
|
||||
```text
|
||||
FROM ./borealis-open-1b.gguf
|
||||
```
|
||||
|
||||
Then create and run the local Ollama model:
|
||||
|
||||
```bash
|
||||
ollama create borealis-open-1b -f Modelfile
|
||||
ollama run borealis-open-1b "Skriv tre korte punkter om norsk språkteknologi."
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- The model may hallucinate or produce incorrect information.
|
||||
- Safety alignment reduces but does not eliminate harmful or inappropriate outputs.
|
||||
- Performance outside Norwegian and English use cases has not been fully characterized.
|
||||
|
||||
## EU AI Act
|
||||
|
||||
The model is a fine-tune of Gemma 3. Using Gemma 3 27B as a conservative upper-bound reference, the original Gemma 3 27B training compute is estimated at approximately 2.1-2.3 x 10^24 FLOPs, based on the disclosed 14T training-token budget and the 27B parameter scale. The fine-tuning run used approximately 3.4 x 10^20 FLOPs, or about 0.015% of the estimated original training compute. This is substantially below the European Commission's indicative one-third threshold for treating a downstream modification as a significant modification that would make the modifier the provider of the modified General Purpose AI (GPAI) model.
|
||||
|
||||
On that basis, the fine-tuning activity is preliminarily assessed as not constituting a substantial modification for the purpose of becoming the provider of a new modified GPAI model under the compute-based criterion. However, the resulting model remains derived from a generative general-purpose AI model and may still be subject to downstream AI-system obligations under the EU AI Act.
|
||||
|
||||
For additional model-level documentation, see the [Model Documentation Form](./Model_Documentation_Form.pdf).
|
||||
|
||||
## License
|
||||
|
||||
The license of this model is an adaptation of the Apache 2.0 license with additional use-based restrictions. In particular, users of the model are required to refrain from intentionally using the model to recreate data the model has been trained on. The license also requires users not to use the model or its output to provide end-user services whose primary purpose is to give access to licensed press publications in the training data.
|
||||
|
||||
For more information, see the [LICENSE](./LICENSE) and the [License FAQ](./LICENSE_FAQ.pdf).
|
||||
|
||||
## Authenticity
|
||||
|
||||
This model release is signed by the National Library of Norway. The signed manifest in `signing/SHA256SUMS` covers the model-runtime artifacts, including model weights, configuration, tokenizer files, and chat template.
|
||||
|
||||
To verify model authenticity and file integrity after downloading the repository, run:
|
||||
|
||||
```bash
|
||||
bash signing/verify.sh
|
||||
```
|
||||
|
||||
For more verification instructions, see [ai.nb.no/verify](https://ai.nb.no/verify).
|
||||
|
||||
## Weights
|
||||
|
||||
This repository contains the Transformers (safetensors) release of **NbAiLab/borealis-open-1b**.
|
||||
|
||||
Companion formats:
|
||||
- GGUF: [NbAiLab/borealis-open-1b-gguf](https://huggingface.co/NbAiLab/borealis-open-1b-gguf)
|
||||
- MLX: [NbAiLab/borealis-open-1b-mlx](https://huggingface.co/NbAiLab/borealis-open-1b-mlx)
|
||||
- MLX 8-bit: [NbAiLab/borealis-open-1b-mlx-8bits](https://huggingface.co/NbAiLab/borealis-open-1b-mlx-8bits)
|
||||
|
||||
## Citation and Contributors
|
||||
|
||||
The Borealis family of models is a joint effort of multiple teams at the National Library of Norway. Led by Javier de la Rosa ([@versae](https://huggingface.co/versae)), key contributors include (in alphabetical order) Rolv-Arild Braaten, Magnus Breder Birkenes, Lucas Charpentier, Pawel Cyrta, Tita Enstad, Markus Sverdvik Heiervang, Arne Martinus Lindstad, Marthe Løken Midtgaard, Marie Roald, Marie Røsok, Thea Tollersrud, and Angelina Zanardi. Olaus Ingskog Bergstrøm contributed with legal advice. And Yngvil Beyer, Svein Arne Brygfjeld, and Wilfred Østgulen helped with strategic oversight.
|
||||
|
||||
A tecnical report will be released soon.
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
Thanks to the Gemma team at Google for releasing Gemma 3, and to everyone contributing to the Norwegian language technology ecosystem.
|
||||
|
||||
## Disclaimer
|
||||
|
||||
The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions. When third parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence. In no event shall the owner of the models (The National Library of Norway) be liable for any results arising from the use made by third parties of these models.
|
||||
|
||||
## Contact
|
||||
|
||||
For feedback, technical concerns, or collaboration inquiries, please contact <a rel="noopener nofollow" href="mailto:ailab@nb.no">ailab@nb.no</a>.
|
||||
3
borealis.png
Normal file
3
borealis.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:5e6d03db57f4f12b359910620c4ba80b70f4fe0711d8125600434663137eb528
|
||||
size 778162
|
||||
3
borealis_evals_202605.png
Normal file
3
borealis_evals_202605.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:5f2b6e74b343ba10c6ee5e8a7913a70d8b8e5335a4138ef583ee2baeb9e07655
|
||||
size 130041
|
||||
47
chat_template.jinja
Normal file
47
chat_template.jinja
Normal file
@@ -0,0 +1,47 @@
|
||||
{{ bos_token }}
|
||||
{%- if messages[0]['role'] == 'system' -%}
|
||||
{%- if messages[0]['content'] is string -%}
|
||||
{%- set first_user_prefix = messages[0]['content'] + '
|
||||
|
||||
' -%}
|
||||
{%- else -%}
|
||||
{%- set first_user_prefix = messages[0]['content'][0]['text'] + '
|
||||
|
||||
' -%}
|
||||
{%- endif -%}
|
||||
{%- set loop_messages = messages[1:] -%}
|
||||
{%- else -%}
|
||||
{%- set first_user_prefix = "" -%}
|
||||
{%- set loop_messages = messages -%}
|
||||
{%- endif -%}
|
||||
{%- for message in loop_messages -%}
|
||||
{%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
|
||||
{{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
|
||||
{%- endif -%}
|
||||
{%- if (message['role'] == 'assistant') -%}
|
||||
{%- set role = "model" -%}
|
||||
{%- else -%}
|
||||
{%- set role = message['role'] -%}
|
||||
{%- endif -%}
|
||||
{{ '<start_of_turn>' + role + '
|
||||
' + (first_user_prefix if loop.first else "") }}
|
||||
{%- if message['content'] is string -%}
|
||||
{{ message['content'] | trim }}
|
||||
{%- elif message['content'] is iterable -%}
|
||||
{%- for item in message['content'] -%}
|
||||
{%- if item['type'] == 'image' -%}
|
||||
{{ '<start_of_image>' }}
|
||||
{%- elif item['type'] == 'text' -%}
|
||||
{{ item['text'] | trim }}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{%- else -%}
|
||||
{{ raise_exception("Invalid content type") }}
|
||||
{%- endif -%}
|
||||
{{ '<end_of_turn>
|
||||
' }}
|
||||
{%- endfor -%}
|
||||
{%- if add_generation_prompt -%}
|
||||
{{'<start_of_turn>model
|
||||
'}}
|
||||
{%- endif -%}
|
||||
72
config.json
Normal file
72
config.json
Normal file
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"_sliding_window_pattern": 6,
|
||||
"architectures": [
|
||||
"Gemma3ForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"attn_logit_softcapping": null,
|
||||
"bos_token_id": 2,
|
||||
"cache_implementation": "hybrid",
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 106,
|
||||
"final_logit_softcapping": null,
|
||||
"head_dim": 256,
|
||||
"hidden_activation": "gelu_pytorch_tanh",
|
||||
"hidden_size": 1152,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 6912,
|
||||
"layer_types": [
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention"
|
||||
],
|
||||
"max_position_embeddings": 32768,
|
||||
"model_type": "gemma3_text",
|
||||
"num_attention_heads": 4,
|
||||
"num_hidden_layers": 26,
|
||||
"num_key_value_heads": 1,
|
||||
"pad_token_id": 0,
|
||||
"query_pre_attn_scalar": 256,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_parameters": {
|
||||
"full_attention": {
|
||||
"rope_theta": 1000000,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"sliding_attention": {
|
||||
"rope_theta": 10000,
|
||||
"rope_type": "default"
|
||||
}
|
||||
},
|
||||
"sliding_window": 512,
|
||||
"sliding_window_pattern": 6,
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "5.8.0",
|
||||
"use_bidirectional_attention": false,
|
||||
"use_cache": false,
|
||||
"vocab_size": 262144
|
||||
}
|
||||
14
generation_config.json
Normal file
14
generation_config.json
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"bos_token_id": 2,
|
||||
"cache_implementation": "hybrid",
|
||||
"do_sample": true,
|
||||
"eos_token_id": [
|
||||
106,
|
||||
1,
|
||||
106
|
||||
],
|
||||
"pad_token_id": 0,
|
||||
"top_k": 64,
|
||||
"top_p": 0.95,
|
||||
"transformers_version": "5.8.0"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:cc7932c393bec8a57a5b0f1c93121bae2615dd34e7040ee89cd8d7f698881d39
|
||||
size 1999811208
|
||||
8
signing/SHA256SUMS
Normal file
8
signing/SHA256SUMS
Normal file
@@ -0,0 +1,8 @@
|
||||
7de1c58e208eda46e9c7f86397df37ec49883aeece39fb961e0a6b24088dd3c4 chat_template.jinja
|
||||
f3ec312ba0a893f7adf275d4203edaa4348d74bd6047a43b4812b1ff787b19c6 config.json
|
||||
d658664f05444ad13f0b7f1298cea34e5b745583f0c720d74531c49549fa1fa1 generation_config.json
|
||||
cc7932c393bec8a57a5b0f1c93121bae2615dd34e7040ee89cd8d7f698881d39 model.safetensors
|
||||
2f7b0adf4fb469770bb1490e3e35df87b1dc578246c5e7e6fc76ecf33213a397 special_tokens_map.json
|
||||
daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726 tokenizer.json
|
||||
1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c tokenizer.model
|
||||
d9aedd7a33a5aff2d3b8426eb18aed48e701213b0945564537ef6cd8844b9150 tokenizer_config.json
|
||||
BIN
signing/SHA256SUMS.sig
Normal file
BIN
signing/SHA256SUMS.sig
Normal file
Binary file not shown.
72
signing/ca-chain.pem
Normal file
72
signing/ca-chain.pem
Normal file
@@ -0,0 +1,72 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIG1jCCBL6gAwIBAgIQSIcdqlEaPQ81vpaZKLkkvzANBgkqhkiG9w0BAQsFADB1
|
||||
MQswCQYDVQQGEwJHUjE3MDUGA1UECgwuSGVsbGVuaWMgQWNhZGVtaWMgYW5kIFJl
|
||||
c2VhcmNoIEluc3RpdHV0aW9ucyBDQTEtMCsGA1UEAwwkSEFSSUNBIENvZGUgU2ln
|
||||
bmluZyBSU0EgUm9vdCBDQSAyMDIxMB4XDTIxMDMxOTA5MjEzNFoXDTM2MDMxNTA5
|
||||
MjEzM1owaDELMAkGA1UEBhMCR1IxNzA1BgNVBAoMLkhlbGxlbmljIEFjYWRlbWlj
|
||||
IGFuZCBSZXNlYXJjaCBJbnN0aXR1dGlvbnMgQ0ExIDAeBgNVBAMMF0hBUklDQSBD
|
||||
b2RlIFNpZ25pbmcgUlNBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA
|
||||
31Aluoq9mggXHg7TLFGOUTVDufEJjAKMhSYM4vh98DxFUPCE0soUcvj4AcbfOq0E
|
||||
rXG93MW2o/8byyISWPDTu7CKSYzF2x67GEcjAlOvdQn5UZOvI2fw86727Q0GHZ77
|
||||
+EfBRkLcLHPEGeVs4mkchtLydKjv42/Lsqu9/5AcYWro03lYf1q++LOmaJjSscvY
|
||||
LdAPPnzm+Aa9FIZc3K3JeZ1dxNhT6fAheWeAO4JxnBT2k7Qa1oJqJ1u6t3RYiadj
|
||||
shRVAwDq6/WVCW0hysPB9GuQEFkRKgkpQjcvZqG2yCMIVy9cCRQjIQWG9LSnLezv
|
||||
wAan4sUIzzgoABiodCU7Gc6uJsshdjHUBiMxQzsbd5rKPZYPk+uiiXaN6V5LP079
|
||||
9treRIh6h3h111HITXE2rQDEBsdKrandn4oUTUz3VZMO6XLQA6xpPAinx7ISDdkO
|
||||
3VOkEYebMxDF/7lo1GSgrgPvFPRVNWhPxHdVrRkB8/ZKg90HyP0g339NkmDwhVqo
|
||||
ZK4EiL5J3033w+U+HNowegNKxn8b1g/vB6ZHlVBGCugsO+77M6Z8/ouk755Jr8TI
|
||||
gcyn/CKKadsb748oy/AvqmviFOM+ilORjZVWOFXy82QdjBDapfMOM6JqtSFUTvf5
|
||||
OmVXDdIHMvosCWurRYXbVKgTNgOXVgJRAaquQBahtc0CAwEAAaOCAW0wggFpMBIG
|
||||
A1UdEwEB/wQIMAYBAf8CAQAwHwYDVR0jBBgwFoAUtGQWSOj8WkszKYnrmUC5ILT2
|
||||
YRowXAYIKwYBBQUHAQEEUDBOMEwGCCsGAQUFBzAChkBodHRwOi8vcmVwby5oYXJp
|
||||
Y2EuZ3IvY2VydHMvSEFSSUNBLUNvZGVTaWduaW5nLVJvb3QtMjAyMS1SU0EuY2Vy
|
||||
MEQGA1UdIAQ9MDswOQYEVR0gADAxMC8GCCsGAQUFBwIBFiNodHRwOi8vcmVwby5o
|
||||
YXJpY2EuZ3IvZG9jdW1lbnRzL0NQUzATBgNVHSUEDDAKBggrBgEFBQcDAzBKBgNV
|
||||
HR8EQzBBMD+gPaA7hjlodHRwOi8vY3JsLmhhcmljYS5nci9IQVJJQ0EtQ29kZVNp
|
||||
Z25pbmctUm9vdC0yMDIxLVJTQS5jcmwwHQYDVR0OBBYEFKnsNRTrbmcFDsRtCQyg
|
||||
cgYhtMouMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsFAAOCAgEAhcU6MtmA
|
||||
N9i3luIBF/JqB+CyCqlc9TFuu0d/Gi1PU86Mf6sqIqAObiDcM4J/uL2QWogUk5GZ
|
||||
b1bNCRBrmhPv9C8rFsbeNKJYdDEWtslXQrnnd9vpAKkurG9imNW4RE/jsPV28T2I
|
||||
iHWEYA0zGiP0qyAyEVvwSntEpFigaslhLFtVqf0uCGIvHRMfdD53WTXq37sInwcG
|
||||
/W2C0zcno3PUK8qtCvF7cO1jujGxsLG/h0blm+M6b09doO1iSq3SAO4kwue+7AvB
|
||||
97ppo4XKgcp1Kq6LGl4Rzac23KeMRkucwofNILRbIBwxdaeudozb6XIvgGeNPmFq
|
||||
71HdLzBS0i9Tyxsf4VjlJj1vazncSFxz56RDNyendlYR39bZrQ6FMr6W5/vt1cwe
|
||||
e7HlVnnQvdNMiFX/+uWH8kz29rP5q1NEQx8xjV7WnDqADnbTzGoUg9P3GR5Mv4m9
|
||||
tSktUu6WqserABWf8e0L3ClvUKcnCeG/lyXbmQWxRIGrWAMmUSP1/BVajITUrEDW
|
||||
qrrBmUlCoSpa0OZ7lfmlQ/QtaWvwrGRNgbLx7RgqlxsjeJtZUoBwC3nFNO6j80QB
|
||||
CQEdsxUWKYZmXTnSuiG63O/wON+TmWrM7SbxqbiZRIGw90aHT2bK+8CfMgSdxl1k
|
||||
JT844f3ByJb2qHA+z2lw0JqwVp07ORkrwwE=
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIFtjCCA56gAwIBAgIQFcKuKk2ZmmOM07oTGXYI9TANBgkqhkiG9w0BAQsFADB1
|
||||
MQswCQYDVQQGEwJHUjE3MDUGA1UECgwuSGVsbGVuaWMgQWNhZGVtaWMgYW5kIFJl
|
||||
c2VhcmNoIEluc3RpdHV0aW9ucyBDQTEtMCsGA1UEAwwkSEFSSUNBIENvZGUgU2ln
|
||||
bmluZyBSU0EgUm9vdCBDQSAyMDIxMB4XDTIxMDIxOTEwNTk1NFoXDTQ1MDIxMzEw
|
||||
NTk1M1owdTELMAkGA1UEBhMCR1IxNzA1BgNVBAoMLkhlbGxlbmljIEFjYWRlbWlj
|
||||
IGFuZCBSZXNlYXJjaCBJbnN0aXR1dGlvbnMgQ0ExLTArBgNVBAMMJEhBUklDQSBD
|
||||
b2RlIFNpZ25pbmcgUlNBIFJvb3QgQ0EgMjAyMTCCAiIwDQYJKoZIhvcNAQEBBQAD
|
||||
ggIPADCCAgoCggIBAIpq7qoKI1UMiwZC45VbVHhfxY4GLHw8Mb6vDamh7EogWAWd
|
||||
4miyu+tffyozufJVnG+qpB7tEL6DKRE25p4/+m17UeHVd6W9y2kOOyIglAwxZUAN
|
||||
Ca8QNXqb6nkIRKSLZ6krTcHn0Nen9rU6jdmjqXm4pGVcvPM+95+Z9rjDZWgtq4Mu
|
||||
3YWZBKn10VzVUUIBuZ9BtUsisgD0y2cQ72nEEK36lAZ2UBJXgq7FFK08fbud8XPh
|
||||
fPCucd5b9xLd94Dx3D7xYLrJGZdvdXSFTP05Q8NkbjSsE8EcZbZSQvFG6y/6XabP
|
||||
Dkmd93R8eA0GOiwS++JLJuRviyGBM0I5E+hCq1tV1Bu7N7YSkffzfbzS/+yxw9Wg
|
||||
zLIrx49dfkxSQkj7j/1akHMQ1alyVI1J2zhSqkZIDi/6ACzMcww2SyTOE2ympKSj
|
||||
1nyb6TgLxiTbTaxnIUlbQTdk5mBrH+0qL2AZMB7YPJwZQ4ffwAr05MpgiHrWo7nk
|
||||
JS554v7Byzy3989OWEz7w62rfarKiPsOOA0fnlw567gHxFAiTfiFf27sjPq2cU3e
|
||||
fZZpxN0+Ht4mkAosTRWVqaI+3D0Od3yNQShPuINRvj63j5A6cDGJWvqTU/xgyY11
|
||||
kO5aLx2EnwCp5sOGI6Ie3RLjoUZgG2e/UBUiknxKtIyPbpyVwizcOzogtryLAgMB
|
||||
AAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFLRkFkjo/FpLMymJ65lA
|
||||
uSC09mEaMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsFAAOCAgEAK6xZi+Y5
|
||||
RNz7rUqI4GT/q8Py0s5wLwPlb7LE9TbWU2uHqx2LmQfIMI5Hcv2huHXWF6EWvWRz
|
||||
kTBXrX39BUCGk9FJlrIL1HxkE4vTIUk4u/vozD4x8a57TCrfggajjgWEnH9wzsC7
|
||||
RRffXWUM5FBIB0SLpC9W2sZ69mCNpDjxnYcWMLLwX3pCVSBxz8zuALESR79CueSz
|
||||
tR0ZhLmYbeBptBVBteJQSMcf6pq3eZHd1dVTGfyuGGxp261ZKJtkmKc8x1WO0TAA
|
||||
z9QynmKxwyruNSLGu/B8u4O4AIn13BqX2TiaKVNhGaan9DtH/d1nlQ+OpGYsgpq3
|
||||
LXHm3674+GjBuw/qUKBFl7calRJvyLO2BWjaah4ONYWErHQMirT0dZ8ir7BUu7ae
|
||||
IvnZ69WKDcfc+JgxX576xpf0QRB104G2MV963YiFCK9wRwI3e+JK7F3y3SkSRMiK
|
||||
qt3SVXgXda9xaQ13cEqwH39C28dx3FjWGL1QxblyBMdne3xTYMpJGBW8QHOuK6gr
|
||||
rG8RRDnsgkh+Ecr71j+2wLm4BpN1vZMndxcNXKWe6sFbAC0MqDVgx+JvNRs9drXn
|
||||
rd10I0+GskfvyC/QhXw5ljehKykCvT6Hz6j2HXUyzzjjc7HuEIG1KrGIBlEeWjpI
|
||||
UfM2WWLfQmZZULdYvvV2QJoSFuYOqtdsoNU=
|
||||
-----END CERTIFICATE-----
|
||||
39
signing/cert.pem
Normal file
39
signing/cert.pem
Normal file
@@ -0,0 +1,39 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIG6TCCBNGgAwIBAgIQRH6DOxTQI6Hp8NKCckMm7DANBgkqhkiG9w0BAQsFADBo
|
||||
MQswCQYDVQQGEwJHUjE3MDUGA1UECgwuSGVsbGVuaWMgQWNhZGVtaWMgYW5kIFJl
|
||||
c2VhcmNoIEluc3RpdHV0aW9ucyBDQTEgMB4GA1UEAwwXSEFSSUNBIENvZGUgU2ln
|
||||
bmluZyBSU0EwHhcNMjYwNTA4MDc1NjU1WhcNMjcwNTA4MDc1NjU1WjBlMQswCQYD
|
||||
VQQGEwJOTzENMAsGA1UEBwwET3NsbzEcMBoGA1UECgwTTmFzam9uYWxiaWJsaW90
|
||||
ZWtldDELMAkGA1UECwwCSVQxHDAaBgNVBAMME05hc2pvbmFsYmlibGlvdGVrZXQw
|
||||
ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCgChY1uSq4zwWzCCPkSMVo
|
||||
+nCvBwBvGIuew3AHO1EVDc2wCr0HMI6U0KP9/DrPbtPUMFVRFT7RHn3GN8wY9GKb
|
||||
tSBqeiFdHOBTesPqoi5QGutvGJfjFXcODk8LtUcPwfj11rV20rupjK5XOWlRURDd
|
||||
rHjPgnWqOe2LuQqhhtD6l/k63cCi3pC4//qC3txaiOhYujTZpCaUi/9jMvPVao5Z
|
||||
1YTfFzIR8F6oAYXACIdyI8raltCVtqh0Fav/gY4oj2WNUFyluPHEuaKkief65JRQ
|
||||
NprwC/XpwdZiVTLlAivngJ+HS+HDqtT4R2SQNkwpouM5g/Vdz6Oxi9v5LO4yeF1Z
|
||||
zXxk5SM7CIS8dw4RVJH2+H0cc6OQg4G2RUhaFlQQwuGrGT6zWGRmav2lr+sNmhfT
|
||||
9tgD9RVzzxPb9/lyEY3z5t/Aq9DTDp980qtloj4Wn6LUPmUv42tCLOdFU6BESOmI
|
||||
G1Wgvxr8qGomxUAIlGABDGwekmaqVRBqpGkxvWvoSHgez/rYophyxBAx+nAdYo+O
|
||||
xhk9d1ipzVUihCkQ6GNogRPpQ2N5ga6id0ULX+iDieU6xVA4lD8xoCkL38I64TxS
|
||||
/8ixkiC604DAAQvfoP3mqmyCYbXLDnmyGFQ9S8cxbVD+OV+EWTY+GwGyfZ7A1fZF
|
||||
o5+5ey8X4EfW3HA5/8SULQIDAQABo4IBkDCCAYwwCQYDVR0TBAIwADAfBgNVHSME
|
||||
GDAWgBSp7DUU625nBQ7EbQkMoHIGIbTKLjBxBggrBgEFBQcBAQRlMGMwPgYIKwYB
|
||||
BQUHMAKGMmh0dHA6Ly9jcnQuaGFyaWNhLmdyL0hBUklDQS1Db2RlU2lnbmluZy1T
|
||||
dWItUjEuY2VyMCEGCCsGAQUFBzABhhVodHRwOi8vb2NzcC5oYXJpY2EuZ3IwYgYD
|
||||
VR0gBFswWTAIBgZngQwBBAEwCAYGBACPegECMEMGDSsGAQQBgc8RAQEDAQIwMjAw
|
||||
BggrBgEFBQcCARYkaHR0cHM6Ly9yZXBvLmhhcmljYS5nci9kb2N1bWVudHMvQ1BT
|
||||
MBMGA1UdJQQMMAoGCCsGAQUFBwMDMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9j
|
||||
cmwuaGFyaWNhLmdyL0hBUklDQS1Db2RlU2lnbmluZy1TdWItUjEuY3JsMB0GA1Ud
|
||||
DgQWBBRkiwQILl9eSkYw+Nbx5KokmIBoUjAOBgNVHQ8BAf8EBAMCB4AwDQYJKoZI
|
||||
hvcNAQELBQADggIBABE2vKd3Q6JMFSxZUXr/kH88h40CsVG5mZ0Hr3tgm9iITjzY
|
||||
6ICVO2YPQXKn7Bt6hddsDWEVjQgYGArAxFV0rS9/Olr1YKdYOE/ANF7TGvbGCcVE
|
||||
qNIisEuHqoah406LW6Om+nSlIzizpiKWPcm4RSL0LQrwNj/Z31aNAbQNKu97eI3x
|
||||
uy6ZAU1xrRbRy5HNxbmXJGfy84wvogaGqHsWkYtbPL/uqzWJoj/yxLKl7YKiyfZ0
|
||||
h604HpGlv8Xq6CjgMWTmHyGgxIIX+EBd3tvPpGHdswGvGhX8HFkBjH2uWAmFdYbI
|
||||
DMTyqh4UzV3WLguaHWHsZ9QzFOcETXOFTwmH5NCckeGGaaPVxjuuAu9jqyenU84k
|
||||
a8rZ1qmFJzUNbjrWXK9pjsLxY+ePMu0UW9dnNGppsPCh9Ac7gBzHHDj3pQMZvIWN
|
||||
UKMmMw/a6hKV6VYnmYRJa3K0nEXObJv53vgEEF0H9WePUEcFAp9VKDeTGULWyT+b
|
||||
6iaWQqgXlkWz3isiaJgmYnyqct//aXOo+HP4TRmM6Azpd3/IlTal9T400rzc2OkQ
|
||||
V89AmB9F8G7ZuHduRA/TKhKL/xFq+rQs0G2S41IgqTRX0Ss6wgNv/pBDWr+3kYgV
|
||||
4M7Vr6zpk7ZogqJKcXZpwnegxchSdjMn62WESKM636yJYdsZAikCs8T+sopK
|
||||
-----END CERTIFICATE-----
|
||||
98
signing/verify.sh
Normal file
98
signing/verify.sh
Normal file
@@ -0,0 +1,98 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Verify the integrity and authenticity of this model release.
|
||||
#
|
||||
# Usage: bash signing/verify.sh
|
||||
#
|
||||
# This script verifies:
|
||||
# 1. The signing certificate is issued by a trusted CA
|
||||
# 2. The SHA256SUMS manifest was signed by Nasjonalbiblioteket
|
||||
# 3. All file checksums match the manifest
|
||||
#
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
NC='\033[0m'
|
||||
|
||||
pass() { echo -e "${GREEN}[PASS]${NC} $*"; }
|
||||
fail() { echo -e "${RED}[FAIL]${NC} $*" >&2; }
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
MODEL_DIR="$(dirname "$SCRIPT_DIR")"
|
||||
SIGNING_DIR="$SCRIPT_DIR"
|
||||
|
||||
cd "$MODEL_DIR"
|
||||
|
||||
errors=0
|
||||
|
||||
# Check required files exist
|
||||
for f in "$SIGNING_DIR/SHA256SUMS" "$SIGNING_DIR/SHA256SUMS.sig" \
|
||||
"$SIGNING_DIR/cert.pem" "$SIGNING_DIR/ca-chain.pem"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
fail "Missing file: $f"
|
||||
errors=$((errors + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ $errors -gt 0 ]]; then
|
||||
echo ""
|
||||
fail "Required signing files are missing. Cannot verify."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Nasjonalbiblioteket Model Verification ==="
|
||||
echo ""
|
||||
|
||||
# Show certificate info
|
||||
echo "Certificate subject:"
|
||||
openssl x509 -in "$SIGNING_DIR/cert.pem" -subject -noout 2>/dev/null | sed 's/^subject=/ /'
|
||||
echo "Certificate issuer:"
|
||||
openssl x509 -in "$SIGNING_DIR/cert.pem" -issuer -noout 2>/dev/null | sed 's/^issuer=/ /'
|
||||
echo "Certificate fingerprint (SHA-256):"
|
||||
openssl x509 -in "$SIGNING_DIR/cert.pem" -fingerprint -sha256 -noout 2>/dev/null | sed 's/^.*=/ /'
|
||||
echo ""
|
||||
|
||||
# 1. Verify certificate chain
|
||||
echo "--- Step 1: Verify certificate chain ---"
|
||||
if openssl verify -CAfile "$SIGNING_DIR/ca-chain.pem" "$SIGNING_DIR/cert.pem" > /dev/null 2>&1; then
|
||||
pass "Certificate chain is valid."
|
||||
else
|
||||
fail "Certificate chain verification failed!"
|
||||
errors=$((errors + 1))
|
||||
fi
|
||||
|
||||
# 2. Verify signature
|
||||
echo "--- Step 2: Verify manifest signature ---"
|
||||
PUBKEY=$(mktemp)
|
||||
trap "rm -f '$PUBKEY'" EXIT
|
||||
openssl x509 -in "$SIGNING_DIR/cert.pem" -pubkey -noout > "$PUBKEY" 2>/dev/null
|
||||
|
||||
if openssl dgst -sha256 -verify "$PUBKEY" \
|
||||
-signature "$SIGNING_DIR/SHA256SUMS.sig" \
|
||||
"$SIGNING_DIR/SHA256SUMS" > /dev/null 2>&1; then
|
||||
pass "Manifest signature is valid."
|
||||
else
|
||||
fail "Manifest signature verification failed!"
|
||||
errors=$((errors + 1))
|
||||
fi
|
||||
|
||||
# 3. Verify file checksums
|
||||
echo "--- Step 3: Verify file checksums ---"
|
||||
if sha256sum -c "$SIGNING_DIR/SHA256SUMS" 2>/dev/null; then
|
||||
pass "All file checksums match."
|
||||
else
|
||||
fail "One or more file checksums do not match!"
|
||||
errors=$((errors + 1))
|
||||
fi
|
||||
|
||||
# Summary
|
||||
echo ""
|
||||
if [[ $errors -eq 0 ]]; then
|
||||
echo -e "${GREEN}✅ Verification successful. All files are authentic and unmodified.${NC}"
|
||||
exit 0
|
||||
else
|
||||
echo -e "${RED}❌ Verification failed with $errors error(s).${NC}"
|
||||
exit 1
|
||||
fi
|
||||
33
special_tokens_map.json
Normal file
33
special_tokens_map.json
Normal file
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"boi_token": "<start_of_image>",
|
||||
"bos_token": {
|
||||
"content": "<bos>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eoi_token": "<end_of_image>",
|
||||
"eos_token": {
|
||||
"content": "<eos>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"image_token": "<image_soft_token>",
|
||||
"pad_token": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726
|
||||
size 33384567
|
||||
3
tokenizer.model
Normal file
3
tokenizer.model
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
|
||||
size 4689074
|
||||
27
tokenizer_config.json
Normal file
27
tokenizer_config.json
Normal file
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"backend": "tokenizers",
|
||||
"boi_token": "<start_of_image>",
|
||||
"bos_token": "<bos>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eoi_token": "<end_of_image>",
|
||||
"eos_token": "<end_of_turn>",
|
||||
"image_token": "<image_soft_token>",
|
||||
"is_local": false,
|
||||
"local_files_only": false,
|
||||
"mask_token": "<mask>",
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"model_specific_special_tokens": {
|
||||
"boi_token": "<start_of_image>",
|
||||
"eoi_token": "<end_of_image>",
|
||||
"image_token": "<image_soft_token>"
|
||||
},
|
||||
"pad_token": "<pad>",
|
||||
"padding_side": "right",
|
||||
"processor_class": "Gemma3Processor",
|
||||
"sp_model_kwargs": null,
|
||||
"spaces_between_special_tokens": false,
|
||||
"split_special_tokens": false,
|
||||
"tokenizer_class": "GemmaTokenizer",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
Reference in New Issue
Block a user