初始化项目，由ModelHub XC社区提供模型
Model: sbintuitions/sarashina-embedding-v2-1b Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
10
1_Pooling/config.json
Normal file
10
1_Pooling/config.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"word_embedding_dimension": 1792,
|
||||
"pooling_mode_cls_token": false,
|
||||
"pooling_mode_mean_tokens": false,
|
||||
"pooling_mode_max_tokens": false,
|
||||
"pooling_mode_mean_sqrt_len_tokens": false,
|
||||
"pooling_mode_weightedmean_tokens": false,
|
||||
"pooling_mode_lasttoken": true,
|
||||
"include_prompt": false
|
||||
}
|
||||
96
LICENSE
Normal file
96
LICENSE
Normal file
@@ -0,0 +1,96 @@
|
||||
Sarashina Model NonCommercial License Agreement
|
||||
Released Date by SB Intuitions Corp.: November 8, 2024
|
||||
|
||||
This Sarashina Model NonCommercial License Agreement (this "Agreement") is a legal agreement between Licensee (as defined in Article 1) and SB Intuitions Corp. ("SB Intuitions"), and governs Licensee's use of the Models (as defined in Article 1) that SB Intuitions provides to Licensee under this Agreement.
|
||||
|
||||
The Models released under this Agreement are intended to be used under this Agreement and contribute to the development of AI technologies.
|
||||
|
||||
By using, reproducing, modifying, distributing, performing or displaying any portion or element of the Model or Derivative Models, or otherwise accepting the terms of this Agreement, you agree to be bound by this Agreement. If you do not agree to this Agreement, you may not download the Model or Derivative Models.
|
||||
|
||||
1. Definition
|
||||
i. "Model" means the deliverables including machine learning models, software, learnt weights, algorithms, parameters and configuration files provided by SB Intuitions and shared under this Agreement.
|
||||
ii. "Output Data" means the text, images, sounds, or other information output of the Model that results from operating or otherwise using the Model.
|
||||
iii. "Derivative Model" means all modifications to the Model, or deliverables created based on the Model, and any other machine learning models which have been trained, tuned, or improved by using Output Data. For clarification, Output Data is not the Derivative Model.
|
||||
iv. "Licensee" means the person or entity exercising the rights granted by this Agreement.
|
||||
v. "Affiliate" means any entity that now or hereafter controls, is controlled by, or is under common control with, SB Intuitions Corp. For the purposes of this definition, "control" shall mean the direct or indirect ownership over or the legal right or power to, directly or indirectly, direct or alter the direction of the management and affairs of the entity.
|
||||
vi. "License" means the rights granted by Licensee under Article 2 （Right）ii of this Agreement.
|
||||
vii. "Intellectual Property Rights" means copyrights, patents, utility model rights, design rights, trademarks and other intellectual property rights including the right to acquire such rights or to apply for the registration, etc. of such rights.
|
||||
|
||||
2. Right
|
||||
i. Condition
|
||||
Any use, reproduction, modification, distribution, performance or display of the Model or Derivative Models shall be subject to the following conditions:
|
||||
(a) Upon renewal of the License, Licensee shall agree to comply with the renewed License or to cease using, copying, modifying, distributing, performing or displaying the Model or Derivative Models.
|
||||
(b) Licensee shall comply with the redistribution regulations set out in Article 3.
|
||||
(c) Licensee shall comply with the prohibition regulations set out in Article 6.
|
||||
(d) Licensee shall not use, copy, modify, distribute, perform or display the Model for commercial purpose (except for research and study purpose).
|
||||
ii. Grant of License
|
||||
Subject to Licensee's compliance with this Agreement, SB Intuitions grants Licensee a non-exclusive and non-transferable license to use, copy, distribute, modify, perform or display the Model. This license also includes the creation of Derivative Models from the Model.
|
||||
iii. Reservation of Rights
|
||||
(a) SB Intuitions reserves all rights, interests and remedies in and to the Model or Derivatives Models by SB Intuitions, and no other rights are granted to Licensee except as expressly set forth in this Agreement.
|
||||
(b) Licensee reserves the copyright to all Derivative Models created by Licensee.
|
||||
(c) SB Intuitions does not claim any rights in the Output Data generated from the Model or Derivative Models. To the extent permitted by law, SB Intuitions acknowledges that the Output Data belongs to Licensee. Licensee is responsible for the use of Output Data.
|
||||
|
||||
3. Redistribution
|
||||
i. Condition of Redistribution
|
||||
Licensee may only reproduce or redistribute the Model or Derivative Models if Licensee complies with all the conditions set out in Article 2, as well as the followings:
|
||||
(a) Licensee shall provide all third-party recipients of the Model or Model Derivatives with a copy of this Agreement;
|
||||
(b) Licensee shall, in all copies of the Model or Derivative Models that Licensee distributes, retain the following attribution notice within a “LICENSE” text file distributed as a part of such copies:
|
||||
"Sarashina is licensed under the Sarashina Model NonCommercial License Agreement, Copyright ©SB Intuitions Corp. All Rights Reserved.";
|
||||
(c) If Licensee distributes the Model or Derivative Models, Licensee shall also include "Sarashina" at the beginning of the name of any such Model or Derivative Models;
|
||||
(d) If Licensee uses the Model or Derivative Models as a component of the system, Licensee shall clearly indicate “Built with Sarashina” in relevant materials of the system such as documents, user interfaces, and/or websites.
|
||||
|
||||
4. Trademark
|
||||
Nothing in this Agreement grant the Licensee permission to use the trade names, trademarks, service marks, or product names owned by SB Intuitions. However, the use within customary scope or the necessity to implement the content of this Agreement, such as quoting the deliverables, describing the origin of the deliverables and reproducing the content of the LICENSE file, is permitted.
|
||||
|
||||
5. Termination and Changes of the License
|
||||
i. Litigation
|
||||
If Licensee institutes a lawsuit (including a cross-claim or counterclaim in a lawsuit) or other legal proceeding against any party in connection with the Model or Derivative Models, this Agreement will terminate as of the date such lawsuit is filed.
|
||||
ii. Changes to the License
|
||||
SB Intuitions may revise this Agreement at any time, for security and/or safety reasons, to adapt to new technologies, to comply with legal and regulatory requirements, or for any other reasons. Upon the revision, SB Intuitions will replace this Agreement on the posted sites.
|
||||
iii. Breach of the License
|
||||
SB Intuitions may terminate this Agreement if Licensee is in breach of any term or condition of this Agreement. Upon termination of this Agreement, Licensee shall delete and cease use of the Model and any Derivative Model.
|
||||
iv. Separability
|
||||
If any part or provision of this Agreement is held invalid, illegal or unenforceable, the validity, legality, and enforceability of the remaining provisions shall not in any way be affected or impaired thereby.
|
||||
|
||||
6. Prohibited activities
|
||||
Licensee shall not engage in any of the prohibited activities set out in the "Sarashina Prohibited Activities Policy".
|
||||
|
||||
7. DISCLAIMER
|
||||
i. FUNCTIONALITY AND PERFORMANCE
|
||||
THE MODEL IS PROVIDED ON AN “AS IS” BASIS. SB INTUITIONS MAKES NO WARRANTY AS TO FUNCTIONALITY OR PERFORMANCE.
|
||||
ii. SUITABILITY
|
||||
SB INTUITIONS MAKES NO WARRANTY, EXPRESSED OR IMPLIED, WITH RESPECT TO THE MODEL OR DERIVATIVE MODELS, AS TO ANY MATTER INCLUDING BUT NOT LIMITED TO, NON-INFRINGEMENT OF THIRD-PARTY INTELLECTUAL PROPERTY RIGHTS AND OTHER RIGHTS, MERCHANTABILITY, INTEGRITY, USABILITY, AND COMPATIBILITY FOR ANY PARTICULAR PURPOSE.
|
||||
iii. APPROPRIATENESS OF USE OR REDISTRIBUTION
|
||||
THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE MODEL OR DERIVATIVE MODELS SHALL BE DETERMINED BY THE LICENSEE WHO SHALL BEAR FULL RESPONSIBILITY FOR SUCH DETERMINATION AND ACTIONS.
|
||||
iv. TECHNICAL SERVICES
|
||||
SB INTUITIONS HAS NO OBLIGATION TO PROVIDE LICENSEE WITH ANY TECHNICAL SERVICES WITH RESPECT TO THE MODEL OR ANY DERIVATIVE MODELS.
|
||||
|
||||
8. LIMITATION OF LIABILITY
|
||||
SB INTUITIONS (INCLUDING ITS AFFILIATES FOR THE PURPOSE OF THIS ARTICLE 8) SHALL NOT BE LIABLE UNDER ANY THEORY OF LIABILITY ARISING FROM THIS AGREEMENT, WHETHER IN AN ACTION BASED ON CONTRACT, TORT, NEGLIGENCE, PRODUCT LIABILITY OR OTHERWISE, EXCEPT IN CASES OF WILFUL MISCONDUCT OR GROSS NEGLIGNECE BY SB INTUITIONS, OR WHERE LIABILITY IS MANDATED BY LAW.
|
||||
|
||||
9. Compensation
|
||||
Licensee shall indemnify and hold SB Intuitions (including its Affiliate for the purpose of this Article 9) harmless from and against all claims, demands, damages, losses, liabilities and expenses including, without limitation, reasonable attorneys' fees and expenses which SB Intuitions may suffer or incur as a result or as a consequence of, arising out of or in connection with the use, duplication, modification, distribution, execution or display of the Model, Derivative Models or Output Data by Licensee.
|
||||
|
||||
10. Trade
|
||||
The Model, Derivative Models or the technology contained therein may be subject to export controls and sanctions laws and regulations of Japan and other jurisdictions (the "Export Controls"). Licensee shall comply with all applicable Export Controls. Licensee shall not access, use, download or send the Model or Derivative Models in any manner that would cause any party to violate any Export Controls.
|
||||
|
||||
11. Elimination of Anti-Social Forces
|
||||
i. Licensee represents and warrants that it is not a gang, a gang member, an individual that left a gang within the last five (5) years, a quasi-member of a gang, a gang affiliate, corporate racketeer, a blackmailer camouflaged as a social movement activist, crime groups specialized in intellectual crimes or any other persons or groups equivalent to these (collectively "Anti-Social Forces") and that it does not fall under any of the following categories, presently and in the future:
|
||||
(a) Licensee has relationships in which Anti-Social Forces are recognized to control the management;
|
||||
(b) Licensee has relationships in which Anti-Social Forces are recognized to be substantially involved in the management;
|
||||
(c) Licensee has relationships in which it is recognized to be engaged in unlawful use of Anti-Social Forces;
|
||||
(d) Licensee provides funds or benefits to Anti-Social Forces; or
|
||||
(e) Licensee’s officers or persons substantially involved in its management have socially reprehensible relationships with Anti-Social Forces.
|
||||
ii. Licensee shall not engage in any of the following actions either by themselves or through a third party:
|
||||
(a) violent demands;
|
||||
(b) unreasonable demands beyond legal liability;
|
||||
(c) threatening actions or use of violence in connection with transactions;
|
||||
(d) spreading rumors, damaging the reputation of the counterparty by using fraudulent means or force, or obstructing the service of the other Party; or
|
||||
(e) any other actions equivalent to any of the preceding items.
|
||||
iii. SB Intuitions shall be entitled to unilaterally terminate this Agreement without giving any notice to Licensee when it is found that Licensee falls under the definition of Licensee or any of the items set forth in paragraph i of this Article 11, or has committed an action that falls under any of the items in paragraph ii of this Article 11, or has made a false declaration with regard to the representation or definitive agreement stated in paragraph i of this Article 11, regardless of any breach attributable to SB Intuitions. SB Intuitions shall not be responsible for any damage caused by the termination in accordance with this paragraph and incurred by Licensee.
|
||||
|
||||
12. Governing Law and Jurisdiction
|
||||
i. Governing Law
|
||||
This Agreement shall be governed by and construed in accordance with the laws of Japan.
|
||||
ii. Jurisdiction
|
||||
Any dispute arising from or in connection with this Agreement shall be subject to the exclusive jurisdiction of the Tokyo District Court in the first instance.
|
||||
122
README.md
Normal file
122
README.md
Normal file
@@ -0,0 +1,122 @@
|
||||
---
|
||||
language:
|
||||
- ja
|
||||
license_name: sarahina-non-commercial-license
|
||||
license_link: LICENSE
|
||||
base_model:
|
||||
- sbintuitions/sarashina2.2-1b
|
||||
tags:
|
||||
- transformers
|
||||
- sentence-similarity
|
||||
- feature-extraction
|
||||
- sentence-transformers
|
||||
inference: false
|
||||
---
|
||||
|
||||
# Sarashina-Embedding-v2-1B
|
||||
|
||||
**[日本語のREADME/Japanese README](https://huggingface.co/sbintuitions/sarashina-embedding-v2-1b/blob/main/README_JA.md)**
|
||||
|
||||
"Sarashina-Embedding-v2-1B" is a Japanese text embedding model, based on the Japanese LLM "[Sarashina2.2-1B](https://huggingface.co/sbintuitions/sarashina2.2-1b)".
|
||||
We trained this model with multi-stage contrastive learning. We achieved the state-of-the-art average score across 28 datasets in [JMTEB](https://huggingface.co/datasets/sbintuitions/JMTEB) (Japanese Massive Text Embedding Benchmark).(Benchmarked on July 28, 2025. )
|
||||
|
||||
This model maps sentences & paragraphs to a 1792-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and other applications.
|
||||
|
||||
## Model Details
|
||||
|
||||
### Model Description
|
||||
|
||||
- **Model Type:** Sentence Transformer
|
||||
- **Base model:** [Sarashina2.2-1B](https://huggingface.co/sbintuitions/sarashina2.2-1b)
|
||||
- **Maximum Sequence Length:** 8,192 tokens
|
||||
- **Output Dimensionality:** 1,792 dimensions
|
||||
- **Similarity Function:** Cosine Similarity
|
||||
- **Language:** Japanese
|
||||
- **License:** [Sarashina Model NonCommercial License Agreement](https://huggingface.co/sbintuitions/sarashina-embedding-v2-1b/blob/main/LICENSE)
|
||||
|
||||
### Full Model Architecture
|
||||
|
||||
```
|
||||
SentenceTransformer(
|
||||
(0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: LlamaModel
|
||||
(1): Pooling({'word_embedding_dimension': 1792, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': False})
|
||||
)
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
First install the [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) library:
|
||||
|
||||
```bash
|
||||
pip install sentence-transformers==4.0.2
|
||||
```
|
||||
|
||||
Then you can load this model and run inference.
|
||||
|
||||
```python
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
# Download from the 🤗 Hub
|
||||
model = SentenceTransformer("sbintuitions/sarashina-embedding-v2-1b")
|
||||
# Run inference
|
||||
query = [
|
||||
'task: クエリを与えるので、与えられたWeb検索クエリに答える関連文章を検索してください。\nquery: Sarashinaのテキスト埋め込みモデルはありますか?'
|
||||
]
|
||||
texts = [
|
||||
'text: 更級日記は、平安時代中期に菅原孝標女によって書かれた回想録です。',
|
||||
'text: Sarashinaは、SB Intuitionsが開発した日本語大規模言語モデルです。これまでに7B, 13B, 70B, 8x70Bのモデルが公開されています。',
|
||||
'text: サラシナエンベディングは日本語言語モデルをベースにした日本語埋め込みモデルです。'
|
||||
]
|
||||
query_embedding = model.encode(query)
|
||||
text_embeddings = model.encode(texts)
|
||||
# Get the similarity scores between the embeddings
|
||||
similarities = model.similarity(query_embedding, text_embeddings)
|
||||
print(similarities)
|
||||
# tensor([[0.7403, 0.8651, 0.8775]])
|
||||
```
|
||||
### How to add instructions and prefixes
|
||||
|
||||
For both the query and document sides, use different prefix formats. On the query side, add the prefix `task:` followed by instructions. (Only for STS task, both sentences are considered as query, and should be prefixed with the same instruction.)
|
||||
|
||||
- Query Side: ```task: {Instrcution}\nquery: {Query}```
|
||||
- Document Side: ```text: {Document}```
|
||||
|
||||
### Templates for instructions and prefixes
|
||||
|
||||
The table below provides instruction and prefix templates for five main tasks.
|
||||
|Task|Query Side|Document Side|
|
||||
|:-:|:-|:-|
|
||||
|Retrieval<br>Reranking|task: 質問を与えるので、その質問に答えるのに役立つ関連文書を検索してください。\nquery: |text: |
|
||||
|Clustering|task: 与えられたドキュメントのトピックまたはテーマを特定してください。\nquery: | - |
|
||||
|Classification|task: 与えられたレビューを適切な評価カテゴリに分類してください。\nquery: | - |
|
||||
|STS|task: クエリを与えるので，もっともクエリに意味が似ている一節を探してください。\nquery: |task: クエリを与えるので，もっともクエリに意味が似ている一節を探してください。\nquery: |
|
||||
|
||||
## Training
|
||||
|
||||
Sarashina-Embedding-v2-1B is created through the following three-stage learning process:
|
||||
|
||||
### Stage 1: Weakly-supervised Learning
|
||||
To build a general-purpose and high-performance embedding model for a wide range of domains, we employed contrastive learning using weak supervision data, which consists of our own web-crawled data and open datasets.
|
||||
|
||||
### Step2: Supervised Fine-tuning
|
||||
To further train the model to better understand the similarity between queries and documents, we performed fine-tuning using higher-quality data than that used in Stage 1. Additionally, we trained multiple models by modifying parts of the data.
|
||||
|
||||
### Stage 3: Model Merging
|
||||
To enhance performance, we merged the weights of the two models that yielded the highest JMTEB scores in Stage 2 through linear merging.
|
||||
|
||||
## Evaluation Results (*) with [JMTEB](https://huggingface.co/datasets/sbintuitions/JMTEB)
|
||||
|
||||
|Model|Avg.|Retrieval|STS|Classification|Reranking|Clustering|
|
||||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|
||||
|Sarashina-Embedding-v2-1B (This model)|**76.38**|**76.48**|**84.22**|77.14|**86.28**|52.56|
|
||||
|[cl-nagoya/ruri-v3-310m](https://huggingface.co/cl-nagoya/ruri-v3-310m)|75.85|76.03|81.59|**77.65**|85.84|50.52|
|
||||
|[sbintuitions/sarashina-embedding-v1-1b](https://huggingface.co/sbintuitions/sarashina-embedding-v1-1b)|74.87|74.53|81.71|77.20|84.36|50.30|
|
||||
|[OpenAI/text-embedding-3-large](https://openai.com/ja-JP/index/new-embedding-models-and-api-updates/)|73.86|71.95|82.52|77.27|83.06|51.82|
|
||||
|
||||
(*) Evaluated on July 28, 2025.
|
||||
|
||||
## License
|
||||
|
||||
This model is licensed under [Sarashina Model NonCommercial License Agreement](https://huggingface.co/sbintuitions/sarashina-embedding-v2-1b/blob/main/LICENSE).
|
||||
|
||||
**If you are interested in using this model for commercial purposes, please feel free to contact us through our [contact page](https://www.sbintuitions.co.jp/contact/).**
|
||||
129
README_JA.md
Normal file
129
README_JA.md
Normal file
@@ -0,0 +1,129 @@
|
||||
---
|
||||
language:
|
||||
- ja
|
||||
license_name: sarahina-non-commercial-license
|
||||
license_link: LICENSE
|
||||
base_model:
|
||||
- sbintuitions/sarashina2.2-1b
|
||||
tags:
|
||||
- transformers
|
||||
- sentence-similarity
|
||||
- feature-extraction
|
||||
- sentence-transformers
|
||||
inference: false
|
||||
---
|
||||
|
||||
# Sarashina-Embedding-v2-1B
|
||||
|
||||
「Sarashina-Embedding-v2-1b」は、日本語LLM「[Sarashina2.2-1B](https://huggingface.co/sbintuitions/sarashina2.2-1b)」をベースにした日本語テキスト埋め込みモデルです。
|
||||
|
||||
このモデルは、マルチステージの対照学習で訓練され、 [JMTEB](https://huggingface.co/datasets/sbintuitions/JMTEB) （Japanese Massive Text Embedding Benchmark）の28個のデータセットの平均で、最高水準の平均スコア(2025/07/28時点)を達成しました。
|
||||
|
||||
このモデルは、文や文章を1792次元の高密度ベクトル空間にマッピングし、意味的テキスト類似度、意味的検索、paraphrase mining、テキスト分類、クラスタリングなどに使用できます。
|
||||
|
||||
## モデル詳細
|
||||
|
||||
### モデル説明
|
||||
|
||||
- **モデルタイプ:** Sentence Transformer
|
||||
- **ベースモデル:** [Sarashina2.2-1B](https://huggingface.co/sbintuitions/sarashina2.2-1b)
|
||||
- **最大シーケンス長:** 8,192トークン
|
||||
- **出力次元数:** 1,792次元
|
||||
- **類似度関数:** コサイン類似度
|
||||
- **言語:** 日本語
|
||||
- **ライセンス:** [Sarashina Model NonCommercial License Agreement](https://huggingface.co/sbintuitions/sarashina-embedding-v2-1b/blob/main/LICENSE)
|
||||
|
||||
### モデルアーキテクチャ
|
||||
|
||||
```
|
||||
SentenceTransformer(
|
||||
(0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: LlamaModel
|
||||
(1): Pooling({'word_embedding_dimension': 1792, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': False})
|
||||
)
|
||||
```
|
||||
|
||||
## 使用方法
|
||||
|
||||
まず、[Sentence Transformers](https://github.com/UKPLab/sentence-transformers)ライブラリをインストールします。
|
||||
|
||||
```bash
|
||||
pip install sentence-transformers==4.0.2
|
||||
```
|
||||
|
||||
次に、このモデルをロードし、推論を実行します。
|
||||
|
||||
```python
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
# 🤗 Hubからモデルをダウンロードする
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
model = SentenceTransformer("sbintuitions/sarashina-embedding-v2-1b")
|
||||
# 推論を実行する
|
||||
query = [
|
||||
'task: クエリを与えるので、与えられたWeb検索クエリに答える関連文章を検索してください。\nquery: Sarashinaのテキスト埋め込みモデルはありますか?'
|
||||
]
|
||||
texts = [
|
||||
'text: 更級日記は、平安時代中期に菅原孝標女によって書かれた回想録です。',
|
||||
'text: Sarashinaは、SB Intuitionsが開発した日本語大規模言語モデルです。これまでに7B, 13B, 70B, 8x70Bのモデルが公開されています。',
|
||||
'text: サラシナエンベディングは日本語言語モデルをベースにした日本語埋め込みモデルです。'
|
||||
]
|
||||
query_embedding = model.encode(query)
|
||||
text_embeddings = model.encode(texts)
|
||||
|
||||
# 埋め込みの類似度スコアを取得する
|
||||
similarities = model.similarity(query_embedding, text_embeddings)
|
||||
print(similarities)
|
||||
# tensor([[0.7403, 0.8651, 0.8775]])
|
||||
```
|
||||
|
||||
### instrcution, prefixの付け方
|
||||
|
||||
クエリ側とドキュメント側で形式の異なるprefixをつけ、クエリ側には```task:```というprefixの後に指示を付与します。(*ただしSTSタスクのみ両方のテキストをクエリとみなし同じ形式のinstrcutionとprefixを使用します)
|
||||
- クエリ側: ```task: {インストラクション}\nquery: {クエリ}```
|
||||
- ドキュメント側: ```text: {ドキュメント}```
|
||||
|
||||
|
||||
### instrcution, prefixのテンプレート
|
||||
|
||||
テキスト埋め込みモデルが用いられる5つの主なタスクで使用できるinstrcutionとprefixのテンプレートを下記の表に示します。
|
||||
|
||||
|タスク|クエリ側|ドキュメント側|
|
||||
|:-:|:-|:-|
|
||||
|Retrieval<br>Reranking|task: 質問を与えるので、その質問に答えるのに役立つ関連文書を検索してください。\nquery: |text: |
|
||||
|Clustering|task: 与えられたドキュメントのトピックまたはテーマを特定してください。\nquery: | - |
|
||||
|Classification|task: 与えられたレビューを適切な評価カテゴリに分類してください。\nquery: | - |
|
||||
|STS|task: クエリを与えるので，もっともクエリに意味が似ている一節を探してください。\nquery: |task: クエリを与えるので，もっともクエリに意味が似ている一節を探してください。\nquery: |
|
||||
|
||||
## 学習
|
||||
|
||||
Sarashina-Embedding-v2-1Bは、以下の3段階の学習ステージによって学習されています。
|
||||
|
||||
### Stage 1: 弱教師あり学習
|
||||
|
||||
幅広いドメインに対して汎用的かつ高い性能を持つ埋め込みモデルを構築するため、独自のwebクロールデータとオープンデータで構成された弱教師データによる対照学習を行いました。
|
||||
|
||||
### Stage 2: ファインチューニング
|
||||
|
||||
より正確なクエリ-ドキュメント間の類似度をモデルに学習させるために、Stage 1で使用したデータよりも高品質なデータを用いてファインチューニングを行いました。
|
||||
またデータの一部を変更して複数のモデルを学習しました。
|
||||
|
||||
### Stage 3: モデルマージ
|
||||
Stage 2においてJMTEBのスコアが最も高かった2つのモデルの重みを線形マージすることで性能向上を図りました。
|
||||
|
||||
## [JMTEB](https://huggingface.co/datasets/sbintuitions/JMTEB)による評価 (*)
|
||||
|
||||
|Model|Avg.|Retrieval|STS|Classification|Reranking|Clustering|
|
||||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|
||||
|Sarashina-Embedding-v2-1B (This model)|**76.38**|**76.48**|**84.22**|77.14|**86.28**|52.56|
|
||||
|[cl-nagoya/ruri-v3-310m](https://huggingface.co/cl-nagoya/ruri-v3-310m)|75.85|76.03|81.59|**77.65**|85.84|50.52|
|
||||
|[sbintuitions/sarashina-embedding-v1-1b](https://huggingface.co/sbintuitions/sarashina-embedding-v1-1b)|74.87|74.53|81.71|77.20|84.36|50.30|
|
||||
|[OpenAI/text-embedding-3-large](https://openai.com/ja-JP/index/new-embedding-models-and-api-updates/)|73.86|71.95|82.52|77.27|83.06|51.82|
|
||||
|
||||
(*) 評価日:2025.07.28
|
||||
|
||||
## ライセンス
|
||||
|
||||
このモデルは[Sarashina Model NonCommercial License Agreement](https://huggingface.co/sbintuitions/sarashina-embedding-v2-1b/blob/main/LICENSE)に基づいて公開されています。
|
||||
|
||||
**もしこのモデルの商用利用にご興味がある場合は、お気軽に[コンタクトページ](https://www.sbintuitions.co.jp/contact/)へご連絡ください。**
|
||||
29
config.json
Normal file
29
config.json
Normal file
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaModel"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"head_dim": 112,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 1792,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 6272,
|
||||
"max_position_embeddings": 8192,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 16,
|
||||
"num_hidden_layers": 24,
|
||||
"num_key_value_heads": 8,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 500000,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.50.3",
|
||||
"use_cache": true,
|
||||
"vocab_size": 102400
|
||||
}
|
||||
10
config_sentence_transformers.json
Normal file
10
config_sentence_transformers.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"__version__": {
|
||||
"sentence_transformers": "4.0.2",
|
||||
"transformers": "4.50.3",
|
||||
"pytorch": "2.6.0+cu126"
|
||||
},
|
||||
"prompts": {},
|
||||
"default_prompt_name": null,
|
||||
"similarity_fn_name": "cosine"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:be5307e805b214153dd0005aef4b1df97ecca74b2d85e3da67914fa127a7de0a
|
||||
size 4896176272
|
||||
14
modules.json
Normal file
14
modules.json
Normal file
@@ -0,0 +1,14 @@
|
||||
[
|
||||
{
|
||||
"idx": 0,
|
||||
"name": "0",
|
||||
"path": "",
|
||||
"type": "sentence_transformers.models.Transformer"
|
||||
},
|
||||
{
|
||||
"idx": 1,
|
||||
"name": "1",
|
||||
"path": "1_Pooling",
|
||||
"type": "sentence_transformers.models.Pooling"
|
||||
}
|
||||
]
|
||||
4
sentence_bert_config.json
Normal file
4
sentence_bert_config.json
Normal file
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"max_seq_length": 8192,
|
||||
"do_lower_case": false
|
||||
}
|
||||
51
special_tokens_map.json
Normal file
51
special_tokens_map.json
Normal file
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"cls_token": {
|
||||
"content": "<cls>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"mask_token": {
|
||||
"content": "<mask>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"sep_token": {
|
||||
"content": "<sep>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
409855
tokenizer.json
Normal file
409855
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
3
tokenizer.model
Normal file
3
tokenizer.model
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:008293028e1a9d9a1038d9b63d989a2319797dfeaa03f171093a57b33a3a8277
|
||||
size 1831879
|
||||
171
tokenizer_config.json
Normal file
171
tokenizer_config.json
Normal file
@@ -0,0 +1,171 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_dummy_prefix_space": false,
|
||||
"add_eos_token": true,
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"3": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"4": {
|
||||
"content": "<sep>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"5": {
|
||||
"content": "<mask>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"6": {
|
||||
"content": "<cls>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"7": {
|
||||
"content": "<|system|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"8": {
|
||||
"content": "<|assistant|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"9": {
|
||||
"content": "<|user|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"10": {
|
||||
"content": "<|available_tools|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"11": {
|
||||
"content": "<|tool_calls|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"12": {
|
||||
"content": "<|tool_results|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"13": {
|
||||
"content": "<|code|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"14": {
|
||||
"content": "<|file|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"102397": {
|
||||
"content": "<|prefix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"102398": {
|
||||
"content": "<|suffix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"102399": {
|
||||
"content": "<|middle|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
}
|
||||
},
|
||||
"bos_token": "<s>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"cls_token": "<cls>",
|
||||
"do_lower_case": false,
|
||||
"eos_token": "</s>",
|
||||
"extra_ids": 0,
|
||||
"extra_special_tokens": {},
|
||||
"keep_accents": true,
|
||||
"legacy": false,
|
||||
"mask_token": "<mask>",
|
||||
"model_max_length": 8192,
|
||||
"pad_token": "<pad>",
|
||||
"padding_side": "left",
|
||||
"sep_token": "<sep>",
|
||||
"sp_model_kwargs": {},
|
||||
"spaces_between_special_tokens": false,
|
||||
"tokenizer_class": "LlamaTokenizer",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
Reference in New Issue
Block a user