初始化项目，由ModelHub XC社区提供模型

Model: setu4993/LaBSE Source: Original Platform
2026-11-02 22:40:24 +08:00
commit 5159e96050
11 changed files with 501447 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,11 @@
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tar.gz filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,223 @@
+---
+pipeline_tag: sentence-similarity
+language:
+  - af
+  - am
+  - ar
+  - as
+  - az
+  - be
+  - bg
+  - bn
+  - bo
+  - bs
+  - ca
+  - ceb
+  - co
+  - cs
+  - cy
+  - da
+  - de
+  - el
+  - en
+  - eo
+  - es
+  - et
+  - eu
+  - fa
+  - fi
+  - fr
+  - fy
+  - ga
+  - gd
+  - gl
+  - gu
+  - ha
+  - haw
+  - he
+  - hi
+  - hmn
+  - hr
+  - ht
+  - hu
+  - hy
+  - id
+  - ig
+  - is
+  - it
+  - ja
+  - jv
+  - ka
+  - kk
+  - km
+  - kn
+  - ko
+  - ku
+  - ky
+  - la
+  - lb
+  - lo
+  - lt
+  - lv
+  - mg
+  - mi
+  - mk
+  - ml
+  - mn
+  - mr
+  - ms
+  - mt
+  - my
+  - ne
+  - nl
+  - no
+  - ny
+  - or
+  - pa
+  - pl
+  - pt
+  - ro
+  - ru
+  - rw
+  - si
+  - sk
+  - sl
+  - sm
+  - sn
+  - so
+  - sq
+  - sr
+  - st
+  - su
+  - sv
+  - sw
+  - ta
+  - te
+  - tg
+  - th
+  - tk
+  - tl
+  - tr
+  - tt
+  - ug
+  - uk
+  - ur
+  - uz
+  - vi
+  - wo
+  - xh
+  - yi
+  - yo
+  - zh
+  - zu
+tags:
+  - bert
+  - sentence_embedding
+  - multilingual
+  - google
+  - sentence-similarity
+license: apache-2.0
+datasets:
+  - CommonCrawl
+  - Wikipedia
+---
+
+# LaBSE
+
+## Model description
+
+Language-agnostic BERT Sentence Encoder (LaBSE) is a BERT-based model trained for sentence embedding for 109 languages. The pre-training process combines masked language modeling with translation language modeling. The model is useful for getting multilingual sentence embeddings and for bi-text retrieval.
+
+- Model: [HuggingFace's model hub](https://huggingface.co/setu4993/LaBSE).
+- Paper: [arXiv](https://arxiv.org/abs/2007.01852).
+- Original model: [TensorFlow Hub](https://tfhub.dev/google/LaBSE/2).
+- Blog post: [Google AI Blog](https://ai.googleblog.com/2020/08/language-agnostic-bert-sentence.html).
+- Conversion from TensorFlow to PyTorch: [GitHub](https://github.com/setu4993/convert-labse-tf-pt).
+
+This is migrated from the v2 model on the TF Hub, which uses dict-based input. The embeddings produced by both the versions of the model are [equivalent](https://github.com/setu4993/convert-labse-tf-pt/blob/ec3a019159a54ed6493181a64486c2808c01f216/tests/test_conversion.py#L31).
+
+## Usage
+
+Using the model:
+
+```python
+import torch
+from transformers import BertModel, BertTokenizerFast
+
+
+tokenizer = BertTokenizerFast.from_pretrained("setu4993/LaBSE")
+model = BertModel.from_pretrained("setu4993/LaBSE")
+model = model.eval()
+
+english_sentences = [
+    "dog",
+    "Puppies are nice.",
+    "I enjoy taking long walks along the beach with my dog.",
+]
+english_inputs = tokenizer(english_sentences, return_tensors="pt", padding=True)
+
+with torch.no_grad():
+    english_outputs = model(**english_inputs)
+```
+
+To get the sentence embeddings, use the pooler output:
+
+```python
+english_embeddings = english_outputs.pooler_output
+```
+
+Output for other languages:
+
+```python
+italian_sentences = [
+    "cane",
+    "I cuccioli sono carini.",
+    "Mi piace fare lunghe passeggiate lungo la spiaggia con il mio cane.",
+]
+japanese_sentences = ["犬", "子犬はいいです", "私は犬と一緒にビーチを散歩するのが好きです"]
+italian_inputs = tokenizer(italian_sentences, return_tensors="pt", padding=True)
+japanese_inputs = tokenizer(japanese_sentences, return_tensors="pt", padding=True)
+
+with torch.no_grad():
+    italian_outputs = model(**italian_inputs)
+    japanese_outputs = model(**japanese_inputs)
+
+italian_embeddings = italian_outputs.pooler_output
+japanese_embeddings = japanese_outputs.pooler_output
+```
+
+For similarity between sentences, an L2-norm is recommended before calculating the similarity:
+
+```python
+import torch.nn.functional as F
+
+
+def similarity(embeddings_1, embeddings_2):
+    normalized_embeddings_1 = F.normalize(embeddings_1, p=2)
+    normalized_embeddings_2 = F.normalize(embeddings_2, p=2)
+    return torch.matmul(
+        normalized_embeddings_1, normalized_embeddings_2.transpose(0, 1)
+    )
+
+
+print(similarity(english_embeddings, italian_embeddings))
+print(similarity(english_embeddings, japanese_embeddings))
+print(similarity(italian_embeddings, japanese_embeddings))
+```
+
+## Details
+
+Details about data, training, evaluation and performance metrics are available in the [original paper](https://arxiv.org/abs/2007.01852).
+
+### BibTeX entry and citation info
+
+```bibtex
+@misc{feng2020languageagnostic,
+      title={Language-agnostic BERT Sentence Embedding},
+      author={Fangxiaoyu Feng and Yinfei Yang and Daniel Cer and Naveen Arivazhagan and Wei Wang},
+      year={2020},
+      eprint={2007.01852},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
--- a/config.json
+++ b/config.json
@@ -0,0 +1,23 @@
+{
+  "architectures": ["BertModel"],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.29.2",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 501153
+}
--- a/flax_model.msgpack
+++ b/flax_model.msgpack
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cbe50771a6b147d2da0beb6da1d80908a706cec2e2e06a09873649ed183e884
+size 1883714625
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77d8e1f2dbab6eb5d3c261ce9d3dbf1e3c69e02938c95f934f94f42c22dfa31f
+size 1883734344
--- a/pytorch_model.bin
+++ b/pytorch_model.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f45b674a2e511b204647f9e2cb056da656eedaee063a5cef435777b5a5ca1bf
+size 1883775789
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}
--- a/tf_model.h5
+++ b/tf_model.h5
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2965837da030339e5e05b7fd1be813cb17b8ddaec5ed9f0a6f0ac9881019b5e
+size 1883974632
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5aab105881afc3a73d5c8445cdc5c0302b1c3efdecd71a1a34fa0cf4e5b7bf43
+size 13631023
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,15 @@
+{
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}
--- a/vocab.txt
+++ b/vocab.txt