初始化项目，由ModelHub XC社区提供模型

Model: Stffens/bge-small-rrf-v2 Source: Original Platform
2026-05-14 14:59:44 +08:00
commit 136b550965
11 changed files with 31006 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/1_Pooling/config.json
+++ b/1_Pooling/config.json
@@ -0,0 +1,10 @@
+{
+    "word_embedding_dimension": 384,
+    "pooling_mode_cls_token": true,
+    "pooling_mode_mean_tokens": false,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false,
+    "pooling_mode_weightedmean_tokens": false,
+    "pooling_mode_lasttoken": false,
+    "include_prompt": true
+}
--- a/README.md
+++ b/README.md
@@ -0,0 +1,181 @@
+---
+language: en
+license: apache-2.0
+library_name: sentence-transformers
+tags:
+  - sentence-transformers
+  - embedding
+  - retrieval
+  - hybrid-search
+  - self-supervised
+  - fine-tuned
+  - BEIR
+  - information-retrieval
+base_model: BAAI/bge-small-en-v1.5
+datasets:
+  - BeIR/scifact
+  - BeIR/nfcorpus
+  - BeIR/fiqa
+pipeline_tag: feature-extraction
+model-index:
+  - name: bge-small-rrf-v2
+    results:
+      - task:
+          type: Retrieval
+        dataset:
+          name: BEIR SciFact
+          type: BeIR/scifact
+        metrics:
+          - type: ndcg_at_10
+            value: 0.6945
+      - task:
+          type: Retrieval
+        dataset:
+          name: BEIR NFCorpus
+          type: BeIR/nfcorpus
+        metrics:
+          - type: ndcg_at_10
+            value: 0.3949
+      - task:
+          type: Retrieval
+        dataset:
+          name: BEIR SciDocs
+          type: BeIR/scidocs
+        metrics:
+          - type: ndcg_at_10
+            value: 0.1875
+---
+
+# bge-small-rrf-v2: A 33M Parameter Model That Beats ColBERTv2 on 3/5 BEIR Datasets
+
+**Paper:** [vstash: Local-First Hybrid Retrieval with Adaptive Fusion for LLM Agents](https://arxiv.org/abs/2604.15484) (arXiv:2604.15484)
+
+**Trained with zero human labels using a novel self-supervised signal: hybrid retrieval disagreement.**
+
+When vector search and keyword search disagree on what's relevant for a query, that disagreement reveals where the embedding model fails. We exploit this signal to fine-tune BGE-small, producing a model that better distinguishes "semantically close" from "actually relevant."
+
+## Key Result
+
+A 33M parameter model, fine-tuned for $0 with zero human labels, **surpasses ColBERTv2 (110M parameters) on 3 out of 5 standard BEIR benchmarks**:
+
+| Dataset | Docs | ColBERTv2 (110M) | BGE-small base (33M) | **This model (33M)** | vs ColBERTv2 |
+|---------|:----:|:-:|:-:|:-:|:-:|
+| SciFact | 5K | 0.693 | 0.646 | **0.695** | **+0.2%** |
+| NFCorpus | 3.6K | 0.344 | 0.330 | **0.395** | **+14.8%** |
+| SciDocs | 25K | 0.154 | 0.178 | **0.188** | **+21.8%** |
+| FiQA | 57K | 0.356 | 0.328 | 0.328 | -7.8% |
+| ArguAna | 8.6K | 0.463 | 0.419 | 0.424 | -8.4% |
+
+**Up to +19.5% NDCG improvement over the base model, with zero additional inference cost.**
+
+## Why This Matters
+
+Most embedding improvements require either:
+- A larger model (more compute, more latency)
+- Human-labeled training data (expensive, slow)
+- A teacher model for distillation (adds complexity)
+
+This model needs none of that. The training signal comes from running the existing hybrid retrieval pipeline and observing where its two components (vector search and keyword search) disagree. **The system improves itself.**
+
+## The Training Signal: Hybrid Retrieval Disagreement
+
+We discovered that **82% of queries produce disagreement** between vector and keyword search in the top-5 results. These disagreements fall into two categories:
+
+- **Vector blind spots** (51%): chunks the vector search ranks high but keyword search ignores. These are semantically similar but not actually relevant.
+- **Keyword blind spots** (49%): chunks keyword search finds but vector search misses. These contain relevant terms but the embedding doesn't recognize their relevance.
+
+Fine-tuning on these disagreement pairs teaches the model to fix both types of blind spots.
+
+## Training Details
+
+| Parameter | Value |
+|-----------|-------|
+| Base model | [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) |
+| Parameters | 33M (unchanged) |
+| Embedding dimension | 384 (unchanged) |
+| Loss function | MultipleNegativesRankingLoss with explicit hard negatives |
+| Training data | 76K (query, positive, hard_negative) triples |
+| Data source | RRF signal disagreement on SciFact, NFCorpus, FiQA |
+| Human labels | **Zero** |
+| Epochs | 2 |
+| Learning rate | 3e-6 |
+| Batch size | 64 |
+| Training time | ~30 min on T4 GPU |
+| Training cost | $0 (Colab free tier) |
+
+### Why MNRL, not TripletLoss?
+
+We tested TripletLoss first. It **destroyed the model** (-84% NDCG after 3 epochs). TripletLoss pushes individual negatives away with brute force, distorting the embedding space. MNRL adjusts relationships across 64 documents simultaneously per batch, preserving the model's general knowledge while learning from disagreements.
+
+| Loss Function | NDCG@10 on SciFact | Result |
+|---|:-:|---|
+| TripletLoss (3 epochs, lr=2e-5) | 0.055 | -84% (destroyed) |
+| TripletLoss (1 epoch, lr=1e-6) | 0.347 | -0.03% (no effect) |
+| MNRL batch-only negatives (v1) | 0.683 | +5.6% |
+| **MNRL + explicit hard negatives (this model)** | **0.695** | **+7.4%** |
+
+## Usage
+
+### With sentence-transformers
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("Stffens/bge-small-rrf-v2")
+embeddings = model.encode(["your query", "your document"])
+similarity = embeddings[0] @ embeddings[1]
+```
+
+### With vstash (hybrid retrieval system)
+```bash
+pip install vstash
+vstash reindex --model Stffens/bge-small-rrf-v2
+vstash search "your query"
+```
+
+### Train your own version on your data
+```bash
+pip install vstash sentence-transformers torch
+vstash retrain  # generates disagreement pairs from YOUR corpus and fine-tunes
+vstash reindex --model ~/.vstash/models/retrained
+```
+
+## Reproduce From Scratch
+
+```bash
+git clone https://github.com/stffns/vstash
+cd vstash
+pip install -e . sentence-transformers torch
+
+# Generate disagreement triples
+python -m experiments.rrf_training_pairs --datasets scifact nfcorpus fiqa
+
+# Train (GPU recommended)
+python -m experiments.finetune_rrf --epochs 2 --lr 3e-6 --batch-size 64
+
+# Evaluate
+python -m experiments.finetune_rrf --evaluate-only
+```
+
+## Limitations
+
+- **ArguAna regression**: queries with 200+ words show -8.4% vs ColBERTv2. Long argumentative queries produce only 1.1% signal disagreement, leaving no training signal.
+- **FiQA neutral**: financial queries show +0.1% vs base but -7.8% vs ColBERTv2. The disagreement signal exists (86.7%) but doesn't translate to NDCG gains on this dataset.
+- **English only**: inherited from BGE-small-en-v1.5.
+- **Not tested beyond BEIR**: performance on domain-specific corpora may vary.
+
+## Citation
+
+```bibtex
+@software{vstash2026,
+  author = {Steffens, Jayson},
+  title = {vstash: Local-First Hybrid Retrieval with Adaptive Fusion for LLM Agents},
+  url = {https://github.com/stffns/vstash},
+  year = {2026}
+}
+```
+
+## Related
+
+- [vstash paper](https://github.com/stffns/vstash/blob/main/paper/vstash-paper.md) (Section 8.10: Self-Supervised Embedding Refinement)
+- [vstash GitHub](https://github.com/stffns/vstash)
+- [Base model: BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)
--- a/config.json
+++ b/config.json
@@ -0,0 +1,35 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "id2label": {
+    "0": "LABEL_0"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "is_decoder": false,
+  "label2id": {
+    "LABEL_0": 0
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}
--- a/config_sentence_transformers.json
+++ b/config_sentence_transformers.json
@@ -0,0 +1,14 @@
+{
+  "__version__": {
+    "sentence_transformers": "5.3.0",
+    "transformers": "5.0.0",
+    "pytorch": "2.10.0+cu128"
+  },
+  "model_type": "SentenceTransformer",
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}
--- a/model.onnx
+++ b/model.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19dea7cb49f02465f22de7cb5be63d2943e731852bd29eb7e9050fd026fd9a10
+size 1262826
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69f84f688ffc976c96ea0ab3f936c0c2cea32d4581b00013962cc1eea477f110
+size 133462104
--- a/modules.json
+++ b/modules.json
@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]
--- a/sentence_bert_config.json
+++ b/sentence_bert_config.json
@@ -0,0 +1,4 @@
+{
+    "max_seq_length": 512,
+    "do_lower_case": true
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,17 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "is_local": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}