初始化项目，由ModelHub XC社区提供模型

Model: deepakdsoni/antahkarana-7B Source: Original Platform
2026-07-18 07:31:10 +08:00
commit 5e6ce064f6
17 changed files with 268768 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,226 @@
+---
+license: apache-2.0
+base_model: mistralai/Mistral-7B-v0.1
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - continual-learning
+  - lifelong-learning
+  - catastrophic-forgetting
+  - self-supervised-learning
+  - antahkarana
+  - vedic-ai
+language:
+  - en
+---
+
+# Antahkarana-7B
+
+> **A lifelong-learning language model that learns new domains without forgetting old ones, and abstains
+> instead of hallucinating when it is unsure** — built on Mistral-7B with an AI architecture derived from the
+> 2,500-year-old Vedic model of mind (the *antaḥkaraṇa*, "inner instrument").
+
+**Author:** Deepak Soni · **License:** Apache-2.0 · **Base model:** Mistral-7B-v0.1
+
+This is a **standalone, full-weights 7B model**: load it with a plain `from_pretrained` — no adapter, no PEFT.
+The continual-learning architecture was trained in via LoRA and then **merged into the base weights**.
+
+---
+
+## 📦 Model family
+
+| Model | What |
+|---|---|
+| [**antahkarana-v1**](https://huggingface.co/deepakdsoni/antahkarana-v1) | the original architecture + v1 vision models — the most stable continual learner (only positive backward transfer) |
+| [**antahkarana-v2**](https://huggingface.co/deepakdsoni/antahkarana-v2) | accuracy-recovering v2 (36.5M) — matches SOTA accuracy at ~3× less forgetting |
+| [**antahkarana-7B**](https://huggingface.co/deepakdsoni/antahkarana-7B) | the architecture scaled to a **7B language model** |
+
+## At a glance — what makes this different
+
+Standard fine-tuning suffers **catastrophic forgetting**: teach a model a new task and it loses the old one.
+Antahkarana-7B is trained with a small set of cognitive "faculties," each derived from a Vedic concept and
+implemented as a concrete mechanism:
+
+| Faculty (Vedic) | Mechanism (ML) | What it does |
+|---|---|---|
+| **saṃskāra** | Fisher-importance consolidation + decay over LoRA | protects what mattered for old domains → don't forget |
+| **vijñāna-smṛti** | dark-knowledge / exemplar replay | rehearses past domains while learning new ones |
+| **pramāṇa** | calibrated-confidence gate | abstains ("I'm not sure") instead of hallucinating |
+| **manas / buddhi** | two decorrelated views, cross-teaching | safe self-learning from unlabeled data (research track) |
+
+### How it works
+
+The borrowed mind (**Mistral-7B**) stays frozen as the stable core (*śruti*); a small trainable instrument
+(*chitta* = LoRA, ~0.2% of params) learns new domains, guided by the faculties — and the **pramāṇa** gate
+decides whether to answer or abstain:
+
+![Antahkarana-7B architecture](architecture.png)
+
+**Measured outcome (continual instruction-tuning, 4 domains, 3 seeds): ~3.8× less forgetting than naive LoRA,
+with higher and far more stable accuracy.**
+
+![Antahkarana-7B vs naive LoRA](benchmark_7b.png)
+
+---
+
+## The journey: from a 2,500-year-old architecture to a 7B model
+
+This model is the production endpoint of a multi-stage research-to-engineering program.
+
+**1. The architecture.** The Vedic tradition describes the mind as an *antaḥkaraṇa* — an "inner instrument" of
+distinct faculties (chitta/memory, manas/perception, buddhi/discernment, ahaṃkāra/identity, plus pramāṇa/valid
+knowledge and the guṇa dynamics). Each faculty was mapped to a concrete, testable ML mechanism.
+
+**2. Research validation (vision, 36–52M params).** The mechanisms were first proven on continual-learning
+image benchmarks (Split-CIFAR-100, Split-Tiny-ImageNet) against the field's standard methods (EWC, ER, DER++):
+the architecture was the **most stable learner tested and the only one with positive backward transfer**, with
+a clean ablation showing each Vedic-derived component adds value.
+
+**3. Scaling on a frozen modern backbone (E1–E2).** On a **frozen ViT-B/16**, the consolidation works in
+*adapter space*, matching the SOTA (DER++) on accuracy while forgetting less, and extends to the harder
+**class-incremental** setting with label-free novelty detection (avidyā).
+
+**4. Self-learning and memory (E-S, śruti/smṛti/nidrā).** The model learns from **unlabeled** data via
+decorrelated co-training and reaches near-supervised accuracy from ~2% labels; a complementary study showed an
+external "smṛti" memory + periodic "sleep" consolidation retains knowledge ~2.4× better than holding it in
+weights.
+
+**5. The 7B model (E7).** The architecture was ported to language: **frozen Mistral-7B + LoRA + saṃskāra +
+vijñāna-smṛti + pramāṇa**, continually instruction-tuned across four domains with checkpointing, then **merged**
+into the standalone 7B model published here.
+
+---
+
+## Results
+
+### Continual instruction-tuning — naive LoRA vs Antaḥkaraṇa-LoRA (3-seed mean ± std)
+Four text-classification domains learned in sequence (AG News → DBpedia → Emotion → SST-2), each with its own
+label space, so forgetting is meaningful.
+
+| Metric | naive LoRA | **Antaḥkaraṇa (this model)** |
+|---|---|---|
+| Final accuracy ↑ | 0.849 ± .029 | **0.882 ± .003** |
+| **Forgetting** ↓ | 0.053 ± .032 | **0.014 ± .009** (~3.8× less) |
+| Confidence on known domains | 0.841 | **0.954** |
+| Known − unknown confidence gap ↑ | 0.467 | **0.494** |
+
+### Live deployment test (this merged model)
+- **General language preserved** — correct world-knowledge answers (e.g. capital of Japan → Tokyo; a fluent
+  one-sentence definition of photosynthesis).
+- **Continual retention: 8/8 correct across all four domains, including the first one learned** — no
+  catastrophic forgetting, demonstrated live.
+- **pramāṇa abstention** — on a factually neutral input (no sentiment to extract), confidence drops to 0.53 and
+  the model abstains rather than guessing; on clear inputs it stays 0.97–0.99 and answers.
+
+---
+
+## Why this is an innovation in today's AI
+
+Most of modern AI is **static**: a model is trained once, frozen, and shipped. Teaching it something new means
+expensive retraining — and naive fine-tuning *overwrites* old knowledge (catastrophic forgetting). The field's
+strongest continual-learning methods buy stability only by trading away accuracy, or vice-versa.
+
+Antaḥkaraṇa breaks that trade-off. Across a rigorous benchmark vs the standard methods (EWC, LwF, ER, DER++), it
+is the **only method that lands in the "ideal corner" — high accuracy *and* very low forgetting** — matching the
+SOTA's accuracy while forgetting ~3× less:
+
+![Accuracy vs forgetting frontier](benchmark_frontier.png)
+
+That combination is what makes a model genuinely *lifelong*: it can keep learning in deployment without
+expensive retraining and without losing what it already knew — while the **pramāṇa** gate lets it say
+*"I don't know"* instead of hallucinating. A static, occasionally-confident model becomes a **living, honest**
+one. That is the shift the architecture is reaching for.
+
+## Potential — and where it needs to adapt
+
+**What this architecture could unlock:**
+- **Lifelong enterprise models** — absorb new products, policies, and data continuously, without retraining the
+  base or forgetting prior knowledge.
+- **Trustworthy / high-stakes AI** — calibrated abstention (pramāṇa) for medical, legal, and financial settings
+  where "I'm not sure" is safer than a confident guess.
+- **Label-efficient & self-learning** — learns from *unlabeled* data (co-training), reaching near-supervised
+  accuracy from as little as ~2% labels — cutting annotation cost dramatically.
+- **Personal / on-device AI** — a tiny adapter (~160 MB) + external memory personalizes a frozen base to a user,
+  privacy-preserving, with no full retraining.
+- **Agentic memory** — the śruti (stable core) / smṛti (external memory) / nidrā (sleep-consolidation) design
+  gives agents that accumulate experience over time.
+
+**Where it still needs to adapt (honest roadmap):**
+- **Beyond classification** — the LLM evaluation here is classification framed as generation; it needs extension
+  to open-ended instruction-following and longer, more realistic domain streams.
+- **Sharper pramāṇa** — the abstention gate works but is over-confident on adversarial nonsense; it needs
+  stronger calibration (e.g. conformal / ensemble methods) at scale.
+- **Scale & breadth** — validated on 4 domains and 7B; next is longer continual streams, established
+  continual-LLM benchmarks, and larger models (13B–70B).
+- **Self-learning + memory at LLM scale** — co-training and the smṛti/nidrā memory are proven in vision and small
+  setups; integrating them into the LLM continual loop is the next build.
+- **Conditional compute** — a guṇa-driven mixture-of-experts / early-exit layer (efficiency) is designed but not
+  yet implemented.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("deepakdsoni/antahkarana-7B")
+model = AutoModelForCausalLM.from_pretrained(
+    "deepakdsoni/antahkarana-7B", dtype=torch.bfloat16, device_map="auto")
+
+prompt = ("Classify the sentiment of this movie review (negative, positive).\n"
+          "Text: a heartfelt, beautifully acted triumph.\nAnswer:")
+out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
+                     max_new_tokens=4, pad_token_id=tok.eos_token_id)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+Requires a GPU for full-precision inference (~15 GB in bf16); 4-bit quantization (bitsandbytes) runs in ~5 GB.
+
+---
+
+## Training details
+
+| | |
+|---|---|
+| **Base model** | `mistralai/Mistral-7B-v0.1` (frozen) |
+| **Adapter** | LoRA (r=16, α=32) on `q/k/v/o_proj`; ~13.6M trainable (0.19%) |
+| **Method** | saṃskāra (Fisher Ω + decay) on LoRA · vijñāna-smṛti exemplar replay · pramāṇa confidence gate |
+| **Curriculum** | 4 classification domains in sequence, per-task checkpointing (resumable) |
+| **Merge** | LoRA folded into base via `merge_and_unload` → standalone full-weights 7B |
+| **Precision** | bfloat16 |
+
+To **continue lifelong-learning** (add new domains with saṃskāra protection), use the LoRA adapter + resume
+workflow rather than this merged checkpoint — merging flattens the LoRA structure.
+
+---
+
+## Limitations & honest notes
+
+- Continual evaluation is on **classification framed as generation** (clean, measurable), not open-ended
+  instruction following — a natural next extension.
+- The pramāṇa gate is **not perfect**: it abstains well on genuinely under-determined input but can still be
+  over-confident on adversarial nonsense; the robust evidence is the calibration AUROC and the
+  in-distribution-vs-unfamiliar confidence gap across many examples.
+- The model inherits the capabilities, biases, and knowledge cutoff of **Mistral-7B-v0.1**.
+
+---
+
+## License & attribution
+
+Released under the **Apache License 2.0**. This is a **derivative work of Mistral-7B-v0.1** (© Mistral AI,
+Apache-2.0) — see the `NOTICE` file. The base 7B weights were used as a **frozen foundation and were not trained
+from scratch**. The Antaḥkaraṇa architecture, continual training, and merging are the contribution of the author.
+
+*Built on the Upaniṣads, Sāṃkhya, Yoga, Nyāya, and modern ML (PyTorch · Transformers · PEFT).*
+
+## Citation
+
+```bibtex
+@misc{antahkarana7b2026,
+  title  = {Antahkarana-7B: Lifelong Learning with a Vedic-Derived Cognitive Architecture},
+  author = {Deepak Soni},
+  year   = {2026},
+  note   = {Built on Mistral-7B-v0.1 (Apache-2.0)},
+  url    = {https://huggingface.co/deepakdsoni/antahkarana-7B}
+}
+```