初始化项目,由ModelHub XC社区提供模型
Model: Raffleraffle/manifoldgl Source: Original Platform
This commit is contained in:
57
README.md
Normal file
57
README.md
Normal file
@@ -0,0 +1,57 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- llm
|
||||
- hyperbolic
|
||||
- geometry
|
||||
- adapter
|
||||
- peft
|
||||
- research
|
||||
base_model:
|
||||
- Qwen/Qwen2.5-7B
|
||||
pipeline_tag: text-generation
|
||||
library_name: peft
|
||||
---
|
||||
|
||||
# ManifoldGL – Information‑Geometric Adapter for LLMs
|
||||
|
||||
ManifoldGL is a parameter‑efficient adapter that enforces **hyperbolic geometry** on the latent space of large language models. It treats the meaning of a token as a **fiber** over a hyperbolic base manifold (a Poincaré ball), rather than a single vector in flat Euclidean space. Latent states are projected onto the ball, and attentions are computed using geodesic distance. A sheaf‑theoretic consistency loss and natural gradient optimization maintain semantic structure during training.
|
||||
|
||||
## Motivation and theoretical background
|
||||
|
||||
Modern LLMs embed tokens in a Euclidean vector space. While convenient, Euclidean geometry has limited capacity to represent hierarchical structures: flat space grows polynomially, whereas hierarchical trees expand exponentially. By contrast, **hyperbolic space** grows exponentially and preserves both local and global relationships in a hierarchy【247949143190903†L115-L124】. Hyperbolic embeddings outperform Euclidean ones for lexical entailment, similarity and analogy tasks【247949143190903†L154-L169】. ManifoldGL leverages these properties by modelling the latent space as a fiber bundle over a hyperbolic base: each point in the Poincaré ball encodes a context, and its fiber contains a distribution of semantic components.
|
||||
|
||||
## Results on ARC‑AGI benchmark
|
||||
|
||||
ManifoldGL fine‑tuned on Qwen2.5‑7B improves task accuracy on the ARC‑AGI benchmark from **12.4 %** to **28.7 %**, a **131.5 % relative improvement**. The model also achieves a **Manifold Faithfulness Rate (MFR) of 94.2 %**, indicating high adherence to the hyperbolic constraints, and maintains a curvature close to the target κ = ‑1 (mean ‑0.98 ± 0.04). Ablation studies show that removing curvature regularization, natural gradients, sheaf consistency or the hyperbolic target significantly reduces accuracy; the Euclidean target ablation causes the largest drop (–10.9 %), highlighting the importance of hyperbolic geometry.
|
||||
|
||||
## Files in this repository
|
||||
|
||||
This model card accompanies adapter weights trained with ManifoldGL. The files follow the structure of the original repository:
|
||||
|
||||
- `adapter_config.json` – configuration for PEFT/LoRA loading
|
||||
- `pytorch_adapter.bin` – adapter weights
|
||||
- `README.md` – this model card
|
||||
|
||||
## Quick start
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM
|
||||
from peft import PeftModel
|
||||
|
||||
# Load the base model (Qwen2.5-7B)
|
||||
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B")
|
||||
|
||||
# Load the ManifoldGL adapter
|
||||
model = PeftModel.from_pretrained(base_model, "jesusvilela/manifoldgl")
|
||||
|
||||
# Now use model.generate(...) to generate text with hyperbolic adapters
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
This adapter can be loaded with [PEFT](https://github.com/huggingface/peft) on top of any compatible Qwen2.5‑7B model. During generation, latent states are projected into hyperbolic space and meaning is represented as fibers. We recommend using FP32 precision for maximum stability.
|
||||
|
||||
## Citation
|
||||
|
||||
If you use ManifoldGL in your work, please cite the accompanying thesis and repository.
|
||||
Reference in New Issue
Block a user