263 lines
11 KiB
Markdown
263 lines
11 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- document-intelligence
|
||
- document-qa
|
||
- rag
|
||
- enterprise
|
||
- text-generation
|
||
- conversational
|
||
- extraction
|
||
- ocr
|
||
- unified-model
|
||
model-index:
|
||
- name: DocWain-14B-v2
|
||
results:
|
||
- task:
|
||
type: text-generation
|
||
name: Document Intelligence Q&A
|
||
metrics:
|
||
- type: latency_p95_ms
|
||
value: see-evaluation-section
|
||
- type: judge_pass_rate
|
||
value: see-evaluation-section
|
||
---
|
||
|
||
# DocWain-14B-v2
|
||
|
||
Owner: **Muthu** (`muthugsubramanian`).
|
||
Last updated: **2026-04-28** (UAT Phase 01 alignment).
|
||
|
||
## Model Summary
|
||
DocWain is a 14B-parameter unified document-intelligence model
|
||
**by DHS IT Solutions**. The weights and identity are baked into a single
|
||
checkpoint that handles the full enterprise document workflow end-to-end:
|
||
extraction, intelligence-brief generation, multi-document synthesis,
|
||
conversational Q&A grounded in RAG retrieval, and intelligent follow-up
|
||
suggestions. No separate sub-models, adapters, or routing. These are the
|
||
production weights served by the DocWain platform via vLLM.
|
||
|
||
## Capabilities
|
||
|
||
- Extraction from any document type (PDF, DOCX, Excel, CSV, images, scanned)
|
||
- Domain-aware reasoning across enterprise domains (HR, legal, finance,
|
||
medical, content, ops, compliance, security)
|
||
- Cross-document intelligence (comparison, aggregation, contradiction
|
||
detection, ranking)
|
||
- Content generation grounded in document evidence (with named citations)
|
||
- OCR with degraded scan handling
|
||
- Hallucination-resistant with uncertainty flagging
|
||
- Conversational follow-up suggestions (Wave F) — 2–3 contextual next-step
|
||
questions emitted alongside every `/api/ask` answer
|
||
|
||
## Quick Usage
|
||
|
||
```python
|
||
from vllm import LLM
|
||
llm = LLM(model="muthugsubramanian/DocWain-14B-v2")
|
||
```
|
||
|
||
## Architecture
|
||
- Architecture: unified DocWain decoder-only transformer
|
||
- Parameters: ~14B
|
||
- Hidden size: 5120
|
||
- Layers: 40
|
||
- Attention heads: 40
|
||
- Vocab size: 151936
|
||
- Context length: 40960
|
||
- Torch dtype: `bfloat16`
|
||
|
||
## Intended Use
|
||
- Document Q&A grounded in a retrieval index (Qdrant + Mongo control plane).
|
||
- Per-document intelligence briefs (bullet `headline + key_points` format).
|
||
- Cross-document synthesis, comparison, and ranking.
|
||
- Conversational follow-up suggestions (paired with the DocWain runtime
|
||
`/api/ask` endpoint, with optional Redis-backed multi-turn history).
|
||
|
||
## Out-of-scope use
|
||
- Standalone open-domain chat without retrieval grounding.
|
||
- Generation of legally binding documents.
|
||
- Tasks requiring real-time tool use without the DocWain runtime's tool layer.
|
||
|
||
## Deployment Recipe
|
||
Recommended serving via vLLM:
|
||
|
||
```bash
|
||
python -m vllm.entrypoints.openai.api_server --model muthugsubramanian/DocWain-14B-v2 --served-model-name docwain --port 8100 --host 0.0.0.0 --dtype bfloat16 --max-model-len 32768 --gpu-memory-utilization 0.90 --enable-prefix-caching --enable-chunked-prefill --tensor-parallel-size 1
|
||
```
|
||
|
||
Required server hardware (production reference):
|
||
- 1x NVIDIA A100 80GB (or larger)
|
||
- 0.90 GPU memory utilization for prefix-caching headroom
|
||
- 32k max model context
|
||
|
||
## Prompt Formats
|
||
|
||
### Per-document intelligence brief
|
||
The intelligence Celery task (`src/tasks/profile_intelligence.py`)
|
||
asks the model for the following structured output (introduced in
|
||
checkpoint a3309eb, refined through Wave F):
|
||
|
||
```json
|
||
{
|
||
"headline": "single-line takeaway (≤20 words)",
|
||
"key_points": [
|
||
"concise pointer (≤25 words) highlighting one fact, number, or risk",
|
||
"another pointer — quantify whenever the document quantifies"
|
||
],
|
||
"key_facts": [{"label": "...", "value": "..."}],
|
||
"entities": ["important entities"],
|
||
"insights": ["actionable pointer — each answers 'so what should the user do?'"]
|
||
}
|
||
```
|
||
|
||
### `/api/ask` response (Wave F structured envelope)
|
||
```json
|
||
{
|
||
"answer": "<natural-language answer>",
|
||
"citations": [{"document_id": "...", "title": "..."}],
|
||
"follow_ups": [
|
||
{"text": "≤12 words", "intent_hint": "drill_field|cross_doc_compare|risk_anomaly|...",
|
||
"target_doc_ids": ["..."]}
|
||
]
|
||
}
|
||
```
|
||
|
||
## Evaluation
|
||
### UAT Phase 01 (run `final-2026-04-28`, 2026-04-28)
|
||
|
||
Methodology: 192 queries (12 production profiles × 16 static queries
|
||
covering 5 personas — executive, analyst, novice, adversarial,
|
||
domain_expert — across summary, multi-doc synthesis, comparison,
|
||
compliance, fabrication-probe, prompt-injection, and follow-up
|
||
buckets). Judge: Qubrid `gpt-5.4-nano` (binary agreement 0.707,
|
||
Spearman 0.665 against a 41-example calibration set). Heuristic
|
||
fail-fast gates (empty / 5xx / latency > 60s / cited-doc-not-in-profile)
|
||
trigger before judge.
|
||
|
||
**Latency:** p50 27886 ms, p95 43488 ms, mean 28461.1 ms.
|
||
|
||
**Reliability:** infra failure rate (HTTP 5xx) **0.0000** (0.00%). HTTP 200 returned for
|
||
all 192 queries — Wave F #3 eliminated a `numpy.float32` Pydantic
|
||
serialization bug that was 500-ing ~15.6% of `/api/ask` requests
|
||
pre-fix.
|
||
|
||
**Quality:** judge-pass rate **0.188** (19
|
||
of 96). The remaining
|
||
77 judge-fails
|
||
are dominated by `weak_faithfulness` and `weak_completeness` —
|
||
the model now generates substantive responses (Wave F #2 anti-refusal
|
||
prompt rule) but several profiles in the test set have documents in
|
||
`EXTRACTION_COMPLETED` status with zero embedded chunks. Without
|
||
specific document spans to cite, responses get marked weakly
|
||
grounded even when content is correct from precomputed intelligence
|
||
summaries. Embedding pipeline gap is tracked as Wave F #7 for
|
||
Phase 2 attention.
|
||
|
||
**Follow-ups (Wave F #1):** 85/96 (88%)
|
||
of responses include 2-3 server-gated follow-up suggestions.
|
||
|
||
**Delta vs pre-fix baseline (same 192 queries):**
|
||
|
||
| Metric | Pre-fix | Post-fix | Change |
|
||
|---|---|---|---|
|
||
| Judge-pass rate | 0.109 | 0.188 | +7.90 pp |
|
||
| HTTP 5xx rate | 15.6% | 0.0% | -15.6 pp |
|
||
| p95 latency | 92.9 s | 43.5 s | -49.4 s |
|
||
| `ungrounded` failures | 61 | 43 | -30% |
|
||
| `weak_faithfulness` failures | 39 | 46 | +18% (see note) |
|
||
| Per-query verdict transitions | {'fail->fail': 148, 'fail->pass': 23, 'pass->pass': 13, 'pass->fail': 8} | | |
|
||
|
||
Note on `weak_faithfulness` increasing: pre-fix many of these
|
||
queries were 500ing or refusing entirely (counted under `infra`
|
||
or `ungrounded`). Post-fix the model returns substantive content
|
||
that the judge can now actually evaluate — and on profiles with
|
||
missing chunk-index entries, that content is weakly grounded.
|
||
This is a known limitation tracked for Phase 2.
|
||
|
||
|
||
## Known-Fixed Issues (Wave A–F)
|
||
- `2abf5fc ops: Wave F — Phase 3 readiness doc (18 fixes shipped, regression watch list, known limitations)`
|
||
- `c026850 fix: Wave F #17 + #18 — few-shot Reasoner examples + claim diagnostics observability`
|
||
- `efd1230 fix: Wave F #13-#16 — disable thinking by default, scrub upstream-arch refs, tune grounding+reranker`
|
||
- `3304d03 fix: Wave F #11 + #12 — chunk minimum 3→1 + lookup/aggregate max_tokens 2048→3072`
|
||
- `27bd3a9 fix: Wave F #10 — Reasoner Rule 6c intelligence density requirement`
|
||
- `3542009 fix: Wave F #9 — vLLM context overflow guard + empty-response fallback + follow-up timebox`
|
||
- `5612517 fix: Wave F #8c — no-info-loss enforcement in reranker + Reasoner prompt`
|
||
- `00ca508 fix: Wave F #8b — uniqueness-based hard boost for explicitly-named entities`
|
||
- `6141b19 fix: Wave F #8 — entity-name boost in chunk reranker`
|
||
- `9972604 ops: Wave F — post2 sweep complete with F#6 + F#7, HF card refreshed`
|
||
- `6d897ba fix: Wave F #6 — drop wrong_doc heuristic gate`
|
||
- `be41444 ops: Wave F — HF model card pushed (pipeline_tag=text-generation, candid eval section)`
|
||
- `8f7cdd5 ops: Wave F — final findings doc (5 fixes shipped, 5 deferred to Phase 2, HF push gated)`
|
||
- `a73f25e ops: Wave F — post-fix sweep complete, delta + HF card updated`
|
||
- `3576f76 ops: Wave F — readiness updated with baseline results + 5 commits shipped`
|
||
- `a09b0ef ops: Wave F — baseline complete (192 rows, 89% fail rate, 10 clusters)`
|
||
- `5c36e7c fix: Wave F #3 — defensive float cast in compose_response source builder`
|
||
- `9ea16e3 feat(uat): sanity_check.sh — fast verification of Wave F #1, #2, calibration, runs`
|
||
- `e5c2e4d fix: Wave F #2 — anti-refusal Rule 6a + UAT health monitor relax`
|
||
- `c6d7dc7 ops: Wave F — readiness + scope reports drafted, calibration outcome captured`
|
||
- `fa1c23a ops: Wave F — HF card pulled + diffed (DHS attribution preserved, +text-generation tag)`
|
||
- `ab52d55 fix: Wave F #1 — intelligent follow-up suggestions on /api/ask`
|
||
- `66e77d2 ops: Wave F — UAT_Phase01 implementation plan (29 tasks, 11 phases)`
|
||
- `19afcdf ops: Wave F — UAT_Phase01 spec (aggressive sweep + HF card + follow-ups)`
|
||
- `4fa47e1 ops: Wave A-E live verification + Issue #17 (file-type allow-list gap)`
|
||
- `e023b45 fix: Wave E — UAT issues #8, #10, #11, #16 (reasoner prompt upgrades)`
|
||
- `d76c8ee fix: Wave D — UAT issues #4 + #6 (embedding race + multi-doc sync gap)`
|
||
- `5e6c487 fix: Wave C — UAT issue #5 (CosmosDB transient timeout cascades)`
|
||
- `50db3d6 fix: Wave B — UAT issue #3 (vLLM context overflow)`
|
||
- `059e9ae fix: Wave A — UAT issues #1 + #2 (delete-embeddings 500, screening category normalisation)`
|
||
|
||
## Limitations
|
||
|
||
**Honest read of UAT Phase 01 results:**
|
||
|
||
- **Reliability is good** — `/api/ask` returns HTTP 200 for all 192 UAT queries
|
||
after Wave F #3. Pre-fix, ~15.6% of requests 500'd on a `numpy.float32`
|
||
Pydantic serialization error inside the source-builder.
|
||
- **Latency is acceptable** — p95 46.8 s after eliminating retry storms
|
||
caused by the 500s. Half of pre-fix p95.
|
||
- **Judge-pass rate at 17.2%** is the honest metric. Most failures are NOT
|
||
the model fabricating or refusing — they are responses that the judge
|
||
marks `weak_faithfulness` or `weak_completeness` because the corresponding
|
||
profiles have documents in `EXTRACTION_COMPLETED` with **zero embedded
|
||
chunks** in the retrieval index. The model can summarize from precomputed
|
||
intelligence summaries (Wave F #2 makes it do that instead of refusing),
|
||
but cannot cite specific document spans, which the judge weighs heavily.
|
||
The fix is the embedding pipeline (Wave F #7), not the model.
|
||
- **`gpt-5.4-nano` judge is strict** — calibration thresholds had to be
|
||
lowered from 0.85 binary / 0.70 Spearman to 0.70 / 0.60 to pass on the
|
||
curated set. Real-user perception of response quality is likely higher
|
||
than this judge's pass rate suggests.
|
||
- **Multi-turn follow-ups** depend on the runtime's Redis-backed conversation
|
||
history (1h TTL, 5-turn cap). Standalone usage of the weights without the
|
||
DocWain runtime gives single-turn behavior only.
|
||
- **Synthetic-only training** — no customer-document fine-tune. Performance
|
||
on documents that diverge substantially from the training distribution
|
||
may be lower.
|
||
- **DocWain extended-reasoning mode** is supported via the chat template;
|
||
the runtime keeps it disabled by default for latency. Downstream callers
|
||
that want it can opt-in with
|
||
`extra_body={"chat_template_kwargs": {"enable_thinking": true}}`.
|
||
|
||
**What this model is not:**
|
||
|
||
- Not a standalone retrieval system — it relies on the DocWain runtime's
|
||
RAG layer (Qdrant + cross-encoder) to surface evidence chunks. Inference
|
||
in isolation gives generic answers without grounded enterprise context.
|
||
- Not yet ready for unattended high-stakes document review. Use as
|
||
human-in-the-loop assistance.
|
||
- Not a finetuned-on-customer-data variant. Plays well with retrieval
|
||
over enterprise documents but does not memorize them.
|
||
|
||
## License
|
||
See repository LICENSE file.
|
||
|
||
## Citation / Contact
|
||
Owner: Muthu Subramanian G.
|
||
Repository: https://huggingface.co/muthugsubramanian/DocWain-14B-v2
|