初始化项目，由ModelHub XC社区提供模型

Model: MainStack/marvy-1-14B Source: Original Platform
2026-08-05 22:37:23 +08:00
commit 7a6b67a053
24 changed files with 2270 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,350 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-14B-Instruct
+base_model_relation: finetune
+library_name: transformers
+pipeline_tag: text-generation
+language:
+  - en
+tags:
+  - servicenow
+  - itsm
+  - csdm
+  - itom
+  - delivery
+  - solution-design
+  - user-stories
+  - business-analysis
+  - qwen2.5
+  - lora
+  - sft
+  - mlx
+model-index:
+  - name: marvy-1-14B
+    results:
+      - task:
+          type: text-generation
+          name: Text Generation
+        dataset:
+          type: custom
+          name: ServiceNow Delivery SFT (project-disjoint test split)
+        metrics:
+          - type: perplexity
+            value: 13.107
+            name: Test perplexity
+          - type: loss
+            value: 2.573
+            name: Test cross-entropy loss
+---
+
+# marvy-1-14B
+
+**The first open, fine-tuned LLM for the full ServiceNow delivery lifecycle — from business analysis to validation.**
+
+marvy-1-14B is an open-source language model fine-tuned for the complete ServiceNow delivery lifecycle: business analysis, requirements, stakeholder mapping, systems inventory, Solution Design Documents, user stories with acceptance criteria, implementation planning, test cases, and validation. Where general-purpose models treat ServiceNow as one topic among many, marvy is built to draft the actual artifacts a delivery team produces — in the structure and sequence real engagements follow. It is a first-draft specialist, not a consultant replacement, and it is not an agentic or tool-use fine-tune.
+
+It was built by [MainStack](https://huggingface.co/MainStack), a consultancy specializing in ServiceNow Agentic Delivery. marvy is a LoRA SFT fine-tune of [Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) (Apache-2.0), trained on ~1,958 anonymized artifacts from real engagements (~887k tokens), rigorously redacted to zero residual PII per an automated leakage scanner. Its test perplexity of 13.107 was measured on a project- and customer-disjoint held-out split — the model generalizes to unseen work rather than memorizing the training set.
+
+> Released under **Apache-2.0**. Built with Qwen — see `NOTICE`.
+
+## Why marvy-1-14B
+
+- **Drafts the full lifecycle, not just snippets.** Business analysis through validation — the artifacts and sequence real delivery teams actually work in.
+- **OOTB-first and implementation-grade.** Tuned to favor out-of-the-box correctness and produce drafts you can review, not rewrite.
+- **Runs locally and privately.** Merged FP16, a LoRA adapter, and GGUF quants — run it on Apple Silicon via LM Studio or Ollama, with your engagement data never leaving your machine.
+- **Trained on real, anonymized delivery work.** ~1,958 redacted engagement artifacts (~887k tokens), with zero residual PII verified by an automated leakage scanner.
+- **Open and Apache-2.0.** Built on Qwen2.5-14B-Instruct — inspect it, fine-tune it, and deploy it on your own terms.
+
+📖 **Full docs:** [`USAGE.md`](./USAGE.md) (every runtime + OpenCode wiring) ·
+[`VALIDATION.md`](./VALIDATION.md) (prove the fine-tune works) ·
+[`validate.sh`](./validate.sh) (one-command probe harness)
+
+---
+
+## Quick start
+
+### Transformers
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_id = "MainStack/marvy-1-14B"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
+
+SYSTEM = (
+  "You are a senior ServiceNow delivery consultant. You produce precise, "
+  "implementation-grade artifacts: business analyses, requirements, solution "
+  "design documents, user stories with acceptance criteria, test cases, and "
+  "validation reviews. You favor out-of-the-box capabilities, cite concrete "
+  "tables/plugins/sys_ids when relevant, and write in clear professional English."
+)
+
+messages = [
+  {"role": "system", "content": SYSTEM},
+  {"role": "user", "content": "Write a ServiceNow user story with acceptance criteria for SLA escalation on P1 incidents."},
+]
+inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
+out = model.generate(inputs, max_new_tokens=1024, temperature=0.4)
+print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
+```
+
+### vLLM
+
+```bash
+pip install vllm
+vllm serve MainStack/marvy-1-14B
+```
+
+### Ollama (via GGUF)
+
+Use the companion repo [`MainStack/marvy-1-14B-GGUF`](https://huggingface.co/MainStack/marvy-1-14B-GGUF):
+
+```bash
+ollama run hf.co/MainStack/marvy-1-14B-GGUF:Q4_K_M
+```
+
+### MLX (Apple Silicon native)
+
+```bash
+pip install mlx-lm
+python -m mlx_lm generate --model MainStack/marvy-1-14B \
+  --system-prompt "You are a senior ServiceNow delivery consultant..." \
+  --prompt "Draft the Platform Architecture section of an ITSM SDD." \
+  --max-tokens 1024 --temp 0.4
+```
+
+### LoRA-only (apply on top of the base)
+
+If you prefer a tiny adapter (~175 MB) on top of the BF16 base, see [`MainStack/marvy-1-14B-lora`](https://huggingface.co/MainStack/marvy-1-14B-lora).
+
+---
+
+## Intended use
+
+marvy-1-14B is designed to produce implementation-grade first drafts across the ServiceNow delivery lifecycle — accelerating the artifacts a practitioner would otherwise write from scratch, then review and refine. Built for solution architects, business analysts, technical consultants, and project managers. Typical tasks:
+
+| Task family            | What it produces                                                                |
+|------------------------|---------------------------------------------------------------------------------|
+| `business_analysis`    | Structured BA reports from SOWs / discovery notes                               |
+| `requirements_extraction` | Functional/non-functional requirements with acceptance bullets               |
+| `stakeholder_mapping`  | RACI / influence-interest grids from raw notes                                  |
+| `systems_inventory`    | CMDB-shaped systems inventories from architecture inputs                        |
+| `sdd_design`           | Solution Design Document sections (architecture, integrations, data model)      |
+| `story_authoring`      | User stories with crisp acceptance criteria                                     |
+| `implementation_planning` | Story-level implementation plans citing tables/plugins                       |
+| `test_case_generation` | Test cases per story, mapped to acceptance criteria                             |
+| `validation_critique`  | Gap analysis, follow-up questions, assumption checks against source docs        |
+| `delivery_chain`       | Multi-turn: story → implementation → test, end-to-end                           |
+
+### Recommended system prompt
+
+```
+You are a senior ServiceNow delivery consultant. You produce precise, implementation-grade
+artifacts: business analyses, requirements, solution design documents, user stories with
+acceptance criteria, test cases, and validation reviews. You favor out-of-the-box
+capabilities, cite concrete tables/plugins/sys_ids when relevant, and write in clear
+professional English.
+```
+
+### Recommended generation settings
+
+| Use case                    | temperature | top_p | max_new_tokens |
+|-----------------------------|-------------|-------|----------------|
+| Structured artifacts (SDD, stories) | 0.3 – 0.5 | 0.9 | 1024 – 4096 |
+| Exploratory brainstorming   | 0.7 – 0.9   | 0.95  | 1024           |
+| Validation / critique       | 0.2 – 0.4   | 0.9   | 1024 – 2048    |
+
+---
+
+## Training data
+
+> **The training dataset is proprietary to MainStack and is not publicly
+> released.** It is derived from confidential, anonymized client engagement
+> artifacts. The statistics below describe the corpus for transparency; the data
+> itself is not distributed with the model.
+
+| Item | Value |
+|---|---|
+| Source | Anonymized real engagement artifacts (`.md`, `.csv`, `.json`, `.mmd`, `.txt`) |
+| Availability | **Proprietary — not released** |
+| Total records | **1,958** (after schema + exact-dedupe) |
+| Estimated tokens | **~887k** |
+| Splits (project-disjoint) | train 1,359 · val 347 · test 252 |
+| Tasks | 11 task families (see table above) |
+| Multi-turn share | `delivery_chain` (158 records) — story→implementation→test |
+
+### Privacy & redaction
+
+- All customer/partner names → stable aliases (e.g. `Customer-FIN-03`, `Customer-ENERGY-01`).
+- Emails → `user@example.com`; hostnames → `instance.example.service-now.com`; IPs → RFC 5737 range; `key: value` secrets → `[REDACTED]`.
+- Credential/login/VPN files excluded entirely; bulk CMDB dumps >1.5 MB excluded.
+- ServiceNow `sys_id`s and table/plugin names preserved (instance-local, technically valuable, low risk).
+- A leakage scanner asserts **0** residual emails, hostnames, or mapped real names in message content.
+
+### Split integrity
+
+Train / val / test are split **by project**, so no customer appears in more than one split. The largest project is forced into `train` to keep eval honest:
+- val projects: `Customer-ENERGY-01`
+- test projects: `Customer-CHEM-01`, `Customer-FININST-01`
+
+---
+
+## Training procedure
+
+| Setting | Value |
+|---|---|
+| Method | LoRA SFT (QLoRA-style: LoRA on 4-bit base) |
+| Base model | `mlx-community/Qwen2.5-14B-Instruct-4bit` (training) → fused onto `Qwen/Qwen2.5-14B-Instruct` BF16 (release) |
+| Framework | [MLX-LM](https://github.com/ml-explore/mlx-lm) 0.31.3 |
+| Hardware | Apple Silicon (M-series), Metal |
+| Max sequence length | 8,192 |
+| Batch size / grad accum | 1 / 16 (effective batch 16) |
+| Iterations | 350 (~4 epochs over 1,359 train records) |
+| Optimizer | AdamW, cosine decay, warmup 20, lr 1e-4 → 1e-6 |
+| LoRA rank / scale / dropout | 32 / 20.0 / 0.0 |
+| LoRA target keys | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
+| Adapted layers | top 16 transformer layers |
+| Prompt masking | yes — loss computed only on assistant turns |
+| Seed | 42 |
+
+---
+
+## Evaluation
+
+### Fine-tuned vs. base — efficiency on the held-out test set
+
+The cleanest measure of the fine-tune's value is to score the **same base
+model twice** — plain vs. with the marvy adapter — on the **project-disjoint**
+test split (252 records from two customers never seen in training/val), using
+per-token cross-entropy/perplexity on the **assistant tokens only**
+(prompt-masked, the same objective used in training). Lower perplexity = the
+model assigns higher probability to the real, human-authored delivery artifact.
+
+![marvy-1-14B vs base — perplexity by task](./marvy_vs_base_ppl.png)
+
+![How much fine-tuning improved each task](./marvy_improvement.png)
+
+**Overall: perplexity 8.91 → 6.03, a 32.3% reduction** on unseen customers.
+
+| Task | Base ppl | marvy-1-14B ppl | Improvement |
+|---|---:|---:|---:|
+| Systems inventory | 77.07 | 10.53 | **−86.3%** |
+| Requirements extraction | 46.76 | 9.39 | **−79.9%** |
+| Stakeholder mapping | 27.81 | 6.91 | **−75.2%** |
+| Story authoring | 15.38 | 7.86 | **−48.9%** |
+| Validation / critique | 9.72 | 8.23 | −15.3% |
+| Business analysis | 7.14 | 6.66 | −6.6% |
+| SDD design | 4.48 | 4.40 | −1.7% |
+| **Overall** | **8.91** | **6.03** | **−32.3%** |
+
+The gains are largest on **structured, format-heavy artifacts** (inventories,
+requirements, stakeholder registers, stories) where the base model wanders from
+the expected schema; they are smaller on long-form prose (SDD sections, business
+analysis) where the base was already competent. This is the honest, expected
+shape of a domain SFT.
+
+> Notes: the test customers (`Customer-CHEM-01`, `Customer-FININST-01`) appear in
+> neither train nor val, so this reflects generalization, not memorization. The
+> test split happens to cover 7 of the 11 task families. An earlier MLX
+> batch-eval reported aggregate ppl ≈ 13.1 with 2,048-token truncation; the
+> figures above recompute per-task with full assistant-token masking, so the
+> base-vs-marvy **delta** is the result of interest.
+
+Reproduce it yourself: `bash benchmark/run_benchmark.sh` (see
+[`VALIDATION.md`](./VALIDATION.md) for qualitative probes too).
+
+---
+
+## Limitations & known issues
+
+- **Text-only sources.** SOWs/SDDs/workbooks in `.docx/.pptx/.pdf/.xlsx` are not parsed in this build. Coverage of binary-only engagements is therefore thin.
+- **Project concentration.** ~95% of records come from ~12 data-rich projects; the long tail contributes a single case study each. Some task families (e.g. `case_study`, `validation_critique`) are smaller and may exhibit higher variance.
+- **Synthetic instructions.** User prompts are templated paraphrases (3–5 variants per task); assistant outputs are the original human-authored artifacts.
+- **English-only.** The corpus is English.
+- **Not a replacement for a consultant.** Output is first-draft, implementation-grade content that requires expert review before client delivery or production use.
+- **No tool use / function calling fine-tune.** `marvy-1-14B` is a text-completion specialist; agentic tool use is left to the orchestrator.
+- **Hallucination risk on instance-specific facts.** The model will confidently invent `sys_id`s, plugin IDs, and table fields if asked about specifics it has not seen. Always verify against an actual ServiceNow instance.
+- **No safety fine-tune beyond the base.** Inherits Qwen2.5-14B-Instruct safety behavior; no additional RLHF.
+
+---
+
+## License
+
+marvy-1-14B is **dual-licensed** — see [`LICENSING.md`](./LICENSING.md) for the full breakdown:
+
+| Component | License |
+|---|---|
+| **Model weights** (safetensors / GGUF / LoRA) | **Apache-2.0** (`LICENSE`) — inherited from the Qwen2.5-14B-Instruct base; free to use, fine-tune, and redistribute, with `NOTICE` retained. |
+| **MainStack contributions** (model cards, docs, benchmark, charts, training methodology) | **CC-BY-4.0** (`LICENSE-CC-BY-4.0`) — reuse requires attribution to MainStack. |
+
+The model weights are a derivative of **Qwen2.5-14B-Instruct** (Apache-2.0).
+Per Apache-2.0, the weights cannot be placed under a more restrictive license;
+MainStack's protection is the CC-BY-4.0 license on our own authored materials
+plus the mandatory `NOTICE` retention. See `NOTICE` for attribution.
+
+## Attribution
+
+`marvy-1-14B` is free to use, fine-tune, and redistribute under Apache-2.0.
+**If you use marvy-1-14B as a baseline, fine-tune it, distill from it, evaluate
+against it, or otherwise build on it, please credit MainStack** and link back to
+this model:
+
+> Built on / evaluated against **marvy-1-14B** by **MainStack** —
+> https://huggingface.co/MainStack/marvy-1-14B
+
+Concretely, we ask that derivatives and comparisons:
+
+- keep the `NOTICE` file intact (this is **required** by Apache-2.0 §4),
+- name `MainStack/marvy-1-14B` in the model card, paper, or README, and
+- cite the entry below.
+
+Per Apache-2.0, you must also continue to attribute the upstream base model
+(Qwen2.5-14B-Instruct) — see `NOTICE`.
+
+## Citation
+
+If you use marvy-1-14B (as a baseline, a starting point, or in evaluation),
+please cite:
+
+```bibtex
+@software{marvy_1_14b_2026,
+  title     = {marvy-1-14B: An open fine-tuned model for the full ServiceNow delivery lifecycle},
+  author    = {MainStack},
+  year      = {2026},
+  publisher = {Hugging Face},
+  url        = {https://huggingface.co/MainStack/marvy-1-14B},
+  note      = {Fine-tune of Qwen2.5-14B-Instruct},
+  license   = {Apache-2.0}
+}
+
+@misc{qwen2.5,
+  title  = {Qwen2.5: A Party of Foundation Models},
+  author = {Qwen Team},
+  year   = {2024},
+  url    = {https://qwenlm.github.io/blog/qwen2.5/}
+}
+```
+
+```bibtex
+@software{marvy_14b_2026,
+  title  = {marvy-1-14B: A ServiceNow delivery lifecycle fine-tune of Qwen2.5-14B-Instruct},
+  author = {MainStack},
+  year   = {2026},
+  url    = {https://huggingface.co/MainStack/marvy-1-14B},
+  license= {Apache-2.0}
+}
+
+@misc{qwen2.5,
+  title  = {Qwen2.5: A Party of Foundation Models},
+  author = {Qwen Team},
+  year   = {2024},
+  url    = {https://qwenlm.github.io/blog/qwen2.5/}
+}
+```
+
+## Acknowledgements
+
+- **Qwen team** at Alibaba Cloud for the Qwen2.5 family.
+- **Apple MLX team** for `mlx` and `mlx-lm`, enabling native Apple Silicon training.
+- **Hugging Face** for hosting and the surrounding ecosystem.