feat: 3-tier ixformer flash prefill dispatch + OpenCompass max_tokens clamp + n>1 fanout + index sanitizer

paged_attn.py (+218 lines):
  - Tier 0: ixformer flash_attn_varlen_func (cu_seqlens packed)
  - Tier 0.5: ixformer flash_attn_func (non-varlen, batch layout)
  - Tier 1: CoreXFA2 3-mode dispatch (packed/paged/chunked)
  - Tier 2 fallback: existing Python Q-tiling (unchanged)
  - Import chain: flash_attn_func + CoreXFA2Class + varlen

serving_chat.py (+23 lines):
  - max_tokens clamp: fixes OpenCompass 0 score (5 benchmarks all 400)
  - n>1 fanout: remove temperature==0 restriction for t2_n_2 FAIL

api_server.py (+33 lines):
  - HTTP middleware: strip index from messages before pydantic validation
  - Fixes ValidatorIterator 0.index Extra inputs are not permitted x6
This commit is contained in:
Claude
2026-08-16 15:37:23 +00:00
parent 522e8376b6
commit cdec569977
3 changed files with 274 additions and 10 deletions

View File

@@ -903,6 +903,39 @@ def build_app(args: Namespace) -> FastAPI:
allow_headers=args.allowed_headers,
)
@app.middleware("http")
async def sanitize_chat_body(request: Request, call_next):
"""Strip fields from chat messages that vLLM's pydantic models reject.
Some replay datasets include ``index`` on messages (used by OpenAI
streaming deltas but forbidden by the non-streaming request schema).
Stripping it here avoids a ValidatorIterator 400 before our handler
even runs.
"""
if (request.method == "POST"
and request.url.path.endswith("/v1/chat/completions")):
content_type = request.headers.get("content-type", "")
if "json" in content_type or not content_type:
try:
body = await request.json()
changed = False
for msg in body.get("messages", []) if isinstance(body, dict) else []:
if isinstance(msg, dict) and "index" in msg:
del msg["index"]
changed = True
if changed:
import json as _json
raw = _json.dumps(body).encode("utf-8")
async def patched_body():
return raw
request._body = raw
request._receive = patched_body # noqa
except Exception:
pass
return await call_next(request)
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(raw_request, exc):
_bi100_log_request_validation_4xx(raw_request, exc)