295
docs/source/_templates/Model-Deployment-Tutorial-Template.md
Normal file
295
docs/source/_templates/Model-Deployment-Tutorial-Template.md
Normal file
@@ -0,0 +1,295 @@
|
||||
# Technical Documentation Template for Deployment Tutorials Based on the XXX Model
|
||||
|
||||
<p align="center">
|
||||
<a href="Model-Deployment-Tutorial-Template.md"><b>English</b></a> | <a href="Model-Deployment-Tutorial-Template.zh.md"><b>中文</b></a>
|
||||
</p>
|
||||
|
||||
This template is based on deployment tutorials for models such as DeepSeek-V3.2 and Qwen-VL-Dense, and is intended to serve as a reference for technical documentation writing. Users can systematically construct relevant technical documentation by following the guidelines provided in this template.
|
||||
|
||||
## 1 Introduction
|
||||
|
||||
**Content Writing Requirements:**
|
||||
|
||||
- Provide a one-sentence description of the model's basic architecture, core features, and primary application scenarios.
|
||||
- Provide a one-sentence description of the document's purpose and the objectives to be achieved.
|
||||
- Specify the version of vLLM-Ascend used in the document and the version support status of the model.
|
||||
|
||||
**Example 1: Model Introduction**
|
||||
|
||||
DeepSeek-V3.2 is a sparse attention model. Its core architecture is similar to that of DeepSeek-V3.1, but it employs a sparse attention mechanism, aiming to explore and validate optimization solutions for training and inference efficiency in long-context scenarios.
|
||||
|
||||
**Example 2: Document Purpose**
|
||||
|
||||
This document will demonstrate the primary validation steps for the model, including supported features, feature configuration, environment preparation, single-node and multi-node deployment, as well as accuracy and performance evaluation.
|
||||
|
||||
**Example 3: Version Information**
|
||||
|
||||
This document is validated and written based on **vLLM-Ascend v0.13.0**. The current model (XXX) is fully supported in this version, and all **v0.13.0 and later versions** can run stably. To use the latest features (e.g., PD separation, MTP), it is recommended to use the latest release candidate or official version.
|
||||
|
||||
## 2 Supported Features
|
||||
|
||||
This section introduces the features supported by the model, including supported hardware, quantization methods, data parallelism, long-sequence features, etc.
|
||||
|
||||
**Content Writing Requirements:**
|
||||
|
||||
- Present the support status of models and features in a table format.
|
||||
- Or provide cross-references with jump links (recommended).
|
||||
|
||||
**Example 1: Feature Support List**
|
||||
|
||||
| Model Name | Support Status | Remarks | BF16 | Supported Hardware | W8A8 | Chunked Prefill | Automatic Prefix Caching | LoRA | Speculative Decoding | Asynchronous Scheduling | Tensor Parallelism | Pipeline Parallelism | Expert Parallelism | Data Parallelism | Prefill-Decode Separation | Segmented ACL Graph Execution | Full ACL Graph Execution | Max Model Length | MLP Weight Prefetch | Documentation |
|
||||
| ------ | ---------- | ------ | ------ | ---------- | ------ | ------------ | -------------- | ------ | ---------- | ---------- | ---------- | ------------ | ---------- | ---------- | ------------------- | ----------- | ----------- | ------------- | ------------- | ---------- |
|
||||
| DeepSeek V3/3.1 | ✅ | | ✅ | Atlas 800I A2:<br>Minimum card requirement: xx | ✅ | ✅ | ✅ | | ✅ | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 240k | | [DeepSeek-V3.1](../../tutorials/models/DeepSeek-V3.1.md) |
|
||||
| DeepSeek V3.2 | ✅ | | ✅ | Atlas 800I A2:<br>Minimum card requirement: xx | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 160k | ✅ | [DeepSeek-V3.2](../../tutorials/models/DeepSeek-V3.2.md) |
|
||||
| Qwen3 | ✅ | | ✅ | Atlas 800I A2:<br>Minimum card requirement: xx | ✅ | ✅ | ✅ | | | ✅ | ✅ | | | ✅ | | ✅ | ✅ | 128k | ✅ | [Qwen3-Dense](../../tutorials/models/Qwen3-Dense.md) |
|
||||
|
||||
>**Note**: This is a simplified example. Please refer to the complete feature matrix for the full table.
|
||||
|
||||
**Example 2: Reference Citation**
|
||||
|
||||
Please refer to the [Supported Features List](../user_guide/support_matrix/supported_models.md) for the model support matrix.
|
||||
|
||||
Please refer to the [Feature Guide](../user_guide/feature_guide/index.md) for feature configuration information.
|
||||
|
||||
## 3 Prerequisites
|
||||
|
||||
### 3.1 Model Weight
|
||||
|
||||
**Content Writing Requirements:** Describe the hardware resources, software environment, and model files required for deployment.
|
||||
|
||||
**Example:**
|
||||
|
||||
- `DeepSeek-V3.2-Exp-W8A8` (Quantized version): requires 1 Atlas 800 A3 (64G × 16) node or 2 Atlas 800 A2 (64G × 8) nodes. [Model Weight](https://www.modelscope.cn/models/vllm-ascend/DeepSeek-V3.2-Exp-W8A8)
|
||||
- `DeepSeek-V3.2-w8a8` (Quantized version): requires 1 Atlas 800 A3 (64G × 16) node or 2 Atlas 800 A2 (64G × 8) nodes. [Model Weight](https://www.modelscope.cn/models/vllm-ascend/DeepSeek-V3.2-W8A8/)
|
||||
|
||||
It is recommended to download the model weight to a shared directory across multiple nodes.
|
||||
|
||||
### 3.2 Verify Multi-node Communication (Optional)
|
||||
|
||||
**Example:**
|
||||
|
||||
If multi-node deployment is required, please follow the [Verify Multi-node Communication Environment](../installation.md#verify-multi-node-communication) guide for communication verification.
|
||||
|
||||
## 4 Installation
|
||||
|
||||
**Content Writing Requirements:**
|
||||
|
||||
- Provide specific installation steps and commands (parameters should be explained with meaning, value range, units, etc.).
|
||||
- **Version Number Writing Specification:** Prefer using placeholders (values are centrally configured). If a fixed value is used and it differs from the documented validation version, a comment MUST be added stating: "Please replace with your actual version."
|
||||
- Provide verification commands and expected status: guide users to check the installation result by executing commands (e.g., docker ps), specifying success criteria such as status codes or output characteristics.
|
||||
- When content involves multiple hardware series (e.g., A3/A2), the `tab-set` markup syntax must be used to present them in separate tabs,and the tabs should be arranged with the newest models first.
|
||||
|
||||
### 4.1 Docker Image Installation
|
||||
|
||||
**Example:**
|
||||
|
||||
:::::{tab-set}
|
||||
:sync-group: install
|
||||
|
||||
::::{tab-item} A3 series
|
||||
:sync: A3
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3
|
||||
docker run ...
|
||||
```
|
||||
|
||||
::::
|
||||
|
||||
::::{tab-item} A2 series
|
||||
:sync: A2
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
docker run ...
|
||||
```
|
||||
|
||||
::::
|
||||
:::::
|
||||
|
||||
### 4.2 Source Code Installation
|
||||
|
||||
**Example:** Omitted
|
||||
|
||||
## 5 Online Service Deployment
|
||||
|
||||
### 5.1 Single-Node Online Deployment
|
||||
|
||||
**Content Writing Requirements:**
|
||||
|
||||
- Describe the architectural characteristics and applicable scenarios of single-node deployment.
|
||||
- Provide startup command templates and key parameter descriptions.
|
||||
- Provide service verification methods (e.g., curl commands) and expected results, specifying success indicators (e.g., 200 OK).
|
||||
- When content involves multiple hardware series (e.g., A3/A2), the `tab-set` markup syntax must be used to present them in separate tabs,and the tabs should be arranged with the newest models first.
|
||||
- Below the startup command, provide guidance on common issues; if already described in the public FAQ, a direct link may be provided.
|
||||
|
||||
**Example:**
|
||||
|
||||
Single-node deployment completes both Prefill and Decode within the same node, suitable for XXX scenarios.
|
||||
|
||||
Startup Command:
|
||||
|
||||
```bash
|
||||
# Omitted
|
||||
```
|
||||
|
||||
Common Issues Tip: If you encounter XXX issues, please refer to the [Public FAQ](https://docs.vllm.ai/projects/ascend/en/latest/faqs.html) for troubleshooting.
|
||||
|
||||
Service Verification:
|
||||
|
||||
```bash
|
||||
# Omitted
|
||||
```
|
||||
|
||||
Expected Result: Omitted (fill in according to actual output).
|
||||
|
||||
### 5.2 Multi-Node PD Separation Deployment
|
||||
|
||||
**Content Writing Requirements:**
|
||||
|
||||
- Describe the principles of PD separation architecture and applicable scenarios.
|
||||
- Provide startup procedures, key configurations, and **deployment verification instructions**, and indicate performance metrics.
|
||||
- Below the startup command, provide guidance on common issues; if already described in the public FAQ, a direct link may be provided.
|
||||
- When content involves multiple hardware series (e.g., A3/A2), the `tab-set` markup syntax must be used to present them in separate tabs,and the tabs should be arranged with the newest models first.
|
||||
|
||||
**Example:** Omitted
|
||||
|
||||
### 5.3 Special Deployment Modes (Optional)
|
||||
|
||||
**Content Writing Requirements:**
|
||||
|
||||
- If the model features non‑standard deployment modes (e.g., offline batch processing for embedding models, low‑latency online serving for reranker models), the corresponding deployment solutions must be explicitly documented.
|
||||
- Section 5.1 and 5.2 above can be referenced for extension.
|
||||
|
||||
## 6 Functional Verification
|
||||
|
||||
**Content Writing Requirements:**
|
||||
|
||||
- Guide users on how to test the basic functionality of the model through simple interface calls after the service is started.
|
||||
- Provide expected results, specifying success indicators (e.g., HTTP 200, JSON response containing a choices field).
|
||||
|
||||
**Example:**
|
||||
|
||||
After the service is started, the model can be invoked by sending a prompt:
|
||||
|
||||
```shell
|
||||
curl http://<node0_ip>:<port>/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "deepseek_v3.2",
|
||||
"prompt": "The future of AI is",
|
||||
"max_tokens": 50,
|
||||
"temperature": 0
|
||||
}'
|
||||
```
|
||||
|
||||
Expected Result: Omitted (fill in according to actual output).
|
||||
|
||||
## 7 Accuracy Evaluation
|
||||
|
||||
**Content Writing Requirements:** Introduce standardized methods and tools for evaluating model output quality (accuracy). Two accuracy evaluation methods are provided below as examples; alternatively, provide direct links to existing documentation.
|
||||
|
||||
### Using AISBench
|
||||
|
||||
For details, please refer to [Using AISBench](../developer_guide/evaluation/using_ais_bench.md).
|
||||
|
||||
### Using Language Model Evaluation Harness
|
||||
|
||||
Using the `gsm8k` dataset as an example test dataset, run the accuracy evaluation for `DeepSeek-V3.2-W8A8` in online mode.
|
||||
|
||||
1. For `lm_eval` installation, please refer to [Using lm_eval](../developer_guide/evaluation/using_lm_eval.md).
|
||||
2. Run `lm_eval` to execute the accuracy evaluation.
|
||||
|
||||
```shell
|
||||
lm_eval \
|
||||
--model local-completions \
|
||||
--model_args model=/root/.cache/Eco-Tech/DeepSeek-V3.2-w8a8-mtp-QuaRot,base_url=http://127.0.0.1:8000/v1/completions,tokenized_requests=False,trust_remote_code=True \
|
||||
--tasks gsm8k \
|
||||
--output_path ./
|
||||
```
|
||||
|
||||
## 8 Performance Evaluation
|
||||
|
||||
Omitted. Requirements are the same as for Accuracy Evaluation.
|
||||
|
||||
## 9 Performance Tuning
|
||||
|
||||
### 9.1 Recommended Configurations
|
||||
|
||||
**Content Writing Requirements:**
|
||||
|
||||
Provide recommended configurations for three typical scenarios (long context, low latency, high throughput). Clearly state that the configurations are not globally optimal and guide users to perform tuning based on their actual circumstances.
|
||||
|
||||
**Example:**
|
||||
|
||||
> **Note**: The following configurations are validated in specific test environments and are for reference only. The optimal configuration depends on factors such as maximum input/output length, prefix cache hit rate, precision requirements, and deployment machine ratios. It is recommended to refer to Section 9.2 for tuning based on actual conditions.
|
||||
|
||||
#### Table 1: Scenario Overview
|
||||
|
||||
| Scenario | Deployment Mode | *Total NPUs | Weight Version | Key Considerations |
|
||||
|----------|----------------|-------------|----------------|------------------------|
|
||||
| High Throughput<br>(32K context → 1K output) | 1P1D deployment | 16 (A3) | glm5.1w4a8 | For short-sequence high throughput, try adjusting xxx parameters |
|
||||
| Long Context | | | | |
|
||||
| Low Latency | | | | |
|
||||
|
||||
> `*Total NPUs` indicates the total number of NPUs used across all nodes.
|
||||
|
||||
#### Table 2: Detailed Node Configuration
|
||||
|
||||
| Scenario | Configuration | NPUs | TP | DP | Max Num Seqs | Max Num Batched Tokens | Max Model Len | MTP Speculation Num | FUSED_MC2 | EP Switch | FC+CP Switch | Async Scheduling |
|
||||
|----------|---------------|-------|----|----|----|-------------|--------------------|---------------------|-----------|-----------|--------------|------------------|
|
||||
| High Throughput (32K→1K) | Server-P Node / Single Machine | 8 | 8 | 2 | 32 | 4096 | 30k | 3 | Off | On | On | On |
|
||||
| High Throughput (32K→1K) | Server-D Node | 8 | 2 | 8 | 8 | 4096 | 30k | 12 | Off | On | Off | On |
|
||||
| Long Context | Server-P Node / Single Machine | | | | | | | | | | | |
|
||||
| Long Context | Server-D Node | | | | | | | | | | | |
|
||||
| Low Latency | Server-P Node / Single Machine | | | | | | | | | | | |
|
||||
| Low Latency | Server-D Node | | | | | | | | | | | |
|
||||
|
||||
> For complete startup commands and parameter descriptions, please refer to the deployment examples in Chapter 5.
|
||||
|
||||
### 9.2 Tuning Guidelines
|
||||
|
||||
#### 9.2.1 General Tuning Reference
|
||||
|
||||
**Content Writing Requirements:**
|
||||
|
||||
If no special tuning is involved, directly provide a feature combination table and a link to the public performance tuning documentation.
|
||||
|
||||
**Example:**
|
||||
|
||||
Please refer to the [Public Performance Tuning Documentation](../../developer_guide/performance_and_debug/optimization_and_tuning.md) for tuning methods.
|
||||
Please refer to the [Feature Guide](../../user_guide/support_matrix/feature_matrix.md) for detailed feature descriptions.
|
||||
|
||||
#### 9.2.2 Model-Specific Optimizations (Optional)
|
||||
|
||||
**Documentation Requirements:**
|
||||
|
||||
If the model has specific optimizations, summarize the key optimization techniques and tuning experience for this model.
|
||||
|
||||
**Example:**
|
||||
|
||||
#### Optimizations Enabled by Default
|
||||
|
||||
The following optimizations are enabled by default and require no additional configuration:
|
||||
|
||||
| Optimization Technique | Technical Principle | Performance Benefit |
|
||||
| --------- | --------- | --------- |
|
||||
| Rope Optimization | The cos_sin_cache and indexing operations of positional encoding are executed only in the first layer, and subsequent layers reuse them directly | Reduces redundant computation during the decoding phase, accelerating inference |
|
||||
| AddRMSNormQuant Fusion | Merges address-wise multi-scale normalization and quantization operations into a single operator | Optimizes memory access patterns, improving computational efficiency |
|
||||
| Zero-like Elimination | Removes unnecessary zero-tensor operations in Attention forward pass | Reduces memory footprint, improves matrix operation efficiency |
|
||||
| FullGraph Optimization | Captures and replays the entire decoding graph at once using `compilation_config={"cudagraph_mode":"FULL_DECODE_ONLY"}` | Significantly reduces scheduling latency, stabilizes multi-device performance |
|
||||
|
||||
#### Optimizations That Require Explicit Enabling
|
||||
|
||||
| Optimization Technique | Applicable Scenarios | Enablement Method | Technical Principle | Precautions |
|
||||
| --------------------- | -------------------- | ----------------- | ------------------- | ----------- |
|
||||
| FlashComm_v1 | High-concurrency, Tensor Parallelism (TP) scenarios | `export VLLM_ASCEND_ENABLE_FLASHCOMM1=1` | Decomposes traditional Allreduce into Reduce-Scatter and All-Gather, reducing RMSNorm computation dimensions | Threshold protection: Only takes effect when the actual number of tokens exceeds the threshold to avoid performance degradation in low-concurrency scenarios |
|
||||
| Matmul-ReduceScatter Fusion | Large-scale distributed environments | Automatically enabled after enabling FlashComm_v1 | Fuses matrix multiplication and Reduce-Scatter operations to achieve pipelined parallel processing | Same as FlashComm_v1, has threshold protection |
|
||||
| Weight Prefetch | MLP-intensive scenarios (Dense models) | `export VLLM_ASCEND_ENABLE_PREFETCH_MLP=1` | Utilizes vector computation time to prefetch MLP weights into L2 cache in advance | Requires coordination with prefetch buffer size adjustment |
|
||||
|
||||
## 10 FAQ
|
||||
|
||||
**Content Writing Requirements:**
|
||||
|
||||
- Add a note at the beginning of the section: For common environment, installation, and general parameter issues, please refer to the [Public FAQs](https://docs.vllm.ai/projects/ascend/en/latest/faqs.html); this chapter only covers model-specific issues.
|
||||
- For **model-specific issues**, provide the following elements: problem phenomenon description, cause analysis, and solution measures.
|
||||
296
docs/source/_templates/Model-Deployment-Tutorial-Template.zh.md
Normal file
296
docs/source/_templates/Model-Deployment-Tutorial-Template.zh.md
Normal file
@@ -0,0 +1,296 @@
|
||||
# 基于XXX模型部署教程的技术文档模板
|
||||
|
||||
<p align="center">
|
||||
<a href="Model-Deployment-Tutorial-Template.md"><b>English</b></a> | <a href="Model-Deployment-Tutorial-Template.zh.md"><b>中文</b></a>
|
||||
</p>
|
||||
|
||||
本模板基于DeepSeek-V3.2、Qwen-VL-Dense等部署教程，旨在为技术文档撰写提供参考。使用者可遵循模板指引，系统性完成相关技术文档的构建工作。
|
||||
|
||||
## 1 简介
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
- 一句话介绍模型的基本架构、核心特性及主要应用场景。
|
||||
- 一句话写清楚文档要干什么，要达成的目的。
|
||||
- 说明文档使用的vLLM-Ascend版本及模型的版本支持情况。
|
||||
|
||||
**示例1：模型介绍**
|
||||
|
||||
DeepSeek-V3.2 是一种稀疏注意力模型。其主要架构与 DeepSeek-V3.1 类似，但采用了稀疏注意力机制，旨在探索和验证在长上下文场景下训练和推理效率的优化方案。
|
||||
|
||||
**示例2：文档目的**
|
||||
|
||||
本文档将展示模型的主要验证步骤，包括支持的功能、功能配置、环境准备、单节点和多节点部署、准确性和性能评估。
|
||||
|
||||
**示例3：版本信息**
|
||||
|
||||
本文档基于 **vLLM-Ascend v0.13.0** 版本进行验证和编写。当前模型（XXX）在该版本中已完整支持，**v0.13.0 及更高版本**均可稳定运行。如需使用最新特性（如PD分离、MTP等），建议使用最新的候选版本或正式版本。
|
||||
|
||||
## 2 支持的特性
|
||||
|
||||
介绍该模型支持的特性，包括支持的硬件、量化方式、数据并行、长序列特性等。
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
- 采用表格形式，呈现模型和特性的支持情况。
|
||||
- 或提供可跳转的交叉引用（推荐）。
|
||||
|
||||
**示例1：特性支持列表**
|
||||
|
||||
| 模型名称 | 支持状态 | 备注 | BF16 | 支持的硬件 | W8A8 | 分块预填充 | 自动前缀缓存 | LoRA | 推测解码 | 异步调度 | 张量并行 | 流水线并行 | 专家并行 | 数据并行 | Prefill-Decode分离 | 分段式ACL图执行 | 整图ACL图执行 | 最大模型长度 | MLP权重预取 | 文档 |
|
||||
| ------ | ---------- | ------ | ------ | ---------- | ------ | ------------ | -------------- | ------ | ---------- | ---------- | ---------- | ------------ | ---------- | ---------- | ------------------- |----------- | ----------- | ------------- | ------------- | ---------- |
|
||||
| DeepSeek V3/3.1 | ✅ | | ✅ | Atlas 800I A2:<br>最低卡数要求为xx | ✅ | ✅ | ✅ | | ✅ | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 240k | | [DeepSeek-V3.1](../../tutorials/models/DeepSeek-V3.1.md) |
|
||||
| DeepSeek V3.2 | ✅ | | ✅ | Atlas 800I A2:<br>最低卡数要求为xx | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 160k | ✅ | [DeepSeek-V3.2](../../tutorials/models/DeepSeek-V3.2.md)|
|
||||
| Qwen3 | ✅ | | ✅ | Atlas 800I A2:<br>最低卡数要求为xx | ✅ | ✅ | ✅ | | | ✅ | ✅ | | | ✅ | | ✅ | ✅ | 128k | ✅ | [Qwen3-Dense](../../tutorials/models/Qwen3-Dense.md) |
|
||||
|
||||
>**注意**：此为简化示例，完整表格请参考完整特性矩阵。
|
||||
|
||||
**示例2：引用**
|
||||
|
||||
请参考[支持的模型](../user_guide/support_matrix/supported_models.md)，获取模型支持的功能矩阵。
|
||||
|
||||
请参考[特性指南](../user_guide/feature_guide/index.md)获取功能配置信息。
|
||||
|
||||
## 3 前置准备
|
||||
|
||||
### 3.1 模型权重
|
||||
|
||||
**资料写作要求：** 说明部署所需的硬件资源、软件环境和模型文件。
|
||||
|
||||
**示例：**
|
||||
|
||||
- `DeepSeek-V3.2-Exp-W8A8`（量化版）：需要 1 台 Atlas 800 A3（64G × 16）节点或 2 台 Atlas 800 A2（64G × 8）节点。 [模型权重](https://www.modelscope.cn/models/vllm-ascend/DeepSeek-V3.2-Exp-W8A8)
|
||||
- `DeepSeek-V3.2-w8a8`（量化版）：需要 1 台 Atlas 800 A3（64G × 16）节点或 2 台 Atlas 800 A2（64G × 8）节点。 [模型权重](https://www.modelscope.cn/models/vllm-ascend/DeepSeek-V3.2-W8A8/)
|
||||
|
||||
建议将模型权重下载至多节点共享目录。
|
||||
|
||||
### 3.2 验证多节点通信（可选）
|
||||
|
||||
**示例：**
|
||||
|
||||
若需部署多节点环境，请依据[验证多节点通信环境](../installation.md#verify-multi-node-communication)指南进行通信验证。
|
||||
|
||||
## 4 安装
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
- 提供具体的安装步骤与命令（参数需解释含义、取值范围、单位等）。
|
||||
- 版本号书写规范：优先使用占位符（值统一配置）；若使用固定值且该值与文档验证版本不一致，须加注释“请按实际版本替换”。
|
||||
- 提供验证命令及预期状态：指导用户通过执行命令（如 docker ps）检查安装结果，说明成功时的状态码或输出特征。
|
||||
- 当涉及多硬件系列（如 A3/A2 系列）时，须使用`tab-set`标记语法进行分标签呈现,标签顺序按新机型优先排列。
|
||||
|
||||
### 4.1 Docker镜像安装
|
||||
|
||||
**示例：**
|
||||
|
||||
:::::{tab-set}
|
||||
:sync-group: install
|
||||
|
||||
::::{tab-item} A3 series
|
||||
:sync: A3
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3
|
||||
docker run ...
|
||||
```
|
||||
|
||||
::::
|
||||
|
||||
::::{tab-item} A2 series
|
||||
:sync: A2
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
docker run ...
|
||||
```
|
||||
|
||||
::::
|
||||
:::::
|
||||
|
||||
### 4.2 源码安装
|
||||
|
||||
**示例：** 略
|
||||
|
||||
## 5 在线服务化部署（Online service deployment）
|
||||
|
||||
### 5.1 单机在线部署
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
- 说明单机部署的架构特点与适用场景
|
||||
- 提供启动命令模板和关键参数说明
|
||||
- 提供服务验证方法（如 curl 命令）及预期结果，说明成功特征（如 200 OK）。
|
||||
- 当涉及多硬件系列（如 A3/A2 系列）时，须使用`tab-set`标记语法进行分标签呈现,标签顺序按新机型优先排列。
|
||||
- 在启动命令下方提供常见问题指引，如公共FAQ中已有描述可直接链接呈现。
|
||||
|
||||
**示例：**
|
||||
|
||||
单机部署将Prefill与Decode在同一节点内完成，适用于XXX场景。
|
||||
|
||||
启动命令：
|
||||
|
||||
```bash
|
||||
# 略
|
||||
```
|
||||
|
||||
常见问题提示：如遇xxx问题，请参考[公共FAQ](https://docs.vllm.ai/projects/ascend/en/latest/faqs.html)进行检查。
|
||||
|
||||
服务验证：
|
||||
|
||||
```bash
|
||||
# 略
|
||||
```
|
||||
|
||||
预期结果：略（按实际输出书写即可）。
|
||||
|
||||
### 5.2 多机PD分离部署
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
- 说明PD分离架构的原理与适用场景。
|
||||
- 提供启动流程、关键配置及**部署验证说明**，并注明性能指标。
|
||||
- 在启动命令下方提供常见问题指引，如公共FAQ中已有描述可直接链接呈现。
|
||||
- 当涉及多硬件系列（如 A3/A2 系列）时，须使用`tab-set`标记语法进行分标签呈现,标签顺序按新机型优先排列。
|
||||
|
||||
**示例：** 略
|
||||
|
||||
### 5.3 特殊部署形态（可选）
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
- 若模型存在非标准部署形态（如embedding模型的离线批处理、reranker模型的低延迟在线服务等），需在文档中明确体现对应部署方案。
|
||||
- 可参考本章5.1和5.2节进行扩展。
|
||||
|
||||
## 6 功能验证
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
- 指导用户如何在服务启动后，通过简单接口测试模型的基本功能是否正常。
|
||||
- 提供预期结果，说明成功特征（如 HTTP 200、返回包含 choices 字段的 JSON）。
|
||||
|
||||
**示例：**
|
||||
|
||||
服务启动后，即可通过发送提示词来调用模型：
|
||||
|
||||
```shell
|
||||
curl http://<node0_ip>:<port>/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "deepseek_v3.2",
|
||||
"prompt": "The future of AI is",
|
||||
"max_tokens": 50,
|
||||
"temperature": 0
|
||||
}'
|
||||
```
|
||||
|
||||
预期结果：略（按实际输出书写即可）。
|
||||
|
||||
## 7 精度评估
|
||||
|
||||
**资料写作要求：** 介绍评估模型输出质量（精度）的标准化方法及工具，以下提供两种精度评估方法作为示例；或直接链接现有文档进行呈现。
|
||||
|
||||
### AISBench的使用
|
||||
|
||||
详情请参考[Using AISBench](../developer_guide/evaluation/using_ais_bench.md)。
|
||||
|
||||
### Language Model Evaluation Harness的使用
|
||||
|
||||
以`gsm8k`数据集作为测试数据集为例，在线模式下运行`DeepSeek-V3.2-W8A8`的精度评估。
|
||||
|
||||
1. `lm_eval`安装请参考[Using lm_eval](../developer_guide/evaluation/using_lm_eval.md)。
|
||||
2. 运行`lm_eval`执行精度评估。
|
||||
|
||||
```shell
|
||||
lm_eval \
|
||||
--model local-completions \
|
||||
--model_args model=/root/.cache/Eco-Tech/DeepSeek-V3.2-w8a8-mtp-QuaRot,base_url=http://127.0.0.1:8000/v1/completions,tokenized_requests=False,trust_remote_code=True \
|
||||
--tasks gsm8k \
|
||||
--output_path ./
|
||||
```
|
||||
|
||||
## 8 性能评估
|
||||
|
||||
略，要求同精度评估
|
||||
|
||||
## 9 性能调优
|
||||
|
||||
### 9.1推荐配置
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
提供模型在三个典型场景下的推荐配置（长序列、低时延、高吞吐），需明确说明配置的非全局最优性，并引导用户根据实际进行调优。
|
||||
|
||||
**示例：**
|
||||
|
||||
> **说明**：以下配置基于特定测试环境验证，仅作参考。实际最优配置取决于最大输入输出长度、前缀缓存命中率、精度要求、部署机器配比等因素，建议根据实际参考9.2章节进行调优。
|
||||
|
||||
#### 表1：场景概览
|
||||
|
||||
| 场景 | 部署形态 | *总卡数 | 权重版本 | 场景要点 |
|
||||
|------|------|---------|----------|----------|
|
||||
| 高吞吐<br>(32K推1K) | 1P1D部署 | 16（A3） | glm5.1w4a8 | 短序列高吞吐情况下，尝试调整xxx参数 |
|
||||
| 长序列 | | | | |
|
||||
| 低时延 | | | | |
|
||||
|
||||
> `*总卡数` 表示所有节点使用的 NPU 总数。
|
||||
|
||||
#### 表2：节点详细配置
|
||||
|
||||
| 场景 | 配置 | 卡数 | TP | DP | 最大序列数 | 最大批量Token数 | 最大上下文 | MTP投机数 | FUSED_MC2 | EP开关 | FC+CP开关 | 异步调度 |
|
||||
|------|------|------|----|----|----|------|----------|---------|---------------|--------|-------|------|
|
||||
| 高吞吐(32K推1K) | 服务端-P节点/单机 | 8 | 8 | 2 | 32 | 4096 | 30k | 3 | 关 | 开 | 开 | 开 |
|
||||
| 高吞吐(32K推1K) | 服务端-D节点 | 8 | 2 | 8 | 8 | 4096 | 30k | 12 | 关 | 开 | 关 | 开 |
|
||||
| 长序列 | 服务端-P节点/单机 | | | | | | | | | | | |
|
||||
| 长序列 | 服务端-D节点 | | | | | | | | | | | |
|
||||
| 低时延 | 服务端-P节点/单机 | | | | | | | | | | | |
|
||||
| 低时延 | 服务端-D节点 | | | | | | | | | | | |
|
||||
|
||||
> 完整启动命令及参数含义请参考第5章部署示例。
|
||||
|
||||
### 9.2 调优思路
|
||||
|
||||
#### 9.2.1 通用调优参考
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
若不涉及特殊调优可直接给出特性叠加表和公共性能调优文档链接供参考。
|
||||
|
||||
**示例：**
|
||||
|
||||
请参考[公共性能调优文档](../../developer_guide/performance_and_debug/optimization_and_tuning.md)获得调优方法。
|
||||
请参考[特性指南](../../user_guide/support_matrix/feature_matrix.md)获得详细特性说明。
|
||||
|
||||
#### 9.2.2 模型特有优化（可选）
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
若该模型存在特有优化，需总结针对该模型的关键优化技术和调参经验。
|
||||
|
||||
**示例：**
|
||||
|
||||
#### 默认启用的优化
|
||||
|
||||
以下优化默认启用，无需额外配置：
|
||||
|
||||
| 优化技术 | 技术原理 | 性能收益 |
|
||||
| --------- | --------- | --------- |
|
||||
| Rope优化 | 位置编码的cos_sin_cache及索引操作仅在第一层执行，后续层直接复用 | 减少解码阶段重复计算，加速推理 |
|
||||
| AddRMSNormQuant融合 | 将逐地址多尺度归一化与量化操作合并为单算子 | 优化内存访问模式，提升计算效率 |
|
||||
| Zero-like Elimination | 移除Attention前向中的非必要零张量操作 | 减少内存占用，提高矩阵运算效率 |
|
||||
| FullGraph优化 | 通过`compilation_config={"cudagraph_mode":"FULL_DECODE_ONLY"}`将整个解码图一次性捕获重放 | 显著降低调度延迟，稳定多设备性能 |
|
||||
|
||||
#### 需显式开启的优化
|
||||
|
||||
| 优化技术 | 适用场景 | 启用方式 | 技术原理 | 注意事项 |
|
||||
| --------- | --------- | --------- | --------- | --------- |
|
||||
| FlashComm_v1 | 大并发、张量并行(TP)场景 | `export VLLM_ASCEND_ENABLE_FLASHCOMM1=1` | 将传统Allreduce分解为Reduce-Scatter和All-Gather，减少RMSNorm计算维度 | 阈值保护：仅当实际token数超过阈值时生效，避免小并发场景性能倒退|
|
||||
| Matmul-ReduceScatter融合 | 大型分布式环境 | 启用FlashComm_v1后自动开启 | 将矩阵乘法与Reduce-Scatter操作融合，实现流水线并行处理 | 同FlashComm_v1，有阈值保护 |
|
||||
| 权重预取 | MLP密集型场景（Dense模型）| `export VLLM_ASCEND_ENABLE_PREFETCH_MLP=1` | 利用向量计算时间，提前将MLP权重加载到L2 Cache | 需配合预取缓冲区大小调整 |
|
||||
| 异步调度 | 大规模模型、高并发场景 | `--async-scheduling` | 非阻塞任务调度，提升并发处理能力 | 与FullGraph优化协同使用 |
|
||||
|
||||
## 10 FAQ
|
||||
|
||||
**资料写作要求：**
|
||||
|
||||
- 在章节开头添加说明：常见环境、安装、通用参数问题请参考[公共FAQs](https://docs.vllm.ai/projects/ascend/en/latest/faqs.html)；本章仅收录本模型特有疑难问题。
|
||||
- 针对**本模型特有疑难问题** ，提供以下要素：问题现象描述、原因分析、解决措施。
|
||||
@@ -54,5 +54,5 @@
|
||||
</style>
|
||||
|
||||
<div class="notification-bar">
|
||||
<p>You are viewing the latest developer preview docs. <a href="https://vllm-ascend.readthedocs.io/en/v0.9.1-dev">Click here</a> to view docs for the latest stable release(v0.9.1).</p>
|
||||
<p>You are viewing the stable release (v0.23.0) documentation. <a href="https://docs.vllm.ai/projects/ascend/en/latest/">Click here</a> to view the latest developer preview documentation.</p>
|
||||
</div>
|
||||
BIN
docs/source/assets/ci_log_summary.png
Normal file
BIN
docs/source/assets/ci_log_summary.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 226 KiB |
BIN
docs/source/assets/cp/blocktable.png
Normal file
BIN
docs/source/assets/cp/blocktable.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 289 KiB |
BIN
docs/source/assets/cp/chunkedprefill.png
Normal file
BIN
docs/source/assets/cp/chunkedprefill.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 378 KiB |
BIN
docs/source/assets/cp/dcp-decode.png
Normal file
BIN
docs/source/assets/cp/dcp-decode.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 146 KiB |
BIN
docs/source/assets/cp/dcp-prefill.png
Normal file
BIN
docs/source/assets/cp/dcp-prefill.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 324 KiB |
BIN
docs/source/assets/cp/device_world.png
Normal file
BIN
docs/source/assets/cp/device_world.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
BIN
docs/source/assets/cp/head-tail-style.png
Normal file
BIN
docs/source/assets/cp/head-tail-style.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 21 KiB |
BIN
docs/source/assets/cp/overview.png
Normal file
BIN
docs/source/assets/cp/overview.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
BIN
docs/source/assets/cp/pcp-decode.png
Normal file
BIN
docs/source/assets/cp/pcp-decode.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 209 KiB |
BIN
docs/source/assets/cp/pcp-prefill.png
Normal file
BIN
docs/source/assets/cp/pcp-prefill.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 285 KiB |
BIN
docs/source/assets/deployment.png
Normal file
BIN
docs/source/assets/deployment.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
BIN
docs/source/assets/disaggregated_prefill_pull.png
Normal file
BIN
docs/source/assets/disaggregated_prefill_pull.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 198 KiB |
BIN
docs/source/assets/disaggregated_prefill_push.png
Normal file
BIN
docs/source/assets/disaggregated_prefill_push.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 212 KiB |
BIN
docs/source/assets/eplb.png
Normal file
BIN
docs/source/assets/eplb.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
BIN
docs/source/assets/quantization/get_quant_method.png
Normal file
BIN
docs/source/assets/quantization/get_quant_method.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
BIN
docs/source/assets/quantization/quant_algorithm_overview.png
Normal file
BIN
docs/source/assets/quantization/quant_algorithm_overview.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
BIN
docs/source/assets/quantization/quant_method_base_class.png
Normal file
BIN
docs/source/assets/quantization/quant_method_base_class.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 3.5 KiB |
BIN
docs/source/assets/quantization/quant_method_call_flow.png
Normal file
BIN
docs/source/assets/quantization/quant_method_call_flow.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
BIN
docs/source/assets/quantization/quant_methods_overview.png
Normal file
BIN
docs/source/assets/quantization/quant_methods_overview.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
BIN
docs/source/assets/sp_moe.png
Normal file
BIN
docs/source/assets/sp_moe.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 576 KiB |
BIN
docs/source/assets/workflow.png
Normal file
BIN
docs/source/assets/workflow.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
@@ -1,138 +1,543 @@
|
||||
# Maintainers and contributors
|
||||
# Committers and Contributors
|
||||
|
||||
## Maintainers
|
||||
## Committers
|
||||
|
||||
| Name | Github ID | Date |
|
||||
| Name | GitHub ID | Date |
|
||||
|:-----------:|:-----:|:-----:|
|
||||
| Xiyuan Wang| [@wangxiyuan](https://github.com/wangxiyuan) | 2025/01 |
|
||||
| Xiyuan Wang | [@wangxiyuan](https://github.com/wangxiyuan) | 2025/01 |
|
||||
| Yikun Jiang| [@Yikun](https://github.com/Yikun) | 2025/02 |
|
||||
| Yi Gan| [@ganyi1996ppo](https://github.com/ganyi1996ppo) | 2025/02 |
|
||||
| Shoujian Zheng| [@jianzs](https://github.com/jianzs) | 2025/06 |
|
||||
| Wengang Chen | [@ApsarasX](https://github.com/ApsarasX) | 2025/08 |
|
||||
| Mengqing Cao | [@MengqingCao](https://github.com/MengqingCao) | 2025/08 |
|
||||
| Peng Yu| [@paulyu12](https://github.com/paulyu12) | 2025/10 |
|
||||
| Yizhou Liu| [@yiz-liu](https://github.com/yiz-liu) | 2025/10 |
|
||||
| Jinqian Wei | [@weijinqian0](https://github.com/weijinqian0) | 2025/10 |
|
||||
| Chuanyu Qin | [@nalinaly](https://github.com/nalinaly) | 2025/10 |
|
||||
| Jie Wen| [@zzzzwwjj](https://github.com/zzzzwwjj) | 2025/12 |
|
||||
| Chao Lei| [@LCAIZJ](https://github.com/LCAIZJ) | 2025/12 |
|
||||
| JiaXu Liu| [@realliujiaxu](https://github.com/realliujiaxu) | 2025/12 |
|
||||
| HeXiang Wang| [@whx-sjtu](https://github.com/whx-sjtu) | 2026/01 |
|
||||
| LinFeng Yuan| [@linfeng-yuan](https://github.com/linfeng-yuan) | 2026/05 |
|
||||
|
||||
## Contributors
|
||||
<!-- last_commit: 1471b7702767ac351c8955fa286b4913b631bdd7 -->
|
||||
|
||||
vLLM Ascend every release would not have been possible without the following contributors:
|
||||
Every release of vLLM Ascend would not have been possible without the following contributors:
|
||||
|
||||
Updated on 2025-09-03:
|
||||
Updated on 2026-07-31:
|
||||
|
||||
| Number | Contributor | Date | Commit ID |
|
||||
|:------:|:-----------:|:-----:|:---------:|
|
||||
| 117 | [@panchao-hub](https://github.com/panchao-hub) | 2025/8/30 | [7215454](https://github.com/vllm-project/vllm-ascend/commit/7215454de6df78f4f9a49a99c5739f8bb360f5bc) |
|
||||
| 116 | [@lidenghui1110](https://github.com/lidenghui1110) | 2025/8/29 | [600b08f](https://github.com/vllm-project/vllm-ascend/commit/600b08f7542be3409c2c70927c91471e8de33d03) |
|
||||
| 115 | [@NSDie](https://github.com/NSDie) | 2025/8/28 | [1191a64](https://github.com/vllm-project/vllm-ascend/commit/1191a64ae508183d5613711bc98a90250963f83a) |
|
||||
| 114 | [@s-jiayang](https://github.com/s-jiayang) | 2025/8/27 | [6a4ec18](https://github.com/vllm-project/vllm-ascend/commit/6a4ec186e731b9516235f4fd30b5b98227513fe7) |
|
||||
| 113 | [@LookAround0301](https://github.com/LookAround0301) | 2025/8/22 | [e9fb895](https://github.com/vllm-project/vllm-ascend/commit/e9fb895b10cef37ea634f4d4af71686b09ca9f20) |
|
||||
| 112 | [@ZhaoJiangJiang](https://github.com/ZhaoJiangJiang) | 2025/8/22 | [3629bc4](https://github.com/vllm-project/vllm-ascend/commit/3629bc4431d3edb4224761f9036b3bddb16158d6) |
|
||||
| 111 | [@NicholasTao](https://github.com/NicholasTao) | 2025/8/20 | [7bec1a9](https://github.com/vllm-project/vllm-ascend/commit/7bec1a9b9c372785551d45682bf11063ec42b216) |
|
||||
| 110 | [@gameofdimension](https://github.com/gameofdimension) | 2025/8/19 | [27d038d](https://github.com/vllm-project/vllm-ascend/commit/27d038dc663bf550a35a8f15659493b2abefda07) |
|
||||
| 109 | [@liuchenbing](https://github.com/liuchenbing) | 2025/8/19 | [3648d18](https://github.com/vllm-project/vllm-ascend/commit/3648d18e673f15a33a82d6ea95d3a9dd891ff1f5) |
|
||||
| 108 | [@LCAIZJ](https://github.com/LCAIZJ) | 2025/8/18 | [03ca2b2](https://github.com/vllm-project/vllm-ascend/commit/03ca2b26ca9ab6b9a12f021b0595a726ee35e223) |
|
||||
| 107 | [@haojiangzheng](https://github.com/haojiangzheng) | 2025/8/11 | [eb43a47](https://github.com/vllm-project/vllm-ascend/commit/eb43a475f429192e7509e85e28b1c65d5097f373) |
|
||||
| 106 | [@QwertyJack](https://github.com/QwertyJack) | 2025/8/11 | [9c6d108](https://github.com/vllm-project/vllm-ascend/commit/9c6d108330574176f79eea52f989ea6049336af8) |
|
||||
| 105 | [@SlightwindSec](https://github.com/SlightwindSec) | 2025/8/5 | [f3b50c5](https://github.com/vllm-project/vllm-ascend/commit/f3b50c54e8243ad8ccefb9b033277fbdd382a9c4) |
|
||||
| 104 | [@CaveNightingale](https://github.com/CaveNightingale) | 2025/8/4 | [957c7f1](https://github.com/vllm-project/vllm-ascend/commit/957c7f108d5f0aea230220ccdc18d657229e4030) |
|
||||
| 103 | [@underfituu](https://github.com/underfituu) | 2025/8/4 | [e38fab0](https://github.com/vllm-project/vllm-ascend/commit/e38fab011d0b81f3a8e40d9bbe263c283dd4129b) |
|
||||
| 102 | [@yangqinghao-cmss](https://github.com/yangqinghao-cmss) | 2025/8/1 | [99fa0ac](https://github.com/vllm-project/vllm-ascend/commit/99fa0ac882c79ae9282940125b042a44ea422757) |
|
||||
| 101 | [@pjgao](https://github.com/pjgao) | 2025/7/31 | [6192bc9](https://github.com/vllm-project/vllm-ascend/commit/6192bc95c0e47097836e9be1f30f2a0a6fdca088) |
|
||||
| 100 | [@Liccol](https://github.com/Liccol) | 2025/7/31 | [7c90ba5](https://github.com/vllm-project/vllm-ascend/commit/7c90ba5fe8e420b891fdd30df050a33e3767835d) |
|
||||
| 99 | [@1024daniel](https://github.com/1024daniel) | 2025/7/31 | [db310c6](https://github.com/vllm-project/vllm-ascend/commit/db310c6ec97b056296f7c2348b90c1d96d0b562a) |
|
||||
| 98 | [@zhoux77899](https://github.com/zhoux77899) | 2025/7/30 | [4fcca13](https://github.com/vllm-project/vllm-ascend/commit/4fcca137a70c11daa4070ae014288be154715939) |
|
||||
| 97 | [@YuanCheng-coder](https://github.com/YuanCheng-coder) | 2025/7/30 | [34dd24a](https://github.com/vllm-project/vllm-ascend/commit/34dd24adf21fb85a2c413292754b1599832efae2) |
|
||||
| 96 | [@hongfugui](https://github.com/hongfugui) | 2025/7/30 | [1dbb888](https://github.com/vllm-project/vllm-ascend/commit/1dbb8882759e4326f5706f6e610674423376c2f3) |
|
||||
| 95 | [@Irving11-BKN](https://github.com/Irving11-BKN) | 2025/7/29 | [ca8007f](https://github.com/vllm-project/vllm-ascend/commit/ca8007f584141d3a59b2bcbd4f8ba269c9b7e252) |
|
||||
| 94 | [@taoxudonghaha](https://github.com/taoxudonghaha) | 2025/7/29 | [540336e](https://github.com/vllm-project/vllm-ascend/commit/540336edc9db09072a9aaa486fbf7ce625da5b9e) |
|
||||
| 93 | [@loukong33](https://github.com/loukong33) | 2025/7/28 | [1a25b0a](https://github.com/vllm-project/vllm-ascend/commit/1a25b0a2ddb23bf4d731ebac4503efaf237b191f) |
|
||||
| 92 | [@Ronald1995](https://github.com/Ronald1995) | 2025/7/25 | [e561a2c](https://github.com/vllm-project/vllm-ascend/commit/e561a2c6ec4493b490b13a4a9007d8f451ae0d0f) |
|
||||
| 91 | [@ZrBac](https://github.com/ZrBac) | 2025/7/24 | [2ffe051](https://github.com/vllm-project/vllm-ascend/commit/2ffe051859d585df8353d1b9eefb64c44078175a) |
|
||||
| 90 | [@SunnyLee151064](https://github.com/SunnyLee151064) | 2025/7/24 | [34571ea](https://github.com/vllm-project/vllm-ascend/commit/34571ea5ae69529758edf75f0252f86ccb4c7184) |
|
||||
| 89 | [@shiyuan680](https://github.com/shiyuan680) | 2025/7/23 | [ac0bf13](https://github.com/vllm-project/vllm-ascend/commit/ac0bf133f47ead20f18bf71f9be6dbe05fbd218f) |
|
||||
| 88 | [@aidoczh](https://github.com/aidoczh) | 2025/7/21 | [c32eea9](https://github.com/vllm-project/vllm-ascend/commit/c32eea96b73d26268070f57ef98416decc98aff7) |
|
||||
| 87 | [@nuclearwu](https://github.com/nuclearwu) | 2025/7/20 | [54f2b31](https://github.com/vllm-project/vllm-ascend/commit/54f2b311848badc86371d269140e729012a60f2c) |
|
||||
| 86 | [@pkking](https://github.com/pkking) | 2025/7/18 | [3e39d72](https://github.com/vllm-project/vllm-ascend/commit/3e39d7234c0e5c66b184c136c602e87272b5a36e) |
|
||||
| 85 | [@lianyiibo](https://github.com/lianyiibo) | 2025/7/18 | [53d2ea3](https://github.com/vllm-project/vllm-ascend/commit/53d2ea3789ffce32bf3ceb055d5582d28eadc6c7) |
|
||||
| 84 | [@xudongLi-cmss](https://github.com/xudongLi-cmss) | 2025/7/2 | [7fc1a98](https://github.com/vllm-project/vllm-ascend/commit/7fc1a984890bd930f670deedcb2dda3a46f84576) |
|
||||
| 83 | [@ZhengWG](https://github.com/) | 2025/7/7 | [3a469de](https://github.com/vllm-project/vllm-ascend/commit/9c886d0a1f0fc011692090b0395d734c83a469de) |
|
||||
| 82 | [@wm901115nwpu](https://github.com/) | 2025/7/7 | [a2a47d4](https://github.com/vllm-project/vllm-ascend/commit/f08c4f15a27f0f27132f4ca7a0c226bf0a2a47d4) |
|
||||
| 81 | [@Agonixiaoxiao](https://github.com/) | 2025/7/2 | [6f84576](https://github.com/vllm-project/vllm-ascend/commit/7fc1a984890bd930f670deedcb2dda3a46f84576) |
|
||||
| 80 | [@zhanghw0354](https://github.com/zhanghw0354) | 2025/7/2 | [d3df9a5](https://github.com/vllm-project/vllm-ascend/commit/9fb3d558e5b57a3c97ee5e11b9f5dba6ad3df9a5) |
|
||||
| 79 | [@GDzhu01](https://github.com/GDzhu01) | 2025/6/28 | [de256ac](https://github.com/vllm-project/vllm-ascend/commit/b308a7a25897b88d4a23a9e3d583f4ec6de256ac) |
|
||||
| 78 | [@leo-pony](https://github.com/leo-pony) | 2025/6/26 | [3f2a5f2](https://github.com/vllm-project/vllm-ascend/commit/10253449120307e3b45f99d82218ba53e3f2a5f2) |
|
||||
| 77 | [@zeshengzong](https://github.com/zeshengzong) | 2025/6/26 | [3ee25aa](https://github.com/vllm-project/vllm-ascend/commit/192dbbcc6e244a8471d3c00033dc637233ee25aa) |
|
||||
| 76 | [@sharonyunyun](https://github.com/sharonyunyun) | 2025/6/25 | [2dd8666](https://github.com/vllm-project/vllm-ascend/commit/941269a6c5bbc79f6c1b6abd4680dc5802dd8666) |
|
||||
| 75 | [@Pr0Wh1teGivee](https://github.com/Pr0Wh1teGivee) | 2025/6/25 | [c65dd40](https://github.com/vllm-project/vllm-ascend/commit/2fda60464c287fe456b4a2f27e63996edc65dd40) |
|
||||
| 74 | [@xleoken](https://github.com/xleoken) | 2025/6/23 | [c604de0](https://github.com/vllm-project/vllm-ascend/commit/4447e53d7ad5edcda978ca6b0a3a26a73c604de0) |
|
||||
| 73 | [@lyj-jjj](https://github.com/lyj-jjj) | 2025/6/23 | [5cbd74e](https://github.com/vllm-project/vllm-ascend/commit/5177bef87a21331dcca11159d3d1438075cbd74e) |
|
||||
| 72 | [@farawayboat](https://github.com/farawayboat)| 2025/6/21 | [bc7d392](https://github.com/vllm-project/vllm-ascend/commit/097e7149f75c0806774bc68207f0f6270bc7d392)
|
||||
| 71 | [@yuancaoyaoHW](https://github.com/yuancaoyaoHW) | 2025/6/20 | [7aa0b94](https://github.com/vllm-project/vllm-ascend/commit/00ae250f3ced68317bc91c93dc1f1a0977aa0b94)
|
||||
| 70 | [@songshanhu07](https://github.com/songshanhu07) | 2025/6/18 | [5e1de1f](https://github.com/vllm-project/vllm-ascend/commit/2a70dbbdb8f55002de3313e17dfd595e1de1f)
|
||||
| 69 | [@wangyanhui-cmss](https://github.com/wangyanhui-cmss) | 2025/6/12| [40c9e88](https://github.com/vllm-project/vllm-ascend/commit/2a5fb4014b863cee6abc3009f5bc5340c9e88) |
|
||||
| 68 | [@chenwaner](https://github.com/chenwaner) | 2025/6/11 | [c696169](https://github.com/vllm-project/vllm-ascend/commit/e46dc142bf1180453c64226d76854fc1ec696169) |
|
||||
| 67 | [@yzim](https://github.com/yzim) | 2025/6/11 | [aaf701b](https://github.com/vllm-project/vllm-ascend/commit/4153a5091b698c2270d160409e7fee73baaf701b) |
|
||||
| 66 | [@Yuxiao-Xu](https://github.com/Yuxiao-Xu) | 2025/6/9 | [6b853f1](https://github.com/vllm-project/vllm-ascend/commit/6b853f15fe69ba335d2745ebcf14a164d0bcc505) |
|
||||
| 65 | [@ChenTaoyu-SJTU](https://github.com/ChenTaoyu-SJTU) | 2025/6/7 | [20dedba](https://github.com/vllm-project/vllm-ascend/commit/20dedba5d1fc84b7ae8b49f9ce3e3649389e2193) |
|
||||
| 64 | [@zxdukki](https://github.com/zxdukki) | 2025/6/7 | [87ebaef](https://github.com/vllm-project/vllm-ascend/commit/87ebaef4e4e519988f27a6aa378f614642202ecf) |
|
||||
| 63 | [@sdmyzlp](https://github.com/sdmyzlp) | 2025/6/7 | [3640c60](https://github.com/vllm-project/vllm-ascend/commit/3640c60b0eb4d4cb104e20bfa406d3f1d17920a7) |
|
||||
| 62 | [@weijinqian0](https://github.com/weijinqian0) | 2025/6/7 | [e9ada68](https://github.com/vllm-project/vllm-ascend/commit/e9ada685ece798f9fe0d4a287e3f5246a8a7207b) |
|
||||
| 61 | [@hahazhky](https://github.com/hahazhky) | 2025/6/6 | [0b12c2a](https://github.com/vllm-project/vllm-ascend/commit/0b12c2acf7d9fd192beebebf662298067d9a5435) |
|
||||
| 60 | [@depeng1994](https://github.com/depeng1994) | 2025/6/6 | [6b094a2](https://github.com/vllm-project/vllm-ascend/commit/6b094a2bd49a8a41eb3647568b2d9e5b337db81f) |
|
||||
| 59 | [@David9857](https://github.com/David9857) | 2025/6/5 | [78431b3](https://github.com/vllm-project/vllm-ascend/commit/78431b34694dfa3c8f54ed7cc626660318557927) |
|
||||
| 58 | [@momo609](https://github.com/momo609) | 2025/6/5 | [908a851](https://github.com/vllm-project/vllm-ascend/commit/908a851a776cfd9051cc062119e6ec481561c6f7) |
|
||||
| 57 | [@zhangxinyuehfad](https://github.com/zhangxinyuehfad) | 2025/6/5 | [7737aaa](https://github.com/vllm-project/vllm-ascend/commit/7737aaa40f699b233a35fb61e908b687adc1e2e5) |
|
||||
| 56 | [@NINGBENZHE](https://github.com/NINGBENZHE) | 2025/6/3 | [6ec64a3](https://github.com/vllm-project/vllm-ascend/commit/6ec64a3f9686df65b5a23a41aa301e669db19099) |
|
||||
| 55 | [@XWFAlone](https://github.com/XWFAlone) | 2025/5/30 | [3442fbd](https://github.com/vllm-project/vllm-ascend/commit/3442fbdb235b4c6d72c2bc64a49707a7bd89958e) |
|
||||
| 54 | [@YisongJiang](https://github.com/YisongJiang) | 2025/5/29 | [90afaf6](https://github.com/vllm-project/vllm-ascend/commit/90afaf6306f680307462becf3c78585737579851) |
|
||||
| 53 | [@ponix-j](https://github.com/ponix-j) | 2025/5/23 | [df58fb8](https://github.com/vllm-project/vllm-ascend/commit/df58fb80eee24139fc61c495be3ce79cf81b3f73) |
|
||||
| 52 | [@ttanzhiqiang](https://github.com/ttanzhiqiang) | 2025/5/23 | [dc6172e](https://github.com/vllm-project/vllm-ascend/commit/dc6172efd3860ce95b40a7b3e93611f875f06d40) |
|
||||
| 51 | [@yangpuPKU](https://github.com/yangpuPKU) | 2025/5/23 | [46df67a](https://github.com/vllm-project/vllm-ascend/commit/46df67a5e9ab73fade08cbb2d8c0155cee7316d1) |
|
||||
| 50 | [@wonderful199082](https://github.com/wonderful199082) | 2025/5/20 | [5cf9ff1](https://github.com/vllm-project/vllm-ascend/commit/5cf9ff18e91b0b7031c258d71a257b8e24689763) |
|
||||
| 49 | [@22dimensions](https://github.com/22dimensions) | 2025/5/17 | [a8730e7](https://github.com/vllm-project/vllm-ascend/commit/a8730e7a3c4ac6c4b39a5946c943252fdea6cce5) |
|
||||
| 48 | [@cxcxflying](https://github.com/cxcxflying) | 2025/5/13 | [e564470](https://github.com/vllm-project/vllm-ascend/commit/e56447033889ca95df512208cab22ef832bfdf07) |
|
||||
| 47 | [@NeverRaR](https://github.com/NeverRaR) | 2025/5/12 | [efabd72](https://github.com/vllm-project/vllm-ascend/commit/efabd722eb757e49aa309c173bbec91ca8c4ced1) |
|
||||
| 46 | [@chris668899](https://github.com/chris668899) | 2025/5/8 | [6c02088](https://github.com/vllm-project/vllm-ascend/commit/6c020883a8332b5c519f4f6502733edd9b391c2b) |
|
||||
| 45 | [@sunbaosong](https://github.com/sunbaosong) | 2025/5/6 | [d6bfae8](https://github.com/vllm-project/vllm-ascend/commit/d6bfae8eeebedf677b643b712d367a3a69c9cce4) |
|
||||
| 44 | [@ApsarasX](https://github.com/ApsarasX) | 2025/4/29 | [87975fa](https://github.com/vllm-project/vllm-ascend/commit/87975fa058fe3f90d204ded42a08989a8dcb413e) |
|
||||
| 43 | [@zouyida2052](https://github.com/zouyida2052) | 2025/4/28 | [b9528e6](https://github.com/vllm-project/vllm-ascend/commit/b9528e6ecdc417cf444e55a0ce4a2bafdef0ea3b) |
|
||||
| 42 | [@ZhengJun9](https://github.com/ZhengJun9) | 2025/4/28 | [1791113](https://github.com/vllm-project/vllm-ascend/commit/17911138c90d78a76bd691e9dcb56763db35b19f) |
|
||||
| 41 | [@linfeng-yuan](https://github.com/linfeng-yuan) | 2025/4/28 | [2204e4d](https://github.com/vllm-project/vllm-ascend/commit/2204e4d08f8e10cf9c30154a14eaa5ca956c2acd) |
|
||||
| 40 | [@jianzs](https://github.com/jianzs) | 2025/4/27 | [fa4a5d9](https://github.com/vllm-project/vllm-ascend/commit/fa4a5d980e8845a88b9162cf169f0a5ab230f8a5) |
|
||||
| 39 | [@fakeYan](https://github.com/fakeYan) | 2025/4/23 | [05bdcbe](https://github.com/vllm-project/vllm-ascend/commit/05bdcbeae47c7fcb9b1c30cad059abf1d40b5421) |
|
||||
| 38 | [@RongRongStudio](https://github.com/RongRongStudio) | 2025/4/22 | [848e041](https://github.com/vllm-project/vllm-ascend/commit/848e041a54732c923660dd02daf8e9bf439736a2) |
|
||||
| 37 | [@paulyu12](https://github.com/paulyu12) | 2025/4/17 | [697908f](https://github.com/vllm-project/vllm-ascend/commit/697908f5cd7c65a3a917ec1a962b0886efc98c7e) |
|
||||
| 36 | [@heartStrive1998](https://github.com/heartStrive1998) | 2025/4/16 | [2f15503](https://github.com/vllm-project/vllm-ascend/commit/2f155039dc3997640854daef469bbf0cb77dc6ed) |
|
||||
| 35 | [@eeethenQ](https://github.com/eeethenQ) | 2025/4/15 | [44a8301](https://github.com/vllm-project/vllm-ascend/commit/44a8301424ded94dae83e13b837f5bfc0a1bfc15) |
|
||||
| 34 | [@wxsIcey](https://github.com/wxsIcey) | 2025/4/10 | [d05ea17](https://github.com/vllm-project/vllm-ascend/commit/d05ea17427b82a506b97409a7de8359f18f565f7) |
|
||||
| 33 | [@yx0716](https://github.com/yx0716) | 2025/4/8 | [5d62393](https://github.com/vllm-project/vllm-ascend/commit/5d6239306be9b0f5ac6dbaa137048c372a92ff20) |
|
||||
| 32 | [@celestialli](https://github.com/celestialli) | 2025/4/7 | [2b765dc](https://github.com/vllm-project/vllm-ascend/commit/2b765dcc4974b1bafc26ff5da817ce7e652f0eb0) |
|
||||
| 31 | [@hfadzxy](https://github.com/hfadzxy) | 2025/3/30 | [7beb433](https://github.com/vllm-project/vllm-ascend/commit/7beb4339dc8047af9ef64db1d0a8c59ddbb3709f) |
|
||||
| 30 | [@wuhuikx](https://github.com/wuhuikx) | 2025/3/28 | [57a84bb](https://github.com/vllm-project/vllm-ascend/commit/57a84bb7befeaa0dc62aa35fa406e4d6affbfcca) |
|
||||
| 29 | [@zzzzwwjj](https://github.com/zzzzwwjj) | 2025/3/28 | [12390af](https://github.com/vllm-project/vllm-ascend/commit/12390af075962456ecc8233d8dcce7064b75f390) |
|
||||
| 28 | [@ganyi1996ppo](https://github.com/ganyi1996ppo) | 2025/3/28 | [27e86b9](https://github.com/vllm-project/vllm-ascend/commit/27e86b993a6a810d818143ec9dbfc439a419fa77) |
|
||||
| 27 | [@ZhengZhenyu](https://github.com/ZhengZhenyu) | 2025/3/26 | [0b5a964](https://github.com/vllm-project/vllm-ascend/commit/0b5a9643fd6c3240d7ede669e37209d7ff433841) |
|
||||
| 26 | [@baifanxxx](https://github.com/baifanxxx) | 2025/3/26 | [1225052](https://github.com/vllm-project/vllm-ascend/commit/122505208ff6284f409846ca7294f4a4b9883285) |
|
||||
| 25 | [@rjg-lyh](https://github.com/rjg-lyh) | 2025/3/13 | [6512470](https://github.com/vllm-project/vllm-ascend/commit/65124705fb39d4cc2c94c80254421e067a82fe50) |
|
||||
| 24 | [@xiemingda-1002](https://github.com/xiemingda-1002) | 2025/3/12 | [59ea23d](https://github.com/vllm-project/vllm-ascend/commit/59ea23d0d394879d7f33de6fd22242539b9c3cc5) |
|
||||
| 23 | [@yiz-liu](https://github.com/yiz-liu) | 2025/3/11 | [0db6670](https://github.com/vllm-project/vllm-ascend/commit/0db6670bfab8cb1d84c9e7270df0a1d42d6ce7ca) |
|
||||
| 22 | [@new-TonyWang](https://github.com/new-TonyWang) | 2025/3/11 | [dfb4e23](https://github.com/vllm-project/vllm-ascend/commit/dfb4e23e9d820ac992a071c123bbe983c7b01b2e) |
|
||||
| 21 | [@mengwei805](https://github.com/mengwei805) | 2025/3/6 | [8fcf3d1](https://github.com/vllm-project/vllm-ascend/commit/8fcf3d1704084626db35c5dc82ade446508598d4) |
|
||||
| 20 | [@baymax591](https://github.com/baymax591) | 2025/2/28 | [e8131b9](https://github.com/vllm-project/vllm-ascend/commit/e8131b99cf199f50a304e6e6fb125a1b95bcc92b) |
|
||||
| 19 | [@dependabot](https://github.com/dependabot) | 2025/2/27 | [a5564ed](https://github.com/vllm-project/vllm-ascend/commit/a5564ed5d8fd9818936a22d9ea35951a27513b4c) |
|
||||
| 18 | [@shink](https://github.com/shink) | 2025/2/27 | [6aed833](https://github.com/vllm-project/vllm-ascend/commit/6aed83335cbe92fd0b8ef07c28966a753d012ccb) |
|
||||
| 17 | [@wwfu109](https://github.com/wwfu109) | 2025/2/27 | [b074047](https://github.com/vllm-project/vllm-ascend/commit/b07404766bdaf6e3cebc5cb0aba89a247501302e) |
|
||||
| 16 | [@kunpengW-code](https://github.com/kunpengW-code) | 2025/2/26 | [ca807ce](https://github.com/vllm-project/vllm-ascend/commit/ca807ce49ed64aa89242f5ae29b9862a77648b45) |
|
||||
| 15 | [@Yaphets24](https://github.com/Yaphets24) | 2025/2/22 | [d0b3cb4](https://github.com/vllm-project/vllm-ascend/commit/d0b3cb4fa79d5fc7f8245a3c68885ce1fa030ba4) |
|
||||
| 14 | [@noemotiovon](https://github.com/noemotiovon) | 2025/2/21 | [202b39a](https://github.com/vllm-project/vllm-ascend/commit/202b39a38c2869b0ecc3df486550fb555a2eb0c0) |
|
||||
| 13 | [@SidaoY](https://github.com/SidaoY) | 2025/2/18 | [718c763](https://github.com/vllm-project/vllm-ascend/commit/718c7638555d12cd43ea2a9e497e185778b68595) |
|
||||
| 12 | [@ShiyaNiu](https://github.com/ShiyaNiu) | 2025/2/17 | [36ea38f](https://github.com/vllm-project/vllm-ascend/commit/36ea38fde56437ff1745bd95cd8d9e02a6578d38) |
|
||||
| 11 | [@ji-huazhong](https://github.com/ji-huazhong) | 2025/2/12 | [c8b57d1](https://github.com/vllm-project/vllm-ascend/commit/c8b57d10b24efcd9b4fadeb66cfbf66aa3dd5f82) |
|
||||
| 10 | [@Angazenn](https://github.com/Angazenn) | 2025/2/11 | [7637759](https://github.com/vllm-project/vllm-ascend/commit/7637759056028839c74960d9cfd3ce6275ee5d35) |
|
||||
| 9 | [@whx-sjtu](https://github.com/whx-sjtu) | 2025/2/7 | [8fc5dc9](https://github.com/vllm-project/vllm-ascend/commit/8fc5dc966aaf4e174d1ec0d1902c40289411ec0e) |
|
||||
| 8 | [@zouyida2002](https://github.com/zouyida2002) | 2025/2/7 | [4495fc6](https://github.com/vllm-project/vllm-ascend/commit/4495fc68389e3fb1ef14534c202948931e38446b) |
|
||||
| 7 | [@hw_whx](https://github.com/hw_whx) | 2025/2/7 | [7d16772](https://github.com/vllm-project/vllm-ascend/commit/7d1677263bc6628ade33bb780455e0f6e5b9b27a) |
|
||||
| 6 | [@MengqingCao](https://github.com/MengqingCao) | 2025/2/6 | [7d9ae22](https://github.com/vllm-project/vllm-ascend/commit/7d9ae22ecb6dc3ea4e720e5109cf46e1ae7da730) |
|
||||
| 5 | [@Potabk](https://github.com/Potabk) | 2025/2/6 | [8cb5615](https://github.com/vllm-project/vllm-ascend/commit/8cb5615fb010b34c2f4f89e03e6257bfee851f86) |
|
||||
| 4 | [@wangxiyuan](https://github.com/wangxiyuan) | 2025/2/6 | [a48b9ad](https://github.com/vllm-project/vllm-ascend/commit/a48b9addefd292af523644411d4ff4142dd4bc66) |
|
||||
| 3 | [@shen-shanshan](https://github.com/shen-shanshan) | 2025/2/6 | [bfccf73](https://github.com/vllm-project/vllm-ascend/commit/bfccf739e2fe121b54d9b198c2ec205a9379190e) |
|
||||
| 2 | [@Yikun](https://github.com/Yikun) | 2025/2/5 | [d5e7756](https://github.com/vllm-project/vllm-ascend/commit/d5e7756028bd5884ade96b654555c375770a2f64) |
|
||||
| 1 | [@simon-mo](https://github.com/simon-mo) | 2025/1/29 | [eb28342](https://github.com/vllm-project/vllm-ascend/commit/eb283428ddc17207b6866118f9bc15454b5b8801) |
|
||||
| 512 | [@ella1107](https://github.com/ella1107) | 2026/07/30 | [71d136c](https://github.com/vllm-project/vllm-ascend/commit/71d136ca26c4692db6b1e5bd07f507b73f09887b) |
|
||||
| 511 | [@axx-ty911](https://github.com/axx-ty911) | 2026/07/30 | [7890de8](https://github.com/vllm-project/vllm-ascend/commit/7890de8eb72ff669c4962ff379090a14dc19b21a) |
|
||||
| 510 | [@ffggs](https://github.com/ffggs) | 2026/07/29 | [5cf0420](https://github.com/vllm-project/vllm-ascend/commit/5cf0420ab3a4ca5f8ba2a707e4e5c31eef1cc968) |
|
||||
| 509 | [@jiaqi-lee](https://github.com/jiaqi-lee) | 2026/07/25 | [5225ae2](https://github.com/vllm-project/vllm-ascend/commit/5225ae20fb09aa41cc3278b294aad23c9dc33f22) |
|
||||
| 508 | [@Oranbean258](https://github.com/Oranbean258) | 2026/07/23 | [e6f3f49](https://github.com/vllm-project/vllm-ascend/commit/e6f3f4954602a3d2fc41857e0c61806b9c8840ff) |
|
||||
| 507 | [@lyur01](https://github.com/lyur01) | 2026/07/23 | [f9602a1](https://github.com/vllm-project/vllm-ascend/commit/f9602a13c52359fe1b56dc8318436d31f03012ba) |
|
||||
| 506 | [@zhangjiale-zjl](https://github.com/zhangjiale-zjl) | 2026/07/23 | [9d03a3a](https://github.com/vllm-project/vllm-ascend/commit/9d03a3a682d196cb534885a7cbdf86ef0b0694aa) |
|
||||
| 505 | [@aisong1988](https://github.com/aisong1988) | 2026/07/22 | [c926c21](https://github.com/vllm-project/vllm-ascend/commit/c926c213cbf2ca0c1d13d4bdaea97d39a5748b15) |
|
||||
| 504 | [@q664171689](https://github.com/q664171689) | 2026/07/18 | [ebad45e](https://github.com/vllm-project/vllm-ascend/commit/ebad45e3666f8453e045ade27d02b47a7fe1669c) |
|
||||
| 503 | [@qijiajin](https://github.com/qijiajin) | 2026/07/18 | [d42948a](https://github.com/vllm-project/vllm-ascend/commit/d42948ac2c1cba80fe30acd0bd61e4991beb58d2) |
|
||||
| 502 | [@Wyz-134](https://github.com/Wyz-134) | 2026/07/17 | [9b8423e](https://github.com/vllm-project/vllm-ascend/commit/9b8423e15a836f1b011b61aece5305459f939c32) |
|
||||
| 501 | [@zh98530](https://github.com/zh98530) | 2026/07/17 | [a272c88](https://github.com/vllm-project/vllm-ascend/commit/a272c88a5c6560c2ef655b05970a05b895d342e4) |
|
||||
| 500 | [@singzhou](https://github.com/singzhou) | 2026/07/15 | [f924703](https://github.com/vllm-project/vllm-ascend/commit/f924703d7d97a90c5d72d58bf32865501b7f87c0) |
|
||||
| 499 | [@ZhangwenTaoHW](https://github.com/ZhangwenTaoHW) | 2026/07/13 | [b89a491](https://github.com/vllm-project/vllm-ascend/commit/b89a491644cd02a16a5b17c1a124cec254866559) |
|
||||
| 498 | [@zhaochuang001](https://github.com/zhaochuang001) | 2026/07/11 | [7be596c](https://github.com/vllm-project/vllm-ascend/commit/7be596cd3fa93c45a4ef4ff5384d0c852659d83c) |
|
||||
| 497 | [@Liuchenbing-2026](https://github.com/Liuchenbing-2026) | 2026/07/10 | [123c5cc](https://github.com/vllm-project/vllm-ascend/commit/123c5ccc78d3aabe3f793223e096ee5e97606a20) |
|
||||
| 496 | [@KadenZhang3321](https://github.com/KadenZhang3321) | 2026/07/08 | [d17a99c](https://github.com/vllm-project/vllm-ascend/commit/d17a99cd51d535c4026ffb596bafa0a23fa87064) |
|
||||
| 495 | [@TYYTao](https://github.com/TYYTao) | 2026/07/06 | [79928c2](https://github.com/vllm-project/vllm-ascend/commit/79928c2ff1b5521d7f57c58233c87782ec77afc1) |
|
||||
| 494 | [@ShySummer](https://github.com/ShySummer) | 2026/07/05 | [9154baa](https://github.com/vllm-project/vllm-ascend/commit/9154baadc9dbac857fe197d03bccb1731664f3df) |
|
||||
| 493 | [@tyy0829](https://github.com/tyy0829) | 2026/07/05 | [a0219ac](https://github.com/vllm-project/vllm-ascend/commit/a0219ac14caad375c74793539ad230688b58f9d6) |
|
||||
| 492 | [@xinhai9906](https://github.com/xinhai9906) | 2026/07/04 | [f81dd51](https://github.com/vllm-project/vllm-ascend/commit/f81dd51845185d18ea26cfbef2042f9b3bc0de2e) |
|
||||
| 491 | [@shiqiangA](https://github.com/shiqiangA) | 2026/07/03 | [b5fbfff](https://github.com/vllm-project/vllm-ascend/commit/b5fbfffe1dc29261c3c8e0778808cfffb19a95fb) |
|
||||
| 490 | [@LebudiPrince](https://github.com/LebudiPrince) | 2026/07/03 | [76d338a](https://github.com/vllm-project/vllm-ascend/commit/76d338a722521e813e92bd89513181a73ea8b895) |
|
||||
| 489 | [@Sunwish](https://github.com/Sunwish) | 2026/07/03 | [bb2666d](https://github.com/vllm-project/vllm-ascend/commit/bb2666df29c4cddab74fd340103f18bd5bf2bb9f) |
|
||||
| 488 | [@czydyy](https://github.com/czydyy) | 2026/07/02 | [ee8bbbd](https://github.com/vllm-project/vllm-ascend/commit/ee8bbbd5f15958f03c870c3b98af745c02af3fd6) |
|
||||
| 487 | [@RainyStone](https://github.com/RainyStone) | 2026/06/30 | [52f9bb0](https://github.com/vllm-project/vllm-ascend/commit/52f9bb06e0b5682c1b694efeb9a9eb6882935f3c) |
|
||||
| 486 | [@garrygale](https://github.com/garrygale) | 2026/06/30 | [49a4720](https://github.com/vllm-project/vllm-ascend/commit/49a47208b61ab7e7252e48a472a8389c1c07b686) |
|
||||
| 485 | [@guxin108](https://github.com/guxin108) | 2026/06/27 | [0e982ed](https://github.com/vllm-project/vllm-ascend/commit/0e982ed5479470c4f55a6eac0dc96affbf704116) |
|
||||
| 484 | [@AJF-cmd](https://github.com/AJF-cmd) | 2026/06/27 | [43e8b0a](https://github.com/vllm-project/vllm-ascend/commit/43e8b0a92e3e40b65aabeb9d8c263a51a785056e) |
|
||||
| 483 | [@shihan-lin168](https://github.com/shihan-lin168) | 2026/06/27 | [1178b73](https://github.com/vllm-project/vllm-ascend/commit/1178b733b940c4ea90ef96dfac3d825a5ec3e59b) |
|
||||
| 482 | [@jiajinzhu2](https://github.com/jiajinzhu2) | 2026/06/27 | [463a6a1](https://github.com/vllm-project/vllm-ascend/commit/463a6a1d065d246918e4cb4319e07ae63a2d2c9e) |
|
||||
| 481 | [@Alex-stack-hub](https://github.com/Alex-stack-hub) | 2026/06/26 | [8cc7bcd](https://github.com/vllm-project/vllm-ascend/commit/8cc7bcd4c35ab7a4fbd50b36c362b136df7c0d95) |
|
||||
| 480 | [@wangzhishenghw](https://github.com/wangzhishenghw) | 2026/06/26 | [8fd6a0c](https://github.com/vllm-project/vllm-ascend/commit/8fd6a0c2c39d5931e62edc2f342c9dad94d3701d) |
|
||||
| 479 | [@delwen123](https://github.com/delwen123) | 2026/06/26 | [17f14dd](https://github.com/vllm-project/vllm-ascend/commit/17f14dd4c194375b619fe1b348a4f0d71ef0e33b) |
|
||||
| 478 | [@Alexzhang369](https://github.com/Alexzhang369) | 2026/06/25 | [db4cc43](https://github.com/vllm-project/vllm-ascend/commit/db4cc43041407553675ff4a35d1b5758129fc73e) |
|
||||
| 477 | [@TallMessiWu](https://github.com/TallMessiWu) | 2026/06/25 | [419755b](https://github.com/vllm-project/vllm-ascend/commit/419755b4fbb07d36baa947d78cd94cb31931451d) |
|
||||
| 476 | [@bowgneo](https://github.com/bowgneo) | 2026/06/25 | [e2f1028](https://github.com/vllm-project/vllm-ascend/commit/e2f1028f6da43130adfda5ce655c4a290659470b) |
|
||||
| 475 | [@yongfuFang](https://github.com/yongfuFang) | 2026/06/25 | [55c27ec](https://github.com/vllm-project/vllm-ascend/commit/55c27ec8b97fb67b8b67802bd53b8db3760e9468) |
|
||||
| 474 | [@wzx0726](https://github.com/wzx0726) | 2026/06/24 | [a08abea](https://github.com/vllm-project/vllm-ascend/commit/a08abea79ddb62c0fee047f1f09f0e0c6aa236a0) |
|
||||
| 473 | [@recky-c](https://github.com/recky-c) | 2026/06/24 | [5bd4e39](https://github.com/vllm-project/vllm-ascend/commit/5bd4e3950c2c1df9e8d4eabf6accaf6f071eef74) |
|
||||
| 472 | [@liuhanhui](https://github.com/liuhanhui) | 2026/06/22 | [2c88d67](https://github.com/vllm-project/vllm-ascend/commit/2c88d67f465c6578c1d653c2523f7d52d8966cba) |
|
||||
| 471 | [@1478931959](https://github.com/1478931959) | 2026/06/24 | [eb81619](https://github.com/vllm-project/vllm-ascend/commit/eb816194a8d2ce1f4048b4037bb4b6e8a269d307) |
|
||||
| 470 | [@RICHARDNAN](https://github.com/RICHARDNAN) | 2026/06/23 | [9788211](https://github.com/vllm-project/vllm-ascend/commit/9788211ce67a18aa2f76bc3bcb176e378edc13a8) |
|
||||
| 469 | [@LostFox11](https://github.com/LostFox11) | 2026/06/23 | [4467a23](https://github.com/vllm-project/vllm-ascend/commit/4467a23d084ee8c497e663d9caf85c5329851a78) |
|
||||
| 468 | [@yjyang62](https://github.com/yjyang62) | 2026/06/22 | [92eeab2](https://github.com/vllm-project/vllm-ascend/commit/92eeab2fe9a73394d2118de3c8459d69cc9d8e62) |
|
||||
| 467 | [@erase266](https://github.com/erase266) | 2026/06/22 | [682cc2b](https://github.com/vllm-project/vllm-ascend/commit/682cc2b938446b73a67e9ddfcc5ca2203a3f8088) |
|
||||
| 466 | [@boes129](https://github.com/boes129) | 2026/06/22 | [2539c01](https://github.com/vllm-project/vllm-ascend/commit/2539c015aa9754951e99c8478424b96bb329b8c5) |
|
||||
| 465 | [@zhiyu-wa](https://github.com/zhiyu-wa) | 2026/06/17 | [c9a3538](https://github.com/vllm-project/vllm-ascend/commit/c9a3538cfa7a45d3b5f4ebd55161051332fa3cca) |
|
||||
| 464 | [@learning-sketch](https://github.com/learning-sketch) | 2026/06/17 | [ea6e8d2](https://github.com/vllm-project/vllm-ascend/commit/ea6e8d2200fd5fcdd0dcf2e374c4bae60170031a) |
|
||||
| 463 | [@He1pa](https://github.com/He1pa) | 2026/06/17 | [3449318](https://github.com/vllm-project/vllm-ascend/commit/344931832c3f7fb14ce7fe3f8d5c9d8a7b61e2fa) |
|
||||
| 462 | [@x1314aq](https://github.com/x1314aq) | 2026/06/16 | [324dc45](https://github.com/vllm-project/vllm-ascend/commit/324dc45ffbada090b48fb2fc0d69ce681e002490) |
|
||||
| 461 | [@lHrHenry233](https://github.com/lHrHenry233) | 2026/06/16 | [fad6464](https://github.com/vllm-project/vllm-ascend/commit/fad646478dff693840cd2e3d67098cc8b51142a6) |
|
||||
| 460 | [@Dawn952](https://github.com/Dawn952) | 2026/06/16 | [e099eeb](https://github.com/vllm-project/vllm-ascend/commit/e099eeb11ad3247768f3e199e622dded55c99c72) |
|
||||
| 459 | [@2416602906](https://github.com/2416602906) | 2026/06/16 | [79ed380](https://github.com/vllm-project/vllm-ascend/commit/79ed38084b500af6e3945fee023dfefa67f8cce8) |
|
||||
| 458 | [@AceCoder0](https://github.com/AceCoder0) | 2026/06/15 | [f482403](https://github.com/vllm-project/vllm-ascend/commit/f4824036f0cedac5732f404b686aba140a634a9e) |
|
||||
| 457 | [@zhangtao0408](https://github.com/zhangtao0408) | 2026/06/13 | [849a92e](https://github.com/vllm-project/vllm-ascend/commit/849a92e60b1e7b27036d90e0e5f5002c880fb845) |
|
||||
| 456 | [@MINGJING-WU](https://github.com/MINGJING-WU) | 2026/06/12 | [7d768b5](https://github.com/vllm-project/vllm-ascend/commit/7d768b5cedd598d6d368fef9a2c087f7c1fe8759) |
|
||||
| 455 | [@luobicangqiong](https://github.com/luobicangqiong) | 2026/06/12 | [11e0040](https://github.com/vllm-project/vllm-ascend/commit/11e004072710415322f5fd504f2bf846142abcb4) |
|
||||
| 454 | [@Inference1](https://github.com/Inference1) | 2026/06/12 | [e6a9d71](https://github.com/vllm-project/vllm-ascend/commit/e6a9d71d0400e89c0d16409d35fe6bd6c24ad180) |
|
||||
| 453 | [@evan-ai-arg](https://github.com/evan-ai-arg) | 2026/06/11 | [971d50b](https://github.com/vllm-project/vllm-ascend/commit/971d50b3f982fd2ba7d9ffdee35e985fc54cd808) |
|
||||
| 452 | [@robellliu-dev](https://github.com/robellliu-dev) | 2026/06/10 | [bef69f9](https://github.com/vllm-project/vllm-ascend/commit/bef69f9f4ca72093270b79d6b0e37464d87e3228) |
|
||||
| 451 | [@li1how](https://github.com/li1how) | 2026/06/10 | [9914ffa](https://github.com/vllm-project/vllm-ascend/commit/9914ffa27058165634dd9f46d37dc75debe8ce10) |
|
||||
| 450 | [@goodgoodname](https://github.com/goodgoodname) | 2026/06/10 | [ee88e35](https://github.com/vllm-project/vllm-ascend/commit/ee88e356a0703e74cd6499b470f740b319d7dcb5) |
|
||||
| 449 | [@baolongsun](https://github.com/baolongsun) | 2026/06/10 | [fcf5a1d](https://github.com/vllm-project/vllm-ascend/commit/fcf5a1d549896070d9baf2e58240503a4ca24fd3) |
|
||||
| 448 | [@AuroraEmiya](https://github.com/AuroraEmiya) | 2026/06/10 | [55d5bd1](https://github.com/vllm-project/vllm-ascend/commit/55d5bd15974e98a7f813bf3e2a480039461f721b) |
|
||||
| 447 | [@SOMEONEUNSEEN](https://github.com/SOMEONEUNSEEN) | 2026/06/09 | [77ff286](https://github.com/vllm-project/vllm-ascend/commit/77ff28652b329796450f5448c7ab51b054697535) |
|
||||
| 446 | [@zouzy5137](https://github.com/zouzy5137) | 2026/06/08 | [80cd596](https://github.com/vllm-project/vllm-ascend/commit/80cd5964286f2a394e3e6b3d1f7581ce901c812a) |
|
||||
| 445 | [@nofushanquan](https://github.com/nofushanquan) | 2026/06/08 | [2b4a9da](https://github.com/vllm-project/vllm-ascend/commit/2b4a9daa66e78e156d72b2c7eb36f82ccef3735a) |
|
||||
| 444 | [@Fishermanykx](https://github.com/Fishermanykx) | 2026/06/08 | [f2cac7e](https://github.com/vllm-project/vllm-ascend/commit/f2cac7e1f9d295b611a50918d9996b7f94ad1d4a) |
|
||||
| 443 | [@muziyuhui666](https://github.com/muziyuhui666) | 2026/06/05 | [bf6450e](https://github.com/vllm-project/vllm-ascend/commit/bf6450e8792ee4189929651ae7f7b538fa0aedc7) |
|
||||
| 442 | [@ljy19911228](https://github.com/ljy19911228) | 2026/06/05 | [b6afa87](https://github.com/vllm-project/vllm-ascend/commit/b6afa8769a357b47ee6f33d5acbe83173a19a6d0) |
|
||||
| 441 | [@cfq0](https://github.com/cfq0) | 2026/06/04 | [00517cf](https://github.com/vllm-project/vllm-ascend/commit/00517cfb0326bad05e55afc434b912ed9bf4ede7) |
|
||||
| 440 | [@zqs164](https://github.com/zqs164) | 2026/06/03 | [9784bca](https://github.com/vllm-project/vllm-ascend/commit/9784bca3da77450efec1a0883ef91ae98a84d73e) |
|
||||
| 439 | [@yilunh998](https://github.com/yilunh998) | 2026/06/03 | [d7bca90](https://github.com/vllm-project/vllm-ascend/commit/d7bca90301672a14bc9044b4956746c105d55f05) |
|
||||
| 438 | [@nomewang](https://github.com/nomewang) | 2026/06/03 | [fcb30be](https://github.com/vllm-project/vllm-ascend/commit/fcb30bebed4bc493238f2c432844b1f1be16ae6d) |
|
||||
| 437 | [@czc-unac](https://github.com/czc-unac) | 2026/06/03 | [d7d0d29](https://github.com/vllm-project/vllm-ascend/commit/d7d0d29ae5664ea87564ec1f1c4b65be043ec293) |
|
||||
| 436 | [@1-Y-C](https://github.com/1-Y-C) | 2026/06/03 | [4335bd2](https://github.com/vllm-project/vllm-ascend/commit/4335bd2561bdd597475da109ba1fb4d2aa929278) |
|
||||
| 435 | [@ztzx3156](https://github.com/ztzx3156) | 2026/06/02 | [25d8b60](https://github.com/vllm-project/vllm-ascend/commit/25d8b60844e31b185ee217242c0ba07d80c04962) |
|
||||
| 434 | [@Yuli-yx](https://github.com/Yuli-yx) | 2026/06/02 | [c6b853a](https://github.com/vllm-project/vllm-ascend/commit/c6b853acb1be8cb5b219009ac91f71f5f0022137) |
|
||||
| 433 | [@U1stRsouland](https://github.com/U1stRsouland) | 2026/06/02 | [7b7475f](https://github.com/vllm-project/vllm-ascend/commit/7b7475fcde68fd1acf7eaec044f3f59f2ed974a2) |
|
||||
| 432 | [@KaiMa-endeavour](https://github.com/KaiMa-endeavour) | 2026/06/02 | [d525e72](https://github.com/vllm-project/vllm-ascend/commit/d525e72a609843e66afc1d14159bc93aee1ed1e7) |
|
||||
| 431 | [@Bybbbb11](https://github.com/Bybbbb11) | 2026/06/02 | [58c59ca](https://github.com/vllm-project/vllm-ascend/commit/58c59ca5666394c614e955b71f82247c27991b6f) |
|
||||
| 430 | [@zhangc10](https://github.com/zhangc10) | 2026/06/01 | [8b2ec42](https://github.com/vllm-project/vllm-ascend/commit/8b2ec42febdfe53f57b9fcf18dd6f5dd9b2b1f86) |
|
||||
| 429 | [@jyoung6652](https://github.com/jyoung6652) | 2026/06/01 | [d9d9e76](https://github.com/vllm-project/vllm-ascend/commit/d9d9e76912c4553423b512a4a88d981d4ecbf904) |
|
||||
| 428 | [@Biuapha](https://github.com/Biuapha) | 2026/06/01 | [c50023e](https://github.com/vllm-project/vllm-ascend/commit/c50023e347505badaa1c72d8f123d87a70be8d14) |
|
||||
| 427 | [@vladimirevmenoff](https://github.com/vladimirevmenoff) | 2026/05/30 | [928fa1d](https://github.com/vllm-project/vllm-ascend/commit/928fa1d862a99230e3cfc69c394dcb8a01c334be) |
|
||||
| 426 | [@ningjingbengxiaohai](https://github.com/ningjingbengxiaohai) | 2026/05/30 | [2a77209](https://github.com/vllm-project/vllm-ascend/commit/2a77209accf9a49845716cf79d4227bb9237ef32) |
|
||||
| 425 | [@xszbuaa](https://github.com/xszbuaa) | 2026/05/29 | [3d0fafd](https://github.com/vllm-project/vllm-ascend/commit/3d0fafda9a0e46ad275ca2e062936d658f9d319a) |
|
||||
| 424 | [@ZRICHARD9527](https://github.com/ZRICHARD9527) | 2026/05/28 | [7372661](https://github.com/vllm-project/vllm-ascend/commit/737266135913650ca27e6a9726492e99764d225d) |
|
||||
| 423 | [@xuchi-0808](https://github.com/xuchi-0808) | 2026/05/28 | [1133733](https://github.com/vllm-project/vllm-ascend/commit/11337339cc3de74042da731e249f2a0cd10ced2f) |
|
||||
| 422 | [@CXY-Katrina](https://github.com/CXY-Katrina) | 2026/05/28 | [158635c](https://github.com/vllm-project/vllm-ascend/commit/158635cda8fb8de6d008038665b1730e46ca9052) |
|
||||
| 421 | [@internel-error](https://github.com/internel-error) | 2026/05/27 | [a61e7fd](https://github.com/vllm-project/vllm-ascend/commit/a61e7fd3af6529386a489a28848623f250fabac9) |
|
||||
| 420 | [@zhao-stack](https://github.com/zhao-stack) | 2026/05/25 | [59517e8](https://github.com/vllm-project/vllm-ascend/commit/59517e88c211bb75036dcbe4b463553e371de768) |
|
||||
| 419 | [@hzx55906](https://github.com/hzx55906) | 2026/05/25 | [526141c](https://github.com/vllm-project/vllm-ascend/commit/526141cf32ddb29246ed0bff05f87e07a2e90f90) |
|
||||
| 418 | [@ltdo111](https://github.com/ltdo111) | 2026/05/23 | [2331144](https://github.com/vllm-project/vllm-ascend/commit/2331144f07d590e29a2fab5d8e9b2d323b052107) |
|
||||
| 417 | [@Shelleyaaa](https://github.com/Shelleyaaa) | 2026/05/22 | [7726528](https://github.com/vllm-project/vllm-ascend/commit/7726528ec34cec8a72931e65215096e3792cbf4a) |
|
||||
| 416 | [@nanxingMy](https://github.com/nanxingMy) | 2026/05/22 | [c88eac1](https://github.com/vllm-project/vllm-ascend/commit/c88eac1206a13a6bce0f7783b3be066fa514de39) |
|
||||
| 415 | [@MosCloud](https://github.com/MosCloud) | 2026/05/22 | [20e338b](https://github.com/vllm-project/vllm-ascend/commit/20e338bd7c1267799dcf611acee025f37cdc9535) |
|
||||
| 414 | [@Tian-Fantasea](https://github.com/Tian-Fantasea) | 2026/05/21 | [364e665](https://github.com/vllm-project/vllm-ascend/commit/364e6657077044d189d3c2996796a34b0fc9f611) |
|
||||
| 413 | [@panther-zhu](https://github.com/panther-zhu) | 2026/05/21 | [68a4db5](https://github.com/vllm-project/vllm-ascend/commit/68a4db5554475d8e413f13d84016b86f5d2c18b7) |
|
||||
| 412 | [@lizy124](https://github.com/lizy124) | 2026/05/21 | [958daf8](https://github.com/vllm-project/vllm-ascend/commit/958daf83205ecad0afec3784be0b2a8b38725cd8) |
|
||||
| 411 | [@lihaofei-2026](https://github.com/lihaofei-2026) | 2026/05/20 | [c03c1ce](https://github.com/vllm-project/vllm-ascend/commit/c03c1ce422fb011261ffb397386f6aabb198fe8d) |
|
||||
| 410 | [@yuhongming-2026](https://github.com/yuhongming-2026) | 2026/05/19 | [eb7e9b0](https://github.com/vllm-project/vllm-ascend/commit/eb7e9b0f10814e251ae4d7ebc1c87e1ff6d1ba95) |
|
||||
| 409 | [@mccube2000](https://github.com/mccube2000) | 2026/05/19 | [8f5962b](https://github.com/vllm-project/vllm-ascend/commit/8f5962baf97a650dda174af1f0fe5036cdf8e969) |
|
||||
| 408 | [@ZeroFadeAway](https://github.com/ZeroFadeAway) | 2026/05/18 | [8dced20](https://github.com/vllm-project/vllm-ascend/commit/8dced2017ebcf80bf2430043092e011ab795bd2e) |
|
||||
| 407 | [@gaozihao-shy](https://github.com/gaozihao-shy) | 2026/05/18 | [d857777](https://github.com/vllm-project/vllm-ascend/commit/d857777d9ce709877203ebe139d58c3b888b7ee0) |
|
||||
| 406 | [@quancs](https://github.com/quancs) | 2026/05/16 | [a45cdf9](https://github.com/vllm-project/vllm-ascend/commit/a45cdf9b99fb8b931269be14532323cdbf89ec7b) |
|
||||
| 405 | [@vvaen](https://github.com/vvaen) | 2026/05/15 | [ac357ed](https://github.com/vllm-project/vllm-ascend/commit/ac357ed4bbc91c6ad16ba028aa0a76066940d0f4) |
|
||||
| 404 | [@wenjinhust](https://github.com/wenjinhust) | 2026/05/14 | [41e4c70](https://github.com/vllm-project/vllm-ascend/commit/41e4c7043432c65a70cf0478668b234967c60eec) |
|
||||
| 403 | [@ccc000-cell](https://github.com/ccc000-cell) | 2026/05/12 | [540749d](https://github.com/vllm-project/vllm-ascend/commit/540749d35548c13ab75e278991f445cdb3df8c91) |
|
||||
| 402 | [@AlanisZomeg](https://github.com/AlanisZomeg) | 2026/05/12 | [12a3490](https://github.com/vllm-project/vllm-ascend/commit/12a34900052a4d410fcc0707b857b8a1d1743ded) |
|
||||
| 401 | [@zzzzzmeng](https://github.com/zzzzzmeng) | 2026/05/08 | [4b3a2af](https://github.com/vllm-project/vllm-ascend/commit/4b3a2af7170936a0fc1505b85ba246886d433d39) |
|
||||
| 400 | [@Maybe2191](https://github.com/Maybe2191) | 2026/05/07 | [66cc615](https://github.com/vllm-project/vllm-ascend/commit/66cc615237aa0eceaca70fd0f492d83bf506c926) |
|
||||
| 399 | [@ChefWu551](https://github.com/ChefWu551) | 2026/05/07 | [ba074eb](https://github.com/vllm-project/vllm-ascend/commit/ba074eb4bf095264b2d1280447281633ab66e112) |
|
||||
| 398 | [@weixinAc](https://github.com/weixinAc) | 2026/05/06 | [e7eb844](https://github.com/vllm-project/vllm-ascend/commit/e7eb844494a45b2bee2df8e5c6de01bf9d6d01fd) |
|
||||
| 397 | [@Lin-Qingyang-Alec](https://github.com/Lin-Qingyang-Alec) | 2026/05/06 | [b3196cc](https://github.com/vllm-project/vllm-ascend/commit/b3196cc80113402d5aad1045abb5de200315b22c) |
|
||||
| 396 | [@chen-commits](https://github.com/chen-commits) | 2026/05/02 | [a5ec266](https://github.com/vllm-project/vllm-ascend/commit/a5ec26693eec9af9a92b89a18e7111507819e45b) |
|
||||
| 395 | [@alex101-ops](https://github.com/alex101-ops) | 2026/04/30 | [89b739a](https://github.com/vllm-project/vllm-ascend/commit/89b739a31e9b92492c9cebd566a08f872609c39b) |
|
||||
| 394 | [@lihaokun-2026](https://github.com/lihaokun-2026) | 2026/04/22 | [c7fde3b](https://github.com/vllm-project/vllm-ascend/commit/c7fde3b068b3c3223edafba491a7de93f88d39e3) |
|
||||
| 393 | [@KlyzhenkoVadim](https://github.com/KlyzhenkoVadim) | 2026/04/22 | [4b6a559](https://github.com/vllm-project/vllm-ascend/commit/4b6a559872b93ba8b3a66a2504fae09647c5bd5a) |
|
||||
| 392 | [@csoulnd](https://github.com/csoulnd) | 2026/04/22 | [fe47bf3](https://github.com/vllm-project/vllm-ascend/commit/fe47bf31e92535ffcf4df19d463ba84daf97523c) |
|
||||
| 391 | [@baoxuebin-2026](https://github.com/baoxuebin-2026) | 2026/04/22 | [1209583](https://github.com/vllm-project/vllm-ascend/commit/120958309849b36bc617e822cbf2ae384328389b) |
|
||||
| 390 | [@anning-2026](https://github.com/anning-2026) | 2026/04/22 | [c7f6d64](https://github.com/vllm-project/vllm-ascend/commit/c7f6d649dce58444d257e4f083c875423b4ebabe) |
|
||||
| 389 | [@semiba11er](https://github.com/semiba11er) | 2026/04/21 | [87200ac](https://github.com/vllm-project/vllm-ascend/commit/87200ac0b270bf5be555a4a4d8fb57e8d2cb89f4) |
|
||||
| 388 | [@linsheng1](https://github.com/linsheng1) | 2026/04/21 | [84bddfd](https://github.com/vllm-project/vllm-ascend/commit/84bddfdccc9224dbb5af6bbd14db299ebc7d0b4d) |
|
||||
| 387 | [@donpromax](https://github.com/donpromax) | 2026/04/21 | [756029b](https://github.com/vllm-project/vllm-ascend/commit/756029b50dcce61d82d08291ae8dc5f9f88b2cba) |
|
||||
| 386 | [@zzzzzz198](https://github.com/zzzzzz198) | 2026/04/17 | [3cc9bce](https://github.com/vllm-project/vllm-ascend/commit/3cc9bce0f423513474559bf51f55cbb767bffd4a) |
|
||||
| 385 | [@triomino](https://github.com/triomino) | 2026/04/17 | [a837599](https://github.com/vllm-project/vllm-ascend/commit/a83759948ddadf2b2e9915cb34ce05edcf06ddab) |
|
||||
| 384 | [@skf-1999](https://github.com/skf-1999) | 2026/04/17 | [b7f92f3](https://github.com/vllm-project/vllm-ascend/commit/b7f92f37007b1eba5889618d4549c39d2e4b6ab0) |
|
||||
| 383 | [@orangeBoxun](https://github.com/orangeBoxun) | 2026/04/17 | [7c73dba](https://github.com/vllm-project/vllm-ascend/commit/7c73dba4dd6108f4f3d8b391e7dc9c0fe8806b25) |
|
||||
| 382 | [@freyfwt](https://github.com/freyfwt) | 2026/04/17 | [fed5c02](https://github.com/vllm-project/vllm-ascend/commit/fed5c02cd9afd88eb9a2b2fa953d5b3f3da3abfd) |
|
||||
| 381 | [@Goalina](https://github.com/Goalina) | 2026/04/16 | [69eb7ef](https://github.com/vllm-project/vllm-ascend/commit/69eb7efcbec29893b60187aa475402bead9ecb1f) |
|
||||
| 380 | [@yangjiuhua](https://github.com/yangjiuhua) | 2026/04/15 | [73aac48](https://github.com/vllm-project/vllm-ascend/commit/73aac4818165dac8b2bdd49bb5b3b121e80ea895) |
|
||||
| 379 | [@sunshine202600](https://github.com/sunshine202600) | 2026/04/15 | [b414cff](https://github.com/vllm-project/vllm-ascend/commit/b414cff937ff1c8f18a0f08fdcbd063fbd96bf79) |
|
||||
| 378 | [@SumrainChan](https://github.com/SumrainChan) | 2026/04/14 | [69c4499](https://github.com/vllm-project/vllm-ascend/commit/69c4499180c63ae578b6728a47545082c843e811) |
|
||||
| 377 | [@justice-dance](https://github.com/justice-dance) | 2026/04/14 | [9e87895](https://github.com/vllm-project/vllm-ascend/commit/9e87895a8b0d8052b301309395a1afc083faf9a3) |
|
||||
| 376 | [@z00918512](https://github.com/z00918512) | 2026/04/13 | [2f4d55e](https://github.com/vllm-project/vllm-ascend/commit/2f4d55e7e88ca3ff8960bccac749eb380f0c423d) |
|
||||
| 375 | [@csoulnd](https://github.com/csoulnd) | 2026/04/08 | [2dbfe46](https://github.com/vllm-project/vllm-ascend/commit/2dbfe465d522e8793072577673723b0c46805305) |
|
||||
| 374 | [@SijieFu](https://github.com/SijieFu) | 2026/04/03 | [223f647](https://github.com/vllm-project/vllm-ascend/commit/223f647bdeee3381dd07d066926c4db5879f6c7e) |
|
||||
| 373 | [@lcfenglinwan](https://github.com/lcfenglinwan) | 2026/04/03 | [0d768aa](https://github.com/vllm-project/vllm-ascend/commit/0d768aa9434e947fc6e15a5721c90877be325c76) |
|
||||
| 372 | [@AyiStar](https://github.com/AyiStar) | 2026/04/03 | [0fccd72](https://github.com/vllm-project/vllm-ascend/commit/0fccd729be06d12f5b3902be1e3af5541c3143c5) |
|
||||
| 371 | [@Cztk5380](https://github.com/Cztk5380) | 2026/03/30 | [7465131](https://github.com/vllm-project/vllm-ascend/commit/7465131074cddba670efa02cc484eba7d5c9bda7) |
|
||||
| 370 | [@Milkyyx](https://github.com/Milkyyx) | 2026/03/29 | [11e0583](https://github.com/vllm-project/vllm-ascend/commit/11e0583474434de15a8968b7fe76a8d5da40f6e7) |
|
||||
| 369 | [@yangsonglin13](https://github.com/yangsonglin13) | 2026/03/27 | [5ae3ded](https://github.com/vllm-project/vllm-ascend/commit/5ae3ded92f8fca335bad1c269a43d2c5f8fb5ae8) |
|
||||
| 368 | [@YangShuai52](https://github.com/YangShuai52) | 2026/03/27 | [c53c3f5](https://github.com/vllm-project/vllm-ascend/commit/c53c3f58180dcd50fd485926e81532151bb19035) |
|
||||
| 367 | [@YangShuai52](https://github.com/YangShuai52) | 2026/03/26 | [505e2bd](https://github.com/vllm-project/vllm-ascend/commit/505e2bddf5c0fe32f2ae0543fa1f7fd6adecab3e) |
|
||||
| 366 | [@PHOEBEMOON0802](https://github.com/PHOEBEMOON0802) | 2026/03/26 | [55c6800](https://github.com/vllm-project/vllm-ascend/commit/55c680073d07ee65b6683bf5b49039141089a877) |
|
||||
| 365 | [@lenghuixing0330](https://github.com/lenghuixing0330) | 2026/03/26 | [1a7cbdb](https://github.com/vllm-project/vllm-ascend/commit/1a7cbdbaad308ac02f8fce47bb6cbf29dda76e0c) |
|
||||
| 364 | [@HarpsealCC](https://github.com/HarpsealCC) | 2026/03/26 | [fe7097f](https://github.com/vllm-project/vllm-ascend/commit/fe7097f3c663af83e39c19358c184913b10c56a3) |
|
||||
| 363 | [@cvSoldier](https://github.com/cvSoldier) | 2026/03/26 | [71ac6d4](https://github.com/vllm-project/vllm-ascend/commit/71ac6d4f7de55866355351359f4bdb7ead935159) |
|
||||
| 362 | [@zhijie-os](https://github.com/zhijie-os) | 2026/03/25 | [3e07206](https://github.com/vllm-project/vllm-ascend/commit/3e072069e53da0a002229a85c8596f33c4be834b) |
|
||||
| 361 | [@Wangbei25](https://github.com/Wangbei25) | 2026/03/25 | [2db12ed](https://github.com/vllm-project/vllm-ascend/commit/2db12ed935da1b31bcc7865c2d08c841651b2d38) |
|
||||
| 360 | [@894743926](https://github.com/894743926) | 2026/03/25 | [cfe2487](https://github.com/vllm-project/vllm-ascend/commit/cfe2487082b7dac71701cfcf486b265bea5142f0) |
|
||||
| 359 | [@GoMarck](https://github.com/GoMarck) | 2026/03/25 | [17da966](https://github.com/vllm-project/vllm-ascend/commit/17da96658f0b53a7e9b5932e64ced69a334f035c) |
|
||||
| 358 | [@drizzlezyk](https://github.com/drizzlezyk) | 2026/03/24 | [5487946](https://github.com/vllm-project/vllm-ascend/commit/54879467c41784a446aa5b486a391d9bfbf488fa) |
|
||||
| 357 | [@liuhy1213-cell](https://github.com/liuhy1213-cell) | 2026/03/23 | [fb283b5](https://github.com/vllm-project/vllm-ascend/commit/fb283b5820effe930d7f60952aca48177d710e94) |
|
||||
| 356 | [@ZhuQi-seu](https://github.com/ZhuQi-seu) | 2026/03/23 | [e942b62](https://github.com/vllm-project/vllm-ascend/commit/e942b62d742ebc5bf128e85bc086d728df8d4935) |
|
||||
| 355 | [@ksiyuan](https://github.com/ksiyuan) | 2026/03/20 | [a16c991](https://github.com/vllm-project/vllm-ascend/commit/a16c99141b0830240eeff0cbe01bfc3c833c62fb) |
|
||||
| 354 | [@idouba](https://github.com/idouba) | 2026/03/20 | [f39f566](https://github.com/vllm-project/vllm-ascend/commit/f39f566e22b87ee75bd1205f982e4255a882c3a4) |
|
||||
| 353 | [@yesyue-w](https://github.com/yesyue-w) | 2026/03/20 | [c860535](https://github.com/vllm-project/vllm-ascend/commit/c860535246cc751b6be7d1da2092e4380013598c) |
|
||||
| 352 | [@jiangmengyu18](https://github.com/jiangmengyu18) | 2026/03/18 | [305820f](https://github.com/vllm-project/vllm-ascend/commit/305820f1a982ed9597932778891b5da64ecccae9) |
|
||||
| 351 | [@SparrowMu](https://github.com/SparrowMu) | 2026/03/18 | [fb8e22e](https://github.com/vllm-project/vllm-ascend/commit/fb8e22ec00aef2b2d42a5f2d3ae7267848ec5016) |
|
||||
| 350 | [@ppppeng](https://github.com/ppppeng) | 2026/03/17 | [a457d0f](https://github.com/vllm-project/vllm-ascend/commit/a457d0f0e8d91060c62d7ff2b1741bfc74d79560) |
|
||||
| 349 | [@asunxiao](https://github.com/asunxiao) | 2026/03/17 | [a370dfa](https://github.com/vllm-project/vllm-ascend/commit/a370dfa9623e648439b724569931988a852e462e) |
|
||||
| 348 | [@GGGGua](https://github.com/GGGGua) | 2026/03/16 | [b1a7888](https://github.com/vllm-project/vllm-ascend/commit/b1a78886a928cd7b5881026302fba79609972bd2) |
|
||||
| 347 | [@bazingazhou233-hub](https://github.com/bazingazhou233-hub) | 2026/03/14 | [9e6c547](https://github.com/vllm-project/vllm-ascend/commit/9e6c547d9808eb5fa532d49102969c91b79be905) |
|
||||
| 346 | [@tfhddd](https://github.com/tfhddd) | 2026/03/12 | [21fea86](https://github.com/vllm-project/vllm-ascend/commit/21fea86b08edf4a016749a0d637d18cf7017dd2a) |
|
||||
| 345 | [@ZRJ026](https://github.com/ZRJ026) | 2026/03/10 | [a398fa6](https://github.com/vllm-project/vllm-ascend/commit/a398fa6a0b024f59aaa823c483529bcf2357540f) |
|
||||
| 344 | [@xmpp777](https://github.com/xmpp777) | 2026/03/10 | [9216e1b](https://github.com/vllm-project/vllm-ascend/commit/9216e1b0505c7e290d8c02cc64cb8817bfdd49f5) |
|
||||
| 343 | [@wanghuanjun2113](https://github.com/wanghuanjun2113) | 2026/03/09 | [dec04ec](https://github.com/vllm-project/vllm-ascend/commit/dec04ec8d884a45f1946b72dea129bc686cc2f44) |
|
||||
| 342 | [@liuchenbing2026](https://github.com/liuchenbing2026) | 2026/03/09 | [542258a](https://github.com/vllm-project/vllm-ascend/commit/542258ac9d9229aab4e8822de42443245a93f001) |
|
||||
| 341 | [@chenxi-hh](https://github.com/chenxi-hh) | 2026/03/09 | [737dfcf](https://github.com/vllm-project/vllm-ascend/commit/737dfcf638eae71d6c24c340dee20ff205f21ed9) |
|
||||
| 340 | [@xiaocongtou6](https://github.com/xiaocongtou6) | 2026/03/06 | [bc0fd7c](https://github.com/vllm-project/vllm-ascend/commit/bc0fd7ca7217498d5faa91504b0e8c3f822a5cc6) |
|
||||
| 339 | [@wanghengkang](https://github.com/wanghengkang) | 2026/03/06 | [c49ce18](https://github.com/vllm-project/vllm-ascend/commit/c49ce18ea544970510ebb04fff49a484533fe2a3) |
|
||||
| 338 | [@Mind-s](https://github.com/Mind-s) | 2026/03/06 | [0f812dc](https://github.com/vllm-project/vllm-ascend/commit/0f812dcc58514d4e01b683282fd5a22cbbc86036) |
|
||||
| 337 | [@guleo](https://github.com/guleo) | 2026/03/06 | [18b52af](https://github.com/vllm-project/vllm-ascend/commit/18b52afe2bb2e3ac80e988a2788ac1e3bbdb898a) |
|
||||
| 336 | [@songjianquan](https://github.com/songjianquan) | 2026/03/05 | [43c8da3](https://github.com/vllm-project/vllm-ascend/commit/43c8da3574c96b9aaeaf4ef360c9b4aaf6a3e305) |
|
||||
| 335 | [@liuchenbing2026](https://github.com/liuchenbing2026) | 2026/03/05 | [640ecd1](https://github.com/vllm-project/vllm-ascend/commit/640ecd1b772b1c3dcdc57336b762cc02d011eba8) |
|
||||
| 334 | [@Zhujiyang2](https://github.com/Zhujiyang2) | 2026/03/04 | [c3c2656](https://github.com/vllm-project/vllm-ascend/commit/c3c265648f6fb3bf9ea2f6c0e43a4a2e67973d40) |
|
||||
| 333 | [@NJX-njx](https://github.com/NJX-njx) | 2026/03/04 | [c7fd7a2](https://github.com/vllm-project/vllm-ascend/commit/c7fd7a25f7f874dc34149f4c235c952dc7c3e227) |
|
||||
| 332 | [@tanhaoan333](https://github.com/tanhaoan333) | 2026/03/03 | [15f6564](https://github.com/vllm-project/vllm-ascend/commit/15f65649762992b17e10bda2d7f942618a10dfbd) |
|
||||
| 331 | [@Eric-dot](https://github.com/Eric-dot) | 2026/03/02 | [3c66a97](https://github.com/vllm-project/vllm-ascend/commit/3c66a970f29a4a976bbb4836e333bc58995b4a9d) |
|
||||
| 330 | [@wangbj127](https://github.com/wangbj127) | 2026/02/26 | [169e434](https://github.com/vllm-project/vllm-ascend/commit/169e434f78f03b963efb2779d8c64675313f9481) |
|
||||
| 329 | [@Li-Yongwen](https://github.com/Li-Yongwen) | 2026/02/26 | [2870f7c](https://github.com/vllm-project/vllm-ascend/commit/2870f7c8ad20754f5cc09cc5ea25044ffb3c6515) |
|
||||
| 328 | [@LoganJane](https://github.com/LoganJane) | 2026/02/25 | [ed05173](https://github.com/vllm-project/vllm-ascend/commit/ed051737e9984216c93cbea8790710c3485b3cee) |
|
||||
| 327 | [@chenchuw886](https://github.com/chenchuw886) | 2026/02/25 | [3da2ba2](https://github.com/vllm-project/vllm-ascend/commit/3da2ba22ebeef10ed31782488edb8120e3935bf7) |
|
||||
| 326 | [@Bowen-Leee](https://github.com/Bowen-Leee) | 2026/02/25 | [e3927cc](https://github.com/vllm-project/vllm-ascend/commit/e3927cc8f57ee9bdc58db1d0365dfb6af9375b02) |
|
||||
| 325 | [@lijiahang226](https://github.com/lijiahang226) | 2026/02/24 | [ff43537](https://github.com/vllm-project/vllm-ascend/commit/170dcbda6260b819b8b16f4630fb3b1a75373500) |
|
||||
| 324 | [@Spicy-Stick](https://github.com/Spicy-Stick) | 2026/02/14 | [64aea60](https://github.com/vllm-project/vllm-ascend/commit/64aea60f2e0256e6cdfd505cca9243972bcccc7c) |
|
||||
| 323 | [@yejj710](https://github.com/yejj710) | 2026/02/12 | [8b23554](https://github.com/vllm-project/vllm-ascend/commit/8b2355474147374bd1abdb6832510b91ef591b33) |
|
||||
| 322 | [@taoyao1221](https://github.com/taoyao1221) | 2026/02/12 | [41d056f](https://github.com/vllm-project/vllm-ascend/commit/41d056f94716d2ac74b73f26547a109c099e82dd) |
|
||||
| 321 | [@nakairika](https://github.com/nakairika) | 2026/02/12 | [b86ea66](https://github.com/vllm-project/vllm-ascend/commit/b86ea66b0a2de3795609804b3f93538563f6dc8d) |
|
||||
| 320 | [@mikequan0425](https://github.com/mikequan0425) | 2026/02/12 | [7221045](https://github.com/vllm-project/vllm-ascend/commit/7221045777bd4c3d77037fcb20d9bddd7b4dba3c) |
|
||||
| 319 | [@lih827](https://github.com/lih827) | 2026/02/12 | [f718120](https://github.com/vllm-project/vllm-ascend/commit/f71812011d49013ca077cc2441412593b7356052) |
|
||||
| 318 | [@huyq](https://github.com/huyq) | 2026/02/12 | [a0315f6](https://github.com/vllm-project/vllm-ascend/commit/a0315f66972d64cd586cb2ef29d8484f51e3194a) |
|
||||
| 317 | [@yydyzr](https://github.com/yydyzr) | 2026/02/11 | [ff3a50d](https://github.com/vllm-project/vllm-ascend/commit/ff3a50d011dcbea08f87ebed69ff1bf156dbb01e) |
|
||||
| 316 | [@luomin2005](https://github.com/luomin2005) | 2026/02/11 | [0c1cfa2](https://github.com/vllm-project/vllm-ascend/commit/0c1cfa2baca623ae8822d934c5941aac4efd57c4) |
|
||||
| 315 | [@GoCHug](https://github.com/GoCHug) | 2026/02/09 | [80e5812](https://github.com/vllm-project/vllm-ascend/commit/80e5812b39bd3b19efa8a6af65658f599bd6cae0) |
|
||||
| 314 | [@acat-rw](https://github.com/acat-rw) | 2026/02/05 | [8e66299](https://github.com/vllm-project/vllm-ascend/commit/8e66299bf16a7b48cf85b080dc3f8927144d5add) |
|
||||
| 313 | [@Zhang-Bryan](https://github.com/Zhang-Bryan) | 2026/02/04 | [804a9ec](https://github.com/vllm-project/vllm-ascend/commit/804a9ec4e6365aa484e3f9db61e226598b406ee6) |
|
||||
| 312 | [@IWantFight](https://github.com/IWantFight) | 2026/02/04 | [e7a13be](https://github.com/vllm-project/vllm-ascend/commit/e7a13beedb74b39dffd9f9dd1dd47bed32360206) |
|
||||
| 311 | [@SkychenLee](https://github.com/SkychenLee) | 2026/02/02 | [dc225e1](https://github.com/vllm-project/vllm-ascend/commit/dc225e1bb00d0f66100cbd83550b826308162f62) |
|
||||
| 310 | [@LQLlulu](https://github.com/LQLlulu) | 2026/02/02 | [cb1212f](https://github.com/vllm-project/vllm-ascend/commit/cb1212f18eae5611c60523f1bd8ab5ecb2a431a1) |
|
||||
| 309 | [@huangazazaz](https://github.com/huangazazaz) | 2026/02/02 | [c1618a0](https://github.com/vllm-project/vllm-ascend/commit/c1618a04273e967616e40551a86f370e7a76222b) |
|
||||
| 308 | [@wxh571001500](https://github.com/wxh571001500) | 2026/01/30 | [47f5d4b](https://github.com/vllm-project/vllm-ascend/commit/47f5d4b941c8e23821f4e98d0a5e470e6bcb13e3) |
|
||||
| 307 | [@wubin58](https://github.com/wubin58) | 2026/01/30 | [4230bc8](https://github.com/vllm-project/vllm-ascend/commit/4230bc86461e4b879e0b434eaa11cfbd2bf83ef7) |
|
||||
| 306 | [@serlar](https://github.com/serlar) | 2026/01/30 | [77ea873](https://github.com/vllm-project/vllm-ascend/commit/77ea8732241bfd80160824b17d1aee75909e1c24) |
|
||||
| 305 | [@Sergey-Zlobin](https://github.com/Sergey-Zlobin) | 2026/01/29 | [6a7b3bc](https://github.com/vllm-project/vllm-ascend/commit/6a7b3bc29cdd4c0bc25c87396b7567bb46c58607) |
|
||||
| 304 | [@HengjiaL](https://github.com/HengjiaL) | 2026/01/28 | [c1e88a5](https://github.com/vllm-project/vllm-ascend/commit/c1e88a55e8e2812eab4714be1f042eaa3eaf9927) |
|
||||
| 303 | [@pu-zhe](https://github.com/pu-zhe) | 2026/01/27 | [57fd6e4](https://github.com/vllm-project/vllm-ascend/commit/57fd6e4bd9c8025c9d5192b74d71e476b794ddba) |
|
||||
| 302 | [@mengchengTang](https://github.com/mengchengTang) | 2026/01/27 | [41eb71d](https://github.com/vllm-project/vllm-ascend/commit/41eb71d665ab9f0b72b6d3bc15d41dee7fcc0f5f) |
|
||||
| 301 | [@huangfeifei1995](https://github.com/huangfeifei1995) | 2026/01/26 | [ce11fd4](https://github.com/vllm-project/vllm-ascend/commit/ce11fd49f3f636e7d1ceea8787683c0a0ca60f24) |
|
||||
| 300 | [@gjc0824](https://github.com/gjc0824) | 2026/01/26 | [b390e0e](https://github.com/vllm-project/vllm-ascend/commit/b390e0ef78afb31d5ee26fdd26e4f33e89d645b3) |
|
||||
| 299 | [@Agent-Skill-007](https://github.com/Agent-Skill-007) | 2026/01/24 | [8129c42](https://github.com/vllm-project/vllm-ascend/commit/8129c429efa232e38a839654efeb9d708adb66c0) |
|
||||
| 298 | [@yjmyl](https://github.com/yjmyl) | 2026/01/23 | [e90b141](https://github.com/vllm-project/vllm-ascend/commit/e90b14140b5ff2e05fc8c5b812eedea51ffc3ee5) |
|
||||
| 297 | [@simplzyu](https://github.com/simplzyu) | 2026/01/23 | [f8d03d2](https://github.com/vllm-project/vllm-ascend/commit/f8d03d21f1fc94cfe14cd1d9430621624ecad76d) |
|
||||
| 296 | [@maxmgrdv](https://github.com/maxmgrdv) | 2026/01/22 | [ef9d836](https://github.com/vllm-project/vllm-ascend/commit/ef9d8367f5c7d62aeebc3eb790487afca5b49b88) |
|
||||
| 295 | [@guanguan0308](https://github.com/guanguan0308) | 2026/01/21 | [1ed9524](https://github.com/vllm-project/vllm-ascend/commit/1ed9524763590e5ba509c0cd07c80113123e7bf6) |
|
||||
| 294 | [@aipaes](https://github.com/aipaes) | 2026/01/19 | [f58e110](https://github.com/vllm-project/vllm-ascend/commit/f58e110afe65653c50f909f701e8ce7fec20054a) |
|
||||
| 293 | [@Tflowers-0129](https://github.com/Tflowers-0129) | 2026/01/17 | [1ffca86](https://github.com/vllm-project/vllm-ascend/commit/1ffca8673fa460174b891a194560e4fd05eb651d) |
|
||||
| 292 | [@ichaoren](https://github.com/ichaoren) | 2026/01/17 | [d17370b](https://github.com/vllm-project/vllm-ascend/commit/d17370b9233241fd1b771fbbebe59d0d46c23f93) |
|
||||
| 291 | [@brandneway](https://github.com/brandneway) | 2026/01/16 | [d426276](https://github.com/vllm-project/vllm-ascend/commit/d426276b4b9f65c952cfeb4e37022829e8b66dd8) |
|
||||
| 290 | [@Rozwel-dx](https://github.com/Rozwel-dx) | 2026/01/13 | [8d57128](https://github.com/vllm-project/vllm-ascend/commit/8d571286dd223f7ebb58b9aae155d55c57b1a5b6) |
|
||||
| 289 | [@LiuYi-Up](https://github.com/LiuYi-Up) | 2026/01/13 | [dde547e](https://github.com/vllm-project/vllm-ascend/commit/dde547e900e5263e046be665c7664af13a5c7f75) |
|
||||
| 288 | [@Jeaniowang](https://github.com/Jeaniowang) | 2026/01/12 | [4453c60](https://github.com/vllm-project/vllm-ascend/commit/4453c602626c6bce50b376bbb6e803d7b0131a6e) |
|
||||
| 287 | [@gh924](https://github.com/gh924) | 2026/01/11 | [6880c1b](https://github.com/vllm-project/vllm-ascend/commit/6880c1b3834ba24b12f5537e50dcac7c7ca1fa92) |
|
||||
| 286 | [@Feng-xiaosuo](https://github.com/Feng-xiaosuo) | 2026/01/10 | [c316679](https://github.com/vllm-project/vllm-ascend/commit/c316679e658205313c0a547b8ae4365382e3a685) |
|
||||
| 285 | [@zyz111222](https://github.com/zyz111222) | 2026/01/09 | [98c788a](https://github.com/vllm-project/vllm-ascend/commit/98c788a65ae7bdc982b8f5088bcefc4f4c716945) |
|
||||
| 284 | [@wwwumr](https://github.com/wwwumr) | 2026/01/09 | [3ce5a34](https://github.com/vllm-project/vllm-ascend/commit/3ce5a34468e92512670759f7ee0aae0defa4ae94) |
|
||||
| 283 | [@lhchg](https://github.com/lhchg) | 2026/01/09 | [dc99cfd](https://github.com/vllm-project/vllm-ascend/commit/dc99cfdc15bfb094983e49ec14e4eaa98dee7a7a) |
|
||||
| 282 | [@icerain-alt](https://github.com/icerain-alt) | 2026/01/09 | [09682e0](https://github.com/vllm-project/vllm-ascend/commit/09682e075118aaacb0a717f2b7078bad040599a9) |
|
||||
| 281 | [@wangyibo1005](https://github.com/wangyibo1005) | 2026/01/07 | [25baf6d](https://github.com/vllm-project/vllm-ascend/commit/25baf6df095f10c6af29fcafd56ad3ff3e83a9b2) |
|
||||
| 280 | [@starmountain1997](https://github.com/starmountain1997) | 2026/01/07 | [086c093](https://github.com/vllm-project/vllm-ascend/commit/086c093347118d0448df6ac7e80a7b4dd1cdc4f8) |
|
||||
| 279 | [@pacoxu](https://github.com/pacoxu) | 2026/01/07 | [4f98080](https://github.com/vllm-project/vllm-ascend/commit/4f9808002b1bc0f1b75762d430983940ca7f4875) |
|
||||
| 278 | [@zhangguinan](https://github.com/zhangguinan) | 2026/01/05 | [6c1a685](https://github.com/vllm-project/vllm-ascend/commit/6c1a685b30b9c75a6fee29c445e2a04229896694) |
|
||||
| 277 | [@L4-1024](https://github.com/L4-1024) | 2026/01/05 | [c23cf30](https://github.com/vllm-project/vllm-ascend/commit/c23cf30709667f6fdde8d66957567933a590546a) |
|
||||
| 276 | [@frankie-ys](https://github.com/frankie-ys) | 2026/01/05 | [ec35633](https://github.com/vllm-project/vllm-ascend/commit/ec3563334b7f95925131ab758378341d23895863) |
|
||||
| 275 | [@Debonex](https://github.com/Debonex) | 2026/01/05 | [d86021f](https://github.com/vllm-project/vllm-ascend/commit/d86021f7b4f86be6f853b03da82628b4d0d6ea4b) |
|
||||
| 274 | [@chuyuelin](https://github.com/chuyuelin) | 2025/12/31 | [d07d8a4](https://github.com/vllm-project/vllm-ascend/commit/d07d8a4535f499c7af95e1966b86700a693c8253) |
|
||||
| 273 | [@ZCG12345](https://github.com/ZCG12345) | 2025/12/30 | [45c3c27](https://github.com/vllm-project/vllm-ascend/commit/45c3c279e2b31c85c8739c45b43d8c47710e447b) |
|
||||
| 272 | [@hu-qi](https://github.com/hu-qi) | 2025/12/30 | [c85cc04](https://github.com/vllm-project/vllm-ascend/commit/c85cc045f893293e3b44e24d2e1f01ddc5849ea8) |
|
||||
| 271 | [@zhanzy178](https://github.com/zhanzy178) | 2025/12/29 | [d8e15da](https://github.com/vllm-project/vllm-ascend/commit/d8e15dae6c5e563c3284309d4557afb4d4a17feb) |
|
||||
| 270 | [@jiazhengyi](https://github.com/jiazhengyi) | 2025/12/29 | [d5f7283](https://github.com/vllm-project/vllm-ascend/commit/d5f72835e65e6b6b8cf8e173a9048c926890881b) |
|
||||
| 269 | [@Fager10086](https://github.com/Fager10086) | 2025/12/29 | [51da5ea](https://github.com/vllm-project/vllm-ascend/commit/51da5ea5438749fa057a7bb60977750e3ac22392) |
|
||||
| 268 | [@jiangkuaixue123](https://github.com/jiangkuaixue123) | 2025/12/27 | [e91e11d](https://github.com/vllm-project/vllm-ascend/commit/e91e11d3b0a961f2e0e034cd738632653e5f6bdc) |
|
||||
| 267 | [@maoxx241](https://github.com/maoxx241) | 2025/12/26 | [7372225](https://github.com/vllm-project/vllm-ascend/commit/7372225bcb0bd4896f43e989cef8109bfe45b13c) |
|
||||
| 266 | [@wjunLu](https://github.com/wjunLu) | 2025/12/25 | [fca2f94](https://github.com/vllm-project/vllm-ascend/commit/fca2f948c101c9d520f1a381a6b705d9e11c042e) |
|
||||
| 265 | [@cookieyyds](https://github.com/cookieyyds) | 2025/12/25 | [2da8038](https://github.com/vllm-project/vllm-ascend/commit/2da8038dd2d30f813c00b37ba2cd9abeaaf3d0f3) |
|
||||
| 264 | [@chenaoxuan](https://github.com/chenaoxuan) | 2025/12/25 | [6d25372](https://github.com/vllm-project/vllm-ascend/commit/6d25372baaa0ef018a75b427b387fab8dd2e92b4) |
|
||||
| 263 | [@changdawei1](https://github.com/changdawei1) | 2025/12/25 | [a9fccbe](https://github.com/vllm-project/vllm-ascend/commit/a9fccbeb300fd4753f8e54aaddf02abf59d7429c) |
|
||||
| 262 | [@TmacAaron](https://github.com/TmacAaron) | 2025/12/24 | [5018f2d](https://github.com/vllm-project/vllm-ascend/commit/5018f2d8fd6dc11240900e38049150619895570d) |
|
||||
| 261 | [@lengrongfu](https://github.com/lengrongfu) | 2025/12/23 | [c9b5881](https://github.com/vllm-project/vllm-ascend/commit/c9b5881bcd3a7b76d7b631471e4f3eae2ac545dd) |
|
||||
| 260 | [@hzxuzhonghu](https://github.com/hzxuzhonghu) | 2025/12/23 | [cb963c5](https://github.com/vllm-project/vllm-ascend/commit/cb963c53a58a4a7339f3507fcc087d83a0d7df7e) |
|
||||
| 259 | [@pisceskkk](https://github.com/pisceskkk) | 2025/12/22 | [ea6206b](https://github.com/vllm-project/vllm-ascend/commit/ea6206bb183e555345e0fbc38c7cd0a1634fe21f) |
|
||||
| 258 | [@OsirisDuan](https://github.com/OsirisDuan) | 2025/12/22 | [b2c1216](https://github.com/vllm-project/vllm-ascend/commit/b2c121637fd8b8045e66e24ea0f63cb17ffb3b69) |
|
||||
| 257 | [@LJQ142857](https://github.com/LJQ142857) | 2025/12/22 | [55beac9](https://github.com/vllm-project/vllm-ascend/commit/55beac9c91b8399cbe6425bd79f225b5dc72b243) |
|
||||
| 256 | [@YuhanBai](https://github.com/YuhanBai) | 2025/12/20 | [5d02eed](https://github.com/vllm-project/vllm-ascend/commit/5d02eed16f6e4e5187259583f9db16a916095ab3) |
|
||||
| 255 | [@luluxiu520](https://github.com/luluxiu520) | 2025/12/19 | [bc05a81](https://github.com/vllm-project/vllm-ascend/commit/bc05a81bf2d68dc45d9c6115e5d4f3191a528891) |
|
||||
| 254 | [@hukongyi](https://github.com/hukongyi) | 2025/12/19 | [ea8f544](https://github.com/vllm-project/vllm-ascend/commit/ea8f544ce73708aef8c9d48a2916b64fa0e09806) |
|
||||
| 253 | [@ZT-AIA](https://github.com/ZT-AIA) | 2025/12/18 | [39fb9e7](https://github.com/vllm-project/vllm-ascend/commit/39fb9e7c8324c48f967a5f38d77b3e1225ecec2f) |
|
||||
| 252 | [@yuxinshan](https://github.com/yuxinshan) | 2025/12/18 | [b0376ab](https://github.com/vllm-project/vllm-ascend/commit/b0376abd4c8d535953208c1b38c646ac7e8b31f4) |
|
||||
| 251 | [@yuxingcyx](https://github.com/yuxingcyx) | 2025/12/18 | [5a88e33](https://github.com/vllm-project/vllm-ascend/commit/5a88e3333bea5d4c948e14447d2d9e4ca8963e4d) |
|
||||
| 250 | [@TingW09](https://github.com/TingW09) | 2025/12/18 | [879ec2d](https://github.com/vllm-project/vllm-ascend/commit/879ec2d1c4267f6a927ac6d3d2d835fe37e89f80) |
|
||||
| 249 | [@LICO1314](https://github.com/LICO1314) | 2025/12/18 | [9fcaf66](https://github.com/vllm-project/vllm-ascend/commit/9fcaf66646d15671ab4d9bcf3530cccfcc6b2675) |
|
||||
| 248 | [@YzTongNiar](https://github.com/YzTongNiar) | 2025/12/17 | [7671ce1](https://github.com/vllm-project/vllm-ascend/commit/7671ce1bf12b3df45aaa92c6316b40d8f954230a) |
|
||||
| 247 | [@Toneymiller](https://github.com/Toneymiller) | 2025/12/17 | [b1a853b](https://github.com/vllm-project/vllm-ascend/commit/b1a853b0f64961ac9145fd4a510e194fb152992e) |
|
||||
| 246 | [@JeffLee1874](https://github.com/JeffLee1874) | 2025/12/17 | [724d043](https://github.com/vllm-project/vllm-ascend/commit/724d04391e89408f3fc3fb0cef13e5f4cf155dde) |
|
||||
| 245 | [@ader47](https://github.com/ader47) | 2025/12/17 | [eda3cab](https://github.com/vllm-project/vllm-ascend/commit/eda3cabf5b0de9e0e435328e666614f5320e657c) |
|
||||
| 244 | [@UnifiedCacheManager](https://github.com/UnifiedCacheManager) | 2025/12/16 | [195eac6](https://github.com/vllm-project/vllm-ascend/commit/195eac665b2d42b8287c59128490586c6931d54c) |
|
||||
| 243 | [@ming1212](https://github.com/ming1212) | 2025/12/15 | [98b9e2e](https://github.com/vllm-project/vllm-ascend/commit/98b9e2e18e8f8e9f1495eafc8f57e3e1791e0a34) |
|
||||
| 242 | [@knight0528](https://github.com/knight0528) | 2025/12/15 | [e25c57b](https://github.com/vllm-project/vllm-ascend/commit/e25c57b3461b8116d533724013fbc9957716f92d) |
|
||||
| 241 | [@wangyao-i](https://github.com/wangyao-i) | 2025/12/12 | [0983c55](https://github.com/vllm-project/vllm-ascend/commit/0983c5510aa49c7310b79db72657d8a0f92918ec) |
|
||||
| 240 | [@sunchendd](https://github.com/sunchendd) | 2025/12/12 | [5932abc](https://github.com/vllm-project/vllm-ascend/commit/5932abc44686cc0d3fa1c4ce9fdbc30957b8aa51) |
|
||||
| 239 | [@yangshihao6](https://github.com/yangshihao6) | 2025/12/11 | [e1bb6f4](https://github.com/vllm-project/vllm-ascend/commit/e1bb6f47ecabcb382ff3b4dfbd168d0943ce504f) |
|
||||
| 238 | [@xlan-huawei](https://github.com/xlan-huawei) | 2025/12/11 | [5ebb9bd](https://github.com/vllm-project/vllm-ascend/commit/5ebb9bd8d26395e86f9bd01183a327477a966760) |
|
||||
| 237 | [@hwhaokun](https://github.com/hwhaokun) | 2025/12/11 | [a47aa4d](https://github.com/vllm-project/vllm-ascend/commit/a47aa4da2f87dfd2792ef13d31476eccd27685bf) |
|
||||
| 236 | [@Gongdayao](https://github.com/Gongdayao) | 2025/12/11 | [89a8607](https://github.com/vllm-project/vllm-ascend/commit/89a8607b30ca4f123ae405df42701077b776e4af) |
|
||||
| 235 | [@cjy0x](https://github.com/cjy0x) | 2025/12/11 | [c12eb22](https://github.com/vllm-project/vllm-ascend/commit/c12eb22cbe9f55ce72fca3d8231c06d0173e22fb) |
|
||||
| 234 | [@wind-all](https://github.com/wind-all) | 2025/12/10 | [1a443f2](https://github.com/vllm-project/vllm-ascend/commit/1a443f2772ba9a75288e5dc09227ee6bdd54c147) |
|
||||
| 233 | [@Trunrain](https://github.com/Trunrain) | 2025/12/10 | [ba9cda9](https://github.com/vllm-project/vllm-ascend/commit/ba9cda9dfd83765a7e79a074c033818ff5a76bd6) |
|
||||
| 232 | [@ghphotoframe](https://github.com/ghphotoframe) | 2025/12/10 | [0d8c0f1](https://github.com/vllm-project/vllm-ascend/commit/0d8c0f1a24a58011908b17994fffb5fcfebe5e22) |
|
||||
| 231 | [@ChenCangtao](https://github.com/ChenCangtao) | 2025/12/10 | [dd622aa](https://github.com/vllm-project/vllm-ascend/commit/dd622aa6a6ea34a6bc799d52171ebe6c3e062972) |
|
||||
| 230 | [@wangqiankun13](https://github.com/wangqiankun13) | 2025/12/09 | [9567e5d](https://github.com/vllm-project/vllm-ascend/commit/9567e5dd8c61d2e62b3572c9e98dc4f9eaf51c2a) |
|
||||
| 229 | [@zengzengran](https://github.com/zengzengran) | 2025/12/08 | [f0876b5](https://github.com/vllm-project/vllm-ascend/commit/f0876b5d88a04734505bb5cea7bda95b026923ff) |
|
||||
| 228 | [@shiro-zzzz](https://github.com/shiro-zzzz) | 2025/12/08 | [0617d7d](https://github.com/vllm-project/vllm-ascend/commit/0617d7d394a9b6aa47a75f368d6059b1e5788f06) |
|
||||
| 227 | [@lulina](https://github.com/lulina) | 2025/12/08 | [2be0fe2](https://github.com/vllm-project/vllm-ascend/commit/2be0fe269160e97dc9449101e92a98e7f3436668) |
|
||||
| 226 | [@lhp-deep](https://github.com/lhp-deep) | 2025/12/08 | [b230e7e](https://github.com/vllm-project/vllm-ascend/commit/b230e7e987da602a3985a4f297b01decfc1ec40f) |
|
||||
| 225 | [@h1074112368](https://github.com/h1074112368) | 2025/12/06 | [7403399](https://github.com/vllm-project/vllm-ascend/commit/74033999ed98fcf02db0a2512d019177d0843791) |
|
||||
| 224 | [@GuoRen868](https://github.com/GuoRen868) | 2025/12/06 | [4bd1030](https://github.com/vllm-project/vllm-ascend/commit/4bd1030842a3831109ccfb136056459cd83c1441) |
|
||||
| 223 | [@AlvisGong](https://github.com/AlvisGong) | 2025/12/06 | [a5163c8](https://github.com/vllm-project/vllm-ascend/commit/a5163c8c369e046c53e6127bbbcee23392cec069) |
|
||||
| 222 | [@MingYang119](https://github.com/MingYang119) | 2025/12/03 | [18b90b5](https://github.com/vllm-project/vllm-ascend/commit/18b90b501d6aad1d9426dcdee1ccfbe8139dd47d) |
|
||||
| 221 | [@amy-why-3459](https://github.com/amy-why-3459) | 2025/12/03 | [26e8e58](https://github.com/vllm-project/vllm-ascend/commit/26e8e58cea8ba5cc5edaef19e72d0ddc4e9f1c1c) |
|
||||
| 220 | [@coder-fny](https://github.com/coder-fny) | 2025/12/02 | [1b5513a](https://github.com/vllm-project/vllm-ascend/commit/1b5513aa914aa37cb539d410faf6e021e6dc5481) |
|
||||
| 219 | [@swy20190](https://github.com/swy20190) | 2025/12/01 | [f4871c6](https://github.com/vllm-project/vllm-ascend/commit/f4871c6ab98af9ab2766779b2167de04805f38e2) |
|
||||
| 218 | [@fluctlux](https://github.com/fluctlux) | 2025/12/01 | [f1f6370](https://github.com/vllm-project/vllm-ascend/commit/f1f6370ed966bd554f8874ddc430c09f25354c72) |
|
||||
| 217 | [@LHXuuu](https://github.com/LHXuuu) | 2025/11/28 | [bdc6697](https://github.com/vllm-project/vllm-ascend/commit/bdc66972db12cc9f4b63721495fb2b103942dd25) |
|
||||
| 216 | [@ChenxiQ](https://github.com/ChenxiQ) | 2025/11/28 | [554f16a](https://github.com/vllm-project/vllm-ascend/commit/554f16ae1fb89b35ad82b24e7f7fe5eaba0e80d0) |
|
||||
| 215 | [@zjchenn](https://github.com/zjchenn) | 2025/11/25 | [463910e](https://github.com/vllm-project/vllm-ascend/commit/463910e686013acc611a7e3ccb70c3a776c8ffef) |
|
||||
| 214 | [@wujinyuan1](https://github.com/wujinyuan1) | 2025/11/25 | [386a85e](https://github.com/vllm-project/vllm-ascend/commit/386a85eccc9a2247180faad2f8a00ad9c0afa4b1) |
|
||||
| 213 | [@Tjh-UKN](https://github.com/Tjh-UKN) | 2025/11/24 | [00ea61e](https://github.com/vllm-project/vllm-ascend/commit/00ea61ec885e21ed0e51dc8e751cb27cfa539dc3) |
|
||||
| 212 | [@mazhixin000](https://github.com/mazhixin000) | 2025/11/22 | [ab51fce](https://github.com/vllm-project/vllm-ascend/commit/ab51fcea4ca77fdb642a04018df60e18586d52ef) |
|
||||
| 211 | [@shenchuxiaofugui](https://github.com/shenchuxiaofugui) | 2025/11/21 | [019c7de](https://github.com/vllm-project/vllm-ascend/commit/019c7ded91f0c2a9533ace8d87f0cf56e396f979) |
|
||||
| 210 | [@InSec](https://github.com/InSec) | 2025/11/21 | [5a4e8cd](https://github.com/vllm-project/vllm-ascend/commit/5a4e8cdebabd8293e9ff61b7014d758b71ebf32a) |
|
||||
| 209 | [@Delphine-Nic](https://github.com/Delphine-Nic) | 2025/11/19 | [a3e9673](https://github.com/vllm-project/vllm-ascend/commit/a3e9673137dd31d77b7441d657a0fc394ac7bc59) |
|
||||
| 208 | [@wangxiaochao6](https://github.com/wangxiaochao6) | 2025/11/18 | [0d04ad8](https://github.com/vllm-project/vllm-ascend/commit/0d04ad8c8f00081932566576da1e6d1dcd963d8d) |
|
||||
| 207 | [@845473182](https://github.com/845473182) | 2025/11/14 | [f90ed95](https://github.com/vllm-project/vllm-ascend/commit/f90ed95578f81ddd1cb96c9bd5d6ef8be8475667) |
|
||||
| 206 | [@thonean](https://github.com/thonean) | 2025/11/12 | [e38fe92](https://github.com/vllm-project/vllm-ascend/commit/e38fe92f4037fe668f34b23c67daa9695a3fc3c7) |
|
||||
| 205 | [@zhaomingyu13](https://github.com/zhaomingyu13) | 2025/11/11 | [7ffbe73](https://github.com/vllm-project/vllm-ascend/commit/7ffbe73d54d7257c571ddd21bac6543b5ead0dac) |
|
||||
| 204 | [@Apocalypse990923-qshi](https://github.com/Apocalypse990923-qshi) | 2025/11/11 | [71866d5](https://github.com/vllm-project/vllm-ascend/commit/71866d531151b36a9af4db36ba8b3b74162f7028) |
|
||||
| 203 | [@herizhen](https://github.com/herizhen) | 2025/11/10 | [75c3f9a](https://github.com/vllm-project/vllm-ascend/commit/75c3f9a7807daa3346685be88e4f06d6a5f362f0) |
|
||||
| 202 | [@wangx700](https://github.com/wangx700) | 2025/11/08 | [55e37f5](https://github.com/vllm-project/vllm-ascend/commit/55e37f50418f38861768556fdde9e86c2a22aef4) |
|
||||
| 201 | [@Semmer2](https://github.com/Semmer2) | 2025/11/08 | [f984256](https://github.com/vllm-project/vllm-ascend/commit/f9842560cbd5bb7b1106b3f47982adb479a91774) |
|
||||
| 200 | [@Liziqi-77](https://github.com/Liziqi-77) | 2025/11/06 | [25b24c0](https://github.com/vllm-project/vllm-ascend/commit/25b24c02ea86f0f8a673f8f332e800ed72abc083) |
|
||||
| 199 | [@Pz1116](https://github.com/Pz1116) | 2025/11/05 | [e0c23cb](https://github.com/vllm-project/vllm-ascend/commit/e0c23cb011e7ee6e5afce32ab0a0b9c73e825baa) |
|
||||
| 198 | [@MrZ20](https://github.com/MrZ20) | 2025/11/04 | [dc1a6cb](https://github.com/vllm-project/vllm-ascend/commit/dc1a6cb5039b5ee1520fde8cf85b99f7ce33ec7b) |
|
||||
| 197 | [@ForBetterCodeNine](https://github.com/ForBetterCodeNine) | 2025/11/03 | [49d7478](https://github.com/vllm-project/vllm-ascend/commit/49d74785c440c964db1a1212c9b24f695f21a5a2) |
|
||||
| 196 | [@Nagisa125](https://github.com/Nagisa125) | 2025/10/31 | [6764777](https://github.com/vllm-project/vllm-ascend/commit/6764777f000f6ac5acfadbba3797ce966397e1d8) |
|
||||
| 195 | [@Liwx1014](https://github.com/Liwx1014) | 2025/10/30 | [eed1957](https://github.com/vllm-project/vllm-ascend/commit/eed1957f03224bf800191390244c39ff5bb362d1) |
|
||||
| 194 | [@Meihan-chen](https://github.com/Meihan-chen) | 2025/10/29 | [cba69e1](https://github.com/vllm-project/vllm-ascend/commit/cba69e117eaa8c5d31fca64b01aecb22d3162860) |
|
||||
| 193 | [@Levi-JQ](https://github.com/Levi-JQ) | 2025/10/27 | [d64bdd0](https://github.com/vllm-project/vllm-ascend/commit/d64bdd06ae656048040c34c0ff3b909293b5113b) |
|
||||
| 192 | [@QilaiZhang](https://github.com/QilaiZhang) | 2025/10/25 | [d30bb95](https://github.com/vllm-project/vllm-ascend/commit/d30bb95b9040475495dc01f5c9e38576905be621) |
|
||||
| 191 | [@gcanlin](https://github.com/gcanlin) | 2025/10/25 | [8295136](https://github.com/vllm-project/vllm-ascend/commit/829513657518c0077622b1904f49d79657b2537a) |
|
||||
| 190 | [@ck-hw-1018](https://github.com/ck-hw-1018) | 2025/10/25 | [7572939](https://github.com/vllm-project/vllm-ascend/commit/7572939b94e9c2215cf3ebd30bd6f61c4d1b04f6) |
|
||||
| 189 | [@yenuo26](https://github.com/yenuo26) | 2025/10/24 | [d301c56](https://github.com/vllm-project/vllm-ascend/commit/d301c56d1aff5d2f408e54285df5ff36fd28d193) |
|
||||
| 188 | [@luoxiaolin712](https://github.com/luoxiaolin712) | 2025/10/24 | [59bb16b](https://github.com/vllm-project/vllm-ascend/commit/59bb16b75c492aa636fec53f1d32680839e7d1e7) |
|
||||
| 187 | [@lio1226](https://github.com/lio1226) | 2025/10/24 | [cd58a64](https://github.com/vllm-project/vllm-ascend/commit/cd58a643c598622fdf5a853764f851a5074d0328) |
|
||||
| 186 | [@yzy1996](https://github.com/yzy1996) | 2025/10/23 | [f06a6ca](https://github.com/vllm-project/vllm-ascend/commit/f06a6cad1b0c6b142d4a9364c65b4b920b251540) |
|
||||
| 185 | [@wlf-darkmatter](https://github.com/wlf-darkmatter) | 2025/10/23 | [097173e](https://github.com/vllm-project/vllm-ascend/commit/097173e50fc9efe2bf4e46bf7d59b8f5e06f2679) |
|
||||
| 184 | [@HuaJiaHeng](https://github.com/HuaJiaHeng) | 2025/10/23 | [062257f](https://github.com/vllm-project/vllm-ascend/commit/062257f624486d15315013b624ce71e1a388ee20) |
|
||||
| 183 | [@destinysky](https://github.com/destinysky) | 2025/10/23 | [427b17e](https://github.com/vllm-project/vllm-ascend/commit/427b17e2da1d6d967e7581b40b6bd169cdb324c8) |
|
||||
| 182 | [@KyrieDrewWang](https://github.com/KyrieDrewWang) | 2025/10/22 | [60e2be1](https://github.com/vllm-project/vllm-ascend/commit/60e2be1b366975f7183c38cd158da811969b9230) |
|
||||
| 181 | [@HF-001](https://github.com/HF-001) | 2025/10/22 | [bc30874](https://github.com/vllm-project/vllm-ascend/commit/bc30874f8b6212ff8eb45b2546c360660c49c9c1) |
|
||||
| 180 | [@drslark](https://github.com/drslark) | 2025/10/21 | [534f32d](https://github.com/vllm-project/vllm-ascend/commit/534f32d27c0cc48731fbdae3701fbb6c3bb4332a) |
|
||||
| 179 | [@Anionex](https://github.com/Anionex) | 2025/10/21 | [5f8b169](https://github.com/vllm-project/vllm-ascend/commit/5f8b1699ae35bb8c046d8b385d215cd208bc3fcb) |
|
||||
| 178 | [@leijie-ww](https://github.com/leijie-ww) | 2025/10/20 | [6b68579](https://github.com/vllm-project/vllm-ascend/commit/6b6857929d8397fbec1cd132eb4ea5d13712497c) |
|
||||
| 177 | [@ZYang6263](https://github.com/ZYang6263) | 2025/10/19 | [1e78ecb](https://github.com/vllm-project/vllm-ascend/commit/1e78ecbad626968c7e65fab5596c90ea79e7454d) |
|
||||
| 176 | [@yechao237](https://github.com/yechao237) | 2025/10/18 | [4750d45](https://github.com/vllm-project/vllm-ascend/commit/4750d45d86632b7085ab9f0b57070cd17aa9d108) |
|
||||
| 175 | [@Shirley125](https://github.com/Shirley125) | 2025/10/18 | [b4233a2](https://github.com/vllm-project/vllm-ascend/commit/b4233a2ec35a91beca4b8c8402ea7cf6394b62c9) |
|
||||
| 174 | [@DreamerLeader](https://github.com/DreamerLeader) | 2025/10/15 | [aa61547](https://github.com/vllm-project/vllm-ascend/commit/aa6154703aeeb259611326a3fba095b6abc5b791) |
|
||||
| 173 | [@yuzhup](https://github.com/yuzhup) | 2025/10/14 | [7877723](https://github.com/vllm-project/vllm-ascend/commit/78777237a9ae95af88ce6068b000635f96aafe25) |
|
||||
| 172 | [@menogrey](https://github.com/menogrey) | 2025/10/14 | [657c08c](https://github.com/vllm-project/vllm-ascend/commit/657c08cfb266c63946ea61a7be377b6de55d20f3) |
|
||||
| 171 | [@elilzhu](https://github.com/elilzhu) | 2025/10/14 | [5c45c22](https://github.com/vllm-project/vllm-ascend/commit/5c45c227dc254591f4a9345e67a84a0d5fe1c345) |
|
||||
| 170 | [@dsxsteven](https://github.com/dsxsteven) | 2025/10/13 | [847d12a](https://github.com/vllm-project/vllm-ascend/commit/847d12a389217e4cbcc5fff70abd72d9b15ad5c4) |
|
||||
| 169 | [@kiscad](https://github.com/kiscad) | 2025/10/12 | [bcc313e](https://github.com/vllm-project/vllm-ascend/commit/bcc313e8f2bfe9a53ed88dffffd93632db52a4ba) |
|
||||
| 168 | [@jiangyunfan1](https://github.com/jiangyunfan1) | 2025/10/12 | [d05d29f](https://github.com/vllm-project/vllm-ascend/commit/d05d29ff0e7c5d261dbdee5e98ff0ca95135217b) |
|
||||
| 167 | [@huangdong2022](https://github.com/huangdong2022) | 2025/10/09 | [23db56a](https://github.com/vllm-project/vllm-ascend/commit/23db56a34063c5285f7d34e80e45c1888b877bd3) |
|
||||
| 166 | [@zhenwenqi2024](https://github.com/zhenwenqi2024) | 2025/09/30 | [9dd8621](https://github.com/vllm-project/vllm-ascend/commit/9dd8621212a26b52e98de58488c2a6cd34cde76e) |
|
||||
| 165 | [@zhangsicheng5](https://github.com/zhangsicheng5) | 2025/09/30 | [657f065](https://github.com/vllm-project/vllm-ascend/commit/657f065f4247689b95c2edea00696114de5b1c48) |
|
||||
| 164 | [@kkrazy](https://github.com/kkrazy) | 2025/09/29 | [f62c866](https://github.com/vllm-project/vllm-ascend/commit/f62c8660fd80856ccfc4c57fb9ed5b05df8b5654) |
|
||||
| 163 | [@socrahow](https://github.com/socrahow) | 2025/09/28 | [c3fee66](https://github.com/vllm-project/vllm-ascend/commit/c3fee66806f252476796389ea73d13a8aca60146) |
|
||||
| 162 | [@slippersss](https://github.com/slippersss) | 2025/09/28 | [a86ece5](https://github.com/vllm-project/vllm-ascend/commit/a86ece5e399db9aa9d7186ab7e51bc0e0dad4134) |
|
||||
| 161 | [@florenceCH](https://github.com/florenceCH) | 2025/09/26 | [14497b7](https://github.com/vllm-project/vllm-ascend/commit/14497b748d778f95328f55a957e997ad80a492da) |
|
||||
| 160 | [@mfyCn-1204](https://github.com/mfyCn-1204) | 2025/09/25 | [33c118c](https://github.com/vllm-project/vllm-ascend/commit/33c118c80e70cec64c9369b7ba4088c61c44bd31) |
|
||||
| 159 | [@dragondream-chen](https://github.com/dragondream-chen) | 2025/09/25 | [07f4710](https://github.com/vllm-project/vllm-ascend/commit/07f4710216da1d61c521fb3bdf4ba90ea1794474) |
|
||||
| 158 | [@zzhx1](https://github.com/zzhx1) | 2025/09/24 | [4ee58e2](https://github.com/vllm-project/vllm-ascend/commit/4ee58e213b421ece745dba6e94967e6f557263ce) |
|
||||
| 157 | [@Csrayz](https://github.com/Csrayz) | 2025/09/24 | [80524f5](https://github.com/vllm-project/vllm-ascend/commit/80524f571152e978f6e2808e504e1aa8b246a2c1) |
|
||||
| 156 | [@clrs97](https://github.com/clrs97) | 2025/09/24 | [cd1ffbb](https://github.com/vllm-project/vllm-ascend/commit/cd1ffbb6cd88a3f265027424bd3cca74d1efb1ea) |
|
||||
| 155 | [@Clorist33](https://github.com/Clorist33) | 2025/09/24 | [302494c](https://github.com/vllm-project/vllm-ascend/commit/302494c1febfb648ac2a5001390bbd9b59db6bc2) |
|
||||
| 154 | [@booker123456](https://github.com/booker123456) | 2025/09/24 | [c4b976a](https://github.com/vllm-project/vllm-ascend/commit/c4b976af1a6459a82f7556c53dc98c850dd3e3cd) |
|
||||
| 153 | [@Mercykid-bash](https://github.com/Mercykid-bash) | 2025/09/23 | [29c173a](https://github.com/vllm-project/vllm-ascend/commit/29c173ab48001f3d26805db2d833be712687fb1a) |
|
||||
| 152 | [@MaoJianwei](https://github.com/MaoJianwei) | 2025/09/23 | [d586255](https://github.com/vllm-project/vllm-ascend/commit/d586255678d974d74b1fe798838594c0e948d6b6) |
|
||||
| 151 | [@Lucaskabela](https://github.com/Lucaskabela) | 2025/09/20 | [53ecd89](https://github.com/vllm-project/vllm-ascend/commit/53ecd89e8ff405302be040a76effa8c012cbaaeb) |
|
||||
| 150 | [@jesse996](https://github.com/jesse996) | 2025/09/19 | [833cd1b](https://github.com/vllm-project/vllm-ascend/commit/833cd1b698f3d467bb0a6a60cbf20ebc5535f5c9) |
|
||||
| 149 | [@offline893](https://github.com/offline893) | 2025/09/17 | [76844ee](https://github.com/vllm-project/vllm-ascend/commit/76844eec78a23f482a4e0dfe9684898a6ef35fb2) |
|
||||
| 148 | [@1Fire4](https://github.com/1Fire4) | 2025/09/17 | [1f6465c](https://github.com/vllm-project/vllm-ascend/commit/1f6465c399d6e699f88e28419c106573bd6c44f0) |
|
||||
| 147 | [@nwpu-zxr](https://github.com/nwpu-zxr) | 2025/09/13 | [0a27705](https://github.com/vllm-project/vllm-ascend/commit/0a27705917e64993a8a76198ac6e30980578fe60) |
|
||||
| 146 | [@zhaozx-cn](https://github.com/zhaozx-cn) | 2025/09/11 | [b9a0a75](https://github.com/vllm-project/vllm-ascend/commit/b9a0a75c783571caf22129612fb3338272d1782c) |
|
||||
| 145 | [@wyu0-0](https://github.com/wyu0-0) | 2025/09/11 | [eab3635](https://github.com/vllm-project/vllm-ascend/commit/eab3635850ba351af81d76a7b4b3db46ffb7f697) |
|
||||
| 144 | [@wuweiqiang24](https://github.com/wuweiqiang24) | 2025/09/11 | [9615dea](https://github.com/vllm-project/vllm-ascend/commit/9615dea3a71df8ecd2c591f284d9615140dce68a) |
|
||||
| 143 | [@wenba0](https://github.com/wenba0) | 2025/09/11 | [bd3dede](https://github.com/vllm-project/vllm-ascend/commit/bd3dedea6123c9c8c19fe83b6e05716f63b1285d) |
|
||||
| 142 | [@anon189Ty](https://github.com/anon189Ty) | 2025/09/11 | [7b2ecc1](https://github.com/vllm-project/vllm-ascend/commit/7b2ecc1e9a64aeda78e2137aa06abdbf2890c000) |
|
||||
| 141 | [@fffrog](https://github.com/fffrog) | 2025/09/10 | [b7ee3fd](https://github.com/vllm-project/vllm-ascend/commit/b7ee3fdad30d00d9aaa31be04315838c7e2c24ac) |
|
||||
| 140 | [@fan2956](https://github.com/fan2956) | 2025/09/10 | [bfc9cdc](https://github.com/vllm-project/vllm-ascend/commit/bfc9cdc1e3a7c4a148ae3e19b9380443e2f78550) |
|
||||
| 139 | [@realliujiaxu](https://github.com/realliujiaxu) | 2025/09/08 | [d3c3538](https://github.com/vllm-project/vllm-ascend/commit/d3c3538ddc67ba8f4873637e2bc1052f9eb09e93) |
|
||||
| 138 | [@marcobarlo](https://github.com/marcobarlo) | 2025/09/08 | [6666e52](https://github.com/vllm-project/vllm-ascend/commit/6666e5265d40ecafc3cb377233fee840d7fe553b) |
|
||||
| 137 | [@machenglong2025](https://github.com/machenglong2025) | 2025/09/08 | [1a82b16](https://github.com/vllm-project/vllm-ascend/commit/1a82b16355d2ec0ba01c23935092dc0af323b820) |
|
||||
| 136 | [@1092626063](https://github.com/1092626063) | 2025/09/05 | [5b3646a](https://github.com/vllm-project/vllm-ascend/commit/5b3646ab2142131579661ce12e4f0e4ba731ad06) |
|
||||
| 135 | [@WithHades](https://github.com/WithHades) | 2025/09/04 | [0c0789b](https://github.com/vllm-project/vllm-ascend/commit/0c0789be7442122eb1203abbf89a9592648922e0) |
|
||||
| 134 | [@vllm-ascend-ci](https://github.com/vllm-ascend-ci) | 2025/09/04 | [3a2a7d8](https://github.com/vllm-project/vllm-ascend/commit/3a2a7d88dbaf4793fb8d9405040bb28348df28e9) |
|
||||
| 133 | [@baxingpiaochong](https://github.com/baxingpiaochong) | 2025/09/04 | [df88a2e](https://github.com/vllm-project/vllm-ascend/commit/df88a2ecc8116a42d79a13fa1a8a05a03c70324f) |
|
||||
| 132 | [@zzy-ContiLearn](https://github.com/zzy-ContiLearn) | 2025/09/03 | [07d44ad](https://github.com/vllm-project/vllm-ascend/commit/07d44ade194b018ae2cc172482d55cb746c5fd0e) |
|
||||
| 131 | [@panchao-hub](https://github.com/panchao-hub) | 2025/08/30 | [7215454](https://github.com/vllm-project/vllm-ascend/commit/7215454de6df78f4f9a49a99c5739f8bb360f5bc) |
|
||||
| 130 | [@wangxiaoteng888](https://github.com/wangxiaoteng888) | 2025/08/29 | [ee6d141](https://github.com/vllm-project/vllm-ascend/commit/ee6d141dd4aa7f6bcef1b52d49614bd74705822f) |
|
||||
| 129 | [@lidenghui1110](https://github.com/lidenghui1110) | 2025/08/29 | [600b08f](https://github.com/vllm-project/vllm-ascend/commit/600b08f7542be3409c2c70927c91471e8de33d03) |
|
||||
| 128 | [@NSDie](https://github.com/NSDie) | 2025/08/28 | [1191a64](https://github.com/vllm-project/vllm-ascend/commit/1191a64ae508183d5613711bc98a90250963f83a) |
|
||||
| 127 | [@s-jiayang](https://github.com/s-jiayang) | 2025/08/27 | [6a4ec18](https://github.com/vllm-project/vllm-ascend/commit/6a4ec186e731b9516235f4fd30b5b98227513fe7) |
|
||||
| 126 | [@ZhaoJiangJiang](https://github.com/ZhaoJiangJiang) | 2025/08/22 | [3629bc4](https://github.com/vllm-project/vllm-ascend/commit/3629bc4431d3edb4224761f9036b3bddb16158d6) |
|
||||
| 125 | [@LookAround0301](https://github.com/LookAround0301) | 2025/08/22 | [e9fb895](https://github.com/vllm-project/vllm-ascend/commit/e9fb895b10cef37ea634f4d4af71686b09ca9f20) |
|
||||
| 124 | [@NicholasTao](https://github.com/NicholasTao) | 2025/08/20 | [7bec1a9](https://github.com/vllm-project/vllm-ascend/commit/7bec1a9b9c372785551d45682bf11063ec42b216) |
|
||||
| 123 | [@liuchenbing](https://github.com/liuchenbing) | 2025/08/19 | [3648d18](https://github.com/vllm-project/vllm-ascend/commit/3648d18e673f15a33a82d6ea95d3a9dd891ff1f5) |
|
||||
| 122 | [@gameofdimension](https://github.com/gameofdimension) | 2025/08/19 | [27d038d](https://github.com/vllm-project/vllm-ascend/commit/27d038dc663bf550a35a8f15659493b2abefda07) |
|
||||
| 121 | [@LCAIZJ](https://github.com/LCAIZJ) | 2025/08/18 | [03ca2b2](https://github.com/vllm-project/vllm-ascend/commit/03ca2b26ca9ab6b9a12f021b0595a726ee35e223) |
|
||||
| 120 | [@hust17yixuan](https://github.com/hust17yixuan) | 2025/08/14 | [17c2884](https://github.com/vllm-project/vllm-ascend/commit/17c2884f43011724def1bfe8ac68f9952e90b677) |
|
||||
| 119 | [@QwertyJack](https://github.com/QwertyJack) | 2025/08/11 | [9c6d108](https://github.com/vllm-project/vllm-ascend/commit/9c6d108330574176f79eea52f989ea6049336af8) |
|
||||
| 118 | [@haojiangzheng](https://github.com/haojiangzheng) | 2025/08/11 | [eb43a47](https://github.com/vllm-project/vllm-ascend/commit/eb43a475f429192e7509e85e28b1c65d5097f373) |
|
||||
| 117 | [@yangqinghao-cmss](https://github.com/yangqinghao-cmss) | 2025/08/01 | [99fa0ac](https://github.com/vllm-project/vllm-ascend/commit/99fa0ac882c79ae9282940125b042a44ea422757) |
|
||||
| 116 | [@Liccol](https://github.com/Liccol) | 2025/07/31 | [7c90ba5](https://github.com/vllm-project/vllm-ascend/commit/7c90ba5fe8e420b891fdd30df050a33e3767835d) |
|
||||
| 115 | [@1024daniel](https://github.com/1024daniel) | 2025/07/31 | [db310c6](https://github.com/vllm-project/vllm-ascend/commit/db310c6ec97b056296f7c2348b90c1d96d0b562a) |
|
||||
| 114 | [@YuanCheng-coder](https://github.com/YuanCheng-coder) | 2025/07/30 | [34dd24a](https://github.com/vllm-project/vllm-ascend/commit/34dd24adf21fb85a2c413292754b1599832efae2) |
|
||||
| 113 | [@hongfugui](https://github.com/hongfugui) | 2025/07/30 | [1dbb888](https://github.com/vllm-project/vllm-ascend/commit/1dbb8882759e4326f5706f6e610674423376c2f3) |
|
||||
| 112 | [@taoxudonghaha](https://github.com/taoxudonghaha) | 2025/07/29 | [540336e](https://github.com/vllm-project/vllm-ascend/commit/540336edc9db09072a9aaa486fbf7ce625da5b9e) |
|
||||
| 111 | [@wanghanqingLYT](https://github.com/wanghanqingLYT) | 2025/07/26 | [833d33c](https://github.com/vllm-project/vllm-ascend/commit/833d33cf09117a32e17cf4057b29709011f6e815) |
|
||||
| 110 | [@pjgao](https://github.com/pjgao) | 2025/07/26 | [149bed1](https://github.com/vllm-project/vllm-ascend/commit/149bed15d6da19cab194dc842fc12419cbef088b) |
|
||||
| 109 | [@lbk-sys](https://github.com/lbk-sys) | 2025/07/26 | [3c7c284](https://github.com/vllm-project/vllm-ascend/commit/3c7c284b980eb224c6ab231061c7e26b24534922) |
|
||||
| 108 | [@CaveNightingale](https://github.com/CaveNightingale) | 2025/07/26 | [fbb9326](https://github.com/vllm-project/vllm-ascend/commit/fbb9326f8f1e0d69c280655006eebf589e9c8ad2) |
|
||||
| 107 | [@SlightwindSec](https://github.com/SlightwindSec) | 2025/07/25 | [515237a](https://github.com/vllm-project/vllm-ascend/commit/515237ac6ec7b9b65b81602d9e017b9120a889ae) |
|
||||
| 106 | [@Ronald1995](https://github.com/Ronald1995) | 2025/07/25 | [e561a2c](https://github.com/vllm-project/vllm-ascend/commit/e561a2c6ec4493b490b13a4a9007d8f451ae0d0f) |
|
||||
| 105 | [@ZrBac](https://github.com/ZrBac) | 2025/07/24 | [2ffe051](https://github.com/vllm-project/vllm-ascend/commit/2ffe051859d585df8353d1b9eefb64c44078175a) |
|
||||
| 104 | [@SunnyLee151064](https://github.com/SunnyLee151064) | 2025/07/24 | [34571ea](https://github.com/vllm-project/vllm-ascend/commit/34571ea5ae69529758edf75f0252f86ccb4c7184) |
|
||||
| 103 | [@raindaywhu](https://github.com/raindaywhu) | 2025/07/24 | [b235fb7](https://github.com/vllm-project/vllm-ascend/commit/b235fb738df680894604b6a3b68b68ed43f4de9a) |
|
||||
| 102 | [@lilinsiman](https://github.com/lilinsiman) | 2025/07/24 | [718c544](https://github.com/vllm-project/vllm-ascend/commit/718c544e11dc71c8d0ddde6159affcdeeac109b5) |
|
||||
| 101 | [@shiyuan680](https://github.com/shiyuan680) | 2025/07/23 | [ac0bf13](https://github.com/vllm-project/vllm-ascend/commit/ac0bf133f47ead20f18bf71f9be6dbe05fbd218f) |
|
||||
| 100 | [@shaopeng-666](https://github.com/shaopeng-666) | 2025/07/23 | [54b1753](https://github.com/vllm-project/vllm-ascend/commit/54b1753cf56945e88c9d2449a7d6b6adf8638ead) |
|
||||
| 99 | [@loukong33](https://github.com/loukong33) | 2025/07/22 | [b73c701](https://github.com/vllm-project/vllm-ascend/commit/b73c701a6038d14a65db099e0d2da9f61f239044) |
|
||||
| 98 | [@aidoczh](https://github.com/aidoczh) | 2025/07/21 | [c32eea9](https://github.com/vllm-project/vllm-ascend/commit/c32eea96b73d26268070f57ef98416decc98aff7) |
|
||||
| 97 | [@nuclearwu](https://github.com/nuclearwu) | 2025/07/20 | [54f2b31](https://github.com/vllm-project/vllm-ascend/commit/54f2b311848badc86371d269140e729012a60f2c) |
|
||||
| 96 | [@pkking](https://github.com/pkking) | 2025/07/18 | [3e39d72](https://github.com/vllm-project/vllm-ascend/commit/3e39d7234c0e5c66b184c136c602e87272b5a36e) |
|
||||
| 95 | [@lianyiibo](https://github.com/lianyiibo) | 2025/07/18 | [53d2ea3](https://github.com/vllm-project/vllm-ascend/commit/53d2ea3789ffce32bf3ceb055d5582d28eadc6c7) |
|
||||
| 94 | [@zheliuyu](https://github.com/zheliuyu) | 2025/07/15 | [a2a6377](https://github.com/vllm-project/vllm-ascend/commit/a2a6377d45376b4f09bf6141f27284ca013ac075) |
|
||||
| 93 | [@NNUCJ](https://github.com/NNUCJ) | 2025/07/10 | [3b99491](https://github.com/vllm-project/vllm-ascend/commit/3b994919876a40030eb1045e01dd7bc3c7bac6a6) |
|
||||
| 92 | [@ZhengWG](https://github.com/ZhengWG) | 2025/07/07 | [9c886d0](https://github.com/vllm-project/vllm-ascend/commit/9c886d0a1f0fc011692090b0395d734c83a469de) |
|
||||
| 91 | [@wm901115nwpu](https://github.com/wm901115nwpu) | 2025/07/07 | [f08c4f1](https://github.com/vllm-project/vllm-ascend/commit/f08c4f15a27f0f27132f4ca7a0c226bf0a2a47d4) |
|
||||
| 90 | [@JC-ut0](https://github.com/JC-ut0) | 2025/07/04 | [e1d282d](https://github.com/vllm-project/vllm-ascend/commit/e1d282d7cc017f7e8075074a6981532045801a73) |
|
||||
| 89 | [@Irving11-BKN](https://github.com/Irving11-BKN) | 2025/07/03 | [3ea2410](https://github.com/vllm-project/vllm-ascend/commit/3ea24101416675663d2c3c2874054b9dd3451bd0) |
|
||||
| 88 | [@zhanghw0354](https://github.com/zhanghw0354) | 2025/07/02 | [9fb3d55](https://github.com/vllm-project/vllm-ascend/commit/9fb3d558e5b57a3c97ee5e11b9f5dba6ad3df9a5) |
|
||||
| 87 | [@xudongLi-cmss](https://github.com/xudongLi-cmss) | 2025/07/02 | [7fc1a98](https://github.com/vllm-project/vllm-ascend/commit/7fc1a984890bd930f670deedcb2dda3a46f84576) |
|
||||
| 86 | [@GDzhu01](https://github.com/GDzhu01) | 2025/06/28 | [b308a7a](https://github.com/vllm-project/vllm-ascend/commit/b308a7a25897b88d4a23a9e3d583f4ec6de256ac) |
|
||||
| 85 | [@zeshengzong](https://github.com/zeshengzong) | 2025/06/26 | [192dbbc](https://github.com/vllm-project/vllm-ascend/commit/192dbbcc6e244a8471d3c00033dc637233ee25aa) |
|
||||
| 84 | [@leo-pony](https://github.com/leo-pony) | 2025/06/26 | [1025344](https://github.com/vllm-project/vllm-ascend/commit/10253449120307e3b45f99d82218ba53e3f2a5f2) |
|
||||
| 83 | [@sharonyunyun](https://github.com/sharonyunyun) | 2025/06/25 | [941269a](https://github.com/vllm-project/vllm-ascend/commit/941269a6c5bbc79f6c1b6abd4680dc5802dd8666) |
|
||||
| 82 | [@Pr0Wh1teGivee](https://github.com/Pr0Wh1teGivee) | 2025/06/25 | [2fda604](https://github.com/vllm-project/vllm-ascend/commit/2fda60464c287fe456b4a2f27e63996edc65dd40) |
|
||||
| 81 | [@liziyu179](https://github.com/liziyu179) | 2025/06/24 | [6ed3f00](https://github.com/vllm-project/vllm-ascend/commit/6ed3f004274b4c89df2676c888b804e103efaf79) |
|
||||
| 80 | [@xleoken](https://github.com/xleoken) | 2025/06/23 | [4447e53](https://github.com/vllm-project/vllm-ascend/commit/4447e53d7ad5edcda978ca6b0a3a26a73c604de0) |
|
||||
| 79 | [@weiguihua2](https://github.com/weiguihua2) | 2025/06/23 | [e112317](https://github.com/vllm-project/vllm-ascend/commit/e1123172d12afa15f306ba6e1e4c9d0c6d1d799e) |
|
||||
| 78 | [@lyj-jjj](https://github.com/lyj-jjj) | 2025/06/23 | [5177bef](https://github.com/vllm-project/vllm-ascend/commit/5177bef87a21331dcca11159d3d1438075cbd74e) |
|
||||
| 77 | [@zhoux77899](https://github.com/zhoux77899) | 2025/06/21 | [a927bee](https://github.com/vllm-project/vllm-ascend/commit/a927bee4f098e7b95a255a064c95adea73ee2a50) |
|
||||
| 76 | [@yuancaoyaoHW](https://github.com/yuancaoyaoHW) | 2025/06/20 | [00ae250](https://github.com/vllm-project/vllm-ascend/commit/00ae250f3ced68317bc91c93dc1f1a0977aa0b94) |
|
||||
| 75 | [@shikang-hangzhou](https://github.com/shikang-hangzhou) | 2025/06/20 | [f1353d5](https://github.com/vllm-project/vllm-ascend/commit/f1353d51b32ddf22444b0608b3d7f60fed048ac4) |
|
||||
| 74 | [@pichangping](https://github.com/pichangping) | 2025/06/20 | [53ce4a0](https://github.com/vllm-project/vllm-ascend/commit/53ce4a0ad232bf36fd40da059b0cfc02c34669ac) |
|
||||
| 73 | [@farawayboat](https://github.com/farawayboat) | 2025/06/20 | [5ab6fdd](https://github.com/vllm-project/vllm-ascend/commit/5ab6fdd0edf149a14742f3c7a214cf5f9f792441) |
|
||||
| 72 | [@underfituu](https://github.com/underfituu) | 2025/06/19 | [e2c9ecb](https://github.com/vllm-project/vllm-ascend/commit/e2c9ecb8029731c0f86199e8371d3af7555504a7) |
|
||||
| 71 | [@songshanhu07](https://github.com/songshanhu07) | 2025/06/18 | [ebb2a70](https://github.com/vllm-project/vllm-ascend/commit/ebb2a70dbbdb8f55002de3313e17dfd595e1de1f) |
|
||||
| 70 | [@zhuo97](https://github.com/zhuo97) | 2025/06/16 | [f5404dc](https://github.com/vllm-project/vllm-ascend/commit/f5404dc650882c6f0423db9e87f9b38f756211c5) |
|
||||
| 69 | [@henryxuxu0716](https://github.com/henryxuxu0716) | 2025/06/14 | [38692b5](https://github.com/vllm-project/vllm-ascend/commit/38692b5c005fa05625e05f8df600d7df712ecd47) |
|
||||
| 68 | [@fems14](https://github.com/fems14) | 2025/06/14 | [ab5d110](https://github.com/vllm-project/vllm-ascend/commit/ab5d110fcc35ca11330977450141b1d7176f21e7) |
|
||||
| 67 | [@wangyanhui-cmss](https://github.com/wangyanhui-cmss) | 2025/06/12 | [c6e2a5f](https://github.com/vllm-project/vllm-ascend/commit/c6e2a5fb4014b863cee6abc3009f5bc5340c9e88) |
|
||||
| 66 | [@yzim](https://github.com/yzim) | 2025/06/11 | [4153a50](https://github.com/vllm-project/vllm-ascend/commit/4153a5091b698c2270d160409e7fee73baaf701b) |
|
||||
| 65 | [@chenwaner](https://github.com/chenwaner) | 2025/06/11 | [e46dc14](https://github.com/vllm-project/vllm-ascend/commit/e46dc142bf1180453c64226d76854fc1ec696169) |
|
||||
| 64 | [@Yuxiao-Xu](https://github.com/Yuxiao-Xu) | 2025/06/09 | [6b853f1](https://github.com/vllm-project/vllm-ascend/commit/6b853f15fe69ba335d2745ebcf14a164d0bcc505) |
|
||||
| 63 | [@zxdukki](https://github.com/zxdukki) | 2025/06/07 | [87ebaef](https://github.com/vllm-project/vllm-ascend/commit/87ebaef4e4e519988f27a6aa378f614642202ecf) |
|
||||
| 62 | [@weijinqian0](https://github.com/weijinqian0) | 2025/06/07 | [e9ada68](https://github.com/vllm-project/vllm-ascend/commit/e9ada685ece798f9fe0d4a287e3f5246a8a7207b) |
|
||||
| 61 | [@sdmyzlp](https://github.com/sdmyzlp) | 2025/06/07 | [3640c60](https://github.com/vllm-project/vllm-ascend/commit/3640c60b0eb4d4cb104e20bfa406d3f1d17920a7) |
|
||||
| 60 | [@ChenTaoyu-SJTU](https://github.com/ChenTaoyu-SJTU) | 2025/06/07 | [20dedba](https://github.com/vllm-project/vllm-ascend/commit/20dedba5d1fc84b7ae8b49f9ce3e3649389e2193) |
|
||||
| 59 | [@hahazhky](https://github.com/hahazhky) | 2025/06/06 | [0b12c2a](https://github.com/vllm-project/vllm-ascend/commit/0b12c2acf7d9fd192beebebf662298067d9a5435) |
|
||||
| 58 | [@depeng1994](https://github.com/depeng1994) | 2025/06/06 | [6b094a2](https://github.com/vllm-project/vllm-ascend/commit/6b094a2bd49a8a41eb3647568b2d9e5b337db81f) |
|
||||
| 57 | [@momo609](https://github.com/momo609) | 2025/06/05 | [908a851](https://github.com/vllm-project/vllm-ascend/commit/908a851a776cfd9051cc062119e6ec481561c6f7) |
|
||||
| 56 | [@David9857](https://github.com/David9857) | 2025/06/05 | [78431b3](https://github.com/vllm-project/vllm-ascend/commit/78431b34694dfa3c8f54ed7cc626660318557927) |
|
||||
| 55 | [@NINGBENZHE](https://github.com/NINGBENZHE) | 2025/06/03 | [6ec64a3](https://github.com/vllm-project/vllm-ascend/commit/6ec64a3f9686df65b5a23a41aa301e669db19099) |
|
||||
| 54 | [@XWFAlone](https://github.com/XWFAlone) | 2025/05/30 | [3442fbd](https://github.com/vllm-project/vllm-ascend/commit/3442fbdb235b4c6d72c2bc64a49707a7bd89958e) |
|
||||
| 53 | [@YisongJiang](https://github.com/YisongJiang) | 2025/05/29 | [90afaf6](https://github.com/vllm-project/vllm-ascend/commit/90afaf6306f680307462becf3c78585737579851) |
|
||||
| 52 | [@yangpuPKU](https://github.com/yangpuPKU) | 2025/05/23 | [46df67a](https://github.com/vllm-project/vllm-ascend/commit/46df67a5e9ab73fade08cbb2d8c0155cee7316d1) |
|
||||
| 51 | [@ttanzhiqiang](https://github.com/ttanzhiqiang) | 2025/05/23 | [dc6172e](https://github.com/vllm-project/vllm-ascend/commit/dc6172efd3860ce95b40a7b3e93611f875f06d40) |
|
||||
| 50 | [@jiangpeng36](https://github.com/jiangpeng36) | 2025/05/23 | [df58fb8](https://github.com/vllm-project/vllm-ascend/commit/df58fb80eee24139fc61c495be3ce79cf81b3f73) |
|
||||
| 49 | [@wonderful199082](https://github.com/wonderful199082) | 2025/05/20 | [5cf9ff1](https://github.com/vllm-project/vllm-ascend/commit/5cf9ff18e91b0b7031c258d71a257b8e24689763) |
|
||||
| 48 | [@22dimensions](https://github.com/22dimensions) | 2025/05/17 | [a8730e7](https://github.com/vllm-project/vllm-ascend/commit/a8730e7a3c4ac6c4b39a5946c943252fdea6cce5) |
|
||||
| 47 | [@cxcxflying](https://github.com/cxcxflying) | 2025/05/13 | [e564470](https://github.com/vllm-project/vllm-ascend/commit/e56447033889ca95df512208cab22ef832bfdf07) |
|
||||
| 46 | [@NeverRaR](https://github.com/NeverRaR) | 2025/05/12 | [efabd72](https://github.com/vllm-project/vllm-ascend/commit/efabd722eb757e49aa309c173bbec91ca8c4ced1) |
|
||||
| 45 | [@chris668899](https://github.com/chris668899) | 2025/05/08 | [6c02088](https://github.com/vllm-project/vllm-ascend/commit/6c020883a8332b5c519f4f6502733edd9b391c2b) |
|
||||
| 44 | [@sunbaosong](https://github.com/sunbaosong) | 2025/05/06 | [d6bfae8](https://github.com/vllm-project/vllm-ascend/commit/d6bfae8eeebedf677b643b712d367a3a69c9cce4) |
|
||||
| 43 | [@ApsarasX](https://github.com/ApsarasX) | 2025/04/29 | [87975fa](https://github.com/vllm-project/vllm-ascend/commit/87975fa058fe3f90d204ded42a08989a8dcb413e) |
|
||||
| 42 | [@zouyida2052](https://github.com/zouyida2052) | 2025/04/28 | [b9528e6](https://github.com/vllm-project/vllm-ascend/commit/b9528e6ecdc417cf444e55a0ce4a2bafdef0ea3b) |
|
||||
| 41 | [@ZhengJun9](https://github.com/ZhengJun9) | 2025/04/28 | [1791113](https://github.com/vllm-project/vllm-ascend/commit/17911138c90d78a76bd691e9dcb56763db35b19f) |
|
||||
| 40 | [@linfeng-yuan](https://github.com/linfeng-yuan) | 2025/04/28 | [2204e4d](https://github.com/vllm-project/vllm-ascend/commit/2204e4d08f8e10cf9c30154a14eaa5ca956c2acd) |
|
||||
| 39 | [@jianzs](https://github.com/jianzs) | 2025/04/27 | [fa4a5d9](https://github.com/vllm-project/vllm-ascend/commit/fa4a5d980e8845a88b9162cf169f0a5ab230f8a5) |
|
||||
| 38 | [@RongRongStudio](https://github.com/RongRongStudio) | 2025/04/23 | [848e041](https://github.com/vllm-project/vllm-ascend/commit/848e041a54732c923660dd02daf8e9bf439736a2) |
|
||||
| 37 | [@fakeYan](https://github.com/fakeYan) | 2025/04/23 | [05bdcbe](https://github.com/vllm-project/vllm-ascend/commit/05bdcbeae47c7fcb9b1c30cad059abf1d40b5421) |
|
||||
| 36 | [@paulyu12](https://github.com/paulyu12) | 2025/04/17 | [697908f](https://github.com/vllm-project/vllm-ascend/commit/697908f5cd7c65a3a917ec1a962b0886efc98c7e) |
|
||||
| 35 | [@heartStrive1998](https://github.com/heartStrive1998) | 2025/04/16 | [2f15503](https://github.com/vllm-project/vllm-ascend/commit/2f155039dc3997640854daef469bbf0cb77dc6ed) |
|
||||
| 34 | [@eeethenQ](https://github.com/eeethenQ) | 2025/04/15 | [44a8301](https://github.com/vllm-project/vllm-ascend/commit/44a8301424ded94dae83e13b837f5bfc0a1bfc15) |
|
||||
| 33 | [@wxsIcey](https://github.com/wxsIcey) | 2025/04/10 | [d05ea17](https://github.com/vllm-project/vllm-ascend/commit/d05ea17427b82a506b97409a7de8359f18f565f7) |
|
||||
| 32 | [@yx0716](https://github.com/yx0716) | 2025/04/08 | [5d62393](https://github.com/vllm-project/vllm-ascend/commit/5d6239306be9b0f5ac6dbaa137048c372a92ff20) |
|
||||
| 31 | [@antonlisq](https://github.com/antonlisq) | 2025/04/07 | [2b765dc](https://github.com/vllm-project/vllm-ascend/commit/2b765dcc4974b1bafc26ff5da817ce7e652f0eb0) |
|
||||
| 30 | [@zzzzwwjj](https://github.com/zzzzwwjj) | 2025/03/28 | [12390af](https://github.com/vllm-project/vllm-ascend/commit/12390af075962456ecc8233d8dcce7064b75f390) |
|
||||
| 29 | [@wuhuikx](https://github.com/wuhuikx) | 2025/03/28 | [57a84bb](https://github.com/vllm-project/vllm-ascend/commit/57a84bb7befeaa0dc62aa35fa406e4d6affbfcca) |
|
||||
| 28 | [@ZhengZhenyu](https://github.com/ZhengZhenyu) | 2025/03/26 | [0b5a964](https://github.com/vllm-project/vllm-ascend/commit/0b5a9643fd6c3240d7ede669e37209d7ff433841) |
|
||||
| 27 | [@baifanxxx](https://github.com/baifanxxx) | 2025/03/26 | [1225052](https://github.com/vllm-project/vllm-ascend/commit/122505208ff6284f409846ca7294f4a4b9883285) |
|
||||
| 26 | [@zhangxinyuehfad](https://github.com/zhangxinyuehfad) | 2025/03/21 | [608ea3d](https://github.com/vllm-project/vllm-ascend/commit/608ea3d679b3b7a8dfff878ff78599b74f09606d) |
|
||||
| 25 | [@cllouud](https://github.com/cllouud) | 2025/03/18 | [d89d28d](https://github.com/vllm-project/vllm-ascend/commit/d89d28decaf5443051fc76ad2c9f66a2f56d4b9a) |
|
||||
| 24 | [@rjg-lyh](https://github.com/rjg-lyh) | 2025/03/13 | [6512470](https://github.com/vllm-project/vllm-ascend/commit/65124705fb39d4cc2c94c80254421e067a82fe50) |
|
||||
| 23 | [@xiemingda-1002](https://github.com/xiemingda-1002) | 2025/03/12 | [59ea23d](https://github.com/vllm-project/vllm-ascend/commit/59ea23d0d394879d7f33de6fd22242539b9c3cc5) |
|
||||
| 22 | [@yiz-liu](https://github.com/yiz-liu) | 2025/03/11 | [0db6670](https://github.com/vllm-project/vllm-ascend/commit/0db6670bfab8cb1d84c9e7270df0a1d42d6ce7ca) |
|
||||
| 21 | [@new-TonyWang](https://github.com/new-TonyWang) | 2025/03/11 | [dfb4e23](https://github.com/vllm-project/vllm-ascend/commit/dfb4e23e9d820ac992a071c123bbe983c7b01b2e) |
|
||||
| 20 | [@mengwei805](https://github.com/mengwei805) | 2025/03/06 | [8fcf3d1](https://github.com/vllm-project/vllm-ascend/commit/8fcf3d1704084626db35c5dc82ade446508598d4) |
|
||||
| 19 | [@baymax591](https://github.com/baymax591) | 2025/02/28 | [e8131b9](https://github.com/vllm-project/vllm-ascend/commit/e8131b99cf199f50a304e6e6fb125a1b95bcc92b) |
|
||||
| 18 | [@wwfu109](https://github.com/wwfu109) | 2025/02/27 | [b074047](https://github.com/vllm-project/vllm-ascend/commit/b07404766bdaf6e3cebc5cb0aba89a247501302e) |
|
||||
| 17 | [@shink](https://github.com/shink) | 2025/02/27 | [6aed833](https://github.com/vllm-project/vllm-ascend/commit/6aed83335cbe92fd0b8ef07c28966a753d012ccb) |
|
||||
| 16 | [@kunpengW-code](https://github.com/kunpengW-code) | 2025/02/26 | [ca807ce](https://github.com/vllm-project/vllm-ascend/commit/ca807ce49ed64aa89242f5ae29b9862a77648b45) |
|
||||
| 15 | [@Yaphets24](https://github.com/Yaphets24) | 2025/02/22 | [d0b3cb4](https://github.com/vllm-project/vllm-ascend/commit/d0b3cb4fa79d5fc7f8245a3c68885ce1fa030ba4) |
|
||||
| 14 | [@noemotiovon](https://github.com/noemotiovon) | 2025/02/21 | [202b39a](https://github.com/vllm-project/vllm-ascend/commit/202b39a38c2869b0ecc3df486550fb555a2eb0c0) |
|
||||
| 13 | [@SidaoY](https://github.com/SidaoY) | 2025/02/18 | [718c763](https://github.com/vllm-project/vllm-ascend/commit/718c7638555d12cd43ea2a9e497e185778b68595) |
|
||||
| 12 | [@ShiyaNiu](https://github.com/ShiyaNiu) | 2025/02/17 | [36ea38f](https://github.com/vllm-project/vllm-ascend/commit/36ea38fde56437ff1745bd95cd8d9e02a6578d38) |
|
||||
| 11 | [@ji-huazhong](https://github.com/ji-huazhong) | 2025/02/12 | [c8b57d1](https://github.com/vllm-project/vllm-ascend/commit/c8b57d10b24efcd9b4fadeb66cfbf66aa3dd5f82) |
|
||||
| 10 | [@Angazenn](https://github.com/Angazenn) | 2025/02/11 | [7637759](https://github.com/vllm-project/vllm-ascend/commit/7637759056028839c74960d9cfd3ce6275ee5d35) |
|
||||
| 9 | [@ganyi1996ppo](https://github.com/ganyi1996ppo) | 2025/02/08 | [49e5baf](https://github.com/vllm-project/vllm-ascend/commit/49e5baf8596bdf6239b01b4124e9a105d0e6b203) |
|
||||
| 8 | [@zouyida2002](https://github.com/zouyida2002) | 2025/02/07 | [4495fc6](https://github.com/vllm-project/vllm-ascend/commit/4495fc68389e3fb1ef14534c202948931e38446b) |
|
||||
| 7 | [@whx-sjtu](https://github.com/whx-sjtu) | 2025/02/07 | [8fc5dc9](https://github.com/vllm-project/vllm-ascend/commit/8fc5dc966aaf4e174d1ec0d1902c40289411ec0e) |
|
||||
| 6 | [@wangxiyuan](https://github.com/wangxiyuan) | 2025/02/06 | [a48b9ad](https://github.com/vllm-project/vllm-ascend/commit/a48b9addefd292af523644411d4ff4142dd4bc66) |
|
||||
| 5 | [@shen-shanshan](https://github.com/shen-shanshan) | 2025/02/06 | [bfccf73](https://github.com/vllm-project/vllm-ascend/commit/bfccf739e2fe121b54d9b198c2ec205a9379190e) |
|
||||
| 4 | [@Potabk](https://github.com/Potabk) | 2025/02/06 | [8cb5615](https://github.com/vllm-project/vllm-ascend/commit/8cb5615fb010b34c2f4f89e03e6257bfee851f86) |
|
||||
| 3 | [@MengqingCao](https://github.com/MengqingCao) | 2025/02/06 | [7d9ae22](https://github.com/vllm-project/vllm-ascend/commit/7d9ae22ecb6dc3ea4e720e5109cf46e1ae7da730) |
|
||||
| 2 | [@Yikun](https://github.com/Yikun) | 2025/02/05 | [d5e7756](https://github.com/vllm-project/vllm-ascend/commit/d5e7756028bd5884ade96b654555c375770a2f64) |
|
||||
| 1 | [@simon-mo](https://github.com/simon-mo) | 2025/01/29 | [eb28342](https://github.com/vllm-project/vllm-ascend/commit/eb283428ddc17207b6866118f9bc15454b5b8801) |
|
||||
|
||||
@@ -1,48 +1,51 @@
|
||||
# Governance
|
||||
|
||||
## Mission
|
||||
As a vital component of vLLM, the vLLM Ascend project is dedicated to providing an easy, fast, and cheap LLM Serving for Everyone on Ascend NPU, and to actively contribute to the enrichment of vLLM.
|
||||
|
||||
As a vital component of vLLM, the vLLM Ascend project is dedicated to providing an easy, fast, and cheap LLM Serving for everyone on Ascend NPUs and to actively contribute to the enrichment of vLLM.
|
||||
|
||||
## Principles
|
||||
vLLM Ascend follows the vLLM community's code of conduct：[vLLM - CODE OF CONDUCT](https://github.com/vllm-project/vllm/blob/main/CODE_OF_CONDUCT.md)
|
||||
|
||||
vLLM Ascend follows the vLLM community's code of conduct: [vLLM - CODE OF CONDUCT](https://github.com/vllm-project/vllm/blob/main/CODE_OF_CONDUCT.md)
|
||||
|
||||
## Governance - Mechanics
|
||||
|
||||
vLLM Ascend is an open-source project under the vLLM community, where the authority to appoint roles is ultimately determined by the vLLM community. It adopts a hierarchical technical governance structure.
|
||||
|
||||
- Contributor:
|
||||
|
||||
**Responsibility:** Help new contributors on boarding, handle and respond to community questions, review RFCs, code
|
||||
**Responsibility:** Help new contributors with onboarding, handle and respond to community questions, review RFCs and code.
|
||||
|
||||
**Requirements:** Complete at least 1 contribution. Contributor is someone who consistently and actively participates in a project, included but not limited to issue/review/commits/community involvement.
|
||||
**Requirements:** Complete at least 1 contribution. A contributor is someone who consistently and actively participates in a project, including but not limited to issue/review/commits/community involvement.
|
||||
|
||||
Contributors will be empowered [vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend) Github repo `Triage` permissions (`Can read and clone this repository. Can also manage issues and pull requests`) to help community developers collaborate more efficiently.
|
||||
Contributors are granted the `Triage` role in the [vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend) GitHub repository, which allows them to read and clone the repo, manage issues and PRs, and collaborate efficiently with community developers.
|
||||
|
||||
- Maintainer:
|
||||
|
||||
**Responsibility:** Develop the project's vision and mission. Maintainers are responsible for driving the technical direction of the entire project and ensuring its overall success, possessing code merge permissions. They formulate the roadmap, review contributions from community members, continuously contribute code, and actively engage in community activities (such as regular meetings/events).
|
||||
**Responsibility:** Develop the project's vision and mission. Maintainers are responsible for shaping the technical direction of the project and ensuring its long-term success. With code merge permissions, they lead roadmap planning, review community contributions, make ongoing code improvements, and actively participate in community engagement—such as regular meetings and events.
|
||||
|
||||
**Requirements:** Deep understanding of vLLM and vLLM Ascend codebases, with a commitment to sustained code contributions. Competency in design/development/PR review workflows.
|
||||
- **Review Quality:** Actively participate in community code reviews, ensuring high-quality code integration.
|
||||
- **Quality Contribution:** Successfully develop and deliver at least one major feature while maintaining consistent high-quality contributions.
|
||||
- **Community Involvement:** Actively address issues, respond to forum inquiries, participate in discussions, and engage in community-driven tasks.
|
||||
**Requirements:** Deep understanding of vLLM and vLLM Ascend code bases, with a commitment to sustained code contributions and competency in design, development, and PR review workflows.
|
||||
|
||||
Requires approval from existing Maintainers. The vLLM community has the final decision-making authority.
|
||||
- **Review quality:** Actively participate in community code reviews, ensuring high-quality code integration.
|
||||
- **Quality contribution:** Successfully develop and deliver at least one major feature while maintaining consistent high-quality contributions.
|
||||
- **Community involvement:** Actively address issues, respond to forum inquiries, participate in discussions, and engage in community-driven tasks.
|
||||
|
||||
Maintainer will be empowered [vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend) Github repo write permissions (`Can read, clone, and push to this repository. Can also manage issues and pull requests`).
|
||||
The approval from existing Maintainers is required. The vLLM community has the final decision-making authority.
|
||||
Maintainers will be granted write access to the [vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend) GitHub repo. This includes permission to read, clone, and push to the repository, as well as manage issues and pull requests.
|
||||
|
||||
## Nominating and Removing Maintainers
|
||||
|
||||
### The Principles
|
||||
|
||||
- Membership in vLLM Ascend is given to individuals on merit basis after they demonstrated strong expertise of the vLLM / vLLM Ascend through contributions, reviews and discussions.
|
||||
- Membership in vLLM Ascend is given to individuals on a merit basis after they demonstrate their strong expertise in vLLM/vLLM Ascend through contributions, reviews, and discussions.
|
||||
|
||||
- For membership in the maintainer group the individual has to demonstrate strong and continued alignment with the overall vLLM / vLLM Ascend principles.
|
||||
- For membership in the maintainer group, individuals have to demonstrate strong and continued alignment with the overall vLLM/vLLM Ascend principles.
|
||||
|
||||
- Light criteria of moving module maintenance to ‘emeritus’ status if they don’t actively participate over long periods of time.
|
||||
- Maintainers who have been inactive for a long time may be transitioned to **emeritus** status under lenient criteria.
|
||||
|
||||
- The membership is for an individual, not a company.
|
||||
|
||||
### Nomination and Removal
|
||||
|
||||
- Nomination: Anyone can nominate someone to become a maintainer (include self-nominate). All existing maintainers are responsible for evaluating the nomination. The nominator should provide nominee's info around the strength of the candidate to be a maintainer, include but not limited to review quality, quality contribution, community involvement.
|
||||
- Removal: Anyone can nominate a person to be removed from maintainer position (include self-nominate). All existing maintainers are responsible for evaluating the nomination. The nominator should provide nominee's info, include but not limited to lack of activity, conflict with the overall direction and other information that makes them unfit to be a maintainer.
|
||||
- Nomination: Anyone can nominate a candidate to become a maintainer, including self-nominations. All existing maintainers are responsible for reviewing and evaluating each nomination. The nominator should provide relevant information about the nominee's qualifications—such as review quality, quality contribution, and community involvement—among other strengths.
|
||||
- Removal: Anyone may nominate an individual for removal from the maintainer role, including self-nominations. All current maintainers are responsible for reviewing and evaluating such nominations. The nominator should provide relevant information about the nominee—such as prolonged inactivity, misalignment with the project's overall direction, or other factors that may render them unsuitable for the maintainer position.
|
||||
|
||||
BIN
docs/source/community/images/ascend_assistant.png
Normal file
BIN
docs/source/community/images/ascend_assistant.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
BIN
docs/source/community/images/issue_label_workflow.png
Normal file
BIN
docs/source/community/images/issue_label_workflow.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
82
docs/source/community/issue-workflow-guidelines.md
Normal file
82
docs/source/community/issue-workflow-guidelines.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# Issue Workflow Guidelines
|
||||
|
||||
This document defines the standard lifecycle for [Issues](https://github.com/vllm-project/vllm-ascend/issues) in the vLLM Ascend project — from creation through triage, active handling, and final closure. It establishes consistent label usage, owner assignment, and communication expectations to ensure smooth collaboration between contributors and maintainers.
|
||||
|
||||
## 1. Label Categories
|
||||
|
||||
### 1.1 Status Labels
|
||||
|
||||
These labels track where an issue stands in the workflow.
|
||||
|
||||
| Label | Description |
|
||||
|-----------------|----------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `triage review` | Newly filed or unseen issue awaiting initial assessment by a maintainer |
|
||||
| `triaged` | Assessment complete; type, priority, and module have been determined |
|
||||
| `wait-feedback` | Blocked on an external dependency or awaiting a response before work can proceed |
|
||||
| `resolved` | Issue has been closed — either via a merged PR, or through non-code resolution (e.g., answered question, configuration guidance) |
|
||||
| `stale` | No activity for an extended period; parties have been notified and the issue will be auto-closed if there is no response |
|
||||
| `duplicated` | A duplicate of an existing open issue or merged PR |
|
||||
| `invalid` | The issue report is invalid, unclear, or not reproducible |
|
||||
| `wontfix` | The issue will not be addressed; as this issue is out of scope, unviable, or intentionally ignored for the foreseeable future |
|
||||
|
||||
### 1.2 Type Labels
|
||||
|
||||
These labels describe the nature of the issue.
|
||||
|
||||
| Label | Description |
|
||||
|-------------------|--------------------------------------------------------------------------------------------------|
|
||||
| `feature request` | Request for new functionality |
|
||||
| `RFC` | Request for Comments — significant architectural or design change requiring community discussion |
|
||||
| `new model` | Request to add support for a new model on Ascend NPU |
|
||||
| `usage` | A usage question; no code change may be required |
|
||||
| `question` | A general question; no code change may be required |
|
||||
| `documentation` | Improvements or corrections to documentation |
|
||||
| `installation` | Issues related to setup and deployment |
|
||||
| `performance` | Performance regression, bottleneck, or optimization request |
|
||||
| `bug` | Something is not working correctly or behaves unexpectedly |
|
||||
|
||||
### 1.3 Priority Labels (Optional)
|
||||
|
||||
| Label | Description |
|
||||
|----------|----------------------------------------------------------------|
|
||||
| `high` | High priority; should be resolved in the current or next cycle |
|
||||
| `medium` | Normal priority; handled in the regular development flow |
|
||||
| `low` | Low priority; edge case or minor issue that can be deferred |
|
||||
|
||||
### 1.4 Contribution Labels (Optional)
|
||||
|
||||
| Label | Description |
|
||||
|--------------------|------------------------------------------------------------------|
|
||||
| `good first issue` | A well-scoped, low-complexity task suitable for new contributors |
|
||||
| `help wanted` | Community contributions are welcome and encouraged |
|
||||
|
||||
## 2. Workflow
|
||||
|
||||
|
||||
|
||||
### Phase 1 — First Response
|
||||
|
||||
When an issue is first picked up by the on-call maintainer:
|
||||
|
||||
- Apply `triaged` to signal that the issue can be classified and add the relevant module label so the issue can be routed to the appropriate module maintainer for detailed triage.
|
||||
- Apply `triage review` to signal that the issue requires more review and specific analysis before classification.
|
||||
|
||||
### Phase 2 — Triage and Analysis
|
||||
|
||||
After a thorough review of the issue content:
|
||||
|
||||
- Verify and apply the appropriate **issue type** label (`bug`, `feature request`, `RFC`, `question`, `documentation`, `installation`, `performance`, `new model`, etc.).
|
||||
- Handle terminal states:
|
||||
- For duplicates, apply the `duplicated` label, provide an explanation and a link to the existing issue or PR. If there are no further questions, close the issue.
|
||||
- For invalid reports, provide an explanation, apply the `invalid` and `resolved` label, and close the issue. The issue creator can comment or request to reopen if they have further questions.
|
||||
- Optionally apply a **priority** label (`high`, `medium`, or `low`).
|
||||
- If community contributions are welcome, apply `help wanted`. For well-scoped beginner-friendly tasks, also apply `good first issue`.
|
||||
- Assign the issue owner and replace `triage review` with `triaged` to indicate that triage is complete.
|
||||
|
||||
### Phase 3 — Closure
|
||||
|
||||
After triage, the issue moves into tracking and implementation:
|
||||
|
||||
- Keep the issue in progress until it is resolved through a merged PR or another confirmed resolution path.
|
||||
- Once the issue is resolved, apply `resolved` and close it, ideally with a reference to the merged PR or a short explanation of the resolution.
|
||||
- If the issue remains inactive for an extended period, apply `stale` as the final state before auto-closure.
|
||||
166
docs/source/community/slash-commands.md
Normal file
166
docs/source/community/slash-commands.md
Normal file
@@ -0,0 +1,166 @@
|
||||
# Slash Commands
|
||||
|
||||
vLLM Ascend supports slash commands in pull request comments to trigger CI workflows. See the [Permission](#permission) section for who can trigger each command.
|
||||
|
||||
## Available Commands
|
||||
|
||||
### `/e2e`
|
||||
|
||||
Run specific E2E tests under `tests/e2e/pull_request/`. Tests are automatically routed to the appropriate NPU runner based on the test path.
|
||||
|
||||
**Examples:**
|
||||
|
||||
```text
|
||||
# Run a single test on the default runner (a2 single card)
|
||||
/e2e tests/e2e/pull_request/one_card/test_attention.py
|
||||
|
||||
# Run multiple tests across different runners
|
||||
/e2e tests/e2e/pull_request/one_card/test_attention.py tests/e2e/pull_request/two_card/test_parallel.py
|
||||
|
||||
# Run tests on 310P
|
||||
/e2e tests/e2e/pull_request/one_card/_310p/test_310p_ops.py
|
||||
```
|
||||
|
||||
**Routing rules** (matched in order):
|
||||
|
||||
| Test path contains | Runner |
|
||||
|---|---|
|
||||
| `four_card/_310p` | 310P 4-card |
|
||||
| `_310p` (under `one_card`/`two_card`) | 310P single card |
|
||||
| `four_card` | A3 4-card |
|
||||
| `two_card` | A3 2-card |
|
||||
| Others (e.g. `one_card`) | A2 single card |
|
||||
|
||||
> Only test paths under `tests/e2e/pull_request/` are supported. Tests in `tests/e2e/nightly/`, `tests/e2e/models/`, or `tests/e2e/doctests/` are not accepted by `/e2e`. Use `/nightly` for nightly tests.
|
||||
|
||||
Tests are run against both the community vLLM version and the latest release.
|
||||
|
||||
### `/nightly`
|
||||
|
||||
Trigger specific nightly test cases on A2 and A3. Supports only PR comments. Test case names correspond to the `test_config.name` entries defined in `schedule_nightly_test_a2.yaml` and `schedule_nightly_test_a3.yaml`.
|
||||
|
||||
**Usage:**
|
||||
|
||||
| Syntax | Scope |
|
||||
|---|---|
|
||||
| `/nightly <test_cases>` | Runs on `main` branch |
|
||||
| `/nightly <test_cases> --branch <branch>` | Runs on the specified branch |
|
||||
|
||||
Use `--branch <name>` to specify a target branch. Without `--branch`, all arguments are treated as test cases (separated by commas or spaces) and the branch defaults to `main`.
|
||||
|
||||
> **Note**: When commenting on a PR, the tests run on the PR branch automatically in the triggered workflow; the `--branch` flag is primarily used in issue comments.
|
||||
|
||||
**Common test case names (A2):**
|
||||
|
||||
`test_custom_op`, `test_custom_op_multi_card`, `qwen3-vl-32b-instruct-w8a8`, `qwen3-32b-int8`, `MiniMax-M2.5-w8a8-QuaRot-A2`, `Qwen3.5-27B-w8a8-A2`, `Qwen3.5-397B-A17B-w4a8-mtp`, `accuracy-group`
|
||||
|
||||
**Common test case names (A3):**
|
||||
|
||||
`multi-node-deepseek-v3.2-W8A8-EP`, `mtpx-deepseek-r1-0528-w8a8`, `deepseek-r1-0528-w8a8`, `kimi-k2-thinking`, `qwen3-vl-235b-a22b-instruct-w8a8`, `custom-multi-ops`, ...
|
||||
|
||||
**Examples:**
|
||||
|
||||
```text
|
||||
# Run a single test case on main branch
|
||||
/nightly qwen3-vl-32b-instruct-w8a8
|
||||
|
||||
# Run on a specific release branch
|
||||
/nightly qwen3-vl-32b-instruct-w8a8 --branch releases/v0.23.0
|
||||
|
||||
# Run all tests on a specific branch
|
||||
/nightly all --branch my-feature-branch
|
||||
|
||||
# Run multiple test cases (comma-separated)
|
||||
/nightly test_custom_op,multi-node-deepseek-v3.2-W8A8-EP
|
||||
|
||||
# Run multiple test cases (space-separated, also works)
|
||||
/nightly test_custom_op accuracy-group
|
||||
|
||||
# Run accuracy group tests (branch defaults to main)
|
||||
/nightly accuracy-group
|
||||
```
|
||||
|
||||
This triggers `workflow_dispatch` on both `schedule_nightly_test_a2.yaml` and `schedule_nightly_test_a3.yaml`.
|
||||
|
||||
### `/cherry-pick`
|
||||
|
||||
Cherry-pick a PR's commits onto a specified target branch and create a new PR. This is useful for backporting fixes to release branches.
|
||||
|
||||
**Usage:**
|
||||
|
||||
| Syntax | Description |
|
||||
|---|---|
|
||||
| `/cherry-pick <target_branch>` | Cherry-pick onto the specified branch |
|
||||
|
||||
**Examples:**
|
||||
|
||||
```text
|
||||
# Cherry-pick to a release branch
|
||||
/cherry-pick releases/v0.23.0
|
||||
|
||||
# Cherry-pick to main
|
||||
/cherry-pick main
|
||||
```
|
||||
|
||||
A new PR will be created with the title format `[Cherry-pick] <original_title> (from #<PR_NUMBER>)` and a body linking back to the original PR.
|
||||
|
||||
If the cherry-pick encounters merge conflicts, the command will report the failure and the cherry-pick must be done manually.
|
||||
|
||||
### `/revert`
|
||||
|
||||
Revert a merged PR by creating a new PR that reverses its changes. The revert targets the same base branch the original PR was merged into.
|
||||
|
||||
**Usage:**
|
||||
|
||||
| Syntax | Description |
|
||||
|---|---|
|
||||
| `/revert` | Revert this PR (no arguments needed) |
|
||||
|
||||
**Example:**
|
||||
|
||||
```text
|
||||
/revert
|
||||
```
|
||||
|
||||
A new PR will be created with the title format `[Revert] Revert "original_title" (#PR_NUMBER)` and a body linking back to the original PR and its merge commit.
|
||||
|
||||
Only merged PRs can be reverted. If the revert encounters merge conflicts (e.g., because the base branch has diverged significantly), the command will report the failure and the revert must be done manually.
|
||||
|
||||
### `/rerun`
|
||||
|
||||
Re-run all failed workflow runs on the current PR commit. Useful when CI jobs failed due to infrastructure issues.
|
||||
|
||||
**Examples:**
|
||||
|
||||
```text
|
||||
# Re-run all failed CI workflows on this PR
|
||||
/rerun
|
||||
```
|
||||
|
||||
## Behavior
|
||||
|
||||
1. When you comment a slash command, a 👀 reaction is added to your comment to indicate it has been received
|
||||
2. The corresponding CI workflow is triggered asynchronously
|
||||
3. Upon completion, a 🎉 reaction and a summary comment are added
|
||||
|
||||
## Scope
|
||||
|
||||
| Command | PR comments | Issue comments |
|
||||
|---|---|---|
|
||||
| `/e2e` | ✅ | ❌ |
|
||||
| `/rerun` | ✅ | ❌ |
|
||||
| `/cherry-pick` | ✅ | ❌ |
|
||||
| `/revert` | ✅ | ❌ |
|
||||
| `/nightly` | ✅ | ❌ |
|
||||
|
||||
## Permission
|
||||
|
||||
| Command | Who can trigger |
|
||||
|---|---|
|
||||
| `/e2e` | PR author, or users with triage+ permission on the repository |
|
||||
| `/rerun` | PR author, or users with triage+ permission on the repository |
|
||||
| `/cherry-pick` | PR author, or users with triage+ permission on the repository |
|
||||
| `/revert` | PR author, or users with triage+ permission on the repository |
|
||||
| `/nightly` | Users with triage+ permission on the repository only |
|
||||
|
||||
Permission is verified via the GitHub API (`repos/{owner}/{repo}/collaborators/{user}/permission`).
|
||||
@@ -1,16 +1,16 @@
|
||||
# User Stories
|
||||
|
||||
Read case studies on how users and developers solves real, everyday problems with vLLM Ascend
|
||||
Read case studies on how users and developers solve real, everyday problems with vLLM Ascend
|
||||
|
||||
- [LLaMA-Factory](./llamafactory.md) is an easy-to-use and efficient platform for training and fine-tuning large language models, it supports vLLM Ascend to speed up inference since [LLaMA-Factory#7739](https://github.com/hiyouga/LLaMA-Factory/pull/7739), gain 2x performance enhancement of inference.
|
||||
- [LLaMA-Factory](./llamafactory.md) is an easy-to-use and efficient platform for training and fine-tuning large language models. It supports vLLM Ascend to speed up inference since [LLaMA-Factory#7739](https://github.com/hiyouga/LlamaFactory/pull/7739), gaining 2x performance enhancement in inference.
|
||||
|
||||
- [Huggingface/trl](https://github.com/huggingface/trl) is a cutting-edge library designed for post-training foundation models using advanced techniques like SFT, PPO and DPO, it uses vLLM Ascend since [v0.17.0](https://github.com/huggingface/trl/releases/tag/v0.17.0) to support RLHF on Ascend NPU.
|
||||
- [Huggingface/trl](https://github.com/huggingface/trl) is a cutting-edge library designed for post-training foundation models using advanced techniques like SFT, PPO and DPO. It uses vLLM Ascend since [v0.17.0](https://github.com/huggingface/trl/releases/tag/v0.17.0) to support RLHF on Ascend NPUs.
|
||||
|
||||
- [MindIE Turbo](https://pypi.org/project/mindie-turbo) is an LLM inference engine acceleration plug-in library developed by Huawei on Ascend hardware, which includes self-developed large language model optimization algorithms and optimizations related to the inference engine framework. It supports vLLM Ascend since [2.0rc1](https://www.hiascend.com/document/detail/zh/mindie/20RC1/AcceleratePlugin/turbodev/mindie-turbo-0001.html).
|
||||
- [MindIE Turbo](https://pypi.org/project/mindie-turbo) is an LLM inference engine acceleration plugin library developed by Huawei on Ascend hardware, which includes self-developed LLM optimization algorithms and optimizations related to the inference engine framework. It supports vLLM Ascend since [2.0rc1](https://www.hiascend.com/document/detail/zh/mindie/20RC1/AcceleratePlugin/turbodev/mindie-turbo-0001.html).
|
||||
|
||||
- [GPUStack](https://github.com/gpustack/gpustack) is an open-source GPU cluster manager for running AI models. It supports vLLM Ascend since [v0.6.2](https://github.com/gpustack/gpustack/releases/tag/v0.6.2), see more GPUStack performance evaluation info on [link](https://mp.weixin.qq.com/s/pkytJVjcH9_OnffnsFGaew).
|
||||
- [GPUStack](https://github.com/gpustack/gpustack) is an open-source GPU cluster manager for running AI models. It supports vLLM Ascend since [v0.6.2](https://github.com/gpustack/gpustack/releases/tag/v0.6.2). See more GPUStack performance evaluation information at [this link](https://mp.weixin.qq.com/s/pkytJVjcH9_OnffnsFGaew).
|
||||
|
||||
- [verl](https://github.com/volcengine/verl) is a flexible, efficient and production-ready RL training library for large language models (LLMs), uses vLLM Ascend since [v0.4.0](https://github.com/volcengine/verl/releases/tag/v0.4.0), see more info on [verl x Ascend Quickstart](https://verl.readthedocs.io/en/latest/ascend_tutorial/ascend_quick_start.html).
|
||||
- [verl](https://github.com/verl-project/verl) is a flexible, efficient, and production-ready RL training library for LLMs. It uses vLLM Ascend since [v0.4.0](https://github.com/verl-project/verl/releases/tag/v0.4.0). See more information on [Ascend Quickstart](https://verl.readthedocs.io/en/latest/ascend_tutorial/get_start/quick_start.html).
|
||||
|
||||
:::{toctree}
|
||||
:caption: More details
|
||||
|
||||
@@ -1,19 +1,19 @@
|
||||
# LLaMA-Factory
|
||||
|
||||
**About / Introduction**
|
||||
## Introduction
|
||||
|
||||
[LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) is an easy-to-use and efficient platform for training and fine-tuning large language models. With LLaMA-Factory, you can fine-tune hundreds of pre-trained models locally without writing any code.
|
||||
[LLaMA-Factory](https://github.com/hiyouga/LlamaFactory) is an easy-to-use and efficient platform for training and fine-tuning large language models. With LLaMA-Factory, you can fine-tune hundreds of pre-trained models locally without writing any code.
|
||||
|
||||
LLaMA-Facotory users need to evaluate and inference the model after fine-tuning the model.
|
||||
LLaMA-Factory users need to evaluate the model and perform inference after fine-tuning.
|
||||
|
||||
**The Business Challenge**
|
||||
## Business challenge
|
||||
|
||||
LLaMA-Factory used transformers to perform inference on Ascend NPU, but the speed was slow.
|
||||
LLaMA-Factory uses Transformers to perform inference on Ascend NPUs, but the speed is slow.
|
||||
|
||||
**Solving Challenges and Benefits with vLLM Ascend**
|
||||
## Benefits with vLLM Ascend
|
||||
|
||||
With the joint efforts of LLaMA-Factory and vLLM Ascend ([LLaMA-Factory#7739](https://github.com/hiyouga/LLaMA-Factory/pull/7739)), the performance of LLaMA-Factory in the model inference stage has been significantly improved. According to the test results, the inference speed of LLaMA-Factory has been increased to 2x compared to the transformers version.
|
||||
With the joint efforts of LLaMA-Factory and vLLM Ascend ([LLaMA-Factory#7739](https://github.com/hiyouga/LlamaFactory/pull/7739)), LLaMA-Factory has achieved significant performance gains during model inference. Benchmark results show that its inference speed is now up to 2× faster compared to the Transformers implementation.
|
||||
|
||||
**Learn more**
|
||||
## Learn more
|
||||
|
||||
See more about LLaMA-Factory and how it uses vLLM Ascend for inference on the Ascend NPU in the following documentation: [LLaMA-Factory Ascend NPU Inference](https://llamafactory.readthedocs.io/en/latest/advanced/npu_inference.html).
|
||||
For more details about LLaMA-Factory, please refer to the [official documentation](https://llamafactory.readthedocs.io/en/latest/index.html).
|
||||
|
||||
@@ -1,53 +1,105 @@
|
||||
# Versioning policy
|
||||
# Versioning Policy
|
||||
|
||||
Starting with vLLM 0.7.x, the vLLM Ascend Plugin ([vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend)) project follows the [PEP 440](https://peps.python.org/pep-0440/) to publish matching with vLLM ([vllm-project/vllm](https://github.com/vllm-project/vllm)).
|
||||
Starting with vLLM 0.7.x, the vLLM Ascend Plugin ([vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend)) project follows [PEP 440](https://peps.python.org/pep-0440/) to publish versions matching vLLM ([vllm-project/vllm](https://github.com/vllm-project/vllm)).
|
||||
|
||||
## vLLM Ascend Plugin versions
|
||||
|
||||
Each vLLM Ascend release will be versioned: `v[major].[minor].[micro][rcN][.postN]` (such as
|
||||
Each vLLM Ascend release is versioned as `v[major].[minor].[micro][rcN][.postN]` (such as
|
||||
`v0.7.3rc1`, `v0.7.3`, `v0.7.3.post1`)
|
||||
|
||||
- **Final releases**: will typically be released every **3 months**, will take the vLLM upstream release plan and Ascend software product release plan into comprehensive consideration.
|
||||
- **Pre releases**: will typically be released **on demand**, ending with rcN, represents the Nth release candidate version, to support early testing by our users prior to a final release.
|
||||
- **Post releases**: will typically be released **on demand** to support to address minor errors in a final release. It's different from [PEP-440 post release note](https://peps.python.org/pep-0440/#post-releases) suggestion, it will contain actual bug fixes considering that the final release version should be matched strictly with the vLLM final release version (`v[major].[minor].[micro]`). The post version has to be published as a patch version of the final release.
|
||||
- **Final releases**: Typically scheduled every three months, with careful alignment to the vLLM upstream release cycle and the Ascend software product roadmap.
|
||||
- **Pre releases**: Typically issued **on demand**, labeled with rcN to indicate the Nth release candidate. They are intended to support early testing by users ahead of the final release.
|
||||
- **Post releases**: Typically issued **on demand** to address minor errors in a final release. Different from [PEP-440 post release note](https://peps.python.org/pep-0440/#post-releases) convention, these versions include actual bug fixes, as the final release version must strictly align with the vLLM final release format (`v[major].[minor].[micro]`). Any post version must be published as a patch version of the final release.
|
||||
|
||||
For example:
|
||||
- `v0.7.x`: it's the first final release to match the vLLM `v0.7.x` version.
|
||||
- `v0.7.3rc1`: will be the first pre version of vLLM Ascend.
|
||||
- `v0.7.3.post1`: will be the post release if the `v0.7.3` release has some minor errors.
|
||||
|
||||
## Release Compatibility Matrix
|
||||
- `v0.7.x`: first final release to match the vLLM `v0.7.x` version.
|
||||
- `v0.7.3rc1`: first pre version of vLLM Ascend.
|
||||
- `v0.7.3.post1`: post release for the `v0.7.3` release if it has some minor errors.
|
||||
|
||||
Following is the Release Compatibility Matrix for vLLM Ascend Plugin:
|
||||
## Release compatibility matrix
|
||||
|
||||
| vLLM Ascend | vLLM | Python | Stable CANN | PyTorch/torch_npu | MindIE Turbo |
|
||||
|-------------|--------------|------------------|-------------|--------------------|--------------|
|
||||
| v0.11.0rc0 | v0.11.0rc3 | >= 3.9, < 3.12 | 8.2.RC1 | 2.7.1 / 2.7.1.dev20250724 | |
|
||||
| v0.10.2rc1 | v0.10.2 | >= 3.9, < 3.12 | 8.2.RC1 | 2.7.1 / 2.7.1.dev20250724 | |
|
||||
| v0.10.1rc1 | v0.10.1/v0.10.1.1 | >= 3.9, < 3.12 | 8.2.RC1 | 2.7.1 / 2.7.1.dev20250724 | |
|
||||
| v0.10.0rc1 | v0.10.0 | >= 3.9, < 3.12 | 8.2.RC1 | 2.7.1 / 2.7.1.dev20250724 | |
|
||||
| v0.9.2rc1 | v0.9.2 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1.post1.dev20250619 | |
|
||||
| v0.9.1 | v0.9.1 | >= 3.9, < 3.12 | 8.2.RC1 | 2.5.1 / 2.5.1.post1 | |
|
||||
| v0.9.1rc3 | v0.9.1 | >= 3.9, < 3.12 | 8.2.RC1 | 2.5.1 / 2.5.1.post1 | |
|
||||
| v0.9.1rc2 | v0.9.1 | >= 3.9, < 3.12 | 8.2.RC1 | 2.5.1 / 2.5.1.post1| |
|
||||
| v0.9.1rc1 | v0.9.1 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1.post1.dev20250528 | |
|
||||
| v0.9.0rc2 | v0.9.0 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1 | |
|
||||
| v0.9.0rc1 | v0.9.0 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1 | |
|
||||
| v0.8.5rc1 | v0.8.5.post1 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1 | |
|
||||
| v0.8.4rc2 | v0.8.4 | >= 3.9, < 3.12 | 8.0.0 | 2.5.1 / 2.5.1 | |
|
||||
| v0.7.3.post1| v0.7.3 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1 | 2.0rc1 |
|
||||
| v0.7.3 | v0.7.3 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1 | 2.0rc1 |
|
||||
The table below is the release compatibility matrix for vLLM Ascend release.
|
||||
|
||||
| vLLM Ascend | vLLM | Python | Stable CANN | PyTorch/torch_npu | Triton Ascend | Mooncake |
|
||||
|-------------|-------------------|-----------------|-------------|---------------------------------|-------------------|--------------|
|
||||
| v0.23.0 | v0.23.0 | >= 3.10, < 3.13 | 9.0.1 | 2.10.0 / 2.10.0.post2 | 3.2.1 | v0.3.11.post1 |
|
||||
| v0.23.0rc1 | v0.23.0 | >= 3.10, < 3.13 | 9.0.1 | 2.10.0 / 2.10.0.post2 | 3.2.1 | v0.3.11.post1 |
|
||||
| v0.22.1rc1 | v0.22.1 | >= 3.10, < 3.13 | 9.0.0 | 2.10.0 / 2.10.0 | 3.2.1 | v0.3.9 |
|
||||
| v0.21.0rc1 | v0.21.0 | >= 3.10, < 3.13 | 9.0.0 | 2.10.0 / 2.10.0 | 3.2.1 | v0.3.9 |
|
||||
| v0.20.2rc1 | v0.20.2 | >= 3.10, < 3.12 | 9.0.0 | 2.10.0 / 2.10.0 | 3.2.1 | v0.3.8.post1 |
|
||||
| v0.19.1rc1 | v0.19.1 | >= 3.10, < 3.12 | 8.5.1 | 2.9.0 / 2.9.0 | 3.2.0 | v0.3.8.post1 |
|
||||
| v0.18.0 | v0.18.0 | >= 3.10, < 3.12 | 8.5.1 | 2.9.0 / 2.9.0.post1+git4c901a4 | 3.2.0.dev20260322 | v0.3.9 |
|
||||
| v0.18.0rc1 | v0.18.0 | >= 3.10, < 3.12 | 8.5.1 | 2.9.0 / 2.9.0 | 3.2.0 | N/A |
|
||||
| v0.17.0rc1 | v0.17.0 | >= 3.10, < 3.12 | 8.5.1 | 2.9.0 / 2.9.0 | 3.2.0 | N/A |
|
||||
| v0.16.0rc1 | v0.16.0 | >= 3.10, < 3.12 | 8.5.1 | 2.9.0 / 2.9.0 | 3.2.0 | N/A |
|
||||
| v0.15.0rc1 | v0.15.0 | >= 3.10, < 3.12 | 8.5.0 | 2.9.0 / 2.9.0 | 3.2.0 | N/A |
|
||||
| v0.14.0rc1 | v0.14.1 | >= 3.10, < 3.12 | 8.5.0 | 2.9.0 / 2.9.0 | 3.2.0 | N/A |
|
||||
| v0.13.0rc3 | v0.13.0 | >= 3.10, < 3.12 | 8.5.1 | 2.8.0 / 2.8.0.post2 | 3.2.0 | N/A |
|
||||
| v0.13.0 | v0.13.0 | >= 3.10, < 3.12 | 8.5.0 | 2.8.0 / 2.8.0.post2 | 3.2.0 | N/A |
|
||||
| v0.13.0rc2 | v0.13.0 | >= 3.10, < 3.12 | 8.5.0 | 2.8.0 / 2.8.0.post1 | 3.2.0 | N/A |
|
||||
| v0.13.0rc1 | v0.13.0 | >= 3.10, < 3.12 | 8.3.RC2 | 2.8.0 / 2.8.0 | N/A | N/A |
|
||||
| v0.12.0rc1 | v0.12.0 | >= 3.10, < 3.12 | 8.3.RC2 | 2.8.0 / 2.8.0 | N/A | N/A |
|
||||
| v0.11.0 | v0.11.0 | >= 3.9, < 3.12 | 8.3.RC2 | 2.7.1 / 2.7.1.post1 | N/A | N/A |
|
||||
| v0.11.0rc3 | v0.11.0 | >= 3.9, < 3.12 | 8.3.RC2 | 2.7.1 / 2.7.1.post1 | N/A | N/A |
|
||||
| v0.11.0rc2 | v0.11.0 | >= 3.9, < 3.12 | 8.3.RC2 | 2.7.1 / 2.7.1 | N/A | N/A |
|
||||
| v0.11.0rc1 | v0.11.0 | >= 3.9, < 3.12 | 8.3.RC1 | 2.7.1 / 2.7.1 | N/A | N/A |
|
||||
| v0.11.0rc0 | v0.11.0rc3 | >= 3.9, < 3.12 | 8.2.RC1 | 2.7.1 / 2.7.1.dev20250724 | N/A | N/A |
|
||||
| v0.10.2rc1 | v0.10.2 | >= 3.9, < 3.12 | 8.2.RC1 | 2.7.1 / 2.7.1.dev20250724 | N/A | N/A |
|
||||
| v0.10.1rc1 | v0.10.1/v0.10.1.1 | >= 3.9, < 3.12 | 8.2.RC1 | 2.7.1 / 2.7.1.dev20250724 | N/A | N/A |
|
||||
| v0.10.0rc1 | v0.10.0 | >= 3.9, < 3.12 | 8.2.RC1 | 2.7.1 / 2.7.1.dev20250724 | N/A | N/A |
|
||||
| v0.9.2rc1 | v0.9.2 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1.post1.dev20250619 | N/A | N/A |
|
||||
| v0.9.1 | v0.9.1 | >= 3.9, < 3.12 | 8.2.RC1 | 2.5.1 / 2.5.1.post1 | N/A | N/A |
|
||||
| v0.9.1rc3 | v0.9.1 | >= 3.9, < 3.12 | 8.2.RC1 | 2.5.1 / 2.5.1.post1 | N/A | N/A |
|
||||
| v0.9.1rc2 | v0.9.1 | >= 3.9, < 3.12 | 8.2.RC1 | 2.5.1 / 2.5.1.post1 | N/A | N/A |
|
||||
| v0.9.1rc1 | v0.9.1 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1.post1.dev20250528 | N/A | N/A |
|
||||
| v0.9.0rc2 | v0.9.0 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1 | N/A | N/A |
|
||||
| v0.9.0rc1 | v0.9.0 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1 | N/A | N/A |
|
||||
| v0.8.5rc1 | v0.8.5.post1 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1 | N/A | N/A |
|
||||
| v0.8.4rc2 | v0.8.4 | >= 3.9, < 3.12 | 8.0.0 | 2.5.1 / 2.5.1 | N/A | N/A |
|
||||
| v0.7.3.post1| v0.7.3 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1 | N/A | N/A |
|
||||
| v0.7.3 | v0.7.3 | >= 3.9, < 3.12 | 8.1.RC1 | 2.5.1 / 2.5.1 | N/A | N/A |
|
||||
|
||||
:::{note}
|
||||
If you're using v0.7.3, don't forget to install [mindie-turbo](https://pypi.org/project/mindie-turbo) as well.
|
||||
:::
|
||||
|
||||
For main branch of vLLM Ascend, we usually make it compatible with the latest vLLM release and a newer commit hash of vLLM. Please note that this table is usually updated. Please check it regularly.
|
||||
|
||||
| vLLM Ascend | vLLM | Python | Stable CANN | PyTorch/torch_npu | Triton Ascend |
|
||||
|-------------|--------------|------------------|-------------|--------------------|---------------|
|
||||
| main | {{main_vllm_commit}}, {{main_vllm_tag}} | {{main_python_version}} | {{main_cann_version}} | {{main_pytorch_torch_npu_version}} | {{main_triton_ascend_version}} |
|
||||
|
||||
## Release cadence
|
||||
|
||||
### release window
|
||||
### Release window
|
||||
|
||||
| Date | Event |
|
||||
|------------|-------------------------------------------|
|
||||
| 2026.07.31 | v0.23.0 Final release, v0.23.0 |
|
||||
| 2026.07.20 | Release candidates, v0.23.0rc1 |
|
||||
| 2026.06.30 | Release candidates, v0.22.1rc1 |
|
||||
| 2026.06.16 | Release candidates, v0.21.0rc1 |
|
||||
| 2026.06.03 | Release candidates, v0.20.2rc1 |
|
||||
| 2026.04.30 | Release candidates, v0.19.1rc1 |
|
||||
| 2026.04.24 | Release candidates, v0.13.0rc3 |
|
||||
| 2026.04.01 | Release candidates, v0.18.0rc1 |
|
||||
| 2026.03.15 | Release candidates, v0.17.0rc1 |
|
||||
| 2026.03.10 | Release candidates, v0.16.0rc1 |
|
||||
| 2026.02.27 | Release candidates, v0.15.0rc1 |
|
||||
| 2026.02.06 | v0.13.0 Final release, v0.13.0 |
|
||||
| 2026.01.26 | Release candidates, v0.14.0rc1 |
|
||||
| 2026.01.24 | Release candidates, v0.13.0rc2 |
|
||||
| 2025.12.27 | Release candidates, v0.13.0rc1 |
|
||||
| 2025.12.16 | v0.11.0 Final release, v0.11.0 |
|
||||
| 2025.12.13 | Release candidates, v0.12.0rc1 |
|
||||
| 2025.12.03 | Release candidates, v0.11.0rc3 |
|
||||
| 2025.11.21 | Release candidates, v0.11.0rc2 |
|
||||
| 2025.11.10 | Release candidates, v0.11.0rc1 |
|
||||
| 2025.09.30 | Release candidates, v0.11.0rc0 |
|
||||
| 2025.09.16 | Release candidates, v0.10.2rc1 |
|
||||
| 2025.09.04 | Release candidates, v0.10.1rc1 |
|
||||
| 2025.09.03 | v0.9.1 Final release |
|
||||
| 2025.09.03 | v0.9.1 Final release, v0.9.1 |
|
||||
| 2025.08.22 | Release candidates, v0.9.1rc3 |
|
||||
| 2025.08.07 | Release candidates, v0.10.0rc1 |
|
||||
| 2025.08.04 | Release candidates, v0.9.1rc2 |
|
||||
@@ -55,8 +107,8 @@ Following is the Release Compatibility Matrix for vLLM Ascend Plugin:
|
||||
| 2025.06.22 | Release candidates, v0.9.1rc1 |
|
||||
| 2025.06.10 | Release candidates, v0.9.0rc2 |
|
||||
| 2025.06.09 | Release candidates, v0.9.0rc1 |
|
||||
| 2025.05.29 | v0.7.x post release, v0.7.3.post1 |
|
||||
| 2025.05.08 | v0.7.x Final release, v0.7.3 |
|
||||
| 2025.05.29 | v0.7.3 post release, v0.7.3.post1 |
|
||||
| 2025.05.08 | v0.7.3 Final release, v0.7.3 |
|
||||
| 2025.05.06 | Release candidates, v0.8.5rc1 |
|
||||
| 2025.04.28 | Release candidates, v0.8.4rc2 |
|
||||
| 2025.04.18 | Release candidates, v0.8.4rc1 |
|
||||
@@ -66,70 +118,79 @@ Following is the Release Compatibility Matrix for vLLM Ascend Plugin:
|
||||
|
||||
## Branch policy
|
||||
|
||||
vLLM Ascend has main branch and dev branch.
|
||||
vLLM Ascend includes two branches: main and dev.
|
||||
|
||||
- **main**: main branch，corresponds to the vLLM main branch and latest 1 or 2 release version. It is continuously monitored for quality through Ascend CI.
|
||||
- **vX.Y.Z-dev**: development branch, created with part of new releases of vLLM. For example, `v0.7.3-dev` is the dev branch for vLLM `v0.7.3` version.
|
||||
- **main**: corresponds to the vLLM main branch and latest 1 or 2 release version. It is continuously monitored for quality through Ascend CI.
|
||||
- **releases/vX.Y.Z**: development branch, created with part of new releases of vLLM. For example, `releases/v0.13.0` is the dev branch for vLLM `v0.13.0` version.
|
||||
|
||||
Usually, a commit should be ONLY first merged in the main branch, and then backported to the dev branch to reduce maintenance costs as much as possible.
|
||||
Commits should typically be merged into the main branch first, and only then backported to the dev branch, to reduce maintenance costs as much as possible.
|
||||
|
||||
### Maintenance branch and EOL:
|
||||
The branch status will be in one of the following states:
|
||||
### Maintenance branch and EOL
|
||||
|
||||
| Branch | Time frame | Summary |
|
||||
|-------------------|----------------------------------|----------------------------------------------------------------------|
|
||||
| Maintained | Approximately 2-3 minor versions | All bugfixes are appropriate. Releases produced, CI commitment. |
|
||||
| Unmaintained | Community interest driven | All bugfixes are appropriate. No Releases produced, No CI commitment |
|
||||
| End of Life (EOL) | N/A | Branch no longer accepting changes |
|
||||
The table below lists branch states.
|
||||
|
||||
### Branch state
|
||||
| Branch | Time Frame | Summary |
|
||||
| ----------------- | -------------------------------- | --------------------------------------------------------- |
|
||||
| Maintained | Approximately 2-3 minor versions | Bugfixes received; releases produced; CI commitment |
|
||||
| Unmaintained | Community-interest driven | Bugfixes received; no releases produced; no CI commitment |
|
||||
| End of Life (EOL) | N/A | Branch no longer accepting changes |
|
||||
|
||||
Note that vLLM Ascend will only be released for a certain vLLM release version rather than all versions. Hence, You might see only part of versions have dev branches (such as only `0.7.1-dev` / `0.7.3-dev` but no `0.7.2-dev`), this is as expected.
|
||||
### Branch states
|
||||
|
||||
Usually, each minor version of vLLM (such as 0.7) will correspond to a vLLM Ascend version branch and support its latest version (for example, we plan to support version 0.7.3) as following shown:
|
||||
Note that vLLM Ascend will only be released for a certain vLLM release version, not for every version. Hence, you may notice that some versions have corresponding dev branches (e.g. `releases/v0.13.0`), while others do not (e.g. `releases/v0.12.0`). The vLLM Ascend release branch now follows the `releases/vX.Y.Z` naming convention, replacing the previous `vX.Y.Z-dev` format to align with vLLM's branch naming standards.
|
||||
|
||||
| Branch | Status | Note |
|
||||
|------------|--------------|--------------------------------------|
|
||||
| main | Maintained | CI commitment for vLLM main branch and vLLM 0.9.2 branch |
|
||||
| v0.9.1-dev | Maintained | CI commitment for vLLM 0.9.1 version |
|
||||
| v0.7.3-dev | Maintained | CI commitment for vLLM 0.7.3 version |
|
||||
| v0.7.1-dev | Unmaintained | Replaced by v0.7.3-dev |
|
||||
Usually, each minor version of vLLM (such as 0.7) corresponds to a vLLM Ascend version branch and supports its latest version (such as 0.7.3), as shown below:
|
||||
|
||||
| Branch | State | Note |
|
||||
| ---------- | ------------ | -------------------------------------------------------- |
|
||||
| main | Maintained | CI commitment for vLLM main branch and vLLM {{main_vllm_tag}} tag |
|
||||
| releases/v0.23.0 | Maintained | CI commitment for vLLM 0.23.0 version |
|
||||
| releases/v0.18.0 | Maintained | CI commitment for vLLM 0.18.0 version |
|
||||
| releases/v0.13.0 | Maintained | CI commitment for vLLM 0.13.0 version |
|
||||
| v0.11.0-dev| Maintained | CI commitment for vLLM 0.11.0 version |
|
||||
| v0.9.1-dev | Maintained | CI commitment for vLLM 0.9.1 version |
|
||||
| v0.7.3-dev | Maintained | CI commitment for vLLM 0.7.3 version |
|
||||
| v0.7.1-dev | Unmaintained | Replaced by v0.7.3-dev |
|
||||
|
||||
### Feature branches
|
||||
|
||||
| Branch | Status | RFC link | Merge plan | Mentor |
|
||||
| Branch | State | RFC Link | Scheduled Merge Time | Mentor |
|
||||
|------------|--------------|---------------------------------------|------------|--------|
|
||||
|rfc/long_seq_optimization|Maintained|https://github.com/vllm-project/vllm/issues/22693|930|wangxiyuan|
|
||||
|rfc/long_seq_optimization|Maintained|[RFC: long sequence optimization #22693](https://github.com/vllm-project/vllm/issues/22693)|930|wangxiyuan|
|
||||
|
||||
- Branch: The feature branch should be created with a prefix `rfc/` followed by the feature name, such as `rfc/feature-name`.
|
||||
- Status: The status of the feature branch is `Maintained` until it is merged into the main branch or deleted.
|
||||
- RFC link: The feature branch should be created with a corresponding RFC issue. The creation of a feature branch requires an RFC and approval from at least two maintainers.
|
||||
- Merge plan: The final goal of a feature branch is to merge it into the main branch. If it exceeds 3 months, the mentor maintainer should evaluate whether to delete the branch.
|
||||
- State: The state of the feature branch is `Maintained` until it is merged into the main branch or deleted.
|
||||
- RFC Link: The feature branch should be created with a corresponding RFC issue. The creation of a feature branch requires an RFC and approval from at least two maintainers.
|
||||
- Scheduled Merge Time: The final goal of a feature branch is to be merged into the main branch. If it remains unmerged for more than three months, the mentor maintainer should evaluate whether to delete the branch.
|
||||
- Mentor: The mentor should be a vLLM Ascend maintainer who is responsible for the feature branch.
|
||||
|
||||
### Backward compatibility
|
||||
|
||||
For main branch, vLLM Ascend should works with vLLM main branch and latest 1 or 2 release version. So to ensure the backward compatibility, we will do the following:
|
||||
- Both main branch and target vLLM release is tested by Ascend E2E CI. For example, currently, vLLM main branch and vLLM 0.8.4 are tested now.
|
||||
- For code changes, we will make sure that the changes are compatible with the latest 1 or 2 vLLM release version as well. In this case, vLLM Ascend introduced a version check machinism inner the code. It'll check the version of installed vLLM package first to decide which code logic to use. If users hit the `InvalidVersion` error, it sometimes means that they have installed an dev/editable version of vLLM package. In this case, we provide the env variable `VLLM_VERSION` to let users specify the version of vLLM package to use.
|
||||
- For documentation changes, we will make sure that the changes are compatible with the latest 1 or 2 vLLM release version as well. Note should be added if there are any breaking changes.
|
||||
For main branch, vLLM Ascend should work with vLLM main branch and latest 1 or 2 releases. To ensure backward compatibility, do as follows:
|
||||
|
||||
- Both main branch and target vLLM release, such as the vLLM main branch and vLLM 0.8.4, are tested by Ascend E2E CI.
|
||||
- To make sure that code changes are compatible with the latest 1 or 2 vLLM releases, vLLM Ascend introduces a version check mechanism inside the code. It checks the version of the installed vLLM package first to decide which code logic to use. If users hit the `InvalidVersion` error, it may indicate that they have installed a dev or editable version of vLLM package. In this case, we provide the env variable `VLLM_VERSION` to let users specify the version of vLLM package to use.
|
||||
- Document changes should be compatible with the latest 1 or 2 vLLM releases. Notes should be added if there are any breaking changes.
|
||||
|
||||
## Document branch policy
|
||||
|
||||
## Document Branch Policy
|
||||
To reduce maintenance costs, **all branch documentation content should remain consistent, and version differences can be controlled via variables in [docs/source/conf.py](https://github.com/vllm-project/vllm-ascend/blob/main/docs/source/conf.py)**. While this is not a simple task, it is a principle we should strive to follow.
|
||||
|
||||
| Version | Purpose | Code Branch |
|
||||
|-----|-----|---------|
|
||||
| latest | Doc for the latest dev branch | vX.Y.Z-dev (Will be `main` after the first final release) |
|
||||
| version | Doc for historical released versions | Git tags, like vX.Y.Z[rcN] |
|
||||
| stable（not yet released） | Doc for latest final release branch | Will be `vX.Y.Z-dev` after the first official release |
|
||||
| latest | Doc for the latest rc release of main branch | `main` branch |
|
||||
| rc version | Doc for RC released versions | `vX.Y.ZrcN` --> `vX.Y.ZrcN` tag |
|
||||
| version | Doc for historical released versions | `vX.Y.Z` --> `releases/vX.Y.Z` branch |
|
||||
|
||||
As shown above:
|
||||
Notes:
|
||||
|
||||
- `latest` documentation: Matches the current maintenance branch `vX.Y.Z-dev` (Will be `main` after the first final release). Continuously updated to ensure usability for the latest release.
|
||||
- `version` documentation: Corresponds to specific released versions (e.g., `v0.7.3`, `v0.7.3rc1`). No further updates after release.
|
||||
- `stable` documentation (**not yet released**): Official release documentation. Updates are allowed in real-time after release, typically based on vX.Y.Z-dev. Once stable documentation is available, non-stable versions should display a header warning: `You are viewing the latest developer preview docs. Click here to view docs for the latest stable release.`.
|
||||
- `latest` documentation: always points to latest rc release of main branch.
|
||||
- `rc version` documentation: there are no further updates after release.
|
||||
- `version` documentation: keep updating the `releases/vX.Y.Z` branch documentation to fix doc bugs.
|
||||
|
||||
## Software Dependency Management
|
||||
- `torch-npu`: Ascend Extension for PyTorch (torch-npu) releases a stable version to [PyPi](https://pypi.org/project/torch-npu)
|
||||
## Software dependency management
|
||||
|
||||
- `torch-npu`: TorchNPU releases a stable version to [PyPI](https://pypi.org/project/torch-npu)
|
||||
every 3 months, a development version (aka the POC version) every month, and a nightly version every day.
|
||||
The PyPi stable version **CAN** be used in vLLM Ascend final version, the monthly dev version **ONLY CANN** be used in
|
||||
vLLM Ascend RC version for rapid iteration, the nightly version **CANNOT** be used in vLLM Ascend any version and branches.
|
||||
The PyPI stable version **CAN** be used in vLLM Ascend final version, the monthly dev version **ONLY CAN** be used in
|
||||
vLLM Ascend RC version for rapid iteration, and the nightly version **CANNOT** be used in any vLLM Ascend version or branch.
|
||||
|
||||
@@ -25,18 +25,19 @@
|
||||
#
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# import sys
|
||||
# sys.path.insert(0, os.path.abspath('.'))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
# -- Project information -----------------------------------------------------
|
||||
|
||||
project = 'vllm-ascend'
|
||||
copyright = '2025, vllm-ascend team'
|
||||
author = 'the vllm-ascend team'
|
||||
project = "vllm-ascend"
|
||||
copyright = "2025, vllm-ascend team"
|
||||
author = "the vllm-ascend team"
|
||||
|
||||
# The full version, including alpha/beta/rc tags
|
||||
release = ''
|
||||
release = "0.23.0"
|
||||
|
||||
# -- General configuration ---------------------------------------------------
|
||||
|
||||
@@ -48,60 +49,82 @@ release = ''
|
||||
extensions = [
|
||||
"sphinx.ext.napoleon",
|
||||
"sphinx.ext.intersphinx",
|
||||
"sphinx.ext.mathjax",
|
||||
"sphinx_copybutton",
|
||||
"sphinx.ext.autodoc",
|
||||
"sphinx.ext.autosummary",
|
||||
"myst_parser",
|
||||
"sphinxarg.ext",
|
||||
"sphinx_design",
|
||||
"tools.docs_i18n.sphinx_extension",
|
||||
"sphinx_togglebutton",
|
||||
"sphinx_substitution_extensions",
|
||||
"tools.docs_codegen.sphinx_extension",
|
||||
]
|
||||
|
||||
myst_enable_extensions = ["colon_fence", "substitution"]
|
||||
myst_enable_extensions = ["colon_fence", "amsmath", "dollarmath", "substitution"]
|
||||
|
||||
# Change this when cut down release
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
_VLLM_MAIN_VERIFIED_COMMIT_PATH = _REPO_ROOT / ".github" / "vllm-main-verified.commit"
|
||||
_VLLM_RELEASE_TAG_PATH = _REPO_ROOT / ".github" / "vllm-release-tag.commit"
|
||||
_VLLM_MAIN_VERIFIED_COMMIT = _VLLM_MAIN_VERIFIED_COMMIT_PATH.read_text(encoding="utf-8").strip()
|
||||
_VLLM_RELEASE_TAG = _VLLM_RELEASE_TAG_PATH.read_text(encoding="utf-8").strip()
|
||||
|
||||
myst_substitutions = {
|
||||
# the branch of vllm, used in vllm clone
|
||||
# - main branch: 'main'
|
||||
# - vX.Y.Z branch: 'vX.Y.Z'
|
||||
'vllm_version': 'v0.11.0rc3',
|
||||
"vllm_version": "v0.23.0",
|
||||
# the branch of vllm-ascend, used in vllm-ascend clone and image tag
|
||||
# - main branch: 'main'
|
||||
# - vX.Y.Z branch: latest vllm-ascend release tag
|
||||
'vllm_ascend_version': 'v0.11.0rc0',
|
||||
"vllm_ascend_version": "v0.23.0",
|
||||
# the newest release version of vllm-ascend and matched vLLM, used in pip install.
|
||||
# This value should be updated when cut down release.
|
||||
'pip_vllm_ascend_version': "0.11.0rc0",
|
||||
'pip_vllm_version': "0.11.0",
|
||||
# CANN image tag
|
||||
'cann_image_tag': "8.2.rc1-910b-ubuntu22.04-py3.11",
|
||||
# vllm version in ci
|
||||
'ci_vllm_version': 'v0.11.0rc3',
|
||||
"pip_vllm_ascend_version": "0.23.0",
|
||||
"pip_vllm_version": "0.23.0",
|
||||
# CANN image tag paired with the vllm_ascend_version above
|
||||
"cann_image_tag": "9.1.0-910b-ubuntu22.04-py3.12",
|
||||
# vLLM commit hash for main branch
|
||||
"main_vllm_commit": _VLLM_MAIN_VERIFIED_COMMIT,
|
||||
# vLLM tag for main branch
|
||||
"main_vllm_tag": _VLLM_RELEASE_TAG,
|
||||
# Python version for main branch
|
||||
"main_python_version": ">= 3.10, < 3.13",
|
||||
# CANN version for main branch
|
||||
"main_cann_version": "9.1.0",
|
||||
# PyTorch/torch_npu version for main branch
|
||||
"main_pytorch_torch_npu_version": "2.10.0 / 2.10.0.post4",
|
||||
# Triton Ascend version for main branch
|
||||
"main_triton_ascend_version": "3.2.2",
|
||||
}
|
||||
|
||||
# For cross-file header anchors
|
||||
myst_heading_anchors = 5
|
||||
|
||||
# Add any paths that contain templates here, relative to this directory.
|
||||
templates_path = ['_templates']
|
||||
templates_path = ["_templates"]
|
||||
|
||||
# The language for content autogenerated by Sphinx. Refer to documentation
|
||||
# for a list of supported languages.
|
||||
#
|
||||
# This is also used if you do content translation via gettext catalogs.
|
||||
# Usually you set "language" from the command line for these cases.
|
||||
locale_dirs = ['locale/']
|
||||
gettext_compact = False
|
||||
locale_dirs = ["locale/"]
|
||||
gettext_compact = False
|
||||
# List of patterns, relative to source directory, that match files and
|
||||
# directories to ignore when looking for source files.
|
||||
# This pattern also affects html_static_path and html_extra_path.
|
||||
exclude_patterns = [
|
||||
'_build',
|
||||
'Thumbs.db',
|
||||
'.DS_Store',
|
||||
'.venv',
|
||||
'README.md',
|
||||
'user_guide/release.template.md',
|
||||
"_build",
|
||||
"Thumbs.db",
|
||||
".DS_Store",
|
||||
".venv",
|
||||
"README.md",
|
||||
"user_guide/release.template.md",
|
||||
# TODO(yikun): Remove this after zh supported
|
||||
'**/*.zh.md'
|
||||
"**/*.zh.md",
|
||||
]
|
||||
|
||||
# -- Options for HTML output -------------------------------------------------
|
||||
@@ -110,33 +133,50 @@ exclude_patterns = [
|
||||
# a list of builtin themes.
|
||||
#
|
||||
html_title = project
|
||||
html_theme = 'sphinx_book_theme'
|
||||
html_logo = 'logos/vllm-ascend-logo-text-light.png'
|
||||
html_theme = "sphinx_book_theme"
|
||||
html_logo = "logos/vllm-ascend-logo-text-light.png"
|
||||
html_theme_options = {
|
||||
'path_to_docs': 'docs/source',
|
||||
'repository_url': 'https://github.com/vllm-project/vllm-ascend',
|
||||
'use_repository_button': True,
|
||||
'use_edit_page_button': True,
|
||||
"path_to_docs": "docs/source",
|
||||
"repository_url": "https://github.com/vllm-project/vllm-ascend",
|
||||
"use_repository_button": True,
|
||||
"use_edit_page_button": True,
|
||||
}
|
||||
# Add any paths that contain custom static files (such as style sheets) here,
|
||||
# relative to this directory. They are copied after the builtin static files,
|
||||
# so a file named "default.css" will overwrite the builtin "default.css".
|
||||
# html_static_path = ['_static']
|
||||
# Copy llms.txt to site root so it is available as /llms.txt.
|
||||
html_extra_path = ["llms.txt"]
|
||||
|
||||
READTHEDOCS_VERSION_TYPE = os.environ.get('READTHEDOCS_VERSION_TYPE')
|
||||
# -- Options for linkcheck builder -------------------------------------------
|
||||
|
||||
# Check external links without validating remote anchors. Many third-party
|
||||
# sites render anchors dynamically, which makes anchor checks flaky in CI.
|
||||
linkcheck_anchors = False
|
||||
linkcheck_retries = 3
|
||||
linkcheck_timeout = 15
|
||||
linkcheck_workers = 10
|
||||
|
||||
# Example service endpoints in docs are intentionally not reachable from CI.
|
||||
linkcheck_ignore = [
|
||||
r"https?://localhost(:\d+)?($|/.*)",
|
||||
r"https?://127\.0\.0\.1(:\d+)?($|/.*)",
|
||||
r"https?://0\.0\.0\.0(:\d+)?($|/.*)",
|
||||
r"https?://192\.0\.0\.1(:\d+)?($|/.*)",
|
||||
r"https?://<[^>]+>.*",
|
||||
r"https://github\.com/vllm-project/vllm-ascend/issues/new/choose",
|
||||
r"https://github\.com/[^/?#]+/?$",
|
||||
r"https?://.*\$%7B.*%7D.*",
|
||||
]
|
||||
|
||||
READTHEDOCS_VERSION_TYPE = os.environ.get("READTHEDOCS_VERSION_TYPE")
|
||||
if READTHEDOCS_VERSION_TYPE == "tag":
|
||||
# remove the warning banner if the version is a tagged release
|
||||
header_file = os.path.join(os.path.dirname(__file__),
|
||||
"_templates/sections/header.html")
|
||||
header_file = os.path.join(os.path.dirname(__file__), "_templates/sections/header.html")
|
||||
# The file might be removed already if the build is triggered multiple times
|
||||
# (readthedocs build both HTML and PDF versions separately)
|
||||
if os.path.exists(header_file):
|
||||
os.remove(header_file)
|
||||
|
||||
|
||||
def setup(app):
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(json.dumps(myst_substitutions))
|
||||
|
||||
133
docs/source/developer_guide/Design_Documents/ACL_Graph.md
Normal file
133
docs/source/developer_guide/Design_Documents/ACL_Graph.md
Normal file
@@ -0,0 +1,133 @@
|
||||
# ACL Graph
|
||||
|
||||
## Overview
|
||||
|
||||
ACL Graph is the Ascend realization of vLLM static graph execution. Upstream vLLM and PyTorch documents already describe the generic graph model, including `CUDAGraphMode`, runtime dispatch, batch descriptors, bucketing and padding, and the definitions of full graph and piecewise graph. This document focuses on what is specific to Ascend in `vllm-ascend`: the platform integration points, the extra constraints introduced by ACL graph capture, and the mechanisms used to keep attention parameters correct during replay.
|
||||
|
||||
On Ascend, the design goal is the same as upstream static graph execution: reduce host launch overhead for small and medium runtime shapes. The implementation boundary is different. vLLM provides the generic dispatch path, while `vllm-ascend` supplies the platform wrapper, capture-size trimming, and attention-specific update logic needed by ACL graph replay.
|
||||
|
||||
## Prerequisites and References
|
||||
|
||||
- Upstream vLLM design doc for generic graph concepts: [CUDA Graphs](https://docs.vllm.ai/en/latest/design/cuda_graphs/).
|
||||
- PyTorch graph documentation for generic capture and replay semantics: [Accelerating PyTorch with CUDA Graphs](https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/).
|
||||
- Ascend user guide for operational enablement: [Graph Mode Guide](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/graph_mode.html).
|
||||
- Existing repo design note: [ACL Graph](https://docs.vllm.ai/projects/ascend/zh-cn/latest/developer_guide/Design_Documents/ACL_Graph.html)
|
||||
|
||||
This document intentionally does not re-explain upstream topics such as graph mode selection, dispatcher behavior, batch descriptor construction, capture bucketing, padding policy, or the generic meaning of full versus piecewise execution.
|
||||
|
||||
## How ACL Graph Fits into vLLM
|
||||
|
||||
vLLM owns the generic static graph flow. On Ascend, `NPUPlatform.get_static_graph_wrapper_cls()` returns `vllm_ascend.compilation.acl_graph.ACLGraphWrapper`, which is the platform-specific wrapper used when vLLM enables static graph mode.
|
||||
|
||||
`ACLGraphWrapper` is responsible for:
|
||||
|
||||
- reading the runtime mode and `batch_descriptor` from the forward context,
|
||||
- deciding whether to run eagerly, capture a new ACL graph, or replay a cached ACL graph,
|
||||
- caching graph entries per batch descriptor,
|
||||
- preserving the graph pool and replay bookkeeping needed by the Ascend backend.
|
||||
|
||||
The wrapper does not define the upstream dispatch policy. It assumes the runtime mode and batch descriptor have already been chosen correctly by vLLM, then applies Ascend capture or replay to that concrete runtime shape.
|
||||
|
||||
## Capture Sizes and Bucketing
|
||||
|
||||
vLLM graph replay requires stable runtime shapes, so vLLM does not try to capture every possible batch shape. Instead, it prepares a finite set of capture sizes and dispatches a runtime batch to the nearest supported size. If the runtime batch is larger than the largest configured capture size, graph mode is skipped and execution falls back to eager mode.
|
||||
|
||||
By default, vLLM builds capture sizes as:
|
||||
|
||||
- `1`, `2`, `4`
|
||||
- multiples of `8` from `8` up to `255`
|
||||
- multiples of `16` from `256` up to `max_cudagraph_capture_size`
|
||||
|
||||
Conceptually, the default list looks like:
|
||||
|
||||
```text
|
||||
[1, 2, 4, 8, 16, 24, 32, ..., 248, 256, 272, 288, ...]
|
||||
```
|
||||
|
||||
The smaller step at small batch sizes reduces padding overhead where latency is most sensitive, while the larger step at bigger sizes keeps the number of captured graphs under control.
|
||||
|
||||
On Ascend, this generic upstream bucketing strategy is still the starting point, but the final capture sizes may be reduced further by platform-specific constraints:
|
||||
|
||||
- sequence-parallel filtering may remove unsupported sizes,
|
||||
- runtime resource limits may still prevent some configured sizes from being captured,
|
||||
- some runtime modes may be normalized before capture begins.
|
||||
|
||||
## Ascend-Specific Design Constraints
|
||||
|
||||
### Capture breadth is still constrained by runtime resources
|
||||
|
||||
Unlike CUDA Graph on CUDA devices, ACL graph capture on Ascend can still fail when the selected graph sizes consume more runtime resources than the current backend can supply. Piecewise mode is the most sensitive case because it captures many subgraphs and the total capture cost scales with model depth and configured size coverage.
|
||||
|
||||
Older versions of vLLM Ascend applied a local `update_aclgraph_sizes()` heuristic to shrink the PIECEWISE capture-size set before final capture. That heuristic has been removed. The current implementation keeps upstream sizing and dispatch behavior intact, then intercepts the confirmed capture-time stream-resource signature in `vllm_ascend/compilation/acl_graph.py` and re-raises it with clearer mitigation guidance.
|
||||
|
||||
In practice, this means users should treat `cudagraph_capture_sizes` and `max_cudagraph_capture_size` as the primary tuning levers when capture fails. Newer HDK/CANN combinations can materially improve ACL graph capacity, while communication-heavy configurations may still require a smaller configured size set.
|
||||
|
||||
### Platform mode normalization is stricter than generic upstream behavior
|
||||
|
||||
Ascend currently narrows some generic upstream modes in `vllm_ascend.platform.NPUPlatform.check_and_update_config()`.
|
||||
|
||||
- Encoder-decoder models are forced to `PIECEWISE`.
|
||||
- `use_inductor` is disabled for ACL graph paths.
|
||||
- `ASCEND_LAUNCH_BLOCKING=1` is rejected when ACL graph is enabled.
|
||||
- Xlite graph mode can disable ACL graph full mode or fall back to `FULL_DECODE_ONLY`, depending on configuration.
|
||||
|
||||
These checks document the subset of upstream graph behavior that the current Ascend backend can execute safely. Some of them are long-term platform constraints, while others are clearly transitional in the current implementation.
|
||||
|
||||
## Key Ascend-Specific Mechanisms
|
||||
|
||||
### Host-side attention parameter update for full graph replay
|
||||
|
||||
Full graph replay on Ascend has an extra problem that upstream generic documentation does not cover in detail: some attention operators need runtime metadata updates even when the overall graph is static. The Ascend implementation handles this by separating graph capture from host-side task parameter updates.
|
||||
|
||||
The flow is:
|
||||
|
||||
1. During capture, attention backends record per-graph task handles, events, workspaces, and weak references to the tensors or metadata that must be refreshed.
|
||||
2. Before replay, `update_full_graph_params()` calls the backend specific `update_graph_params()` implementation.
|
||||
3. That backend runs parameter refresh on an update stream with `torch.npu.graph_task_update_begin(...)` and `torch.npu.graph_task_update_end(...)` around the underlying attention operator launch.
|
||||
4. `torch.npu.ExternalEvent` objects are used to enforce ordering between the host-side update stream and the replay stream.
|
||||
|
||||
This mechanism is implemented in attention backends such as:
|
||||
|
||||
- `vllm_ascend/attention/attention_v1.py`
|
||||
- `vllm_ascend/attention/mla_v1.py`
|
||||
- `vllm_ascend/attention/context_parallel/attention_cp.py`
|
||||
- `vllm_ascend/attention/context_parallel/mla_cp.py`
|
||||
|
||||
The important design point is that Ascend full graph support depends on backend-provided `update_graph_params()` hooks. Without that hook, capture alone is not enough to replay the correct attention state.
|
||||
|
||||
### Replay ordering and synchronization
|
||||
|
||||
`ACLGraphWrapper` synchronizes the current stream before replay in the common path to ensure that host-side parameter updates stay aligned with the graph execution that will consume them. This is especially relevant in asynchronous scheduling or multi-threaded execution.
|
||||
|
||||
If ordering is not preserved, the parameter update for iteration *i* can be observed by the replay of iteration *i-1*, or the replay of iteration *i* can start before its own parameter update has completed. In practice, this means the attention operator may run with mismatched runtime metadata, which can cause incorrect results, precision issues, or even hangs. The code keeps a narrower path for the main full-graph eagle case, but the general design assumption is the same: replay must not overtake pending parameter update work.
|
||||
|
||||
## Full vs Piecewise on Ascend
|
||||
|
||||
Upstream docs already define full graph and piecewise graph semantically. On Ascend, the practical difference is driven by backend support and resource cost.
|
||||
|
||||
### Piecewise mode
|
||||
|
||||
Piecewise mode is the conservative path. It relies on the generic vLLM split execution strategy, then applies ACL graph capture to the non-attention segments selected by the compilation path. On Ascend, this mode is currently the more widely supported option, but it is also the most sensitive to stream pressure because the number of captured graphs scales with model depth.
|
||||
|
||||
### Full graph mode
|
||||
|
||||
Full graph mode is the more performance-oriented path when the attention backend can support runtime parameter patching through `update_graph_params()`. On Ascend, full graph support is tied to those attention-specific update hooks, workspace caching, and replay ordering guarantees.
|
||||
|
||||
## Diagnostics and Operational Notes
|
||||
|
||||
- The simplest way to confirm that graph mode is active is to enable cudagraph metrics and keep log stats enabled. In CLI usage, use `--cudagraph-metrics` and do not pass `--disable-log-stats`. In Python usage, set `cudagraph_metrics=True` and `disable_log_stats=False`. Then inspect the emitted metrics and logs.
|
||||
- Profiling can also confirm whether replay is happening, and developers can add temporary prints before replay when debugging locally, but those are secondary methods and are not expanded here.
|
||||
- Capture-size selection primarily follows upstream configuration and dispatch behavior; only the confirmed stream-resource capture failure is rewritten with user-facing guidance at runtime.
|
||||
- In debug mode, `ACLGraphWrapper` asserts that replay uses the same tensor addresses recorded during capture.
|
||||
- `ASCEND_LAUNCH_BLOCKING=1` is incompatible with ACL graph enablement in the current implementation.
|
||||
- For debugging inside graph execution, the repo also provides graph-aware print helpers in `vllm_ascend.utils`, but those are developer diagnostics rather than part of the execution design.
|
||||
|
||||
## Related Files
|
||||
|
||||
- `vllm_ascend/platform.py`, mode normalization, platform hooks, and static graph wrapper selection.
|
||||
- `vllm_ascend/compilation/acl_graph.py`, ACL graph wrapper, capture and replay cache, graph parameter containers, and full graph update dispatch.
|
||||
- `vllm_ascend/compilation/acl_graph.py`, runtime ACL graph capture, replay, and capture-failure guidance.
|
||||
- `vllm_ascend/attention/attention_v1.py`, full graph attention parameter capture and update logic.
|
||||
- `vllm_ascend/attention/mla_v1.py`, MLA (Multi-Head Latent Attention) specific full graph parameter capture and update logic.
|
||||
- `vllm_ascend/attention/context_parallel/attention_cp.py`, context parallel attention update path.
|
||||
- `vllm_ascend/attention/context_parallel/mla_cp.py`, context parallel MLA update path.
|
||||
@@ -0,0 +1,91 @@
|
||||
# KV Cache Pool
|
||||
|
||||
## Why KV Cache Pool?
|
||||
|
||||
Prefix caching is an important feature in LLM inference that can reduce prefill computation time drastically.
|
||||
|
||||
However, the performance gain from prefix caching is highly dependent on the cache hit rate, while the cache hit rate can be limited if one only uses on-chip memory for KV cache storage.
|
||||
|
||||
Hence, KV Cache Pool is proposed to utilize various types of storage including on-chip memory, DRAM, and SSD, making a pool for KV Cache storage while making the prefix of requests visible across all nodes, increasing the cache hit rate for all requests.
|
||||
|
||||
vLLM Ascend currently supports [MooncakeStore](https://github.com/kvcache-ai/Mooncake), one of the most recognized KV Cache storage engines.
|
||||
|
||||
While one can utilize MooncakeStore in vLLM V1 engine by setting it as a remote backend of LMCache with GPU (see [Tutorial](https://github.com/LMCache/LMCache/blob/dev/examples/kv_cache_reuse/remote_backends/mooncakestore/README.md)), we find it would be better to integrate a connector that directly supports MooncakeStore and can utilize the data transfer strategy that best fits Huawei NPU hardware.
|
||||
|
||||
Hence, we propose to integrate MooncakeStore with a brand new **MooncakeStoreConnectorV1**, which is indeed largely inspired by **LMCacheConnectorV1** (see the [How is MooncakeStoreConnectorV1 Implemented?](#how-is-mooncakestoreconnectorv1-implemented) section).
|
||||
|
||||
## Usage
|
||||
|
||||
vLLM Ascend currently supports MooncakeStore for KV Cache Pool. To enable MooncakeStore, one needs to configure `kv-transfer-config` and choose `MooncakeStoreConnector` as the KV Connector.
|
||||
|
||||
For step-by-step deployment and configuration, please refer to the [KV Pool User Guide](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/kv_pool.html).
|
||||
|
||||
## How it works?
|
||||
|
||||
The KV Cache Pool integrates multiple memory tiers (on-chip memory, DRAM, SSD, etc.) through a connector-based architecture.
|
||||
|
||||
Each connector implements a unified interface for storing, retrieving, and transferring KV blocks between tiers, depending on access frequency and hardware bandwidth.
|
||||
|
||||
When combined with vLLM's Prefix Caching mechanism, the pool enables efficient caching both locally (in on-chip memory) and globally (via Mooncake), ensuring that frequently used prefixes remain hot while less frequently accessed KV data can spill over to lower-cost memory.
|
||||
|
||||
### 1. Combining KV Cache Pool with on-chip memory Prefix Caching
|
||||
|
||||
Prefix Caching with on-chip memory is already supported by the vLLM V1 Engine.
|
||||
By introducing KV Connector V1, users can seamlessly combine on-chip memory-based Prefix Caching with Mooncake-backed KV Pool.
|
||||
|
||||
The user can enable both features simply by enabling Prefix Caching, which is enabled by default in vLLM V1 unless the `--no-enable-prefix-caching` flag is set, and setting up the KV Connector for KV Pool (e.g., the MooncakeStoreConnector).
|
||||
|
||||
**Workflow**:
|
||||
|
||||
1. The engine first checks for prefix hits in the on-chip memory cache.
|
||||
|
||||
2. After getting the number of hit tokens on on-chip memory, it queries the KV Pool via the connector. If there are additional hits in the KV Pool, we get the **additional blocks only** from the KV Pool, and get the rest of the blocks directly from on-chip memory to minimize the data transfer latency.
|
||||
|
||||
3. After the KV Caches in the KV Pool are loaded into on-chip memory, the remaining process is the same as Prefix Caching in on-chip memory.
|
||||
|
||||
### 2. Combining KV Cache Pool with Mooncake PD Disaggregation
|
||||
|
||||
When used together with Mooncake PD (Prefill-Decode) Disaggregation, the KV Cache Pool can further decouple prefill and decode stages across devices or nodes.
|
||||
|
||||
Currently, we only perform put and get operations of KV Pool for **Prefill Nodes**, and Decode Nodes get their KV Cache from Mooncake P2P KV Connector, i.e., MooncakeConnector.
|
||||
|
||||
The key benefit of doing this is that we can keep the gain in performance by computing less with Prefix Caching from on-chip memory and KV Pool for Prefill Nodes, while not sacrificing the data transfer efficiency between Prefill and Decode nodes with P2P KV Connector that transfers KV Caches between NPU devices directly.
|
||||
|
||||
To enable this feature, we need to set up both Mooncake Connector and MooncakeStore Connector with a Multi Connector, which is a KV Connector class provided by vLLM that can call multiple KV Connectors in a specific order.
|
||||
|
||||
For details, please also refer to the [Mooncake connector deployment guide](https://github.com/vllm-project/vllm-ascend/blob/main/examples/disaggregated_prefill_v1/mooncake_connector_deployment_guide.md).
|
||||
|
||||
## How is MooncakeStoreConnectorV1 Implemented?
|
||||
|
||||
**MooncakeStoreConnectorV1** inherits the KV Connector V1 class in vLLM V1: through implementing the required methods defined in the KV connector V1 base class, one can integrate a third-party KV cache transfer/storage backend into the vLLM framework.
|
||||
|
||||
MooncakeStoreConnectorV1 is also largely inspired by LMCacheConnectorV1 in terms of the `Lookup Engine`/`Lookup Client` design for looking up KV cache keys, and the `ChunkedTokenDatabase` class for processing tokens into prefix-aware hashes as well as other hashing related designs. On top of this, we have also added our own design including `KVTransferThread` that allows async `get` and `put` of KV caches with multi-threading, and NPU-related data transfer optimization such as removing the `LocalBuffer` in LMCache to remove redundant data transfer.
|
||||
|
||||
The KV Connector methods that need to be implemented can be categorized into scheduler-side methods that are called in V1 scheduler and worker-side methods that are called in V1 worker, namely:
|
||||
|
||||
### KV Connector Scheduler-Side Methods
|
||||
|
||||
`get_num_new_matched_tokens`: Get prefix cache hit in number of tokens through looking up into the KV pool.
|
||||
`update_states_after_alloc`: Update KVConnector state after temporary buffer alloc.
|
||||
`build_connector_meta`: Attach the connector metadata to the request object.
|
||||
`request_finished`: Once a request is finished, determine whether request blocks should be freed now or will be sent asynchronously and freed later.
|
||||
|
||||
### Connector Worker-Side Methods
|
||||
|
||||
`register_kv_caches`: Register KV cache buffers needed for KV cache transfer.
|
||||
`start_load_kv`: Perform KV cache load operation that transfers KV cache from storage to device.
|
||||
`wait_for_layer_load`: Optional; Wait for layer load in layerwise + async KV load scenario.
|
||||
`save_kv_layer`: Optional; Do layerwise KV cache put into KV Pool.
|
||||
`wait_for_save`: Wait for KV Save to finish if async KV cache save/put.
|
||||
`get_finished`: Get request that finished KV transfer, `done_sending` if `put` finished, `done_receiving` if `get` finished.
|
||||
|
||||
## DFX
|
||||
|
||||
1. When looking up a key in KV Pool, if we cannot find the key, there is no Cache Hit for this specific block; we return no hit for this block and do not look up further blocks for the current request.
|
||||
2. Similarly, when we are trying to put a block into KV Pool and it fails, we do not put further blocks (subject to change).
|
||||
|
||||
## Limitations
|
||||
|
||||
1. Currently, MooncakeStore for vLLM Ascend only supports DRAM as the storage for KV Cache Pool.
|
||||
|
||||
2. For now, if we successfully looked up a key and found it exists, but failed to get it when calling KV Pool's get function, we just output a log indicating the get operation failed and keep going; hence, the accuracy of that specific request may be affected. We will handle this situation by falling back the request and re-compute everything assuming there's no prefix cache hit (or even better, revert only one block and keep using the Prefix Caches before that).
|
||||
@@ -0,0 +1,286 @@
|
||||
# Prepare inputs for model forwarding
|
||||
|
||||
## Purpose
|
||||
|
||||
Information required to perform model forward pass:
|
||||
|
||||
- the inputs
|
||||
- the corresponding attention metadata of the inputs
|
||||
|
||||
The following diagram shows what we should prepare for model inference.
|
||||
|
||||
```text
|
||||
+---------------+
|
||||
inputs --> | |
|
||||
| model | --> output
|
||||
attn_meta --> | |
|
||||
+---------------+
|
||||
```
|
||||
|
||||
Therefore, as long as we have these two pieces of information mentioned above, we can perform the model's forward propagation.
|
||||
|
||||
This document will explain **how we obtain the inputs and their corresponding attention metadata**.
|
||||
|
||||
## Overview
|
||||
|
||||
### 1. Obtain inputs
|
||||
|
||||
The workflow of obtaining inputs:
|
||||
|
||||
1. Get `token positions`: relative position of each token within its request sequence.
|
||||
|
||||
2. Get `token indices`: index of each scheduled token in the token table.
|
||||
|
||||
3. Get `Token IDs`: using token indices to retrieve the Token IDs from **token id table**.
|
||||
|
||||
At last, these `Token IDs` are required to be fed into a model, and `positions` should also be sent into the model to create `RoPE` (Rotary positional embedding). Both of them are the inputs of the model.
|
||||
|
||||
**Note**: The `Token IDs` are the inputs of a model, so we also call them `Input IDs`.
|
||||
|
||||
### 2. Build inputs attention metadata
|
||||
|
||||
A model requires these attention metadata during the forward pass:
|
||||
|
||||
- `query start location`: start and end location of each request corresponding to the scheduled tokens.
|
||||
- `sequence length`: length of each request including both computed tokens and newly scheduled tokens.
|
||||
- `number of computed tokens`: number of computed tokens for each request.
|
||||
- `number of requests`: number of requests in this batch.
|
||||
- `number of tokens`: total number of scheduled tokens in this batch.
|
||||
- **`block table`**: translates the logical address (within its sequence) of each block to its global physical address in the device's memory.
|
||||
- `max query len`: the longest scheduled tokens length in this request batch.
|
||||
- `slot mapping`: indices of each token that input token will be stored into.
|
||||
- `attention mask`: mask matrix applied to attention scores before softmax to control which tokens can attend to each other (usually a causal attention).
|
||||
|
||||
## Before start
|
||||
|
||||
There are mainly three types of variables.
|
||||
|
||||
- token level: represents one attribute corresponding to each scheduled token, so the length of this variable is the number of scheduled tokens.
|
||||
- request level: represents one attribute of each scheduled request, whose length usually is the number of scheduled requests. (`query start location` is a special case, which has one more element.)
|
||||
- system level:
|
||||
1. **Token IDs table**: stores the token IDs (i.e. the inputs of a model) of each request. The shape of this table is `(max num request, max model len)`. Here, `max num request` is the maximum count of concurrent requests allowed in a forward batch and `max model len` is the maximum token count that can be handled at one request sequence in this model.
|
||||
2. **Block table**: translates the logical address (within its sequence) of each block to its global physical address in the device's memory. The shape of this table is `(max num request, max model len / block size)`
|
||||
|
||||
**Note**: Both of these two tables come from the `_update_states` method before **preparing inputs**. You can take a look if you need more inspiration.
|
||||
|
||||
### Tips
|
||||
|
||||
Simply put, a `token ID` is an **integer** (usually `int32`), which represents a token.
|
||||
Example of `Token ID`:
|
||||
|
||||
```shell
|
||||
| Token ID | Token |
|
||||
|--------------|---------------|
|
||||
| 0 | [PAD] |
|
||||
| 1 | <|endoftext|> |
|
||||
| 2 | <|start|> |
|
||||
| 3 | [SEP] |
|
||||
| 4 | I |
|
||||
| 5 | the |
|
||||
| 6 | be |
|
||||
| 7 | of |
|
||||
| 8 | and |
|
||||
| ... | ... |
|
||||
| ... | ... |
|
||||
| vocab_size-1 | <|im_end|> |
|
||||
```
|
||||
|
||||
## Go through details
|
||||
|
||||
Assumptions:
|
||||
|
||||
- maximum number of tokens that can be scheduled at once: 10
|
||||
- `block size`: 2
|
||||
- Totally schedule 3 requests. Their prompt lengths are 3, 2, and 8 respectively.
|
||||
- `max model length`: 12 (the maximum token count that can be handled at one request sequence in a model).
|
||||
|
||||
These assumptions are configured at the beginning when starting vLLM. They are not fixed, so you can manually set them.
|
||||
|
||||
### Step 1: All requests in the prefill phase
|
||||
|
||||
#### Obtain inputs
|
||||
|
||||
As the maximum number of tokens that can be scheduled is 10, the scheduled tokens of each request can be represented as `{'0': 3, '1': 2, '2': 5}`. Note that `request_2` uses chunked prefill, leaving 3 prompt tokens unscheduled.
|
||||
|
||||
##### 1. Get token positions
|
||||
|
||||
First, determine which request each token belongs to: tokens 0–2 are assigned to **request_0**, tokens 3–4 to **request_1**, and tokens 5–9 to **request_2**. To represent this mapping, we use `request indices`, for example, `request indices`: `[0, 0, 0, 1, 1, 2, 2, 2, 2, 2]`.
|
||||
|
||||
For each request, use **the number of computed tokens** + **the relative position of current scheduled tokens** (`request_0: [0 + 0, 0 + 1, 0 + 2]`, `request_1: [0 + 0, 0 + 1]`, `request_2: [0 + 0, 0 + 1,..., 0 + 4]`) and then concatenate them together (`[0, 1, 2, 0, 1, 0, 1, 2, 3, 4]`).
|
||||
|
||||
Note: there is a more efficient way (using `request indices`) to create positions in actual code.
|
||||
|
||||
Finally, `token positions` can be obtained as `[0, 1, 2, 0, 1, 0, 1, 2, 3, 4]`. This variable is **token level**.
|
||||
|
||||
##### 2. Get token indices
|
||||
|
||||
The shape of the current **Token IDs table** is `(max num request, max model len)`.
|
||||
|
||||
Why are these `T_3_5`, `T_3_6`, `T_3_7` in this table without being scheduled?
|
||||
|
||||
- We fill all Token IDs in one request sequence to this table at once, but we only retrieve the tokens we scheduled this time. Then we retrieve the remaining Token IDs next time.
|
||||
|
||||
```shell
|
||||
| T_0_0 | T_0_1 | T_0_2 | ? | ? | ? | ? | ? | ? | ? | ? | ? |
|
||||
| T_1_0 | T_1_1 | ? | ? | ? | ? | ? | ? | ? | ? | ? | ? |
|
||||
| T_2_0 | T_2_1 | T_3_2 | T_3_3 | T_3_4 | T_3_5 | T_3_6 | T_3_7 | ? | ? | ? | ? |
|
||||
| ? | ? | ? | ? | ? | ? | ? | ? | ? | ? | ? | ? |
|
||||
......
|
||||
......
|
||||
......
|
||||
```
|
||||
|
||||
Note that `T_x_x` is an `int32`.
|
||||
|
||||
Let's say `M = max model len`. Then we can use `token positions` together with `request indices` of each token to construct `token indices`.
|
||||
|
||||
So `token indices` = `[0 + 0 * M, 1 + 0 * M, 2 + 0 * M, 0 + 1 * M, 1 + 1 * M, 0 + 2 * M, 1 + 2 * M, 2 + 2 * M, 3 + 2 * M, 4 + 2 * M]` = `[0, 1, 2, 12, 13, 24, 25, 26, 27, 28]`
|
||||
|
||||
##### 3. Retrieve the Token IDs
|
||||
|
||||
We use `token indices` to select out the corresponding `Input IDs` from the token table. The pseudocode is as follows:
|
||||
|
||||
```shell
|
||||
input_ids = token_table[token_indices]
|
||||
```
|
||||
|
||||
As mentioned before, we refer to these `Token IDs` as `Input IDs`.
|
||||
|
||||
- `Input IDs` = `[T_0_0, T_0_1, T_0_2, T_1_0, T_1_1, T_2_0, T_2_1, T_3_2, T_3_3, T_3_4]`
|
||||
|
||||
#### Build inputs attention metadata
|
||||
|
||||
In the current **Block Table**, we use the first block (i.e. block_0) to mark the unused block. The shape of the block is `(max num request, max model len / block size)`, where `max model len / block size = 12 / 2 = 6`.
|
||||
|
||||
```shell
|
||||
| 1 | 2 | 0 | 0 | 0 | 0 |
|
||||
| 3 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 4 | 5 | 6 | 0 | 0 | 0 |
|
||||
| 0 | 0 | 0 | 0 | 0 | 0 |
|
||||
......
|
||||
......
|
||||
......
|
||||
```
|
||||
|
||||
The KV cache block in the device memory is like:
|
||||
|
||||
```shell
|
||||
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ......
|
||||
```
|
||||
|
||||
Let's say `K = max model len / block size = 6`, and we can get token `device block number`.
|
||||
|
||||
The workflow of achieving slot mapping:
|
||||
|
||||
1. Get `block table indices` using `K`, `positions` and `request indices`.
|
||||
|
||||
Purpose: For each token, it could be used to select `device block number` from `block table`.
|
||||
|
||||
2. Get `device block number` using `block table indices`.
|
||||
|
||||
Purpose: `device block number` indicates which device block each token belongs to.
|
||||
|
||||
3. Get `block offsets` using `positions` and `block size`.
|
||||
|
||||
Purpose: `block offsets` indicates the offsets of each token within a block.
|
||||
|
||||
4. construct `slot mapping` using `device block number` and `block offsets`.
|
||||
|
||||
Purpose: we can use `slot mapping` to store Token IDs into token slots.
|
||||
|
||||
Details:
|
||||
|
||||
1. (**Token level**) Use a simple formula to calculate `block table indices`: `request indices * K + positions / block size`. So it equals `[0 * 6 + 0 / 2, 0 * 6 + 1 / 2, 0 * 6 + 2 / 2, 1 * 6 + 0 / 2, 1 * 6 + 1 / 2, 2 * 6 + 0 / 2, 2 * 6 + 1 / 2, 2 * 6 + 2 / 2, 2 * 6 + 3 / 2, 2 * 6 + 4 / 2] = [0, 0, 1, 6, 6, 12, 12, 13, 13, 14]`. This could be used to select `device block number` from `block table`.
|
||||
2. (**Token level**) Use `block table indices` to select out `device block number` for each scheduled token. The pseudocode is `block_numbers = block_table[block_table_indices]`. So `device block number=[1, 1, 2, 3, 3, 4, 4, 5, 5, 6]`
|
||||
3. (**Token level**) `block offsets` could be computed by `block offsets = positions % block size = [0, 1, 0, 0, 1, 0, 1, 0, 1, 0]`.
|
||||
4. Finally, use `block offsets` and `device block number` to create `slot mapping`: `device block number * block size + block_offsets = [2, 3, 4, 6, 7, 8, 9, 10, 11, 12]`
|
||||
|
||||
(**Request level**) As we know the scheduled token count is `[3, 2, 5]`:
|
||||
|
||||
- (**Request level**) Use prefix sum to calculate `query start location`: `[0, 3, 5, 10]`.
|
||||
- (**Request level**) All tokens in step 1 are in the prefill stage, and the computed tokens count is 0; then `sequence length` = `[3, 2, 5]`.
|
||||
- (**Request level**) As mentioned above, `number of computed tokens` are all 0s: `[0, 0, 0]`.
|
||||
- `number of requests`: `3`
|
||||
- (**Request level**) `number of tokens`: `[3, 2, 5]`
|
||||
- `max query len`: `5`
|
||||
- (**Token level**) `slot mapping`: `[2, 3, 4, 6, 7, 8, 9, 10, 11, 12]`
|
||||
- `attention mask`: For all requests that initiate a prefill process, we simply create only one mask matrix for reuse across different requests. The shape of this mask matrix is `5 * 5`:
|
||||
|
||||
### Step 2: Chunked prefill
|
||||
|
||||
In Step 2, we no longer provide explanations or perform calculations; instead, we directly present the final result.
|
||||
|
||||
#### Obtain inputs
|
||||
|
||||
Scheduled token of each request: `{'0': 1, '1': 1, '2': 3}`
|
||||
|
||||
1. `request indices`: `[0, 1, 2, 2, 2]`
|
||||
2. `token positions`: `[3, 2, 5, 6, 7]`
|
||||
|
||||
Current **Token IDs table**:
|
||||
|
||||
```shell
|
||||
| T_0_0 | T_0_1 | T_0_2 | T_0_3 | ? | ? | ? | ? | ? | ? | ? | ? |
|
||||
| T_1_0 | T_1_1 | T_1_2 | ? | ? | ? | ? | ? | ? | ? | ? | ? |
|
||||
| T_2_0 | T_2_1 | T_3_2 | T_3_3 | T_3_4 | T_3_5 | T_3_6 | T_3_7 | ? | ? | ? | ? |
|
||||
| ? | ? | ? | ? | ? | ? | ? | ? | ? | ? | ? | ? |
|
||||
......
|
||||
......
|
||||
......
|
||||
```
|
||||
|
||||
**Note**: **T_0_3**, **T_1_2** are new Token IDs of **request_0** and **request_1** respectively. They are sampled from the output of the model.
|
||||
|
||||
3. `token indices`: `[3, 14, 29, 30, 31]`
|
||||
4. `Input IDs`: `[T_0_3, T_1_2, T_3_5, T_3_6, T_3_7]`
|
||||
|
||||
#### Build inputs attention metadata
|
||||
|
||||
We allocate the blocks `7` and `8` to `request_1` and `request_2` respectively, as they need more space in device to store KV cache following token generation or chunked prefill.
|
||||
|
||||
Current **Block Table**:
|
||||
|
||||
```shell
|
||||
| 1 | 2 | 0 | 0 | 0 | 0 |
|
||||
| 3 | 7 | 0 | 0 | 0 | 0 |
|
||||
| 4 | 5 | 6 | 8 | 0 | 0 |
|
||||
| 0 | 0 | 0 | 0 | 0 | 0 |
|
||||
......
|
||||
......
|
||||
......
|
||||
```
|
||||
|
||||
KV cache block in the device memory:
|
||||
|
||||
```shell
|
||||
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ......
|
||||
```
|
||||
|
||||
1. (**Token level**) `block table indices`: `[1, 7, 14, 15, 15]`
|
||||
2. (**Token level**) `device block number`: `[2, 7, 6, 8, 8]`
|
||||
3. (**Token level**) `block offsets`: `[1, 0, 1, 0, 1]`
|
||||
4. (**Token level**) `slot mapping`: `[5, 14, 13, 16, 17]`
|
||||
|
||||
Scheduled token count: `[1, 1, 3]`
|
||||
|
||||
- `query start location`: `[0, 1, 2, 5]`
|
||||
|
||||
- `sequence length`: `[4, 3, 8]`
|
||||
|
||||
- `number of computed tokens`: `[3, 2, 5]`
|
||||
|
||||
- `number of requests`: `3`
|
||||
|
||||
- `max query len`: `3`
|
||||
|
||||
- `slot mapping`: `[5, 14, 13, 16, 17]`
|
||||
|
||||
- `attention mask`: `5 * 8`
|
||||
|
||||
Each token has a `1 * 8` vector, and there are 5 scheduled tokens.
|
||||
|
||||
## At last
|
||||
|
||||
If you understand step 1 and step 2, you will know all the following steps.
|
||||
|
||||
Hope this document helps you better understand how vLLM prepares inputs for model forwarding. If you have any good ideas, you are welcome to contribute to us.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Adding a custom aclnn operation
|
||||
|
||||
This document describes how to add a custom aclnn operation to vllm-ascend.
|
||||
|
||||
## How custom aclnn operation works in vllm-ascend?
|
||||
|
||||
Custom aclnn operations are built and installed into `vllm_ascend/cann_ops_custom` directory during the build process of vllm-ascend. Then the aclnn operators are bound to `torch.ops._C_ascend` module, enabling users to invoke them in vllm-ascend python code.
|
||||
|
||||
To enable custom operations, use the following code:
|
||||
|
||||
```python
|
||||
from vllm_ascend.utils import enable_custom_op
|
||||
|
||||
enable_custom_op()
|
||||
```
|
||||
|
||||
## How to add a custom aclnn operation?
|
||||
|
||||
1. Create a new operation folder under `csrc` directory.
|
||||
2. Create `op_host` and `op_kernel` directories for host and kernel source code.
|
||||
3. Add build options in `csrc/build_aclnn.sh` for supported SOC. Note that multiple ops should be separated with `;`, i.e. `CUSTOM_OPS="op1;op2;op3"`.
|
||||
4. Bind aclnn operators to torch.ops._C_ascend module in `csrc/torch_binding.cpp`.
|
||||
5. Write a meta implementation in `csrc/torch_binding_meta.cpp` for the op to be captured into the aclgraph.
|
||||
|
||||
After a successful build of vllm-ascend, the custom aclnn operation can be invoked in python code.
|
||||
197
docs/source/developer_guide/Design_Documents/context_parallel.md
Normal file
197
docs/source/developer_guide/Design_Documents/context_parallel.md
Normal file
@@ -0,0 +1,197 @@
|
||||
# Context Parallel (CP)
|
||||
|
||||
TL;DR: PCP accelerates prefill via sequence splitting. DCP eliminates KV cache redundancy.
|
||||
|
||||
|
||||
|
||||
For the main discussions during the development process, please refer to the [RFC](https://github.com/vllm-project/vllm/issues/25749) and the relevant links referenced by or referencing this RFC.
|
||||
|
||||
## What is CP?
|
||||
|
||||
**Context Parallel (CP)** is a strategy for parallelizing computation along the sequence dimension across multiple devices.
|
||||
|
||||
**Prefill Context Parallel (PCP)** expands the world size of devices and uses dedicated communication domains.
|
||||
Its primary goal is to partition the sequence dimension during the prefill phase, enabling different devices to compute distinct chunks of the sequence simultaneously.
|
||||
The KV cache is sharded along the sequence dimension across devices.
|
||||
This approach impacts the computational logic of both the Prefill and Decode stages to varying degrees.
|
||||
|
||||
**Decode Context Parallel (DCP)** reuses the communication domain of Tensor Parallelism (TP) and does not require additional devices.
|
||||
Its main objective is to eliminate duplicated storage of the KV cache by sharding it along the sequence dimension across devices within the TP domain that would otherwise hold redundant copies.
|
||||
DCP primarily influences the Decode logic, as well as the logic for chunked prefill and cached prefill.
|
||||
|
||||
## How to Use CP?
|
||||
|
||||
Please refer to the [context parallel user guide](../../user_guide/feature_guide/context_parallel.md) for detailed information.
|
||||
|
||||
## How It Works?
|
||||
|
||||
### Device Distribution
|
||||
|
||||
We introduce new communication domains for PCP and reuse TP for DCP, and this is the new layout of devices for PCP2, DCP2, and TP4.
|
||||
|
||||
|
||||
### Block Table
|
||||
|
||||
CP performs sequence sharding on the KV cache storage. To facilitate efficient storage and access, tokens are stored in an interleaved manner across devices, with the interleaving granularity determined by `cp_kv_cache_interleave_size`, whose default value is `cp_kv_cache_interleave_size=1`, a.k.a. 'token interleave'.
|
||||
|
||||
Given that PCP and DCP behave similarly for KV cache sharding, we refer to them collectively as CP. Specifically, `cp_size = pcp_size * dcp_size`, and `cp_rank = pcp_rank * dcp_size + dcp_rank`.
|
||||
|
||||
As illustrated, a virtual block is defined in the block table, where blocks within the same CP device group form a virtual block. The virtual block size is `virtual_block_size = block_size * cp_size`.
|
||||
|
||||
For any token `x`, referencing the following figure, its (virtual) block index is `x // virtual_block_size`, and the offset within the virtual block is `offset_within_virtual_block = x % virtual_block_size`.
|
||||
The local block index is `local_block_index = offset_within_virtual_block // cp_kv_cache_interleave_size`, and the device number is `target_rank = local_block_index % cp_size`.
|
||||
The offset within the local block is `(local_block_index // cp_size) * cp_kv_cache_interleave_size + offset_within_virtual_block % cp_kv_cache_interleave_size`.
|
||||
|
||||
|
||||
|
||||
Based on the logic above, the `slot_mapping` calculation process is adjusted, and the `slot_mapping` values on each device are modified to ensure the KV cache is sharded along the sequence dimension and stored across different devices as expected.
|
||||
|
||||
The current implementation requires that `block_size % cp_kv_cache_interleave_size == 0`.
|
||||
|
||||
### Decode Context Parallel (DCP)
|
||||
|
||||
As mentioned above, the primary function of DCP is to shard the KV cache along the sequence dimension for storage. Its impact lies in the logic of the decode and chunked prefill phases.
|
||||
|
||||
**Prefill Phase:**
|
||||
As illustrated, during the Chunked Prefill computation, two distinct logic implementations are employed for MLA and GQA backends.
|
||||
|
||||
- In the **MLA backend**, a Context KV Cache `all_gather` operation is performed to aggregate the full KV values.
|
||||
These are then used for attention computation with the Q values of the current chunk.
|
||||
Note that in multi-request scenarios, the directly gathered KV results are interleaved across requests.
|
||||
The `reorg_kvcache` function is used to reorganize the KV cache, ensuring that the KV cache of the same request is stored contiguously.
|
||||
|
||||
- In the **GQA backend**, an `all_gather` is performed along the head dimension for Q.
|
||||
This is because DCP overlaps with the TP communication domain, and the Q heads within a DCP group differ.
|
||||
However, they need to exchange results with the locally computed KV cache for online Softmax updates.
|
||||
To ensure correctness during result updates, the Q values are synchronized across the DCP group via head-dimension `all_gather`.
|
||||
During the result update process, `cp_lse_ag_out_rs` is invoked to aggregate `attn_output` and `attn_lse`, update the results, and perform a reduce-scatter operation on the outputs.
|
||||
Alternatively, we can use an all-to-all communication to exchange the output and LSE results, followed by direct local updates. This approach aligns with the logic adapted for PCP compatibility.
|
||||
|
||||
|
||||
|
||||
**Decode Phase:**
|
||||
The logic during the decode phase is consistent with that of GQA's chunked prefill: an all-gather operation is first performed along the Q head dimension to ensure consistency within the DCP group.
|
||||
After computing the results with the local KV cache, the results are updated via the `cp_lse_ag_out_rs` function.
|
||||
|
||||
|
||||
|
||||
### GLM-5.2 SFA DCP Replicated Indexer
|
||||
|
||||
GLM-5.2 uses Sparse Flash Attention (SFA) with a LightningIndexer. For DCP,
|
||||
the indexer needs a full-sequence view to select the same sparse top-k blocks
|
||||
as non-DCP SFA, while the much larger SFA KV cache should remain sharded to
|
||||
retain DCP's memory benefit. The replicated-indexer path provides this split
|
||||
layout:
|
||||
|
||||
- The LightningIndexer cache is replicated on every DCP rank. Index selection
|
||||
therefore uses the complete sequence and produces globally consistent sparse
|
||||
top-k indices.
|
||||
- The SFA KV cache remains DCP-local. The global indices from the replicated
|
||||
indexer view are remapped to local KV indices before SFA runs.
|
||||
- During prefill or a mixed batch, only the KV blocks referenced by the sparse
|
||||
block table are compacted and all-gathered after the current layer has
|
||||
written its KV cache. The gathered KV uses a remapped block table for SFA,
|
||||
so this path does not all-gather Q and does not need LSE or output
|
||||
post-processing.
|
||||
- Decode-only batches retain the DCP SFA Q-gather and result-merge path.
|
||||
|
||||
This mode is selected automatically for SFA sparse models when
|
||||
`prefill_context_parallel_size=1` and `decode_context_parallel_size>1`. It
|
||||
requires `decode_context_parallel_size == tensor_parallel_size`; PCP combined
|
||||
with this replicated-indexer path is not supported.
|
||||
|
||||
For a GLM-5.2 DSA-CP deployment, enable FlashComm1 and DSA CP and keep the CP
|
||||
interleave size equal to the KV-cache block size:
|
||||
|
||||
```bash
|
||||
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
|
||||
|
||||
vllm serve <glm-5.2-model> \
|
||||
--tensor-parallel-size <N> \
|
||||
--prefill-context-parallel-size 1 \
|
||||
--decode-context-parallel-size <N> \
|
||||
--block-size <B> \
|
||||
--cp-kv-cache-interleave-size <B> \
|
||||
--additional-config '{"enable_dsa_cp": true}'
|
||||
```
|
||||
|
||||
The replicated indexer increases indexer-cache memory in proportion to the
|
||||
DCP world size; the SFA KV cache itself remains sharded. For this SFA CP path,
|
||||
`cp_kv_cache_interleave_size` must equal `block_size`. A mismatched setting is
|
||||
overridden during configuration validation, but deployments should set both
|
||||
values explicitly to avoid relying on that fallback.
|
||||
|
||||
### Prefill Context Parallel (PCP)
|
||||
|
||||
**Tokens Partition in Head-Tail Style**
|
||||
|
||||
PCP requires splitting the input sequence and ensuring balanced computational load across devices during the prefill phase.
|
||||
We employ a head-tail style for splitting and concatenation: specifically, the sequence is first padded to a length of `2*pcp_size`, then divided into `2*pcp_size` equal parts.
|
||||
The first part is merged with the last part, the second part with the second last part, and so on, thereby assigning computationally balanced chunks to each device.
|
||||
Additionally, since allgather aggregation of KV or Q results in interleaved chunks from different requests, we compute `pcp_allgather_restore_idx` to quickly restore the original order.
|
||||
|
||||
These logics are implemented in the function `_update_tokens_for_pcp`.
|
||||
|
||||
|
||||
|
||||
**Prefill Phase:**
|
||||
|
||||
During the Prefill phase (excluding chunked prefill), we employ an all-gather KV approach to address the issue of incomplete sequences on individual GPUs.
|
||||
It is important to note that we only aggregate the KV values for the current layer at a time, and these are discarded immediately after use, avoiding excessive peak memory usage.
|
||||
This method can also be directly applied to KV cache storage (since the KV cache partitioning method differs from PCP sequence partitioning, it is inevitable that each GPU requires a complete copy of the KV values).
|
||||
All attention backends maintain consistency in this logic.
|
||||
|
||||
Note: While a Ring Attention approach could also facilitate information exchange with lower peak memory and enable computation-communication overlap, we prioritized the all-gather KV implementation after evaluating that the development complexity was high and the benefits of overlap were limited.
|
||||
|
||||
|
||||
|
||||
**Decode Phase:**
|
||||
|
||||
During the decode phase, we only need to add an allgather within the PCP group after the DCP all-to-all communication exchanges the output and LSE, before proceeding with the output update.
|
||||
|
||||
|
||||
|
||||
**Chunked Prefill:**
|
||||
|
||||
Currently, there are three viable approaches for Chunked Prefill compatibility: **AllGatherQ**, **AllGatherKV**, and **Ring-Attn**.
|
||||
Since PCP performs sequence sharding on both the query sequence and the KV cache, we need to ensure that one side has complete information or employ a method like Ring-Attn to perform computations sequentially.
|
||||
The advantages and disadvantages of Ring-Attn will not be elaborated here.
|
||||
|
||||
We have implemented the **AllGatherQ** approach in the GQA attention backend and the **AllGatherKV** approach in the MLA attention backend.
|
||||
The workflow after **AllGatherQ** is identical to the decode phase, while the workflow after **AllGatherKV** is the same as the standard prefill phase.
|
||||
For details, please refer to the diagram below; specific steps will not be repeated.
|
||||
|
||||
One important note: **AllGatherKV** may lead to significant peak memory usage when the context length becomes excessively long.
|
||||
To mitigate this, we adopt a segmented processing strategy.
|
||||
By predefining the maximum amount of KV cache processed per round, we sequentially complete the attention computation and online softmax updates for each segment.
|
||||
|
||||
|
||||
|
||||
### SFA DSA-CP Mixed `o_proj` Path
|
||||
|
||||
SFA DSA-CP mixed execution intentionally reuses the normal TP-sharded `o_proj`.
|
||||
This is part of the DSA-CP mixed data path, not a standalone user-facing `o_proj` TP switch.
|
||||
The mixed path is used when one instance may handle both decode-only and prefill/mixed batches, so `o_proj` must support two layouts at runtime:
|
||||
|
||||
- **Decode-only batches** keep the decode TP path.
|
||||
SFA outputs are exchanged with an all-to-all in the TP group, then the original TP-sharded `o_proj` runs normally.
|
||||
- **Prefill or mixed batches** produce SFA outputs that are not directly compatible with the TP-sharded `o_proj` input layout.
|
||||
Before `o_proj` forward, each rank all-gathers the TP-sharded `o_proj` weight and all input-sharded quantization parameters into temporary full-weight buffers.
|
||||
The full-weight `o_proj` forward runs once for that batch, and the module is then restored to the TP parameter aliases.
|
||||
|
||||
The storage invariant is that the original TP-sharded `o_proj` parameter remains the only persistent source of truth.
|
||||
`o_proj_tp_*` tensors are aliases of the original parameter storage.
|
||||
`o_proj_full_*` tensors are reusable communication buffers for prefill/mixed full-gather execution only.
|
||||
They must not become a second persistent copy of the TP weight.
|
||||
|
||||
This coupling preserves the existing decode TP behavior, supports prefill/mixed DSA-CP batches, and avoids adding an extra configuration path whose state can drift from DSA-CP mixed execution.
|
||||
|
||||
### Related Files
|
||||
|
||||
- slot_mapping computation: `vllm_ascend/worker/block_table.py`
|
||||
- sequences splitting and metadata prepare: `vllm_ascend/worker/model_runner_v1.py`
|
||||
- PCP token splitting and metadata generation: `vllm_ascend/worker/pcp_utils.py`
|
||||
- GQA backend: `vllm_ascend/attention/context_parallel/attention_cp.py`
|
||||
- MLA backend: `vllm_ascend/attention/context_parallel/mla_cp.py`
|
||||
- DSA backend: `vllm_ascend/attention/context_parallel/dsa_cp.py`
|
||||
- SFA backend: `vllm_ascend/attention/context_parallel/sfa_cp.py`
|
||||
281
docs/source/developer_guide/Design_Documents/cpu_binding.md
Normal file
281
docs/source/developer_guide/Design_Documents/cpu_binding.md
Normal file
@@ -0,0 +1,281 @@
|
||||
# CPU Binding
|
||||
|
||||
## Overview
|
||||
|
||||
CPU binding is an **Ascend-native host-side optimization** for vLLM workers on
|
||||
ARM servers. **Starting from vllm-ascend v0.18.0rc1, it is enabled by default
|
||||
through `enable_cpu_binding=True`.**
|
||||
|
||||
The feature does not change model execution logic or numerical results. It only
|
||||
controls CPU placement for the worker process, key runtime threads, memory
|
||||
pages, and NPU IRQs when the host environment allows it. By keeping the main
|
||||
worker, ACL, and release threads on dedicated CPU ranges, it **helps reduce
|
||||
context-switch overhead from scheduler preemption on busy hosts.**
|
||||
|
||||
## Why CPU Binding?
|
||||
|
||||
On multi-socket ARM systems, the Linux scheduler may place worker threads on
|
||||
CPUs far from the NPU that the worker drives. This can increase cross-NUMA
|
||||
traffic, increase thread preemption, and introduce latency jitter. The Ascend
|
||||
backend therefore owns a CPU allocation policy to **reduce cross-NUMA traffic,
|
||||
reduce thread preemption, and improve latency stability** instead of relying on
|
||||
upstream GPU NUMA binding flags.
|
||||
|
||||
This is also why upstream NUMA flags are adapted on Ascend:
|
||||
|
||||
- `--numa-bind` is converted to `additional_config={"enable_cpu_binding": true}`.
|
||||
- `--numa-bind-nodes` and `--numa-bind-cpus` are ignored because Ascend computes CPU pools from NPU topology or global logical NPU IDs.
|
||||
|
||||
## How It Works?
|
||||
|
||||
The allocator derives its plan from runtime host state:
|
||||
|
||||
| Input | Source | Purpose |
|
||||
| --- | --- | --- |
|
||||
| Allowed CPUs | `/proc/self/status` `Cpus_allowed_list` | The only CPUs eligible for binding. Container cpusets are respected. |
|
||||
| Logical NPU map | `npu-smi info -m` | Maps card/chip IDs to global logical NPU IDs and gives `total_logic_npus`. On Ascend 950, `Chip Logic ID` is not reported, so `NPU ID` is used as the logical ID. |
|
||||
| Running NPUs | `npu-smi info` process table, filtered by `ASCEND_RT_VISIBLE_DEVICES` | Identifies the logical NPUs used by this worker process. A2/A3 process rows use `NPU Chip`; Ascend 950 process rows use `NPU ID`. |
|
||||
| Topology affinity | `npu-smi info -t topo` | Provides NPU-to-CPU affinity for `topo_affinity` mode. |
|
||||
| CPU NUMA map | `lscpu -e=CPU,NODE` | Used to extend single-NUMA affinity pools to the next NUMA node. |
|
||||
| Thread topology | `lscpu` `Thread(s) per core` | Determines Ascend 950 cluster size: 8 CPUs for 1 thread per core, 16 CPUs for 2 threads per core. |
|
||||
| UVB polling threads | `ps -Te` | Finds host `uvb_poll_window_thread` threads for Ascend 950 UVB CPU binding. Docker containers must use `--pid=host` to see these host threads. |
|
||||
|
||||
### Strategy Selection
|
||||
|
||||
The binding strategy is selected by Ascend device type:
|
||||
|
||||
| Device type | Strategy | Reason |
|
||||
| --- | --- | --- |
|
||||
| A3 | `global_slice` | A3 uses HCCS card-to-card interconnect. Each NPU is nearly equidistant from all NUMA nodes, so there is no strong NPU-to-NUMA affinity signal. Global logical NPU ID based slicing gives deterministic, non-overlapping CPU pools and CPU/NUMA isolation between workers. |
|
||||
| Ascend 950 | `topo_affinity` | Ascend 950 uses NPU-to-CPU affinity from `npu-smi info -t topo` to choose an affinity NUMA node, then assigns one CPU cluster from that NUMA node to each worker. It also reports process rows by `NPU ID` instead of `NPU Chip`, skips IRQ binding, and binds host UVB polling threads. |
|
||||
| A2 and Atlas 300 inference products | `topo_affinity` | A2 and Atlas 300 inference products provide NPU-to-CPU affinity information through `npu-smi info -t topo`, so they use this topology signal when available. |
|
||||
|
||||
If `topo_affinity` is selected but topo affinity is unavailable, the allocator falls back to `global_slice`.
|
||||
|
||||
### CPU Pool Construction
|
||||
|
||||
#### global_slice
|
||||
|
||||
`global_slice` is designed for devices without a useful NPU-to-CPU affinity
|
||||
signal, including A3. Because A3's **HCCS interconnect makes the distance
|
||||
from each NPU to each NUMA node nearly the same**, topology affinity is not a
|
||||
useful placement signal. The allocator therefore partitions the sorted
|
||||
`allowed_cpus` list by global logical NPU ID.
|
||||
|
||||
1. Determine `total_npus` in this order:
|
||||
- `total_logic_npus` from `npu-smi info -m`
|
||||
- number of topo affinity entries
|
||||
- number of running NPUs
|
||||
2. Compute:
|
||||
- `base = len(allowed_cpus) // total_npus`
|
||||
- `extra = len(allowed_cpus) % total_npus`
|
||||
3. Each logical NPU gets a deterministic slice:
|
||||
- NPU IDs `< extra` receive `base + 1` CPUs.
|
||||
- Remaining NPU IDs receive `base` CPUs.
|
||||
4. Only running NPUs are materialized into `npu_cpu_pool`.
|
||||
|
||||
This is the key property: two independent worker processes with the same cpuset
|
||||
but different visible NPU IDs still get **non-overlapping CPU pools** because
|
||||
both processes slice against the same global NPU ID space. With a NUMA-aligned
|
||||
cpuset, this also provides **CPU/NUMA isolation between workers**, so one worker
|
||||
does not share the same CPU or NUMA slice with another worker.
|
||||
|
||||
`global_slice` requires enough CPUs for the selected device's role split:
|
||||
|
||||
- Devices with IRQ binding require `base >= 5`:
|
||||
2 CPUs for SQ/CQ IRQ binding, at least 1 CPU for the main worker, 1 CPU for
|
||||
ACL thread, and 1 CPU for release thread.
|
||||
|
||||
#### topo_affinity
|
||||
|
||||
`topo_affinity` is designed for A2, Atlas 300 inference products, Ascend 950,
|
||||
and other non-A3 device types. A2 and Atlas 300 inference products expose
|
||||
**meaningful NPU-to-CPU affinity information**, so the allocator starts from NPU
|
||||
topology affinity when it is available and then avoids overlap for shared
|
||||
affinity groups.
|
||||
|
||||
1. Build candidate NPUs from all logical NPUs:
|
||||
- always include running NPUs
|
||||
- include non-running NPUs only when their affinity overlaps this process's allowed cpuset
|
||||
2. For each candidate NPU, intersect topo affinity with `allowed_cpus`.
|
||||
3. If the intersection is empty for a candidate, binding fails for this rank.
|
||||
4. If the affinity CPUs are all on one NUMA node, extend the pool with CPUs from the next NUMA node, constrained by `allowed_cpus`.
|
||||
5. Group NPUs with identical extended pools and split each shared pool evenly across that group.
|
||||
6. Keep only running NPUs in the final `npu_cpu_pool`.
|
||||
|
||||
The non-running candidate step is intentional. It prevents two independent
|
||||
single-card workers from selecting the same CPU range when their visible NPUs
|
||||
share the same topology affinity.
|
||||
|
||||
For Ascend 950, topology affinity is used differently:
|
||||
|
||||
1. Bind all visible host `uvb_poll_window_thread` threads to NUMA0 CPUs except CPU0, constrained by `allowed_cpus`. Docker containers must use `--pid=host` to make these host threads visible.
|
||||
2. Use topo affinity to identify each NPU's single affinity NUMA node.
|
||||
3. Parse `Thread(s) per core` from `lscpu` and set cluster size to 8 CPUs when it is 1, or 16 CPUs when it is 2.
|
||||
4. Split each affinity NUMA's sorted allowed CPU list into contiguous clusters.
|
||||
5. Assign clusters by sorted logical NPU ID, including hidden NPUs that share the same affinity NUMA.
|
||||
6. Keep only running NPUs in the final `npu_cpu_pool`.
|
||||
|
||||
If Ascend 950 topo affinity is missing, spans multiple NUMA nodes, has too few
|
||||
clusters, or reports an unsupported `Thread(s) per core`, worker CPU binding is
|
||||
skipped without raising to the worker process.
|
||||
|
||||
### Role Split
|
||||
|
||||
After a CPU pool is built, the allocator splits it by role:
|
||||
|
||||
For devices with IRQ binding:
|
||||
|
||||
| Role | CPUs |
|
||||
| --- | --- |
|
||||
| SQ/CQ IRQ | `pool[0]`, `pool[1]` |
|
||||
| Main worker process and subthreads | `pool[2:-2]` |
|
||||
| ACL thread | `pool[-2]` |
|
||||
| Release thread | `pool[-1]` |
|
||||
|
||||
For Ascend 950:
|
||||
|
||||
| Role | CPUs |
|
||||
| --- | --- |
|
||||
| Main worker process and subthreads | the whole assigned cluster |
|
||||
| ACL thread | not separately pinned |
|
||||
| Release thread | not separately pinned |
|
||||
|
||||
If a final pool has fewer CPUs than the selected role split requires, binding
|
||||
fails for this rank and the worker logs a warning from the caller. The minimum
|
||||
is 5 CPUs per NPU for devices with IRQ binding. Ascend 950 requires one full
|
||||
cluster per worker.
|
||||
|
||||
## Conditional Host Tuning
|
||||
|
||||
After CPU affinity is applied, CPU binding can also apply two host-side tuning
|
||||
steps when the environment supports them:
|
||||
|
||||
- Memory migration uses `migratepages` to move the worker process's existing
|
||||
pages to the selected NUMA node. This keeps the worker closer to the memory it
|
||||
reads and reduces remote-NUMA memory read latency.
|
||||
- IRQ binding places NPU IRQ handling on the CPUs reserved for the corresponding
|
||||
NPU when `/proc/irq` is writable and IRQ files can be resolved.
|
||||
Ascend 950 skips this step.
|
||||
|
||||
These are conditional parts of CPU binding, not separate feature switches. If a
|
||||
host prerequisite is missing, that step is skipped while CPU thread binding
|
||||
still proceeds. Missing `migratepages` can still leave pages on remote NUMA
|
||||
nodes, so **latency or throughput may regress compared with a full CPU binding
|
||||
setup.**
|
||||
|
||||
## Examples
|
||||
|
||||
### A3 inference server with 640 CPUs and 16 NPUs
|
||||
|
||||
Inputs:
|
||||
|
||||
- `allowed_cpus = [0..639]`
|
||||
- `total_logic_npus = 16`
|
||||
- `running_npu_list = [0..15]`
|
||||
|
||||
Computation:
|
||||
|
||||
- `base = 640 // 16 = 40`
|
||||
- `extra = 0`
|
||||
- Worker `i` driving logical NPU `i` receives CPU slice
|
||||
`[i * 40 .. i * 40 + 39]`.
|
||||
|
||||
Global slice view:
|
||||
|
||||
```text
|
||||
CPU range: 0 639
|
||||
|-- worker0/NPU0 --|-- worker1/NPU1 --| ... |-- worker15/NPU15 --|
|
||||
| 0-39 | 40-79 | ... | 600-639 |
|
||||
```
|
||||
|
||||
Role split inside each worker slice:
|
||||
|
||||
```text
|
||||
40-CPU worker slice
|
||||
| IRQ CPUs | main worker process and subthreads | ACL thread | release thread |
|
||||
| c0-c1 | c2-c37 | c38 | c39 |
|
||||
```
|
||||
|
||||
Concrete examples:
|
||||
|
||||
| Worker | Logical NPU | CPU pool | IRQ CPUs | Main CPUs | ACL CPU | Release CPU |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| 0 | 0 | 0-39 | 0-1 | 2-37 | 38 | 39 |
|
||||
| 1 | 1 | 40-79 | 40-41 | 42-77 | 78 | 79 |
|
||||
| ... | ... | ... | ... | ... | ... | ... |
|
||||
| 15 | 15 | 600-639 | 600-601 | 602-637 | 638 | 639 |
|
||||
|
||||
This layout remains deterministic even when different worker processes share
|
||||
the same cpuset, because slicing is based on the global logical NPU ID.
|
||||
|
||||
### A2 topo_affinity with hidden same-affinity NPUs
|
||||
|
||||
Inputs from an A2 topology:
|
||||
|
||||
- NPU0 affinity: 144-167
|
||||
- NPU2 affinity: 144-167
|
||||
- Process A sees only NPU0
|
||||
- Process B sees only NPU2
|
||||
- Both processes have `allowed_cpus = [144..191]`
|
||||
|
||||
The allocator includes the hidden same-affinity NPU as a candidate in each
|
||||
process, splits the shared extended pool, and then keeps only the visible NPU in
|
||||
the final pool.
|
||||
|
||||
Final pools:
|
||||
|
||||
| Process | Visible NPU | Final CPU pool |
|
||||
| --- | --- | --- |
|
||||
| A | 0 | 144-167 |
|
||||
| B | 2 | 168-191 |
|
||||
|
||||
This avoids overlapping CPU pools even when the two workers are launched as independent single-card services.
|
||||
|
||||
## Logs
|
||||
|
||||
The allocator logs the selected mode and allocation plan:
|
||||
|
||||
```text
|
||||
[cpu_bind_mode] mode=topo_affinity rank=0 visible_npus=[0]
|
||||
The CPU allocation plan is as follows:
|
||||
NPU0: main=[...] acl=[...] release=[...]
|
||||
```
|
||||
|
||||
Ascend 950 uses a different role split, so its plan log does not include ACL or
|
||||
release fields. UVB polling thread binding is reported separately when matching
|
||||
threads are found:
|
||||
|
||||
```text
|
||||
[cpu_bind_mode] mode=topo_affinity rank=0 visible_npus=[0]
|
||||
The CPU allocation plan is as follows:
|
||||
Ascend 950 NPU0: worker=[...]
|
||||
[cpu_bind_ascend_950] uvb_poll_window_thread tids=[...] cpus=[...]
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- CPU binding runs only on ARM. It is skipped on x86_64.
|
||||
- Each final NPU pool must have enough CPUs for its role split: at least 5 CPUs
|
||||
for devices with IRQ binding. Ascend 950 requires one complete CPU cluster per worker.
|
||||
- `global_slice` is deterministic and provides CPU/NUMA isolation when the
|
||||
cpuset is NUMA-aligned, but it cannot guarantee NUMA-local pools when CPU
|
||||
numbering or cpuset layout crosses NUMA boundaries.
|
||||
- `topo_affinity` depends on usable output from `npu-smi info -t topo`.
|
||||
- IRQ binding requires writable `/proc/irq` and resolvable PCI/IRQ information.
|
||||
Ascend 950 skips IRQ binding even when `/proc/irq` is writable.
|
||||
- Ascend 950 UVB polling thread binding requires visibility into the host PID
|
||||
namespace. Docker containers must be created with `--pid=host`; otherwise
|
||||
`uvb_poll_window_thread` may not be found.
|
||||
- Memory migration requires `migratepages`; otherwise only memory migration is
|
||||
skipped. CPU affinity still applies, but performance may degrade because
|
||||
existing pages are not moved to the target NUMA node and may be read through
|
||||
higher-latency remote NUMA access.
|
||||
- If an exception escapes the binding flow, `NPUWorker` logs a warning and skips CPU binding for that rank.
|
||||
|
||||
## References
|
||||
|
||||
- Implementation: `vllm_ascend/cpu_binding.py`
|
||||
- Worker integration: `vllm_ascend/worker/worker.py`
|
||||
- Config: `vllm_ascend/ascend_config.py` and `docs/source/user_guide/configuration/additional_config.md`
|
||||
- Tests: `tests/ut/device_allocator/test_cpu_binding.py`
|
||||
@@ -0,0 +1,105 @@
|
||||
# Disaggregated-prefill
|
||||
|
||||
## Why disaggregated-prefill?
|
||||
|
||||
This feature addresses the need to optimize the **Time Per Output Token (TPOT)** and **Time To First Token (TTFT)** in large-scale inference tasks. The motivation is two-fold:
|
||||
|
||||
1. **Adjusting Parallel Strategy and Instance Count for P and D Nodes**
|
||||
Using the disaggregated-prefill strategy, this feature allows the system to flexibly adjust the parallelization strategy (e.g., data parallelism (dp), tensor parallelism (tp), and expert parallelism (ep)) and the instance count for both P (Prefiller) and D (Decoder) nodes. This leads to better system performance tuning, particularly for **TTFT** and **TPOT**.
|
||||
|
||||
2. **Optimizing TPOT**
|
||||
Without the disaggregated-prefill strategy, prefill tasks are inserted during decoding, which results in inefficiencies and delays. Disaggregated-prefill solves this by allowing for better control over the system's **TPOT**. By managing chunked prefill tasks effectively, the system avoids the challenge of determining the optimal chunk size and provides more reliable control over the time taken for generating output tokens.
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
vLLM Ascend currently supports two types of connectors for handling KV cache management:
|
||||
|
||||
- **MooncakeConnector**: D nodes pull KV cache from P nodes.
|
||||
- **MooncakeLayerwiseConnector**: P nodes push KV cache to D nodes in a layered manner.
|
||||
|
||||
For step-by-step deployment and configuration, refer to the following guide:
|
||||
[https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_disaggregation_mooncake_multi_node.html](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_disaggregation_mooncake_multi_node.html)
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
### 1. Design Approach
|
||||
|
||||
Under the disaggregated-prefill, a global proxy receives external requests, forwarding prefill to P nodes and decode to D nodes; the KV cache (key-value cache) is exchanged between P and D nodes via peer-to-peer (P2P) communication.
|
||||
|
||||
### 2. Implementation Design
|
||||
|
||||
Our design diagram is shown below, illustrating the pull and push schemes respectively.
|
||||
|
||||
|
||||
|
||||
#### Mooncake Connector
|
||||
|
||||
1. The request is sent to the Proxy's `_handle_completions` endpoint.
|
||||
2. The Proxy calls `select_prefiller` to choose a P node and forwards the request, configuring `kv_transfer_params` with `do_remote_decode=True`, `max_completion_tokens=1`, and `min_tokens=1`.
|
||||
3. After the P node's scheduler finishes prefill, `update_from_output` invokes the schedule connector's `request_finished` to defer KV cache release, constructs `kv_transfer_params` with `do_remote_prefill=True`, and returns to the Proxy.
|
||||
4. The Proxy calls `select_decoder` to choose a D node and forwards the request.
|
||||
5. On the D node, the scheduler marks the request as `RequestStatus.WAITING_FOR_REMOTE_KVS`, pre-allocates KV cache, calls `kv_connector_no_forward` to pull the remote KV cache, then notifies the P node to release KV cache and proceeds with decoding to return the result.
|
||||
|
||||
#### Mooncake Layerwise Connector
|
||||
|
||||
1. The request is sent to the Proxy's `_handle_completions` endpoint.
|
||||
2. The Proxy calls `select_decoder` to choose a D node and forwards the request, configuring `kv_transfer_params` with `do_remote_prefill=True` and setting the `metaserver` endpoint.
|
||||
3. On the D node, the scheduler uses `kv_transfer_params` to mark the request as `RequestStatus.WAITING_FOR_REMOTE_KVS`, pre-allocates KV cache, then calls `kv_connector_no_forward` to send a request to the metaserver and waits for the KV cache transfer to complete.
|
||||
4. The Proxy's `metaserver` endpoint receives the request, calls `select_prefiller` to choose a P node, and forwards it with `kv_transfer_params` set to `do_remote_decode=True`, `max_completion_tokens=1`, and `min_tokens=1`.
|
||||
5. During processing, the P node's scheduler pushes KV cache layer-wise; once all layers pushing is complete, it releases the request and notifies the D node to begin decoding.
|
||||
6. The D node performs decoding and returns the result.
|
||||
|
||||
### 3. Interface Design
|
||||
|
||||
Taking MooncakeConnector as an example, the system is organized into three primary classes:
|
||||
|
||||
- **MooncakeConnector**: Base class that provides core interfaces.
|
||||
- **MooncakeConnectorScheduler**: Interface for scheduling the connectors within the engine core, responsible for managing KV cache transfer requirements and completion.
|
||||
- **MooncakeConnectorWorker**: Interface for managing KV cache registration and transfer in worker processes.
|
||||
|
||||
### 4. Specifications Design
|
||||
|
||||
This feature is flexible and supports various configurations, including setups with MLA and GQA models. It is compatible with A2 and A3 hardware configurations and facilitates scenarios involving equal TP setups and certain unequal TP setups across multiple P and D nodes.
|
||||
|
||||
| Feature | Status |
|
||||
|-------------------------------|----------------|
|
||||
| A2 | 🟢 Functional |
|
||||
| A3 | 🟢 Functional |
|
||||
| equal TP configuration | 🟢 Functional |
|
||||
| unequal TP configuration | 🟢 Functional |
|
||||
| MLA | 🟢 Functional |
|
||||
| GQA | 🟢 Functional |
|
||||
|
||||
- 🟢 Functional: Fully operational, with ongoing optimizations.
|
||||
- 🔵 Experimental: Experimental support, interfaces and functions may change.
|
||||
- 🚧 WIP: Under active development, will be supported soon.
|
||||
- 🟡 Planned: Scheduled for future implementation (some may have open PRs/RFCs).
|
||||
- 🔴 NO plan/Deprecated: No plan or deprecated by vLLM.
|
||||
|
||||
---
|
||||
|
||||
## DFX Analysis
|
||||
|
||||
### 1. Config Parameter Validation
|
||||
|
||||
Validate KV transfer config by checking whether the kv_connector type is supported. On transfer failures, emit clear error logs for diagnostics.
|
||||
|
||||
### 2. Port Conflict Detection
|
||||
|
||||
Before startup, perform a port-usage check on configured ports (e.g., rpc_port, metrics_port, http_port/metaserver) by attempting to bind. If a port is already in use, fail fast and log an error.
|
||||
|
||||
### 3. PD Ratio Validation
|
||||
|
||||
Under non-symmetric PD scenarios, validate the P-to-D tp ratio against expected and scheduling constraints to ensure correct and reliable operation.
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
- Heterogeneous P and D nodes are not supported, for example, running P nodes on A2 and D nodes on A3.
|
||||
|
||||
- In non-symmetric TP configurations, only cases where the P nodes have a higher TP degree than the D nodes and the P TP count is an integer multiple of the D TP count are supported (i.e., P_tp > D_tp and P_tp % D_tp = 0).
|
||||
@@ -0,0 +1,158 @@
|
||||
# Dynamic Chunked Pipeline Parallel (CPP)
|
||||
|
||||
TL;DR CPP uses profiling-based dynamic chunking to equalize per-chunk latency and eliminate pipeline bubbles in PP scenarios.
|
||||
|
||||
## Background
|
||||
|
||||
### Problem Statement
|
||||
|
||||
In Pipeline Parallelism (PP) + Chunked Prefill scenarios, long sequences are split into fixed-size chunks that pass through the pipeline sequentially. Due to the O(n²) computational complexity of Self-Attention, **chunks of the same size take increasingly longer to process as the prefix sequence grows**:
|
||||
|
||||
```text
|
||||
Chunk 1 (history=0): ██████ → Time T1
|
||||
Chunk 2 (history=4K): ████████ → Time T2 > T1
|
||||
Chunk 3 (history=8K): ██████████ → Time T3 > T2
|
||||
Chunk 4 (history=12K): ████████████ → Time T4 > T3
|
||||
```
|
||||
|
||||
This time variance propagates across pipeline stages, causing increased idle waiting (Pipeline Bubble) and significantly reducing GPU utilization.
|
||||
|
||||
### Solution Overview
|
||||
|
||||
Dynamic Chunked Pipeline Parallel uses a **profile-first, then predict** strategy:
|
||||
|
||||
```text
|
||||
Fixed Chunking (equal chunk size, unequal time):
|
||||
|
||||
Stage 0 |■■■■|■■■■■■|■■■■■■■■|■■■■■■■■■■|
|
||||
Stage 1 | |■■■■ |■■■■■■ |■■■■■■■■ |■■■■■■■■■■|
|
||||
↑ bubble ↑ bubble ↑ bubble
|
||||
|
||||
Dynamic Chunking (unequal chunk size, equal time):
|
||||
|
||||
Stage 0 |■■■■■■|■■■■■■|■■■■■■|■■■■■■|
|
||||
Stage 1 | |■■■■■■|■■■■■■|■■■■■■|■■■■■■|
|
||||
↑ no bubble — stages stay in sync
|
||||
```
|
||||
|
||||
The core idea is borrowed from [SGLang's dynamic chunking mechanism](https://lmsys.org/blog/2026-01-15-chunked-pipeline/), with additional enhancements such as online calibration.
|
||||
|
||||
## Design
|
||||
|
||||
### Quadratic Latency Model
|
||||
|
||||
Transformer prefill latency grows quadratically with sequence length due to the O(n²) Self-Attention mechanism:
|
||||
|
||||
$$f(l) = a \cdot l^2 + b \cdot l + c$$
|
||||
|
||||
Where:
|
||||
|
||||
- $a \cdot l^2$: Attention overhead (quadratic)
|
||||
- $b \cdot l$: Linear operations (FFN, projection)
|
||||
- $c$: Fixed overhead (kernel launch)
|
||||
|
||||
### Startup Phase: Profiling
|
||||
|
||||
During engine initialization, the system profiles actual model performance:
|
||||
|
||||
1. **Sampling**: Uniformly sample 64 different chunk sizes from `base_chunk_size` down to near 0
|
||||
2. **Execution**: Perform real model forward passes for each chunk size and precisely measure latency (milliseconds)
|
||||
3. **Fitting**: Fit the quadratic model using least squares
|
||||
4. **Target Setting**: Calculate target per-chunk latency based on `base_chunk_size`
|
||||
|
||||
In PP mode, all workers execute forward passes to stay synchronized, but only the first PP rank's timing results are used for scheduling decisions.
|
||||
|
||||
### Runtime Phase: Dynamic Prediction
|
||||
|
||||
Given current prefix length $L$ and target latency $T = f(\text{base\_chunk\_size}) - f(0)$, the system solves for the next chunk size $x$:
|
||||
|
||||
$$f(L + x) - f(L) = T$$
|
||||
|
||||
Expanding to:
|
||||
|
||||
$$a \cdot x^2 + (2aL + b) \cdot x - T = 0$$
|
||||
|
||||
Solved using the quadratic formula:
|
||||
|
||||
$$x = \frac{-(2aL + b) + \sqrt{(2aL + b)^2 + 4aT}}{2a}$$
|
||||
|
||||
The result goes through post-processing:
|
||||
|
||||
1. **Smoothing**: Blend predicted chunk size with `base_chunk_size` using `smooth_factor`
|
||||
2. **Alignment**: Round down to multiple of `page_size` (minimum 64)
|
||||
3. **Constraints**: Not exceeding `max_model_len - history_len` and `max_num_scheduled_tokens`
|
||||
|
||||
### Online Calibration
|
||||
|
||||
Since profiling only covers sequences up to `max_num_batched_tokens` (typically shorter than real workloads), the system continuously refines the model at runtime.
|
||||
|
||||
**Extended Model (two variables):**
|
||||
|
||||
$$f(C, H) = a \cdot C(C+H) + b \cdot (C+H) + c$$
|
||||
|
||||
Where $C$ is chunk size and $H$ is prefix history length.
|
||||
|
||||
After each batch, feature vectors `[Σ(C+H)·C, Σ(C+H), N]` and actual execution time are recorded. Once enough data points accumulate (5-30), model parameters are updated using least squares.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Key Components
|
||||
|
||||
| Component | Location | Responsibility |
|
||||
|-----------|----------|---------------|
|
||||
| **ChunkSizePredictor** | `vllm_ascend/core/profiling_chunk_predictor.py` | Quadratic model fitting and prediction |
|
||||
| **ProfilingChunkManager** | `vllm_ascend/core/profiling_chunk_predictor.py` | Manage profiling workflow and predictor |
|
||||
| **Scheduler** | `vllm_ascend/core/scheduler_profiling_chunk.py` | Integrate CPP scheduling |
|
||||
| **EngineCore** | `vllm_ascend/patch/platform/patch_profiling_chunk.py` | Startup profiling, record execution time |
|
||||
| **NPUWorker** | `vllm_ascend/worker/worker.py` | Execute real forward pass profiling |
|
||||
| **NPUModelRunner** | `vllm_ascend/worker/model_runner_v1.py` | `profile_cpp=True` mode |
|
||||
|
||||
### Workflow
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Startup Phase │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 1. EngineCore.init() triggers profiling │
|
||||
│ 2. ProfilingChunkManager samples 64 chunk sizes │
|
||||
│ 3. NPUWorker executes forward passes │
|
||||
│ 4. ChunkSizePredictor fits quadratic model │
|
||||
│ 5. Target latency = f(base_chunk_size) - f(0) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Runtime Phase │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ For each prefill chunk: │
|
||||
│ 1. Scheduler queries ChunkSizePredictor │
|
||||
│ 2. Given history length L, solve for optimal chunk size │
|
||||
│ 3. Apply smoothing and alignment │
|
||||
│ 4. Execute chunk │
|
||||
│ 5. Record actual timing for online calibration │
|
||||
│ 6. Update model if enough samples collected │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Comparison with SGLang
|
||||
|
||||
| Feature | SGLang Dynamic Chunking | Dynamic Chunked Pipeline Parallel |
|
||||
|---------|------------------------|-----------------------------------|
|
||||
| Profiling method | Preset quadratic function | Real forward pass profiling at startup |
|
||||
| Model fitting | $f(l) = a \cdot l^2 + b \cdot l + c$ | Same + online calibration $f(C,H)$ |
|
||||
| Online updates | None | History-based fitting |
|
||||
| Accuracy | May deviate on different hardware | Adapts to actual hardware performance |
|
||||
| Startup cost | None | ~64 forward passes (tens of seconds) |
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Pipeline Parallelism Required**: Must set `--pipeline-parallel-size > 1`
|
||||
- **Chunked Prefill Required**: Must enable `--enable-chunked-prefill`
|
||||
- **Incompatible with Balance Scheduling**: Cannot enable `VLLM_ASCEND_BALANCE_SCHEDULING`
|
||||
- **Startup Overhead**: Profiling phase adds tens of seconds to initialization
|
||||
- **Memory**: No additional runtime memory overhead; profiling reuses existing dummy_run mechanism
|
||||
|
||||
## References
|
||||
|
||||
- [SGLang Dynamic Chunking Blog](https://lmsys.org/blog/2026-01-15-chunked-pipeline/)
|
||||
- [User Guide](../../user_guide/feature_guide/dynamic_chunk_pipeline_parallel.md)
|
||||
- [Tutorial](../../tutorials/features/dynamic_chunked_pipeline_parallel.md)
|
||||
@@ -0,0 +1,245 @@
|
||||
# Expert Parallelism Load Balancer (EPLB)
|
||||
|
||||
## Why We Need EPLB?
|
||||
|
||||
When using Expert Parallelism (EP), different experts are assigned to different NPUs. Given that the load of various experts may vary depending on the current workload, it is crucial to maintain balanced loads across different NPUs. We adopt a redundant experts strategy by duplicating heavily-loaded experts. Then, we heuristically pack these duplicated experts onto NPUs to ensure load balancing across them. Moreover, thanks to the group-limited expert routing used in MoE models, we also attempt to place experts of the same group on the same node to reduce inter-node data traffic, whenever possible.
|
||||
|
||||
To facilitate reproduction and deployment, vLLM Ascend supports the deployed EP load balancing algorithm in `vllm_ascend/eplb/core/policy`. The algorithm computes a balanced expert replication and placement plan based on the estimated expert loads. Note that the exact method for predicting expert loads is outside the scope of this repository. A common method is to use a moving average of historical statistics.
|
||||
|
||||
|
||||
|
||||
## How to Use EPLB?
|
||||
|
||||
Please refer to the EPLB section of the user guide for detailed information: [How to Use EPLB](../../user_guide/feature_guide/expert_parallelism_load_balancer.md)
|
||||
|
||||
## How It Works?
|
||||
|
||||
**EPLB Module Architecture**
|
||||
|
||||
```shell
|
||||
vllm_ascend
|
||||
├── eplb
|
||||
│ ├── adaptor
|
||||
│ │ └── vllm_adaptor.py
|
||||
│ ├── core
|
||||
│ │ ├── policy
|
||||
│ │ │ ├── policy_abstract.py
|
||||
│ │ │ ├── policy_default_eplb.py
|
||||
│ │ │ ├── policy_factory.py
|
||||
│ │ │ ├── policy_flashlb.py
|
||||
│ │ │ ├── policy_random.py
|
||||
│ │ │ └── policy_swift_balancer.py
|
||||
│ │ ├── eplb_device_transfer_loader.py
|
||||
│ │ ├── eplb_utils.py
|
||||
│ │ └── eplb_worker.py
|
||||
│ ├── eplb_updator.py
|
||||
│ └── utils.py
|
||||
└───────────
|
||||
```
|
||||
|
||||
**1. Adaptor Module**
|
||||
*Handles registration and adaptation for different MoE model types*
|
||||
|
||||
- `vllm_adaptor.py`
|
||||
Implementation supporting Qwen3-MoE and DeepSeek models, standardizing parameter handling for policy algorithms
|
||||
|
||||
**2. Core Module**
|
||||
*Implements core algorithms, updates, and asynchronous processing*
|
||||
|
||||
- **Policy Submodule**
|
||||
*Load balancing algorithms with factory pattern instantiation*
|
||||
- `policy_abstract.py`
|
||||
Abstract class for load balancing strategy interfaces
|
||||
- `policy_default_eplb.py`
|
||||
Default implementation of open-source EPLB paper algorithm
|
||||
- `policy_swift_balancer.py`
|
||||
Enhanced version optimizing expert swaps for low-bandwidth devices (e.g., A2)
|
||||
- `policy_flashlb.py`
|
||||
Threshold-based adjustment reducing operational costs through layer-wise fluctuation detection
|
||||
- `policy_random.py`
|
||||
Random policy for basic testing
|
||||
- `policy_factory.py`
|
||||
Strategy factory for automatic algorithm instantiation
|
||||
|
||||
- `eplb_device_transfer_loader.py`
|
||||
Manages expert table/weight transmission and updates
|
||||
- `eplb_utils.py`
|
||||
Utilities for expert table initialization and mapping
|
||||
- `eplb_worker.py`
|
||||
Asynchronous algorithm orchestration and result processing
|
||||
|
||||
**3. System Components**
|
||||
|
||||
- `eplb_updator.py`
|
||||
Central coordinator for load balancing during inference workflows
|
||||
- `utils.py`
|
||||
General utilities for EPLB interface registration
|
||||
|
||||
*Key Optimizations:*
|
||||
|
||||
1. Maintained original structure while improving technical clarity
|
||||
2. Standardized terminology
|
||||
3. Enhanced algorithm differentiation through concise descriptors
|
||||
4. Improved scoping through hierarchical presentation
|
||||
5. Preserved file/class relationships while optimizing readability
|
||||
|
||||
### Default Algorithm
|
||||
|
||||
#### Hierarchical Load Balancing
|
||||
|
||||
When the number of server nodes evenly divides the number of expert groups, we use the hierarchical load balancing policy to leverage group-limited expert routing. We first pack the expert groups onto nodes evenly, ensuring balanced loads across different nodes. Then, we replicate the experts within each node. Finally, we pack the replicated experts onto individual NPUs to ensure load balancing across them. The hierarchical load balancing policy can be used in the prefilling stage with a smaller expert-parallel size.
|
||||
|
||||
#### Global Load Balancing
|
||||
|
||||
In other cases, we use the global load balancing policy, which replicates experts globally regardless of expert groups, and packs the replicated experts onto individual NPUs. This policy can be adopted in the decoding stage with a larger expert-parallel size.
|
||||
|
||||
### Add a New EPLB Policy
|
||||
|
||||
If you want to add a new eplb policy to vllm_ascend, you must follow these steps:
|
||||
|
||||
1. Inherit the `EplbPolicy` abstract class of `policy_abstract.py` and override the `rebalance_experts` interface, ensuring consistent input parameters `current_expert_table`, `expert_workload` and return types `newplacement`.
|
||||
For example:
|
||||
|
||||
```python
|
||||
class RandomLoadBalance(EplbPolicy):
|
||||
def rebalance_experts(self, current_expert_table, expert_workload):
|
||||
new_table = copy.deepcopy(current_expert_table)
|
||||
num_layers = len(current_expert_table)
|
||||
|
||||
for i in range(num_layers):
|
||||
# randomly choose two card
|
||||
# indices = random.sample(range(num_card), 2)
|
||||
indices = [3, 1]
|
||||
|
||||
# swap redundant experts
|
||||
expert_id_to_exchange = new_table[i][indices[0]][-1].clone()
|
||||
new_table[i][indices[0]][-1] = new_table[i][indices[1]][-1]
|
||||
new_table[i][indices[1]][-1] = expert_id_to_exchange
|
||||
|
||||
return 1, [-i for i in range(num_layers)], new_table
|
||||
```
|
||||
|
||||
2. To add a new EPLB algorithm, include the policy type and its corresponding implementation class in the `PolicyFactory` of `policy_factory.py`.
|
||||
|
||||
### Add a New MoE Model
|
||||
|
||||
**Implementation Guide for Model Integration**
|
||||
|
||||
1. **Adapter File Modification**
|
||||
- Inherit or modify `vllm_ascend/eplb/adaptor/vllm_adaptor.py`
|
||||
- Add processing logic for key parameters:
|
||||
- `num_dense_layers`
|
||||
- `global_expert_num`
|
||||
- `num_roe_layers`
|
||||
- Ensure parameter synchronization in the `model_register` function.
|
||||
|
||||
For example:
|
||||
|
||||
Modify `__init__` of `vllm_adaptor.py` to add a new moe model eplb params:
|
||||
|
||||
```python
|
||||
if self.model.config.model_type == "qwen3_moe":
|
||||
self.num_dense_layers = 0
|
||||
self.global_expert_num = self.model.config.num_experts
|
||||
```
|
||||
|
||||
Modify `model_register` of `vllm_adaptor.py` to register eplb params for new moe model:
|
||||
|
||||
```python
|
||||
if config.model_type == "qwen3_moe":
|
||||
model.num_moe_layers = config.num_hidden_layers
|
||||
```
|
||||
|
||||
2. **MoE Feature Integration**
|
||||
- Extend `vllm_ascend/eplb/utils.py` with MoE-specific methods
|
||||
- Implement required functionality for expert routing or weight management
|
||||
|
||||
3. **Registration Logic Update**
|
||||
- Add patch logic within the `model_register` function
|
||||
- Maintain backward compatibility with existing model types
|
||||
|
||||
4. **Validation & Testing**
|
||||
- Verify parameter consistency across layers
|
||||
- Test cross-device communication for expert tables
|
||||
- Benchmark against baseline implementations (e.g., Qwen3-MoE)
|
||||
|
||||
*Key Implementation Notes:*
|
||||
|
||||
- Preserve existing interface contracts in abstract classes
|
||||
- Use decorators for non-intrusive patch integration
|
||||
- Leverage `eplb_utils.py` for shared expert mapping operations
|
||||
|
||||
## DFX
|
||||
|
||||
### Parameter Validation
|
||||
|
||||
#### Integer Parameters
|
||||
|
||||
All integer input parameters must explicitly specify their maximum and minimum values and be subject to valid value validation. For example, `expert_heat_collection_interval` must be greater than 0:
|
||||
|
||||
```python
|
||||
@staticmethod
|
||||
def check_iterations(iterations):
|
||||
if not isinstance(iterations, int):
|
||||
raise TypeError(f"The {iterations} is not int.")
|
||||
if iterations <= 0:
|
||||
raise ValueError(
|
||||
f"The {iterations} can not be less than or equal to 0.")
|
||||
if iterations > sys.maxsize:
|
||||
raise ValueError(
|
||||
f"The {iterations} can not be larger than {sys.maxsize}")
|
||||
```
|
||||
|
||||
#### File Path
|
||||
|
||||
The file path for EPLB must be checked for legality, such as whether the file path is valid and whether it has appropriate read and write permissions. For example:
|
||||
|
||||
```python
|
||||
@staticmethod
|
||||
def check_expert_map_path(expert_map):
|
||||
if expert_map is None:
|
||||
return
|
||||
if not isinstance(expert_map, str):
|
||||
raise TypeError("The expert_map is not str.")
|
||||
if not expert_map.strip():
|
||||
raise ValueError("The expert_map is not empty.")
|
||||
_, ext = os.path.splitext(expert_map)
|
||||
if ext.lower() != ".json":
|
||||
raise TypeError("The expert_map is not json.")
|
||||
if not os.path.exists(expert_map):
|
||||
raise ValueError("The expert_map does not exist.")
|
||||
try:
|
||||
with open(expert_map, "w", encoding='utf-8') as f:
|
||||
f.read()
|
||||
except Exception as e:
|
||||
raise IOError(
|
||||
f"Fail read expert info from {expert_map}, please check the reading permission of {expert_map} : {e}"
|
||||
)
|
||||
|
||||
```
|
||||
|
||||
### Function Specifications
|
||||
|
||||
#### Initialization Function
|
||||
|
||||
All EPLB parameters must be initialized by default during initialization, with specified parameter types and default values for proper handling.
|
||||
|
||||
#### General Functions
|
||||
|
||||
All method arguments must specify parameter types and default values, and functions must include default return value handling for default arguments. It is recommended to use `try-except` blocks to handle the function body, specifying the type of exception captured and the failure handling (e.g., logging exceptions or returning a failure status).
|
||||
|
||||
### Consistency
|
||||
|
||||
#### Expert Map
|
||||
|
||||
The expert map must be globally unique during initialization and update. In a multi-node scenario during initialization, distributed communication should be used to verify the consistency of expert maps across each rank. If they are inconsistent, the user should be notified of which ranks have inconsistent maps.
|
||||
During the update process, if only a few layers or the expert table of a certain rank has been changed, the updated expert table must be synchronized with the EPLB's context to ensure global consistency.
|
||||
|
||||
#### Expert Weight
|
||||
|
||||
When updating expert weights, ensure that the memory allocated for the expert weights has been released, or that the expert (referring to the old version) is no longer in use.
|
||||
|
||||
## Limitations
|
||||
|
||||
Before using EPLB, start the script and add `export DYNAMIC_EPLB="true"`.
|
||||
Before performing load data collection (or performance data collection), start the script and add `export EXPERT_MAP_RECORD="true"`.
|
||||
20
docs/source/developer_guide/Design_Documents/index.md
Normal file
20
docs/source/developer_guide/Design_Documents/index.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# Design Documents
|
||||
|
||||
This section provides an overview of the features implemented in vLLM Ascend. Developers can refer to this guide to understand how vLLM Ascend works.
|
||||
|
||||
:::{toctree}
|
||||
:caption: Design Documents
|
||||
:maxdepth: 1
|
||||
patch
|
||||
cpu_binding
|
||||
ModelRunner_prepare_inputs
|
||||
disaggregated_prefill
|
||||
eplb_swift_balancer
|
||||
ACL_Graph
|
||||
KV_Cache_Pool_Guide
|
||||
add_custom_aclnn_op
|
||||
context_parallel
|
||||
dynamic_chunked_pipeline_parallel
|
||||
quantization
|
||||
npugraph_ex
|
||||
:::
|
||||
105
docs/source/developer_guide/Design_Documents/npugraph_ex.md
Normal file
105
docs/source/developer_guide/Design_Documents/npugraph_ex.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# Npugraph_ex
|
||||
|
||||
## How Does It Work?
|
||||
|
||||
This is an optimization based on FX graphs, which can be considered an acceleration solution for the aclgraph mode.
|
||||
|
||||
You can get its code [code](https://gitcode.com/Ascend/torchair)
|
||||
|
||||
```{note}
|
||||
Atlas 300I DUO and Atlas 200I Pro do not support `enable_npugraph_ex`. Set --additional-config '{"ascend_compilation_config": {"enable_npugraph_ex":false}}'.
|
||||
```
|
||||
|
||||
## Default FX Graph Optimization
|
||||
|
||||
### FX Graph pass
|
||||
|
||||
- For the intermediate nodes of the model, replace the non-in-place operators contained in the nodes with in-place operators to reduce memory movement during computation and improve performance.
|
||||
- For the original input parameters of the model, if they include in-place operators, Dynamo's Functionalize process will replace the in-place operators with a form of non-in-place operators + copy operators. npugraph_ex will reverse this process, restoring the in-place operators and reducing memory movement.
|
||||
|
||||
### FX fusion pass
|
||||
|
||||
npugraph_ex now provides some operator fusion passes, and more will be added in the future.
|
||||
|
||||
Operator combinations that meet the replacement rules can be replaced with the corresponding fused operators.
|
||||
|
||||
You can get the default [fusion pass list](https://www.hiascend.com/document/detail/zh/Pytorch/2600/modthirdparty/torchairuseguide/docs/zh/npugraph_ex/basic/pattern_fusion_pass.md#功能简介)
|
||||
|
||||
## Custom fusion pass
|
||||
|
||||
Users can register a custom graph fusion pass in npugraph_ex to modify PyTorch FX graphs. The registration relies on the register_replacement API.
|
||||
|
||||
Below is the declaration of this API and a demo of its usage.
|
||||
|
||||
```python
|
||||
register_replacement(search_fn, replace_fn, example_inputs, trace_fn=fwd_only, extra_check=_return_true, search_fn_pattern=None)
|
||||
```
|
||||
|
||||
|Parameter Name| Input/Output |Explanation|Is necessary|
|
||||
|--|--------------|---|-------|
|
||||
|search_fn|Input|This function is the operator combination or calculation logic that you want to recognize in the FX graph, such as the operator combination that needs to be fused|Yes|
|
||||
|replace_fn|Input|When the combination corresponding to search_fn is found in the target graph, this function's computation logic will replace the original subgraph to achieve operator fusion or optimization.|Yes|
|
||||
|example_inputs|Input|Example input tensors used to track search_fn and replace_fn. The shape and dtype of the input should match the actual scenario.|Yes|
|
||||
|trace_fn|Input|By default, only the forward computation graph is tracked, which is suitable for optimization during the inference phase; if training scenarios need to be supported, a function that supports backward tracking can be provided.|No|
|
||||
|extra_check|Input|Find the extra verification function after operator fusion. The function's input parameter must be a Match object from torch._inductor.pattern_matcher, and it is used for further custom checks on the matching result, such as checking whether the fused operators are on the same stream, checking the device type, checking the input shapes, and so on.|No|
|
||||
|search_fn_pattern|Input|A custom pattern object is generally unnecessary to provide. Its definition follows the rules of the native PyTorch MultiOutputPattern object. After passing this parameter, search_fn will no longer be used to match operator combinations; instead, this parameter will be used directly as the matching rule.|No|
|
||||
|
||||
### Usage Example
|
||||
|
||||
```python
|
||||
import functools
|
||||
import torch, torch_npu, npugraph_ex
|
||||
|
||||
from torch._inductor.pattern_matcher import Match
|
||||
from torch._subclasses.fake_tensor import FakeTensorMode
|
||||
from npugraph_ex.core.utils import logger
|
||||
|
||||
# Assume fusing the add operator and the npu_rms_norm operator into the npu_add_rms_norm operator
|
||||
# Define a search_fn to find the operator combinations in the original FX graph before fusion.
|
||||
def search_fn(x1, x2, gamma):
|
||||
xOut = torch.add(x1, x2)
|
||||
y, _ = torch_npu.npu_rms_norm(xOut, gamma)
|
||||
return y, xOut
|
||||
|
||||
# Define a replace_fn, that is, a fusion operator, used to replace operator combinations in the FX graph
|
||||
def replace_fn(x1, x2, gamma):
|
||||
y, _, xOut = torch_npu.npu_add_rms_norm(
|
||||
x1, x2, gamma
|
||||
)
|
||||
return y, xOut
|
||||
|
||||
# extra_check can pass in additional validation logic. Here, it is used to check whether the last dimension of the first input parameter x1 is a specific value; if it is not the specific value, fusion is not allowed.
|
||||
def extra_check(match: Match):
|
||||
x1 = match.kwargs.get("x1")
|
||||
|
||||
if x1 is None:
|
||||
return False
|
||||
if not hasattr(x1, "meta") or "val" not in x1.meta:
|
||||
return False
|
||||
|
||||
a_shape = x1.meta["val"].shape
|
||||
return a_shape[-1] == 7168
|
||||
|
||||
|
||||
# Define some sample inputs to trace search_fn and replace_fn into an FX graph
|
||||
fake_mode = FakeTensorMode()
|
||||
with fake_mode:
|
||||
# sizes/values don't actually matter for initial trace
|
||||
# once we get a possible match we re-trace with the actual values and verify the match still holds
|
||||
input_tensor = functools.partial(torch.empty, (1, 1, 2), device="npu", dtype=torch.float16)
|
||||
kwargs_tensor = functools.partial(torch.empty, 2, device="npu", dtype=torch.float16)
|
||||
|
||||
# Call the npugraph_ex.register_replacement API with search_fn, replace_fn, and example_inputs. If there are additional validations, you can pass them in as extra_check.
|
||||
npugraph_ex.register_replacement(
|
||||
search_fn=search_fn,
|
||||
replace_fn=replace_fn,
|
||||
example_inputs=(input_tensor(), input_tensor(), kwargs_tensor()),
|
||||
extra_check=extra_check
|
||||
)
|
||||
```
|
||||
|
||||
The default fusion pass in npugraph_ex is also implemented based on this API. You can see more examples of using this API in the vllm-ascend and npugraph_ex code repositories.
|
||||
|
||||
### DFX
|
||||
|
||||
By reusing the TORCH_COMPILE_DEBUG environment variable from the PyTorch community, when TORCH_COMPILE_DEBUG=1 is set, it will output the FX graphs throughout the entire process.
|
||||
75
docs/source/developer_guide/Design_Documents/patch.md
Normal file
75
docs/source/developer_guide/Design_Documents/patch.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# Patch in vLLM Ascend
|
||||
|
||||
vLLM Ascend is a platform plugin for vLLM. Due to the different release cycle of vLLM and vLLM Ascend and their hardware limitations, we need to patch some code in vLLM to make it compatible with vLLM Ascend.
|
||||
|
||||
In vLLM Ascend code, we provide a patch module `vllm_ascend/patch` to adapt to changes in vLLM.
|
||||
|
||||
## Principle
|
||||
|
||||
We should keep in mind that Patch is not the best way to make vLLM Ascend compatible. It's just a temporary solution. The best way is to contribute the change to vLLM to make it compatible with vLLM Ascend initially. In vLLM Ascend, we have the basic principle for Patch strategy:
|
||||
|
||||
1. Less is more. Please do not patch unless it's the only way currently.
|
||||
2. Once a patch is added, it's required to describe the future plan for removing the patch.
|
||||
3. Anytime, cleaning the patch code is welcome.
|
||||
|
||||
## How it works
|
||||
|
||||
In `vllm_ascend/patch`, you can see the code structure as follows:
|
||||
|
||||
```shell
|
||||
vllm_ascend/
|
||||
└── patch/
|
||||
├── platform/
|
||||
│ └── patch_xxx.py
|
||||
└── worker/
|
||||
└── patch_yyy.py
|
||||
```
|
||||
|
||||
- **platform**: The patch code in this directory is for patching the code in vLLM Main process. It's called by `vllm_ascend/platform::NPUPlatform::pre_register_and_update` very early when vLLM is initialized.
|
||||
- For online mode, vLLM process calls the platform patch in `vllm/vllm/engine/arg_utils.py::AsyncEngineArgs.add_cli_args` when parsing the CLI args.
|
||||
- For offline mode, vLLM process calls the platform patch in `vllm/vllm/engine/arg_utils.py::EngineArgs.create_engine_config` when parsing the input parameters.
|
||||
- **worker**: The patch code in this directory is for patching the code in vLLM worker process. It's called by `vllm_ascend/worker/worker::NPUWorker::__init__` when the vLLM Worker process is initialized.
|
||||
- For both online and offline mode, vLLM EngineCore process calls the worker patch in `vllm/vllm/worker/worker_base.py::WorkerWrapperBase.init_worker` when initializing the worker process.
|
||||
|
||||
## How to write a patch
|
||||
|
||||
Before writing a patch, following the principle above, we should patch the least code. If it's necessary, we can patch the code in either **platform** or **worker** folder. Here is an example to patch `distributed` module in vLLM.
|
||||
|
||||
1. Decide which version of vLLM we should patch. For example, after analysis, here we want to patch both `0.10.0` and `main` of vLLM.
|
||||
2. Decide which process we should patch. For example, here `distributed` belongs to the vLLM main process, so we should patch `platform`.
|
||||
3. Create the patch file in the right folder. The file should be named as `patch_{module_name}.py`. The example here is `vllm_ascend/patch/platform/patch_distributed.py`.
|
||||
4. Write your patch code in the new file. Here is an example:
|
||||
|
||||
```python
|
||||
import vllm
|
||||
|
||||
def patch_destroy_model_parallel():
|
||||
# your patch code
|
||||
...
|
||||
|
||||
vllm.distributed.parallel_state.destroy_model_parallel = patch_destroy_model_parallel
|
||||
```
|
||||
|
||||
5. Import the patch file in `__init__.py`. In this example, add `import vllm_ascend.patch.platform.patch_distributed` into `vllm_ascend/patch/platform/__init__.py`.
|
||||
6. Add the description of the patch in `vllm_ascend/patch/__init__.py`. The description format is as follows:
|
||||
|
||||
```python
|
||||
# ** File: <The patch file name> **
|
||||
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
# 1. `<The target patch module in vLLM>`
|
||||
# Why:
|
||||
# <Describe the reason why we need to patch>
|
||||
# How:
|
||||
# <Describe the way to patch>
|
||||
# Related PR (if no, explain why):
|
||||
# <Add a link to the related PR in vLLM. If there is no related PR, explain why>
|
||||
# Future Plan:
|
||||
# <Describe the future plan to remove the patch>
|
||||
```
|
||||
|
||||
7. Add the Unit Test and E2E Test. Any newly added code in vLLM Ascend should contain the Unit Test and E2E Test as well. You can find more details in [test guide](../contribution/testing.md)
|
||||
|
||||
## Limitations
|
||||
|
||||
1. In V1 Engine, vLLM starts three kinds of processes: Main process, EngineCore process and Worker process. Now vLLM Ascend can only patch the code in Main process and Worker process by default. If you want to patch the code running in EngineCore process, you should patch EngineCore process entirely during setup. Find the entire code in `vllm.v1.engine.core`. Please override `EngineCoreProc` and `DPEngineCoreProc` entirely.
|
||||
2. If you are running edited vLLM code, the version of vLLM may be changed automatically. For example, if you run the edited vLLM based on v0.9.n, the version of vLLM may be changed to v0.9.nxxx. In this case, the patch for v0.9.n in vLLM Ascend would not work as expected, because vLLM Ascend can't distinguish the version of the vLLM you're using. In this case, you can set the environment variable `VLLM_VERSION` to specify the version of the vLLM you're using, and then the patch for that version (e.g., v0.9.n) should work.
|
||||
114
docs/source/developer_guide/Design_Documents/quantization.md
Normal file
114
docs/source/developer_guide/Design_Documents/quantization.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# Quantization Adaptation Guide
|
||||
|
||||
This document provides guidance for adapting quantization algorithms and models related to **ModelSlim**.
|
||||
|
||||
## Quantization Feature Introduction
|
||||
|
||||
### Quantization Inference Process
|
||||
|
||||
The current process for registering and obtaining quantization methods in vLLM Ascend is as follows:
|
||||
|
||||
|
||||
|
||||
vLLM Ascend registers a custom Ascend quantization method. By configuring the `--quantization ascend` parameter (or `quantization="ascend"` for offline), the quantization feature is enabled. When constructing the `quant_config`, the registered `AscendModelSlimConfig` is initialized and `get_quant_method` is called to obtain the quantization method corresponding to each weight part, stored in the `quant_method` attribute.
|
||||
|
||||
Currently supported quantization methods include `AscendLinearMethod`, `AscendFusedMoEMethod`, `AscendEmbeddingMethod`, and their corresponding non-quantized methods:
|
||||
|
||||
|
||||
|
||||
The quantization method base class defined by vLLM and the overall call flow of quantization methods are as follows:
|
||||
|
||||
|
||||
|
||||
The `embedding` method is generally not implemented for quantization, focusing only on the other three methods.
|
||||
|
||||
The `create_weights` method is used for weight initialization; the `process_weights_after_loading` method is used for weight post-processing, such as transposition, format conversion, data type conversion, etc.; the `apply` method is used to perform activation quantization and quantized matrix multiplication calculations during the forward process.
|
||||
|
||||
We need to implement the `create_weights`, `process_weights_after_loading`, and `apply` methods for different **layers** (**attention**, **mlp**, **MoE (Mixture of Experts)**).
|
||||
|
||||
**Supplement**: When loading the model, the quantized model's description file **quant_model_description.json** needs to be read. This file describes the quantization configuration and parameters for each part of the model weights, for example:
|
||||
|
||||
```json
|
||||
{
|
||||
"model.layers.0.linear_attn.dt_bias": "FLOAT",
|
||||
"model.layers.0.linear_attn.A_log": "FLOAT",
|
||||
"model.layers.0.linear_attn.conv1d.weight": "FLOAT",
|
||||
"model.layers.0.linear_attn.in_proj_qkvz.weight": "W8A8_DYNAMIC",
|
||||
"model.layers.0.linear_attn.in_proj_qkvz.weight_scale": "W8A8_DYNAMIC",
|
||||
"model.layers.0.linear_attn.in_proj_qkvz.weight_offset": "W8A8_DYNAMIC",
|
||||
"model.layers.0.linear_attn.in_proj_ba.weight": "FLOAT",
|
||||
"model.layers.0.linear_attn.norm.weight": "FLOAT",
|
||||
"model.layers.0.linear_attn.out_proj.weight": "FLOAT",
|
||||
"model.layers.0.mlp.gate.weight": "FLOAT",
|
||||
"model.layers.0.mlp.experts.0.gate_proj.weight": "W8A8_DYNAMIC",
|
||||
"model.layers.0.mlp.experts.0.gate_proj.weight_scale": "W8A8_DYNAMIC",
|
||||
"model.layers.0.mlp.experts.0.gate_proj.weight_offset": "W8A8_DYNAMIC"
|
||||
}
|
||||
```
|
||||
|
||||
Based on the above content, we present a brief description of the adaptation process for quantization algorithms and quantized models.
|
||||
|
||||
### Quantization Algorithm Adaptation
|
||||
|
||||
- **Step 1: Algorithm Design**. Define the algorithm ID (e.g., `W4A8_DYNAMIC`), determine supported layers (linear, moe, attention), and design the quantization scheme (static/dynamic, pertensor/perchannel/pergroup).
|
||||
- **Step 2: Registration**. Use the `@register_scheme` decorator in `vllm_ascend/quantization/methods/registry.py` to register your quantization scheme class.
|
||||
|
||||
```python
|
||||
from vllm_ascend.quantization.methods import register_scheme, AscendLinearScheme, AscendMoEScheme
|
||||
|
||||
@register_scheme("W4A8_DYNAMIC", "linear")
|
||||
class AscendW4A8DynamicLinearMethod(AscendLinearScheme):
|
||||
...
|
||||
|
||||
@register_scheme("W4A8_DYNAMIC", "moe")
|
||||
class AscendW4A8DynamicFusedMoEMethod(AscendMoEScheme):
|
||||
...
|
||||
```
|
||||
|
||||
- **Step 3: Implementation**. Create an algorithm implementation file, such as `vllm_ascend/quantization/methods/w4a8.py`, and implement the method class and logic.
|
||||
- **Step 4: Testing**. Use your algorithm to generate quantization configurations and verify correctness and performance on target models and hardware.
|
||||
|
||||
### Quantized Model Adaptation
|
||||
|
||||
Adapting a new quantized model requires ensuring the following three points:
|
||||
|
||||
- The original model has been successfully adapted in `vLLM Ascend`.
|
||||
- **Fused Module Mapping**: Add the model's `model_type` to `packed_modules_model_mapping` in `vllm_ascend/quantization/modelslim_config.py` (e.g., `qkv_proj`, `gate_up_proj`, `experts`) to ensure sharding consistency and correct loading.
|
||||
|
||||
```python
|
||||
packed_modules_model_mapping = {
|
||||
"qwen3_moe": {
|
||||
"qkv_proj": [
|
||||
"q_proj",
|
||||
"k_proj",
|
||||
"v_proj",
|
||||
],
|
||||
"gate_up_proj": [
|
||||
"gate_proj",
|
||||
"up_proj",
|
||||
],
|
||||
"experts":
|
||||
["experts.0.gate_proj", "experts.0.up_proj", "experts.0.down_proj"],
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
- All quantization algorithms used by the quantized model have been integrated into the `quantization` module.
|
||||
|
||||
## Currently Supported Quantization Algorithms
|
||||
|
||||
vLLM Ascend supports multiple quantization algorithms. The following table provides an overview of each quantization algorithm based on the implementation in the `vllm_ascend.quantization` module:
|
||||
|
||||
| Algorithm | Weight | Activation | Weight Granularity | Activation Granularity | Type | Description |
|
||||
| ------------------------ | ------ | ---------- | ------------------ | ---------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `W4A16` | INT4 | FP16/BF16 | Per-Group | Per-Tensor | Static | 4-bit weight quantization with 16-bit activation precision, specifically designed for MoE model expert layers, supporting int32 format weight packing |
|
||||
| `W8A16` | INT8 | FP16/BF16 | Per-Channel | Per-Tensor | Static | 8-bit weight quantization with 16-bit activation precision, balancing accuracy and performance, suitable for linear layers |
|
||||
| `W8A8` | INT8 | INT8 | Per-Channel | Per-Tensor | Static | Static activation quantization, suitable for scenarios requiring high precision |
|
||||
| `W8A8_DYNAMIC` | INT8 | INT8 | Per-Channel | Per-Token | Dynamic | Dynamic activation quantization with per-token scaling factor calculation |
|
||||
| `W4A8_DYNAMIC` | INT4 | INT8 | Per-Group | Per-Token | Dynamic | Supports both direct per-channel quantization to 4-bit and two-step quantization (per-channel to 8-bit then per-group to 4-bit) |
|
||||
| `W4A4_FLATQUANT_DYNAMIC` | INT4 | INT4 | Per-Channel | Per-Token | Dynamic | Uses FlatQuant for activation distribution smoothing before 4-bit dynamic quantization, with additional matrix multiplications for precision preservation |
|
||||
| `W8A8_MIX` | INT8 | INT8 | Per-Channel | Per-Tensor/Token | Mixed | We support two deployment modes: PD Colocation (dynamic quantization for both P and D) and PD Disaggregation (dynamic-quant P and static-quant D) |
|
||||
|
||||
**Static vs Dynamic:** Static quantization uses pre-computed scaling factors with better performance, while dynamic quantization computes scaling factors on-the-fly for each token/activation tensor with higher precision.
|
||||
|
||||
**Granularity:** Refers to the scope of scaling factor computation (e.g., per-tensor, per-channel, per-group).
|
||||
469
docs/source/developer_guide/contribution/doc_writing.md
Normal file
469
docs/source/developer_guide/contribution/doc_writing.md
Normal file
@@ -0,0 +1,469 @@
|
||||
# Doc writing guide
|
||||
|
||||
## Guide to Writing Model Tutorial Doc
|
||||
|
||||
`docs/source/_templates/Model-Deployment-Tutorial-Template.md` is a template for writing model deployment tutorials. You can copy and modify it to create new docs.
|
||||
|
||||
## Testable doc code block generation (``model-code``)
|
||||
|
||||
- For **documentation authors**: how to insert testable command blocks into docs
|
||||
- For **developers**: how to add a new converter
|
||||
|
||||
Built-in supported `converter_tag` values:
|
||||
|
||||
| converter_tag | Renders | YAML source |
|
||||
| --- | --- | --- |
|
||||
| `single_node` | A single node's env exports + `vllm serve` script | `test_cases[case_index]` |
|
||||
| `multi_node` | One host's env exports + `vllm serve` script | `deployment[host_index]` |
|
||||
| `external_dp_template` | One external-DP node's env exports + `vllm serve` command | `templates[host_index]` |
|
||||
| `external_dp_launch` | One `launch_online_dp.py` line per node | `config[]` |
|
||||
| `external_dp_proxy` | The load-balance proxy launch command | `config[]` + `routing` |
|
||||
|
||||
### For authors: add a block
|
||||
|
||||
:::{important}
|
||||
By default, the generator scans only `.md` files under `docs/source/tutorials/models/` and produces artifacts.
|
||||
If you put ``model-code`` blocks in other directories, Sphinx builds will not automatically generate the corresponding scripts.
|
||||
:::
|
||||
|
||||
All ``model-code`` blocks need:
|
||||
|
||||
| Option | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `block_name` | Yes | Block name; must be unique within the current document |
|
||||
| `converter_tag` | Yes | Selects one of the built-in converters |
|
||||
| `test_case_path` | Yes | Repository-relative YAML path that stays within the repo; the file must exist |
|
||||
|
||||
Use the body of the block to add shell wrapper lines such as `set -eux`. Always
|
||||
place the `{{ generated }}` placeholder where the converter output should be
|
||||
inserted.
|
||||
|
||||
#### converter_tag: `single_node`
|
||||
|
||||
`single_node` reads one item from `test_cases`. The optional `case_index`
|
||||
metadata selects the item; when omitted, it defaults to `0`.
|
||||
|
||||
Only the fields read by this converter are expanded below. Other test metadata
|
||||
can be left in the YAML and is ignored by this converter.
|
||||
|
||||
```yaml
|
||||
test_cases:
|
||||
- name: qwen3-8b-single
|
||||
model: Qwen/Qwen3-8B
|
||||
envs:
|
||||
HCCL_BUFFSIZE: "1024"
|
||||
SERVER_PORT: DEFAULT_PORT
|
||||
server_cmd:
|
||||
- --tensor-parallel-size
|
||||
- "1"
|
||||
- --port
|
||||
- $SERVER_PORT
|
||||
- --trust-remote-code
|
||||
server_cmd_extra:
|
||||
- --enable-expert-parallel
|
||||
benchmarks: ...
|
||||
```
|
||||
|
||||
`envs` is rendered as `export` lines. `SERVER_PORT: DEFAULT_PORT` is resolved
|
||||
to the default single-node port `8000`. `model` becomes `vllm serve <model>`,
|
||||
and `server_cmd` plus optional `server_cmd_extra` become command arguments.
|
||||
Both command fields can be either a shell string or a flat token list.
|
||||
|
||||
Write the doc block like this:
|
||||
|
||||
````md
|
||||
```{model-code}
|
||||
:block_name: qwen3_8b_single_node
|
||||
:converter_tag: single_node
|
||||
:test_case_path: tests/e2e/nightly/single_node/models/configs/your_model.yaml
|
||||
:case_index: 0
|
||||
|
||||
set -eux
|
||||
{{ generated }}
|
||||
```
|
||||
````
|
||||
|
||||
Generated shell script:
|
||||
|
||||
```bash
|
||||
set -eux
|
||||
export HCCL_BUFFSIZE=1024
|
||||
export SERVER_PORT=8000
|
||||
|
||||
vllm serve Qwen/Qwen3-8B \
|
||||
--tensor-parallel-size 1 \
|
||||
--port $SERVER_PORT \
|
||||
--trust-remote-code \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
#### converter_tag: `multi_node`
|
||||
|
||||
`multi_node` reads one item from `deployment`. The required `host_index`
|
||||
metadata selects which host to render.
|
||||
|
||||
```yaml
|
||||
deployment:
|
||||
- envs:
|
||||
SERVER_PORT: "8000"
|
||||
server_cmd: >
|
||||
vllm serve Qwen/Qwen3-235B-A22B
|
||||
--host 0.0.0.0
|
||||
--port $SERVER_PORT
|
||||
--data-parallel-size 2
|
||||
--tensor-parallel-size 8
|
||||
--data-parallel-address $LOCAL_IP
|
||||
- envs:
|
||||
SERVER_PORT: "8000"
|
||||
server_cmd: >
|
||||
vllm serve Qwen/Qwen3-235B-A22B
|
||||
--headless
|
||||
--port $SERVER_PORT
|
||||
--data-parallel-size 2
|
||||
--tensor-parallel-size 8
|
||||
--data-parallel-start-rank 1
|
||||
--data-parallel-address $MASTER_IP
|
||||
benchmarks: ...
|
||||
```
|
||||
|
||||
`server_cmd` must be a complete command starting with `vllm serve <model>`.
|
||||
It can be written as a shell string or a flat token list.
|
||||
|
||||
Write the doc block like this:
|
||||
|
||||
````md
|
||||
```{model-code}
|
||||
:block_name: qwen3_235b_worker_1
|
||||
:converter_tag: multi_node
|
||||
:test_case_path: tests/e2e/nightly/multi_node/internal_dp/config/your_model.yaml
|
||||
:host_index: 1
|
||||
|
||||
set -eux
|
||||
{{ generated }}
|
||||
```
|
||||
````
|
||||
|
||||
Generated shell script for `host_index: 1`:
|
||||
|
||||
```bash
|
||||
set -eux
|
||||
export MASTER_IP=192.168.1.10
|
||||
export SERVER_PORT=8000
|
||||
|
||||
vllm serve Qwen/Qwen3-235B-A22B \
|
||||
--headless \
|
||||
--port $SERVER_PORT \
|
||||
--data-parallel-size 2 \
|
||||
--tensor-parallel-size 8 \
|
||||
--data-parallel-start-rank 1 \
|
||||
--data-parallel-address $MASTER_IP
|
||||
```
|
||||
|
||||
#### converter_tag: `external_dp_template`
|
||||
|
||||
`external_dp_template` reads one item from `templates`. The required
|
||||
`host_index` metadata selects which template to render. The top-level `model`
|
||||
field is also required because the converter builds `vllm serve <model>`.
|
||||
|
||||
```yaml
|
||||
model: Eco-Tech/GLM-Test
|
||||
templates:
|
||||
- node_index: 0
|
||||
envs:
|
||||
HCCL_BUFFSIZE: "1024"
|
||||
ASCEND_RT_VISIBLE_DEVICES: "${VISIBLE_DEVICES}"
|
||||
server_cmd_template:
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- ${PORT}
|
||||
- --data-parallel-size
|
||||
- ${DP_SIZE}
|
||||
- --data-parallel-rank
|
||||
- ${DP_RANK}
|
||||
- --data-parallel-address
|
||||
- ${DP_ADDRESS}
|
||||
- --data-parallel-rpc-port
|
||||
- ${DP_RPC_PORT}
|
||||
- --tensor-parallel-size
|
||||
- ${TP_SIZE}
|
||||
- --trust-remote-code
|
||||
config: ...
|
||||
routing: ...
|
||||
```
|
||||
|
||||
Known braced template variables are rewritten to the positional shell arguments
|
||||
that `run_dp_template.sh` receives from `launch_online_dp.py`:
|
||||
|
||||
| Template variable | Rendered positional |
|
||||
| --- | --- |
|
||||
| `${VISIBLE_DEVICES}` | `$1` |
|
||||
| `${PORT}` | `$2` |
|
||||
| `${DP_SIZE}` | `$3` |
|
||||
| `${DP_RANK}` | `$4` |
|
||||
| `${DP_ADDRESS}` | `$5` |
|
||||
| `${DP_RPC_PORT}` | `$6` |
|
||||
| `${TP_SIZE}` | `$7` |
|
||||
|
||||
Unknown braced variables and unbraced shell references such as `$SERVER_PORT`
|
||||
are left unchanged.
|
||||
|
||||
Write the doc block like this:
|
||||
|
||||
````md
|
||||
```{model-code}
|
||||
:block_name: glm_external_dp_template_node0
|
||||
:converter_tag: external_dp_template
|
||||
:test_case_path: tests/e2e/nightly/multi_node/external_dp/config/your_model.yaml
|
||||
:host_index: 0
|
||||
|
||||
set -eux
|
||||
export HCCL_IF_IP=$local_ip
|
||||
export GLOO_SOCKET_IFNAME=$nic_name
|
||||
export TP_SOCKET_IFNAME=$nic_name
|
||||
export HCCL_SOCKET_IFNAME=$nic_name
|
||||
|
||||
{{ generated }}
|
||||
```
|
||||
````
|
||||
|
||||
Generated shell script for `host_index: 0`:
|
||||
|
||||
```bash
|
||||
set -eux
|
||||
export HCCL_IF_IP=$local_ip
|
||||
export GLOO_SOCKET_IFNAME=$nic_name
|
||||
export TP_SOCKET_IFNAME=$nic_name
|
||||
export HCCL_SOCKET_IFNAME=$nic_name
|
||||
|
||||
export HCCL_BUFFSIZE=1024
|
||||
export ASCEND_RT_VISIBLE_DEVICES=$1
|
||||
|
||||
vllm serve Eco-Tech/GLM-Test \
|
||||
--host 0.0.0.0 \
|
||||
--port $2 \
|
||||
--data-parallel-size $3 \
|
||||
--data-parallel-rank $4 \
|
||||
--data-parallel-address $5 \
|
||||
--data-parallel-rpc-port $6 \
|
||||
--tensor-parallel-size $7 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
#### converter_tag: `external_dp_launch`
|
||||
|
||||
`external_dp_launch` reads the full `config` list and renders one
|
||||
`launch_online_dp.py` command per node. It does not take an index option.
|
||||
|
||||
```yaml
|
||||
config:
|
||||
- node_index: 0
|
||||
port_start: 7100
|
||||
dp_rpc_port: 12321
|
||||
dp_size: 2
|
||||
dp_size_local: 2
|
||||
dp_rank_start: 0
|
||||
tp_size: 8
|
||||
dp_address: "${NODE_0_IP}"
|
||||
- node_index: 1
|
||||
port_start: 7200
|
||||
dp_rpc_port: 12321
|
||||
dp_size: 4
|
||||
dp_size_local: 4
|
||||
dp_rank_start: 0
|
||||
tp_size: 4
|
||||
dp_address: "${NODE_1_IP}"
|
||||
templates: ...
|
||||
routing: ...
|
||||
```
|
||||
|
||||
Write the doc block like this:
|
||||
|
||||
````md
|
||||
```{model-code}
|
||||
:block_name: glm_external_dp_launch
|
||||
:converter_tag: external_dp_launch
|
||||
:test_case_path: tests/e2e/nightly/multi_node/external_dp/config/your_model.yaml
|
||||
|
||||
set -eux
|
||||
{{ generated }}
|
||||
```
|
||||
````
|
||||
|
||||
Generated shell script:
|
||||
|
||||
```bash
|
||||
set -eux
|
||||
python launch_online_dp.py --dp-size 2 --tp-size 8 --dp-size-local 2 --dp-rank-start 0 --dp-address ${NODE_0_IP} --dp-rpc-port 12321 --vllm-start-port 7100
|
||||
|
||||
python launch_online_dp.py --dp-size 4 --tp-size 4 --dp-size-local 4 --dp-rank-start 0 --dp-address ${NODE_1_IP} --dp-rpc-port 12321 --vllm-start-port 7200
|
||||
```
|
||||
|
||||
#### converter_tag: `external_dp_proxy`
|
||||
|
||||
`external_dp_proxy` reads `config` and `routing`. It renders the
|
||||
`load_balance_proxy_server_example.py` command for `routing.type:
|
||||
disaggregated_prefill`. It does not take an index option.
|
||||
|
||||
```yaml
|
||||
routing:
|
||||
type: disaggregated_prefill
|
||||
groups:
|
||||
prefiller: [0]
|
||||
decoder: [1]
|
||||
config:
|
||||
- node_index: 0
|
||||
port_start: 7100
|
||||
dp_size_local: 2
|
||||
dp_rpc_port: 12321
|
||||
dp_size: 2
|
||||
dp_rank_start: 0
|
||||
tp_size: 8
|
||||
dp_address: "${NODE_0_IP}"
|
||||
- node_index: 1
|
||||
port_start: 7200
|
||||
dp_size_local: 4
|
||||
dp_rpc_port: 12321
|
||||
dp_size: 4
|
||||
dp_rank_start: 0
|
||||
tp_size: 4
|
||||
dp_address: "${NODE_1_IP}"
|
||||
templates: ...
|
||||
```
|
||||
|
||||
`routing.groups.prefiller` and `routing.groups.decoder` contain indices into
|
||||
`config`. Each referenced node expands to `dp_size_local` host and port entries.
|
||||
The proxy itself is rendered on `${NODE_0_IP}:1999`.
|
||||
|
||||
Write the doc block like this:
|
||||
|
||||
````md
|
||||
```{model-code}
|
||||
:block_name: glm_external_dp_proxy
|
||||
:converter_tag: external_dp_proxy
|
||||
:test_case_path: tests/e2e/nightly/multi_node/external_dp/config/your_model.yaml
|
||||
|
||||
set -eux
|
||||
{{ generated }}
|
||||
```
|
||||
````
|
||||
|
||||
Generated shell script:
|
||||
|
||||
```bash
|
||||
set -eux
|
||||
python load_balance_proxy_server_example.py \
|
||||
--host ${NODE_0_IP} \
|
||||
--port 1999 \
|
||||
--prefiller-hosts \
|
||||
${NODE_0_IP} \
|
||||
${NODE_0_IP} \
|
||||
--prefiller-ports \
|
||||
7100 \
|
||||
7101 \
|
||||
--decoder-hosts \
|
||||
${NODE_1_IP} \
|
||||
${NODE_1_IP} \
|
||||
${NODE_1_IP} \
|
||||
${NODE_1_IP} \
|
||||
--decoder-ports \
|
||||
7200 \
|
||||
7201 \
|
||||
7202 \
|
||||
7203
|
||||
```
|
||||
|
||||
### Local debugging and generation
|
||||
|
||||
#### Generate only (without building the full site)
|
||||
|
||||
```bash
|
||||
# Generate all model-code artifacts under docs/source/tutorials/models/
|
||||
python3 tools/docs_codegen/cli.py
|
||||
|
||||
# Generate artifacts for a single document
|
||||
python3 tools/docs_codegen/cli.py --doc docs/source/tutorials/models/Kimi-K2-Thinking.md
|
||||
|
||||
# Generate a single block and print it (no files written)
|
||||
python3 tools/docs_codegen/cli.py \
|
||||
--block docs/source/tutorials/models/Kimi-K2-Thinking.md::kimi_k2_thinking_single_node \
|
||||
--dry-run --stdout
|
||||
```
|
||||
|
||||
By default, artifacts are written to: `docs/_build/doc_codegen/<doc_stem>/<block_name>.sh`.
|
||||
|
||||
:::{note}
|
||||
After the script is generated, please make sure to check whether the generated content is runnable, especially key parts such as environment variables and command-line parameters.
|
||||
:::
|
||||
|
||||
#### Build the site & preview locally
|
||||
|
||||
```bash
|
||||
# Install documentation build dependencies
|
||||
python3 -m pip install -r docs/requirements-docs.txt
|
||||
|
||||
# (Optional) Clean previous builds
|
||||
make -C docs clean
|
||||
|
||||
# Build the English site
|
||||
make -C docs html
|
||||
|
||||
# (Optional) Build the Chinese site
|
||||
make -C docs intl
|
||||
|
||||
# Preview locally
|
||||
python3 -m http.server -d docs/_build/html 8000
|
||||
|
||||
# Then open in a browser:
|
||||
# http://localhost:8000
|
||||
```
|
||||
|
||||
### For developers: add a new converter
|
||||
|
||||
A converter turns one loaded YAML file plus one parsed `ModelCodeBlock` into a
|
||||
`GeneratedScript`. The current pipeline is:
|
||||
|
||||
1. `BlockScanner` parses ``model-code`` fences and accepts only options listed
|
||||
in `MODEL_CODE_OPTION_NAMES`.
|
||||
2. `YamlLoader` loads `test_case_path`.
|
||||
3. `get_converter()` looks up `block.converter_tag` from
|
||||
`build_default_converters()`.
|
||||
4. The selected converter returns `GeneratedScript(content=..., language="shell")`.
|
||||
5. `GeneratorService` replaces `{{ generated }}` in the block body, validates
|
||||
that the final script is non-empty, and writes
|
||||
`docs/_build/doc_codegen/<doc_stem>/<block_name>.sh`.
|
||||
|
||||
To add a converter:
|
||||
|
||||
1. In `tools/docs_codegen/converters.py`, add a `BaseConverter` subclass with a
|
||||
unique `name`. That name is the value authors put in `:converter_tag:`.
|
||||
2. Implement `convert(self, loaded_yaml, *, block) -> GeneratedScript`. Use
|
||||
`make_docs_codegen_error(..., block=block)` for user-facing validation
|
||||
errors so the CLI and Sphinx output include document context.
|
||||
3. Reuse helpers from `tools/docs_codegen/utils.py`, such as
|
||||
`require_mapping`, `require_mapping_list`, `require_scalar_mapping`,
|
||||
`require_indexed_mapping`, `require_node_field`, `parse_command_tokens`,
|
||||
`substitute_template_positionals`, and `render_cli_command`.
|
||||
4. Register the converter in `build_default_converters()`. If it is not
|
||||
registered, `get_converter()` will reject the new `converter_tag`.
|
||||
5. If the converter needs new directive metadata, add the option name to
|
||||
`MODEL_CODE_OPTION_NAMES` in `tools/docs_codegen/scanner.py` and to
|
||||
`ModelCodeDirective.option_spec` in
|
||||
`tools/docs_codegen/sphinx_extension.py`. Read the option with
|
||||
`block.get_option("<option_name>")`.
|
||||
6. Add or update tests in `tests/ut/tools/test_docs_codegen.py`. Cover the
|
||||
successful render path, required option validation, YAML shape validation,
|
||||
and any CLI/Sphinx scanner behavior affected by new metadata.
|
||||
7. Add a real ``model-code`` example in a model tutorial, preferably under
|
||||
`docs/source/tutorials/models/`, and point it to an existing YAML file under
|
||||
`tests/`.
|
||||
8. Validate with the CLI:
|
||||
|
||||
```bash
|
||||
python3 tools/docs_codegen/cli.py --doc <your_doc> --dry-run
|
||||
python3 tools/docs_codegen/cli.py --block <your_doc>::<block_name> --dry-run --stdout
|
||||
```
|
||||
|
||||
If a converter should render something other than shell, set
|
||||
`GeneratedScript.language` accordingly so Sphinx can highlight the generated
|
||||
literal block correctly.
|
||||
154
docs/source/developer_guide/contribution/e2e_ci_test.md
Normal file
154
docs/source/developer_guide/contribution/e2e_ci_test.md
Normal file
@@ -0,0 +1,154 @@
|
||||
# E2E CI Test
|
||||
|
||||
This document explains how to trigger specific E2E tests against your PR code via a
|
||||
comment command, without running the full E2E test suite.
|
||||
|
||||
## Background
|
||||
|
||||
The `E2E-Full` workflow ([`pr_test.yaml`](https://github.com/vllm-project/vllm-ascend/blob/main/.github/workflows/pr_test.yaml)) normally runs the complete E2E test suite
|
||||
when a PR has `ready` label. This is expensive in CI resources
|
||||
and time.
|
||||
|
||||
Authorized users can trigger only the specific test files they care about by posting a
|
||||
`/e2e` comment on the PR, then adding the `ready` label.
|
||||
|
||||
## How to Trigger
|
||||
|
||||
### 1. Post a comment
|
||||
|
||||
First, post a comment on the PR specifying which test paths to run:
|
||||
|
||||
```text
|
||||
/e2e [test-path-1] [test-path-2] ...
|
||||
```
|
||||
|
||||
- Each path must be a valid pytest path relative to the repository root.
|
||||
- Multiple paths can be listed in a single comment, separated by spaces.
|
||||
- A specific test case can be targeted using `::` notation.
|
||||
|
||||
| Comment format | Effect |
|
||||
|---|---|
|
||||
| `/e2e tests/e2e/pull_request/one_card/test_foo.py` | Run one test file on one_card |
|
||||
| `/e2e tests/e2e/pull_request/two_card/test_bar.py` | Run one test file on two_card |
|
||||
| `/e2e path1 path2 path3` | Run multiple files, routed by path pattern |
|
||||
| `/e2e tests/e2e/pull_request/one_card/test_foo.py::test_case` | Run a specific test case |
|
||||
|
||||
### 2. Add the label
|
||||
|
||||
After posting the comment, add the **`ready`** label to your PR.
|
||||
Adding the label is what actually **triggers** the workflow — at that point the workflow
|
||||
reads the existing comments to find the `/e2e` command.
|
||||
|
||||
:::{note}
|
||||
Only repository **Contributors** (Triage role) and **Maintainers** (Write role) can add
|
||||
labels. If you do not have this permission, ask a maintainer to add the label for you.
|
||||
You can find the list of maintainers and contributors by checking the
|
||||
[CODEOWNERS](https://github.com/vllm-project/vllm-ascend/blob/main/.github/CODEOWNERS)
|
||||
file.
|
||||
:::
|
||||
|
||||
:::{important}
|
||||
The comment must be posted **before** the label is added. If you add the label first,
|
||||
the workflow will find no `/e2e` comment and will not trigger any per-test runs.
|
||||
:::
|
||||
|
||||
:::{note}
|
||||
Additionally, only the **PR author** or collaborators with **write or admin** repository
|
||||
access can trigger tests via comment. The workflow validates the commenter's permission
|
||||
before proceeding.
|
||||
:::
|
||||
|
||||
### 3. Wait for results
|
||||
|
||||
GitHub Actions will trigger the `E2E-Full` workflow. Only the hardware jobs matching
|
||||
the provided test paths will run, which saves CI resources.
|
||||
|
||||
## Path Routing Rules
|
||||
|
||||
The workflow automatically routes each test path to the correct hardware runner based
|
||||
on path patterns:
|
||||
|
||||
| Path pattern | Hardware | Runner |
|
||||
|---|---|---|
|
||||
| `two_card` in path | two_card A3 NPU | `linux-aarch64-a3-2` |
|
||||
| `four_card` in path | four_card A3 NPU | `linux-aarch64-a3-4` |
|
||||
| `_310p` in filename under one/two_card | Ascend 310P x1 | `linux-aarch64-310p-*` |
|
||||
| `_310p` in filename under four_card | Ascend 310P x4 | `linux-aarch64-310p-*` |
|
||||
| All other paths | one_card A2 NPU | `linux-aarch64-a2b3-1` |
|
||||
|
||||
When paths from multiple categories are listed in a single comment, each category's
|
||||
tests run on its respective hardware in parallel.
|
||||
|
||||
## Test Path Reference
|
||||
|
||||
The `tests/e2e/pull_request/` directory is organized by hardware category:
|
||||
|
||||
```text
|
||||
tests/e2e/pull_request/
|
||||
├── one_card/ # Single card tests → A2 NPU x1 runner
|
||||
├── two_card/ # Two card tests → A3 NPU x2 runner
|
||||
├── four_card/ # Four card tests → A3 NPU x4 runner
|
||||
```
|
||||
|
||||
310P tests use `_310p` subdirectories or `_310p.py` filename suffix under the
|
||||
corresponding card directory:
|
||||
|
||||
```text
|
||||
tests/e2e/pull_request/one_card/_310p/ # 310P single card
|
||||
tests/e2e/pull_request/four_card/_310p/ # 310P four card
|
||||
```
|
||||
|
||||
## Comparison with Full E2E Suite
|
||||
|
||||
| Aspect | Full E2E suite | Per-test comment trigger |
|
||||
|---|---|---|
|
||||
| Trigger | `ready` labels | `/e2e` comment + `ready` label |
|
||||
| Scope | All E2E tests | Only specified test paths |
|
||||
| Who can trigger | Anyone who can add labels | PR author or write/admin collaborator |
|
||||
| Use case | Pre-merge validation | Iterative debugging of specific tests |
|
||||
|
||||
## Examples
|
||||
|
||||
Run a single one_card test:
|
||||
|
||||
```text
|
||||
/e2e tests/e2e/pull_request/one_card/test_offline_inference.py
|
||||
```
|
||||
|
||||
Run a two_card test:
|
||||
|
||||
```text
|
||||
/e2e tests/e2e/pull_request/two_card/test_data_parallel.py
|
||||
```
|
||||
|
||||
Run tests across multiple hardware categories in one comment:
|
||||
|
||||
```text
|
||||
/e2e tests/e2e/pull_request/one_card/test_offline_inference.py tests/e2e/pull_request/two_card/test_data_parallel.py
|
||||
```
|
||||
|
||||
Re-trigger after fixing an issue: just push a new commit. The `synchronize` event
|
||||
re-runs the workflow and picks up the existing `/e2e` comment automatically — no need
|
||||
to post a new comment.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**The workflow did not start after I added the label.**
|
||||
|
||||
- Make sure the `/e2e` comment was posted **before** the label was added.
|
||||
If the label was added first, remove it and re-add it after posting the comment.
|
||||
- Check that the comment starts exactly with `/e2e` followed by at least one path,
|
||||
with no leading spaces or extra characters before the slash.
|
||||
- To re-trigger after fixing an issue, simply push a new commit — the workflow will
|
||||
reuse the existing `/e2e` comment automatically.
|
||||
|
||||
**Tests ran on the wrong hardware.**
|
||||
|
||||
- Check that the path includes the expected directory segment (`one_card`, `two_card`,
|
||||
`four_card`, or `_310p`). Paths that do not match any of these patterns are routed to
|
||||
the one_card runner by default.
|
||||
|
||||
**The `parse-comment` job skipped with a permission error.**
|
||||
|
||||
- Only the PR author or write/admin collaborators can use the comment trigger.
|
||||
Ask a maintainer to post the `/e2e` comment instead.
|
||||
@@ -1,16 +1,17 @@
|
||||
# Contributing
|
||||
|
||||
## Building and testing
|
||||
It's recommended to set up a local development environment to build and test
|
||||
## Building and Testing
|
||||
|
||||
It's recommended to set up a local development environment to build vllm-ascend and run tests
|
||||
before you submit a PR.
|
||||
|
||||
### Setup development environment
|
||||
### Set up a development environment
|
||||
|
||||
Theoretically, the vllm-ascend build is only supported on Linux because
|
||||
`vllm-ascend` dependency `torch_npu` only supports Linux.
|
||||
|
||||
But you can still set up dev env on Linux/Windows/macOS for linting and basic
|
||||
test as following commands:
|
||||
But you can still set up a development environment on Linux/Windows/macOS for linting and running basic
|
||||
tests.
|
||||
|
||||
#### Run lint locally
|
||||
|
||||
@@ -27,20 +28,19 @@ cd vllm-ascend
|
||||
# Install lint requirement and enable pre-commit hook
|
||||
pip install -r requirements-lint.txt
|
||||
|
||||
# Run lint (You need install pre-commits deps via proxy network at first time)
|
||||
# Run lint (You need to install pre-commits deps via proxy network at first time)
|
||||
bash format.sh
|
||||
```
|
||||
|
||||
#### Run CI locally
|
||||
|
||||
After complete "Run lint" setup, you can run CI locally:
|
||||
After completing "Run lint" setup, you can run CI (Continuous integration) locally:
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
|
||||
cd ~/vllm-project/
|
||||
|
||||
# Run CI need vLLM installed
|
||||
# Run CI needs vLLM installed
|
||||
git clone --branch |vllm_version| https://github.com/vllm-project/vllm.git
|
||||
cd vllm
|
||||
pip install -r requirements/build.txt
|
||||
@@ -51,7 +51,7 @@ cd ..
|
||||
cd vllm-ascend
|
||||
# For Linux:
|
||||
pip install -r requirements-dev.txt
|
||||
# For non Linux:
|
||||
# For non-Linux:
|
||||
cat requirements-dev.txt | grep -Ev '^#|^--|^$|^-r' | while read PACKAGE; do pip install "$PACKAGE"; done
|
||||
cat requirements.txt | grep -Ev '^#|^--|^$|^-r' | while read PACKAGE; do pip install "$PACKAGE"; done
|
||||
|
||||
@@ -68,13 +68,13 @@ git commit -sm "your commit info"
|
||||
|
||||
🎉 Congratulations! You have completed the development environment setup.
|
||||
|
||||
### Test locally
|
||||
### Testing locally
|
||||
|
||||
You can refer to [Testing](./testing.md) doc to help you setup testing environment and running tests locally.
|
||||
You can refer to [Testing](./testing.md) to set up a testing environment and running tests locally.
|
||||
|
||||
## DCO and Signed-off-by
|
||||
|
||||
When contributing changes to this project, you must agree to the DCO. Commits must include a `Signed-off-by:` header which certifies agreement with the terms of the DCO.
|
||||
When contributing changes to this project, you must agree to the DCO. Commits must include a `Signed-off-by:` header which certifies agreement with the terms of the DCO (Developer Certificate of Origin).
|
||||
|
||||
Using `-s` with `git commit` will automatically add this header.
|
||||
|
||||
@@ -88,8 +88,8 @@ Only specific types of PRs will be reviewed. The PR title is prefixed appropriat
|
||||
- `[Platform]` for new features or optimization in platform.
|
||||
- `[Worker]` for new features or optimization in worker.
|
||||
- `[Core]` for new features or optimization in the core vllm-ascend logic (such as platform, attention, communicators, model runner)
|
||||
- `[Kernel]` changes affecting compute kernels and ops.
|
||||
- `[Bugfix]` for bug fixes.
|
||||
- `[Kernel]` for changes affecting compute kernels and ops.
|
||||
- `[BugFix]` for bug fixes.
|
||||
- `[Doc]` for documentation fixes and improvements.
|
||||
- `[Test]` for tests (such as unit tests).
|
||||
- `[CI]` for build or continuous integration improvements.
|
||||
@@ -101,11 +101,15 @@ If the PR spans more than one category, please include all relevant prefixes.
|
||||
|
||||
## Others
|
||||
|
||||
You may find more information about contributing to vLLM Ascend backend plugin on [<u>docs.vllm.ai</u>](https://docs.vllm.ai/en/latest/contributing/overview.html).
|
||||
If you find any problem when contributing, you can feel free to submit a PR to improve the doc to help other developers.
|
||||
You may find more information about contributing to vLLM Ascend backend plugin on [<u>docs.vllm.ai</u>](https://docs.vllm.ai/en/latest/contributing).
|
||||
If you encounter any problems while contributing, feel free to submit a PR to improve the documentation to help other developers.
|
||||
|
||||
:::{toctree}
|
||||
:caption: Index
|
||||
:maxdepth: 1
|
||||
testing
|
||||
doc_writing
|
||||
multi_node_test
|
||||
nightly_ci_test
|
||||
e2e_ci_test
|
||||
:::
|
||||
|
||||
553
docs/source/developer_guide/contribution/multi_node_test.md
Normal file
553
docs/source/developer_guide/contribution/multi_node_test.md
Normal file
@@ -0,0 +1,553 @@
|
||||
# Multi Node Test
|
||||
|
||||
Multi-Node CI is designed to test distributed scenarios of very large models, for example, disaggregated_prefill multi DP across multi nodes and so on.
|
||||
|
||||
## How it works
|
||||
|
||||
The following picture shows the basic deployment view of the multi-node CI mechanism. It shows how the GitHub action interacts with [lws](https://lws.sigs.k8s.io/docs/overview/) (a kind of kubernetes crd resource).
|
||||
|
||||
|
||||
|
||||
From the workflow perspective, we can see how the final test script is executed. The key point is that the shared files `tests/e2e/nightly/multi_node/scripts/lws.yaml.jinja2` and `tests/e2e/nightly/multi_node/scripts/run.sh` define the cluster template and pod entry script. Each node executes different logic according to the [LWS_WORKER_INDEX](https://lws.sigs.k8s.io/docs/reference/labels-annotations-and-environment-variables/) environment variable, so that multiple nodes can form a distributed cluster to perform tasks. `run.sh` selects the pytest entrypoint from the config path: internal DP configs use `internal_dp/scripts/test_multi_node.py`, while external DP configs use `external_dp/scripts/test_external_dp.py`.
|
||||
|
||||
|
||||
|
||||
## How to contribute
|
||||
|
||||
1. Upload custom weights
|
||||
|
||||
If you need customized weights, for example, you quantized a w8a8 weight for DeepSeek-V3 and you want your weight to run on CI, uploading weights to ModelScope's [vllm-ascend](https://www.modelscope.cn/organization/vllm-ascend) organization is welcome. If you do not have permission to upload, please contact @Potabk
|
||||
|
||||
2. Add config yaml
|
||||
|
||||
For the normal internal DP multi-node flow, add the config yaml to `tests/e2e/nightly/multi_node/internal_dp/config/`, like `DeepSeek-V3.yaml`. External DP cases use the separate `tests/e2e/nightly/multi_node/external_dp/config/` directory and should pass that directory through `config_base_path` in workflow or `CONFIG_BASE_PATH` locally.
|
||||
|
||||
Suppose you have **2 nodes** running a 1P1D setup (1 Prefillers + 1 Decoder):
|
||||
|
||||
you may add a config file looks like:
|
||||
|
||||
```yaml
|
||||
test_name: "test DeepSeek-V3 disaggregated_prefill"
|
||||
# the model being tested
|
||||
model: "vllm-ascend/DeepSeek-V3-W8A8"
|
||||
# how large the cluster is
|
||||
num_nodes: 2
|
||||
npu_per_node: 16
|
||||
# All env vars you need should add it here
|
||||
env_common: &env_common
|
||||
VLLM_USE_MODELSCOPE: true
|
||||
OMP_PROC_BIND: false
|
||||
OMP_NUM_THREADS: 100
|
||||
HCCL_BUFFSIZE: 1024
|
||||
SERVER_PORT: 8080
|
||||
disaggregated_prefill:
|
||||
enabled: true
|
||||
# node index(a list) which meet all the conditions:
|
||||
# - prefiller
|
||||
# - no headless(have api server)
|
||||
prefiller_host_index: [0]
|
||||
# node index(a list) which meet all the conditions:
|
||||
# - decoder
|
||||
decoder_host_index: [1]
|
||||
|
||||
# Add each node's vllm serve cli command just like you run locally
|
||||
# Add each node's individual envs like follow
|
||||
deployment:
|
||||
- name: prefiller node # optional: just for description, not used in code
|
||||
envs:
|
||||
<<: *env_common
|
||||
VLLM_ASCEND_ENABLE_FLASHCOMM1: 1
|
||||
# Continue to add other envs if needed
|
||||
server_cmd: >
|
||||
vllm serve ...
|
||||
- name: decoder node # optional: just for description, not used in code
|
||||
envs:
|
||||
<<: *env_common
|
||||
VLLM_ASCEND_ENABLE_FLASHCOMM1: 1
|
||||
# Continue to add other envs if needed
|
||||
server_cmd: >
|
||||
vllm serve ...
|
||||
benchmarks:
|
||||
perf:
|
||||
# fill with performance test kwargs
|
||||
acc:
|
||||
# fill with accuracy test kwargs
|
||||
```
|
||||
|
||||
3. Add the case to nightly workflow
|
||||
|
||||
Currently, the multi-node test workflow is defined in `.github/workflows/schedule_nightly_test_a3.yaml`.
|
||||
|
||||
```yaml
|
||||
multi-node-tests:
|
||||
name: multi-node
|
||||
if: always() && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 1
|
||||
matrix:
|
||||
test_config:
|
||||
- name: multi-node-deepseek-pd
|
||||
config_file_path: DeepSeek-V3.yaml
|
||||
size: 2
|
||||
- name: multi-node-qwen3-dp
|
||||
config_file_path: Qwen3-235B-A22B.yaml
|
||||
size: 2
|
||||
- name: GLM5_1-W8A8-EP-external
|
||||
config_file_path: GLM5_1-W8A8-EP-external.yaml
|
||||
config_base_path: tests/e2e/nightly/multi_node/external_dp/config/
|
||||
size: 4
|
||||
uses: ./.github/workflows/_e2e_nightly_multi_node.yaml
|
||||
with:
|
||||
soc_version: a3
|
||||
runner: linux-aarch64-a3-0
|
||||
image: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/vllm-ascend:nightly-a3'
|
||||
replicas: 1
|
||||
size: ${{ matrix.test_config.size }}
|
||||
config_file_path: ${{ matrix.test_config.config_file_path }}
|
||||
config_base_path: ${{ matrix.test_config.config_base_path || '' }}
|
||||
name: ${{ matrix.test_config.name }}
|
||||
secrets:
|
||||
KUBECONFIG_B64: ${{ secrets.KUBECONFIG_B64 }}
|
||||
```
|
||||
|
||||
The matrix above defines all the parameters required to add a multi-machine use
|
||||
case. The parameters worth noting are `size`, `config_file_path`, and
|
||||
`config_base_path`. `size` defines the number of nodes required for your use
|
||||
case. `config_file_path` is the yaml file name, and `config_base_path` tells the
|
||||
loader which config directory to use. For internal DP cases, use an empty
|
||||
`config_base_path` so the loader uses its default internal DP config directory.
|
||||
For external DP cases, set it to
|
||||
`tests/e2e/nightly/multi_node/external_dp/config/`.
|
||||
|
||||
## Run Multi-Node tests locally
|
||||
|
||||
### 1. Use kubernetes
|
||||
|
||||
This section assumes that you already have a [Kubernetes](https://kubernetes.io/docs/setup/) NPU cluster environment locally. Then you can easily start our test with one click.
|
||||
|
||||
- Step 1. Install LWS CRD resources
|
||||
|
||||
See <https://lws.sigs.k8s.io/docs/installation/> Which can be used as a reference
|
||||
|
||||
- Step 2. Deploy the following yaml file `lws.yaml` as needed
|
||||
|
||||
```yaml
|
||||
apiVersion: leaderworkerset.x-k8s.io/v1
|
||||
kind: LeaderWorkerSet
|
||||
metadata:
|
||||
name: test-server
|
||||
namespace: vllm-project
|
||||
spec:
|
||||
replicas: 1
|
||||
leaderWorkerTemplate:
|
||||
size: 2
|
||||
restartPolicy: None
|
||||
leaderTemplate:
|
||||
metadata:
|
||||
labels:
|
||||
role: leader
|
||||
spec:
|
||||
containers:
|
||||
- name: vllm-leader
|
||||
imagePullPolicy: Always
|
||||
image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/vllm-ascend:nightly-a3
|
||||
env:
|
||||
- name: CONFIG_YAML_PATH
|
||||
value: DeepSeek-V3.yaml
|
||||
- name: CONFIG_BASE_PATH
|
||||
value: tests/e2e/nightly/multi_node/internal_dp/config/
|
||||
- name: WORKSPACE
|
||||
value: "/vllm-workspace"
|
||||
- name: FAIL_TAG
|
||||
value: FAIL_TAG
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
bash /vllm-workspace/vllm-ascend/tests/e2e/nightly/multi_node/scripts/run.sh
|
||||
resources:
|
||||
limits:
|
||||
huawei.com/ascend-1980: 16
|
||||
memory: 512Gi
|
||||
ephemeral-storage: 100Gi
|
||||
requests:
|
||||
huawei.com/ascend-1980: 16
|
||||
memory: 512Gi
|
||||
ephemeral-storage: 100Gi
|
||||
cpu: 125
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
# readinessProbe:
|
||||
# tcpSocket:
|
||||
# port: 8080
|
||||
# initialDelaySeconds: 15
|
||||
# periodSeconds: 10
|
||||
volumeMounts:
|
||||
- mountPath: /root/.cache
|
||||
name: shared-volume
|
||||
- mountPath: /usr/local/Ascend/driver/tools
|
||||
name: driver-tools
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
volumes:
|
||||
- name: dshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 15Gi
|
||||
- name: shared-volume
|
||||
persistentVolumeClaim:
|
||||
claimName: nv-action-vllm-benchmarks-v2
|
||||
- name: driver-tools
|
||||
hostPath:
|
||||
path: /usr/local/Ascend/driver/tools
|
||||
workerTemplate:
|
||||
spec:
|
||||
containers:
|
||||
- name: vllm-worker
|
||||
imagePullPolicy: Always
|
||||
image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/vllm-ascend:nightly-a3
|
||||
env:
|
||||
- name: CONFIG_YAML_PATH
|
||||
value: DeepSeek-V3.yaml
|
||||
- name: CONFIG_BASE_PATH
|
||||
value: tests/e2e/nightly/multi_node/internal_dp/config/
|
||||
- name: WORKSPACE
|
||||
value: "/vllm-workspace"
|
||||
- name: FAIL_TAG
|
||||
value: FAIL_TAG
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
bash /vllm-workspace/vllm-ascend/tests/e2e/nightly/multi_node/scripts/run.sh
|
||||
resources:
|
||||
limits:
|
||||
huawei.com/ascend-1980: 16
|
||||
memory: 512Gi
|
||||
ephemeral-storage: 100Gi
|
||||
requests:
|
||||
huawei.com/ascend-1980: 16
|
||||
ephemeral-storage: 100Gi
|
||||
cpu: 125
|
||||
volumeMounts:
|
||||
- mountPath: /root/.cache
|
||||
name: shared-volume
|
||||
- mountPath: /usr/local/Ascend/driver/tools
|
||||
name: driver-tools
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
volumes:
|
||||
- name: dshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 15Gi
|
||||
- name: shared-volume
|
||||
persistentVolumeClaim:
|
||||
claimName: nv-action-vllm-benchmarks-v2
|
||||
- name: driver-tools
|
||||
hostPath:
|
||||
path: /usr/local/Ascend/driver/tools
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: vllm-leader
|
||||
namespace: vllm-project
|
||||
spec:
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
protocol: TCP
|
||||
targetPort: 8080
|
||||
selector:
|
||||
leaderworkerset.sigs.k8s.io/name: vllm
|
||||
role: leader
|
||||
type: ClusterIP
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl apply -f lws.yaml
|
||||
```
|
||||
|
||||
Verify the status of the pods:
|
||||
|
||||
```bash
|
||||
kubectl get pods -n vllm-project
|
||||
```
|
||||
|
||||
Should get an output similar to this:
|
||||
|
||||
```bash
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
vllm-0 1/1 Running 0 2s
|
||||
vllm-0-1 1/1 Running 0 2s
|
||||
```
|
||||
|
||||
Verify that the distributed inference works:
|
||||
|
||||
```bash
|
||||
kubectl logs -f vllm-0 -n vllm-project
|
||||
```
|
||||
|
||||
Should get something similar to this:
|
||||
|
||||
```shell
|
||||
INFO 12-30 11:00:57 [__init__.py:43] Available plugins for group vllm.platform_plugins:
|
||||
INFO 12-30 11:00:57 [__init__.py:45] - ascend -> vllm_ascend:register
|
||||
INFO 12-30 11:00:57 [__init__.py:48] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
|
||||
INFO 12-30 11:00:57 [__init__.py:217] Platform plugin ascend is activated
|
||||
INFO 12-30 11:00:57 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
================================================================================================== test session starts ===================================================================================================
|
||||
platform linux -- Python 3.12.13, pytest-8.4.2, pluggy-1.6.0 -- /usr/local/python3.12.13/bin/python3
|
||||
cachedir: .pytest_cache
|
||||
rootdir: /vllm-workspace/vllm-ascend
|
||||
configfile: pyproject.toml
|
||||
plugins: cov-7.0.0, asyncio-1.3.0, mock-3.15.1, anyio-4.12.0
|
||||
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
|
||||
collected 1 item
|
||||
|
||||
tests/e2e/nightly/multi_node/internal_dp/scripts/test_multi_node.py::test_multi_node [2025-12-30 11:01:01] INFO multi_node_config.py:294: Loading config yaml: tests/e2e/nightly/multi_node/internal_dp/config/DeepSeek-V3.yaml
|
||||
[2025-12-30 11:01:01] INFO multi_node_config.py:348: Resolving cluster IPs via DNS...
|
||||
[2025-12-30 11:01:01] INFO multi_node_config.py:212: Node 0 envs: {'VLLM_USE_MODELSCOPE': 'True', 'OMP_PROC_BIND': 'False', 'OMP_NUM_THREADS': '100', 'HCCL_BUFFSIZE': '1024', 'SERVER_PORT': '8080', 'NUMEXPR_MAX_THREADS': '128', 'DISAGGREGATED_PREFILL_PROXY_SCRIPT': 'examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py', 'HCCL_IF_IP': '10.0.0.102', 'HCCL_SOCKET_IFNAME': 'eth0', 'GLOO_SOCKET_IFNAME': 'eth0', 'TP_SOCKET_IFNAME': 'eth0', 'LOCAL_IP': '10.0.0.102', 'NIC_NAME': 'eth0', 'MASTER_IP': '10.0.0.102'}
|
||||
[2025-12-30 11:01:01] INFO multi_node_config.py:159: Launching proxy: python examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py --host 10.0.0.102 --port 6000 --prefiller-hosts 10.0.0.102 --prefiller-ports 8080 --decoder-hosts 10.0.0.138 --decoder-ports 8080
|
||||
[2025-12-30 11:01:01] INFO conftest.py:107: Starting server with command: vllm serve vllm-ascend/DeepSeek-V3-W8A8 --host 0.0.0.0 --port 8080 --data-parallel-size 2 --data-parallel-size-local 2 --tensor-parallel-size 8 --seed 1024 --enforce-eager --enable-expert-parallel --max-num-seqs 16 --max-model-len 8192 --max-num-batched-tokens 8192 --quantization ascend --trust-remote-code --no-enable-prefix-caching --gpu-memory-utilization 0.9 --kv-transfer-config {"kv_connector": "MooncakeConnectorV1", "kv_role": "kv_producer", "kv_port": "30000",
|
||||
"kv_connector_extra_config": {
|
||||
"prefill": {
|
||||
"dp_size": 2,
|
||||
"tp_size": 8
|
||||
},
|
||||
"decode": {
|
||||
"dp_size": 2,
|
||||
"tp_size": 8
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Test without Kubernetes
|
||||
|
||||
The same `tests/e2e/nightly/multi_node/scripts/run.sh` entrypoint can be used
|
||||
on prepared bare-metal or container hosts. Without LWS, set the values that
|
||||
Kubernetes normally injects yourself:
|
||||
|
||||
- `cluster_hosts` in the config yaml, using IPs reachable from every node.
|
||||
- `LWS_WORKER_INDEX` on each node, starting from `0`.
|
||||
- `CONFIG_YAML_PATH` as the config file name and `CONFIG_BASE_PATH` as the
|
||||
config directory.
|
||||
|
||||
Use the host NIC IPs that can reach each other, for example addresses shown by
|
||||
`ip addr` or `ifconfig` on the active network interface. Do not use per-host
|
||||
Docker bridge addresses such as `172.17.0.1`, because each host has its own
|
||||
local bridge.
|
||||
|
||||
Local `cluster_hosts` edits should be removed before submitting a PR unless the
|
||||
hosts are part of a committed test environment.
|
||||
|
||||
#### 2.1 Internal DP local run
|
||||
|
||||
##### 2.1.1 Add cluster hosts
|
||||
|
||||
Edit the internal DP config you want to run, for example:
|
||||
|
||||
```text
|
||||
tests/e2e/nightly/multi_node/internal_dp/config/DeepSeek-V3.yaml
|
||||
```
|
||||
|
||||
Add `cluster_hosts` as a top-level field, for example near `num_nodes` and
|
||||
`npu_per_node`:
|
||||
|
||||
```yaml
|
||||
cluster_hosts:
|
||||
- "172.22.0.xxx"
|
||||
- "172.22.0.xxx"
|
||||
```
|
||||
|
||||
##### 2.1.2 Prepare the environment
|
||||
|
||||
Install vllm-ascend development dependencies on every cluster host:
|
||||
|
||||
```bash
|
||||
cd /vllm-workspace/vllm-ascend
|
||||
python3 -m pip install -r requirements-dev.txt
|
||||
```
|
||||
|
||||
Install AISBench on the first host, which is the node with
|
||||
`LWS_WORKER_INDEX=0`:
|
||||
|
||||
```bash
|
||||
export AIS_BENCH_TAG="v3.1-20260330-master"
|
||||
export AIS_BENCH_URL="https://github.com/AISBench/benchmark.git"
|
||||
export BENCHMARK_HOME=/vllm-workspace/vllm-ascend/benchmark
|
||||
|
||||
git clone -b ${AIS_BENCH_TAG} --depth 1 ${AIS_BENCH_URL} $BENCHMARK_HOME
|
||||
cd $BENCHMARK_HOME
|
||||
pip install -e . -r requirements/api.txt -r requirements/extra.txt
|
||||
```
|
||||
|
||||
If your local image already contains the model, benchmark data, Ascend runtime,
|
||||
and AISBench, you only need the run-time exports in the next step.
|
||||
|
||||
##### 2.1.3 Start each node
|
||||
|
||||
Run the script on each node separately. Start worker nodes first, then start
|
||||
node 0.
|
||||
|
||||
On node 1:
|
||||
|
||||
```bash
|
||||
export WORKSPACE=/vllm-workspace
|
||||
export IS_PR_TEST=false
|
||||
export CONFIG_YAML_PATH=DeepSeek-V3.yaml
|
||||
export CONFIG_BASE_PATH=tests/e2e/nightly/multi_node/internal_dp/config/
|
||||
export LWS_WORKER_INDEX=1
|
||||
|
||||
cd $WORKSPACE/vllm-ascend
|
||||
bash tests/e2e/nightly/multi_node/scripts/run.sh
|
||||
```
|
||||
|
||||
On node 0:
|
||||
|
||||
```bash
|
||||
export WORKSPACE=/vllm-workspace
|
||||
export IS_PR_TEST=false
|
||||
export CONFIG_YAML_PATH=DeepSeek-V3.yaml
|
||||
export CONFIG_BASE_PATH=tests/e2e/nightly/multi_node/internal_dp/config/
|
||||
export LWS_WORKER_INDEX=0
|
||||
|
||||
cd $WORKSPACE/vllm-ascend
|
||||
bash tests/e2e/nightly/multi_node/scripts/run.sh
|
||||
```
|
||||
|
||||
Internal DP logs are mainly printed to the terminal running `run.sh`. When
|
||||
`LOG_PREFIX` is set, the shared script also backs up Ascend logs to:
|
||||
|
||||
```text
|
||||
$LOG_PREFIX/node_<LWS_WORKER_INDEX>_plogs/
|
||||
```
|
||||
|
||||
#### 2.2 External DP local run
|
||||
|
||||
##### 2.2.1 Add cluster hosts
|
||||
|
||||
Edit the external DP config you want to run. For example:
|
||||
|
||||
```text
|
||||
tests/e2e/nightly/multi_node/external_dp/config/GLM5_1-W8A8-EP-external.yaml
|
||||
```
|
||||
|
||||
Add `cluster_hosts` as a top-level field, for example near `num_nodes` and
|
||||
`npu_per_node`:
|
||||
|
||||
```yaml
|
||||
cluster_hosts:
|
||||
- "172.22.0.xxx"
|
||||
- "172.22.0.xxx"
|
||||
- "172.22.0.xxx"
|
||||
- "172.22.0.xxx"
|
||||
```
|
||||
|
||||
##### 2.2.2 Prepare the environment
|
||||
|
||||
Install vllm-ascend development dependencies on every cluster host:
|
||||
|
||||
```bash
|
||||
cd /vllm-workspace/vllm-ascend
|
||||
python3 -m pip install -r requirements-dev.txt
|
||||
```
|
||||
|
||||
Install AISBench on node 0:
|
||||
|
||||
```bash
|
||||
export AIS_BENCH_TAG="v3.1-20260330-master"
|
||||
export AIS_BENCH_URL="https://github.com/AISBench/benchmark.git"
|
||||
export BENCHMARK_HOME=/vllm-workspace/vllm-ascend/benchmark
|
||||
|
||||
git clone -b ${AIS_BENCH_TAG} --depth 1 ${AIS_BENCH_URL} $BENCHMARK_HOME
|
||||
cd $BENCHMARK_HOME
|
||||
pip install -e . -r requirements/api.txt -r requirements/extra.txt
|
||||
```
|
||||
|
||||
If your local image already contains the model, benchmark data, Ascend runtime,
|
||||
and AISBench, you only need the run-time exports in the next step.
|
||||
|
||||
##### 2.2.3 Start each node
|
||||
|
||||
External DP uses the same shared `run.sh`. Set `CONFIG_BASE_PATH` to the
|
||||
external DP config directory so the script chooses
|
||||
`external_dp/scripts/test_external_dp.py`.
|
||||
|
||||
Then start non-master nodes first, and start node 0 last. The following example
|
||||
uses `GLM5_1-W8A8-EP-external.yaml`, which is a 4-node disaggregated prefill
|
||||
case.
|
||||
|
||||
On node 1, node 2, and node 3, set the matching `LWS_WORKER_INDEX`:
|
||||
|
||||
```bash
|
||||
export WORKSPACE=/vllm-workspace
|
||||
export IS_PR_TEST=false
|
||||
export CONFIG_BASE_PATH=tests/e2e/nightly/multi_node/external_dp/config/
|
||||
export CONFIG_YAML_PATH=GLM5_1-W8A8-EP-external.yaml
|
||||
export LWS_WORKER_INDEX=1 # Use 2 on node 2, and 3 on node 3.
|
||||
|
||||
cd $WORKSPACE/vllm-ascend
|
||||
bash tests/e2e/nightly/multi_node/scripts/run.sh
|
||||
```
|
||||
|
||||
On node 0:
|
||||
|
||||
```bash
|
||||
export WORKSPACE=/vllm-workspace
|
||||
export IS_PR_TEST=false
|
||||
export CONFIG_BASE_PATH=tests/e2e/nightly/multi_node/external_dp/config/
|
||||
export CONFIG_YAML_PATH=GLM5_1-W8A8-EP-external.yaml
|
||||
export LWS_WORKER_INDEX=0
|
||||
|
||||
cd $WORKSPACE/vllm-ascend
|
||||
bash tests/e2e/nightly/multi_node/scripts/run.sh
|
||||
```
|
||||
|
||||
For `GLM5_1-W8A8-EP-external.yaml`, node 0 and node 1 start prefiller ranks,
|
||||
node 2 and node 3 start decoder ranks, and node 0 also starts the proxy and
|
||||
benchmark.
|
||||
|
||||
##### 2.2.4 Read logs while the test is running
|
||||
|
||||
The terminal running `run.sh` prints pytest orchestration logs. For external DP,
|
||||
AISBench output is also printed on node 0, while rank and proxy stdout/stderr
|
||||
are written to `EXTERNAL_DP_LOG_DIR`. The default layout is:
|
||||
|
||||
```text
|
||||
/tmp/external_dp_logs/
|
||||
node-0/
|
||||
rank-0.log
|
||||
rank-1.log
|
||||
proxy.log
|
||||
node-1/
|
||||
rank-0.log
|
||||
rank-1.log
|
||||
```
|
||||
|
||||
The first line of each rank log records the exact command and environment used
|
||||
to start that rank. `proxy.log` exists only on the configured proxy node,
|
||||
usually node 0.
|
||||
|
||||
Use a separate log directory when running multiple local experiments:
|
||||
|
||||
```bash
|
||||
export EXTERNAL_DP_LOG_DIR=/tmp/external_dp_logs_pd_local
|
||||
```
|
||||
|
||||
To watch logs in real time, run these commands in another terminal on the
|
||||
corresponding node:
|
||||
|
||||
```bash
|
||||
# node 0: ranks and proxy
|
||||
tail -F /tmp/external_dp_logs/node-0/rank-0.log \
|
||||
/tmp/external_dp_logs/node-0/rank-1.log \
|
||||
/tmp/external_dp_logs/node-0/proxy.log
|
||||
|
||||
# node 1: ranks
|
||||
tail -F /tmp/external_dp_logs/node-1/rank-0.log \
|
||||
/tmp/external_dp_logs/node-1/rank-1.log
|
||||
```
|
||||
317
docs/source/developer_guide/contribution/nightly_ci_test.md
Normal file
317
docs/source/developer_guide/contribution/nightly_ci_test.md
Normal file
@@ -0,0 +1,317 @@
|
||||
# Nightly CI Test
|
||||
|
||||
This document explains how to trigger nightly hardware CI tests against your own PR code
|
||||
on Ascend NPU hardware (A2/A3), without waiting for the scheduled nightly run.
|
||||
|
||||
## Background
|
||||
|
||||
By default, nightly CI tests run on a fixed schedule using pre-built nightly images.
|
||||
Contributors can self-service trigger these tests directly against their PR changes
|
||||
by combining a GitHub label with a comment command.
|
||||
|
||||
## How to Trigger
|
||||
|
||||
### 1. Post a comment
|
||||
|
||||
Post one of the following comments in the PR to specify which tests to run.
|
||||
The comment itself triggers the workflow — no label is required.
|
||||
|
||||
| Comment | Effect |
|
||||
|---------|--------|
|
||||
| `/nightly` | Run **all** nightly tests |
|
||||
| `/nightly all` | Run **all** nightly tests (same as above) |
|
||||
| `/nightly test1 test2 ...` | Run only the **named** tests |
|
||||
|
||||
:::{note}
|
||||
Only repository **Contributors** (Triage role) and **Maintainers** (Write role) can
|
||||
trigger the `/nightly` command. If you do not have this permission, ask a maintainer
|
||||
to post the comment for you. You can find the list of maintainers and contributors in
|
||||
the project's [Governance](../../community/governance.md) page or by checking the
|
||||
[CODEOWNERS](https://github.com/vllm-project/vllm-ascend/blob/main/.github/CODEOWNERS)
|
||||
file.
|
||||
:::
|
||||
|
||||
### 2. Wait for results
|
||||
|
||||
GitHub Actions will trigger the `Nightly-A2` or `Nightly-A3` workflow. Only tests
|
||||
matching the filter will be dispatched, which saves hardware resources.
|
||||
|
||||
## Differences Between PR and Scheduled Runs
|
||||
|
||||
| | Scheduled / Manual Dispatch | PR-triggered |
|
||||
|---|----------------------------|---|
|
||||
| Trigger | Cron (daily) or `workflow_dispatch` | `/nightly` comment |
|
||||
| Code tested | Pre-built nightly image | Your PR's HEAD commit (source installed fresh) |
|
||||
| Test scope | All tests | Configurable via `/nightly <names>` |
|
||||
| vLLM + vllm-ascend | From image | Checked out and installed from source |
|
||||
| Test matrix | From main branch's matrix YAML | From PR branch's matrix YAML |
|
||||
|
||||
When a PR run is detected (`is_pr_test: true`), the workflow additionally:
|
||||
|
||||
1. Uninstalls any existing vllm packages in the container.
|
||||
2. Checks out the specific vllm version and your PR's vllm-ascend commit from source.
|
||||
3. Installs all dependencies from source.
|
||||
4. Installs the `aisbench` benchmark suite.
|
||||
|
||||
## Test Matrix Data Source
|
||||
|
||||
The set of nightly test cases (their names, runners, test paths, model configs) is
|
||||
declared in a single data file:
|
||||
|
||||
```text
|
||||
.github/workflows/configs/nightly_config.yaml
|
||||
```
|
||||
|
||||
The file is organized as `a2:` and `a3:` top-level keys (one per SoC). Under each
|
||||
SoC, tests are grouped by execution shape (single-node, multi-node, double-node,
|
||||
multi-card, accuracy) and each group holds a `test_config` (or `nightly` / `pr_only`
|
||||
for accuracy) list whose entries carry a `name` plus the fields consumed by the
|
||||
downstream reusable workflows (`os`, `tests`, `config_file_path`, `size`, etc.).
|
||||
|
||||
Both the `Nightly-A2` and `Nightly-A3` workflows dynamically read this file at run
|
||||
time — there is no hardcoded test matrix in the workflow YAMLs. The
|
||||
`/nightly <name>` slash command resolves names by walking the same file from the
|
||||
PR branch, so newly added entries can be exercised on a PR before they land on
|
||||
main.
|
||||
|
||||
## Adding a New Nightly Test Case
|
||||
|
||||
To add a new test case (no need to touch the workflow YAMLs):
|
||||
|
||||
1. Append an entry under the appropriate section in
|
||||
`.github/workflows/configs/nightly_config.yaml`. Each entry needs at least:
|
||||
- `name`: unique identifier used in `/nightly <name>` filters
|
||||
- `os` (for single-node / multi-card pytest+yaml tests) or `runner` is inferred
|
||||
- one of `tests:` (pytest directory) or `config_file_path:` (YAML-driven model config)
|
||||
- `size` (multi-node / double-node only)
|
||||
2. Add the actual test files (pytest modules under `tests/e2e/nightly/...` or
|
||||
YAML model configs in `tests/e2e/nightly/.../configs/`).
|
||||
3. Open a PR. Once CI is green, you can validate the new entry against real NPU
|
||||
hardware **without** merging the PR — see *Examples* below.
|
||||
|
||||
## Available Test Names
|
||||
|
||||
The test names you can pass to `/nightly` correspond to the `name` fields under
|
||||
the matching section in `.github/workflows/configs/nightly_config.yaml`. The
|
||||
tables below mirror the current contents of that file.
|
||||
|
||||
### A2 workflow (`.github/workflows/schedule_nightly_test_a2.yaml`)
|
||||
|
||||
**Single-node tests** (`a2.single_node.test_config`):
|
||||
|
||||
| Test name | Description |
|
||||
|-----------|-------------|
|
||||
| `test_custom_op_multi_card` | Custom operator tests (multi card) |
|
||||
| `qwen3-vl-32b-instruct-w8a8` | Qwen3-VL-32B-Instruct W8A8 |
|
||||
| `qwen3-32b-int8` | Qwen3-32B INT8 quantization |
|
||||
| `Qwen3.5-27B-w8a8-A2` | Qwen3.5-27B W8A8 |
|
||||
| `Qwen3.5-397B-A17B-w4a8-mtp` | Qwen3.5-397B-A17B W4A8 + MTP |
|
||||
|
||||
**Multi-node tests** (`a2.multi_node.test_config`):
|
||||
|
||||
| Test name | Description |
|
||||
|-----------|-------------|
|
||||
| `multi-node-qwen3-235b-dp` | Qwen3-235B-A22B, 2-node DP |
|
||||
| `multi-node-GLM-5.1-w8a8-A2` | GLM-5.1 W8A8, 2 nodes |
|
||||
| `multi-node-Kimi-K2.5-W4A8-A2` | Kimi-K2.5 W4A8, 2 nodes |
|
||||
|
||||
**Accuracy tests** (`a2.accuracy.nightly` and `a2.accuracy.pr_only`):
|
||||
|
||||
| Test name | Description | Scope |
|
||||
|-----------|-------------|-------|
|
||||
| `accuracy-group-1` | Qwen3-VL-8B, Qwen3-8B, Qwen2-Audio-7B, etc. | nightly |
|
||||
| `accuracy-group-2` | ERNIE-4.5, Molmo-7B, Llama-3.2-3B, etc. | nightly |
|
||||
| `accuracy-group-3` | Qwen3-30B-A3B, Qwen3-VL-30B-A3B, etc. | nightly |
|
||||
| `accuracy-group-4` | Qwen3-Next-80B-A3B, Qwen3-Omni-30B-A3B, etc. | nightly |
|
||||
| `pr-accuracy-group-1` | gemma-3-4b-it, internlm3-8b-instruct, etc. | pr_only |
|
||||
| `pr-accuracy-group-2` | Qwen2.5-Math-RM-72B, Hunyuan-A13B-Instruct | pr_only |
|
||||
|
||||
The `pr-accuracy-group-*` entries only run on `/nightly` (PR-triggered) runs;
|
||||
`/nightly all` on the schedule skips them.
|
||||
|
||||
### A3 workflow (`.github/workflows/schedule_nightly_test_a3.yaml`)
|
||||
|
||||
**Multi-node tests** (`a3.multi_node.test_config`, 4-node):
|
||||
|
||||
| Test name | Description |
|
||||
|-----------|-------------|
|
||||
| `multi-node-deepseek-v3.2-W8A8-EP` | DeepSeek-V3.2-W8A8 with EP, 4-node |
|
||||
|
||||
**Double-node tests** (`a3.double_node.test_config`, 2-node, run after multi-node):
|
||||
|
||||
| Test name | Description |
|
||||
|-----------|-------------|
|
||||
| `multi-node-deepseek-r1-w8a8-longseq` | DeepSeek-R1-W8A8 long sequence, 2-node |
|
||||
| `multi-node-qwen3-dp` | Qwen3-235B-A22B, 2-node DP |
|
||||
| `multi-node-qwenw8a8-2node-eplb` | Qwen3-235B-W8A8 with EPLB, 2-node |
|
||||
| `multi-node-dpsk3.2-2node` | DeepSeek-V3.2-W8A8, 2-node |
|
||||
| `multi-node-qwenw8a8-2node-longseq` | Qwen3-235B-W8A8 long sequence, 2-node |
|
||||
| `multi-node-qwen-disagg-pd` | Qwen3-235B disaggregated PD, 2-node |
|
||||
| `multi-node-qwen-vl-disagg-pd` | Qwen3-VL-235B disaggregated PD, 2-node |
|
||||
| `multi-node-deepseek-v3.1` | DeepSeek-V3.1-BF16, 2-node |
|
||||
| `multi-node-deepseek-v3.2-W8A8-EP` | DeepSeek-V3.2-W8A8 with EP, 4-node |
|
||||
| `multi-node-glm-5.2` | GLM-5.1-W8A8, 2-node |
|
||||
|
||||
**Single-node tests** (`a3.single_node.test_config`):
|
||||
|
||||
| Test name | Description |
|
||||
|-----------|-------------|
|
||||
| `mtpx-deepseek-r1-0528-w8a8` | MTP-X + DeepSeek-R1-0528-W8A8 |
|
||||
| `deepseek-r1-0528-w8a8` | DeepSeek-R1-0528-W8A8 |
|
||||
| `kimi-k2-thinking` | Kimi-K2-Thinking |
|
||||
| `qwen3-vl-235b-a22b-instruct-w8a8` | Qwen3-VL-235B-A22B-Instruct-W8A8 |
|
||||
| `deepseek-r1-0528-w8a8-prefix-cache` | DeepSeek-R1-0528-W8A8 prefix cache |
|
||||
| `deepseek-v3-2-w8a8` | DeepSeek-V3.2-W8A8 |
|
||||
| `glm-4.7-w8a8` | GLM-4.7 W8A8 |
|
||||
| `kimi-k2.5` | Kimi-K2.5 |
|
||||
| `qwen3-235b-a22b-w8a8` | Qwen3-235B-A22B-W8A8 |
|
||||
| `Qwen3.5-397B-A17B-w8a8-mtp` | Qwen3.5-397B-A17B W8A8 + MTP |
|
||||
| `MiniMax-M2.5-w8a8-QuaRot-A3` | MiniMax-M2.5 W8A8 + QuaRot |
|
||||
| `Qwen3.5-27B-w8a8-A3` | Qwen3.5-27B W8A8 |
|
||||
| `Qwen3.5-122B-A10B-W8A8-A3` | Qwen3.5-122B-A10B W8A8 |
|
||||
| `DeepSeek-V4-Flash-W8A8-A3` | DeepSeek-V4-Flash W8A8 |
|
||||
|
||||
**Multi-card tests** (`a3.multi_card.test_config`):
|
||||
|
||||
| Test name | Description |
|
||||
|-----------|-------------|
|
||||
| `qwen3-30b-acc` | Qwen3-30B accuracy test |
|
||||
| `qwen3-30b-a3b-w8a8` | Qwen3-30B-A3B-W8A8 |
|
||||
| `qwen3-32b-int8` | Qwen3-32B-Int8 |
|
||||
| `qwen3-32b-int8-prefix-cache` | Qwen3-32B-Int8 prefix cache |
|
||||
| `Qwen3-30B-A3B-W4A8-llm-compressor` | Qwen3-30B-A3B W4A8 via llm-compressor |
|
||||
| `Qwen3-30B-QuaRot` | Qwen3-30B QuaRot + eagle3 |
|
||||
| `Qwen3-32B-QuaRot` | Qwen3-32B QuaRot + eagle3 |
|
||||
|
||||
:::{warning}
|
||||
The A3 resource pool has a maximum concurrency of **5×16 NPUs**. Multi-node tests
|
||||
run with `max-parallel: 2` to avoid resource exhaustion. Running `/nightly all` on
|
||||
A3 will queue a large number of jobs — prefer targeting specific test names when
|
||||
possible.
|
||||
:::
|
||||
|
||||
## Examples
|
||||
|
||||
Run all available nightly tests against your PR:
|
||||
|
||||
```text
|
||||
/nightly
|
||||
```
|
||||
|
||||
Run only the custom operator multi-card test:
|
||||
|
||||
```text
|
||||
/nightly test_custom_op_multi_card
|
||||
```
|
||||
|
||||
Run two specific tests at once (one per SoC):
|
||||
|
||||
```text
|
||||
/nightly test_custom_op_multi_card mtpx-deepseek-r1-0528-w8a8
|
||||
```
|
||||
|
||||
Run a single accuracy group (with all of its models):
|
||||
|
||||
```text
|
||||
/nightly accuracy-group-1
|
||||
```
|
||||
|
||||
Run a single accuracy model (only that model from a group):
|
||||
|
||||
```text
|
||||
/nightly accuracy-group-1/Qwen3-8B
|
||||
```
|
||||
|
||||
Re-trigger after fixing an issue: just push a new commit. The `synchronize` event
|
||||
re-runs the workflow and picks up the existing `/nightly` comment automatically — no
|
||||
need to post a new comment.
|
||||
|
||||
## Adding a New Test Case — Worked Example
|
||||
|
||||
To add `my-new-test` to the A2 single-node section:
|
||||
|
||||
1. Edit `.github/workflows/configs/nightly_config.yaml`, append under
|
||||
`a2.single_node.test_config`:
|
||||
|
||||
```yaml
|
||||
- name: my-new-test
|
||||
os: linux-aarch64-a2b3-4
|
||||
tests: tests/e2e/nightly/single_node/ops/multicard_ops_a2/test_my_new.py
|
||||
```
|
||||
|
||||
2. Commit the new pytest file (`test_my_new.py`) in the same PR.
|
||||
|
||||
3. Trigger from the PR:
|
||||
|
||||
```text
|
||||
/nightly my-new-test
|
||||
```
|
||||
|
||||
The workflow will:
|
||||
|
||||
- `pr_nightly_command.yml` reads your PR's `nightly_config.yaml` and resolves
|
||||
`my-new-test` → dispatch A2 only.
|
||||
- `Nightly-A2` is dispatched at `main`, but `generate-a2-matrix` checks out your
|
||||
PR commit and reads the new entry from the matrix.
|
||||
- `single-node-tests` runs one matrix job for `my-new-test`, with
|
||||
`should_run=true`. The reusable workflow checks out your PR code (via
|
||||
`vllm_ascend_ref`) and runs your pytest.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**The workflow didn't start after I posted the comment.**
|
||||
|
||||
- Check that the comment starts exactly with `/nightly` with no leading spaces or
|
||||
extra characters before the slash.
|
||||
- Confirm you have at least Triage permission on the repository; unauthorized
|
||||
users' comments are ignored.
|
||||
- To re-trigger after fixing an issue, simply push a new commit — the workflow will
|
||||
reuse the existing `/nightly` comment automatically.
|
||||
|
||||
**Only some tests ran, not the ones I expected.**
|
||||
|
||||
- Test names are case-sensitive and must match the `name` field in
|
||||
`.github/workflows/configs/nightly_config.yaml` exactly (see the tables above).
|
||||
- For a PR-triggered run, the matrix is loaded from your PR's
|
||||
`nightly_config.yaml`, not main. If a name isn't in your PR's file, it won't
|
||||
be recognized and the dispatch will be skipped.
|
||||
- Check the `parse-trigger` job output in GitHub Actions for the resolved
|
||||
`test_filter` value.
|
||||
|
||||
**The workflow ran with the scheduled image, not my PR code.**
|
||||
|
||||
- Confirm the workflow was triggered by `repository_dispatch` (slash command),
|
||||
not bare `workflow_dispatch`. The `pr_nightly_command.yml` workflow is what
|
||||
actually dispatches `schedule_nightly_test_a2.yaml` / `_a3.yaml` with
|
||||
`vllm_ascend_ref` pointing at your PR SHA.
|
||||
|
||||
**A new test I added isn't being recognized.**
|
||||
|
||||
- Confirm the entry is well-formed YAML under
|
||||
`.github/workflows/configs/nightly_config.yaml`. The `name` field is required
|
||||
and must be unique within the SoC's section.
|
||||
- The matrix is loaded from your PR branch, so make sure the file is committed
|
||||
to the same branch the `/nightly` comment was posted on.
|
||||
|
||||
**How to obtain more detailed logs to pinpoint problems for multi-node tests**
|
||||
|
||||
- For most issues, the stdout pop-up logs from GitHub actions are sufficient (this log always represents the logs from the first node).
|
||||
- If the logs from a first node are no longer sufficient to provide effective logging information, see the summary of your jobs to download log archive for the corresponding test, which includes the framework-side logs and plog information for each node, structured as follows:
|
||||
|
||||
```shell
|
||||
.
|
||||
├── node0
|
||||
│ ├── root
|
||||
│ │ └── ascend
|
||||
│ │ └── log
|
||||
│ └── var
|
||||
│ └── log
|
||||
│ └── vllm-deepseek-v3-0f233d-0_logs.txt
|
||||
└── node1
|
||||
├── root
|
||||
│ └── ascend
|
||||
│ └── log
|
||||
└── var
|
||||
└── log
|
||||
└── vllm-deepseek-v3-0f233d-0-1_logs.txt
|
||||
```
|
||||
@@ -1,10 +1,10 @@
|
||||
# Testing
|
||||
|
||||
This secition explains how to write e2e tests and unit tests to verify the implementation of your feature.
|
||||
This document explains how to write unit tests, E2E tests, and nightly tests to verify your feature implementation.
|
||||
|
||||
## Setup test environment
|
||||
## Set up a test environment
|
||||
|
||||
The fastest way to setup test environment is to use the main branch container image:
|
||||
The fastest way to set up a test environment is to use the main branch's container image:
|
||||
|
||||
:::::{tab-set}
|
||||
:sync-group: e2e
|
||||
@@ -13,7 +13,7 @@ The fastest way to setup test environment is to use the main branch container im
|
||||
:selected:
|
||||
:sync: cpu
|
||||
|
||||
You can run the unit tests on CPU with the following steps:
|
||||
You can run the unit tests on CPUs with the following steps:
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
@@ -22,39 +22,50 @@ cd ~/vllm-project/
|
||||
# ls
|
||||
# vllm vllm-ascend
|
||||
|
||||
# Use mirror to speedup download
|
||||
# docker pull quay.nju.edu.cn/ascend/cann:|cann_image_tag|
|
||||
# Use mirror to speed up download
|
||||
# docker pull m.daocloud.io/quay.io/ascend/cann:|cann_image_tag|
|
||||
export IMAGE=quay.io/ascend/cann:|cann_image_tag|
|
||||
docker run --rm --name vllm-ascend-ut \
|
||||
-v $(pwd):/vllm-project \
|
||||
-v ~/.cache:/root/.cache \
|
||||
-ti $IMAGE bash
|
||||
|
||||
# (Optional) Configure mirror to speedup download
|
||||
# (Optional) Configure mirror to speed up download
|
||||
sed -i 's|ports.ubuntu.com|mirrors.huaweicloud.com|g' /etc/apt/sources.list
|
||||
pip config set global.index-url https://mirrors.huaweicloud.com/repository/pypi/simple/
|
||||
|
||||
# For torch-npu dev version or x86 machine
|
||||
# For TorchNPU dev version or x86 machine
|
||||
export PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu/ https://mirrors.huaweicloud.com/ascend/repos/pypi"
|
||||
|
||||
# src path
|
||||
export SRC_WORKSPACE=/vllm-workspace
|
||||
mkdir -p $SRC_WORKSPACE
|
||||
cd $SRC_WORKSPACE
|
||||
|
||||
apt-get update -y
|
||||
apt-get install -y python3-pip git vim wget net-tools gcc g++ cmake libnuma-dev curl gnupg2
|
||||
|
||||
# Install vllm
|
||||
cd /vllm-project/vllm
|
||||
VLLM_TARGET_DEVICE=empty python3 -m pip -v install .
|
||||
git clone -b |vllm_ascend_version| --depth 1 https://github.com/vllm-project/vllm-ascend.git
|
||||
git clone --depth 1 https://github.com/vllm-project/vllm.git
|
||||
|
||||
# Install vllm-ascend
|
||||
cd /vllm-project/vllm-ascend
|
||||
# [IMPORTANT] Import LD_LIBRARY_PATH to enumerate the CANN environment under CPU
|
||||
# vllm
|
||||
cd $SRC_WORKSPACE/vllm
|
||||
VLLM_TARGET_DEVICE=empty python3 -m pip install .
|
||||
python3 -m pip uninstall -y triton
|
||||
|
||||
# vllm-ascend
|
||||
cd $SRC_WORKSPACE/vllm-ascend
|
||||
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/ascend-toolkit/latest/$(uname -m)-linux/devlib
|
||||
# For cpu environment, set SOC_VERSION for different chips.
|
||||
# See https://github.com/vllm-project/vllm-ascend/blob/3cb0af0bcf3299089ca7e72159fa36e825a470f8/setup.py#L132 for detail.
|
||||
export SOC_VERSION="ascend910b1"
|
||||
python3 -m pip install .
|
||||
python3 -m pip install -r requirements-dev.txt
|
||||
python3 -m pip install -v .
|
||||
```
|
||||
|
||||
::::
|
||||
|
||||
::::{tab-item} Single card
|
||||
::::{tab-item} Single-card
|
||||
:sync: single
|
||||
|
||||
```{code-block} bash
|
||||
@@ -66,6 +77,7 @@ export DEVICE=/dev/davinci0
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:main
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--shm-size=1g \
|
||||
--device $DEVICE \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
@@ -86,13 +98,16 @@ After starting the container, you should install the required packages:
|
||||
# Prepare
|
||||
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
|
||||
|
||||
# Switch to the /vllm-workspace/vllm-ascend directory
|
||||
cd /vllm-workspace/vllm-ascend/
|
||||
|
||||
# Install required packages
|
||||
pip install -r requirements-dev.txt
|
||||
```
|
||||
|
||||
::::
|
||||
|
||||
::::{tab-item} Multi cards
|
||||
::::{tab-item} Multi-cards
|
||||
:sync: multi
|
||||
|
||||
```{code-block} bash
|
||||
@@ -101,6 +116,7 @@ pip install -r requirements-dev.txt
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:main
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--shm-size=1g \
|
||||
--device /dev/davinci0 \
|
||||
--device /dev/davinci1 \
|
||||
--device /dev/davinci2 \
|
||||
@@ -136,13 +152,13 @@ pip install -r requirements-dev.txt
|
||||
|
||||
## Running tests
|
||||
|
||||
### Unit test
|
||||
### Unit tests
|
||||
|
||||
There are several principles to follow when writing unit tests:
|
||||
|
||||
- The test file path should be consistent with source file and start with `test_` prefix, such as: `vllm_ascend/worker/worker_v1.py` --> `tests/ut/worker/test_worker_v1.py`
|
||||
- The vLLM Ascend test are using unittest framework, see [here](https://docs.python.org/3/library/unittest.html#module-unittest) to understand how to write unit tests.
|
||||
- All unit tests can be run on CPU, so you must mock the device-related function to host.
|
||||
- The test file path should be consistent with the source file and start with the `test_` prefix, such as: `vllm_ascend/worker/worker.py` --> `tests/ut/worker/test_worker.py`
|
||||
- The vLLM Ascend test uses unittest framework. See [the Python unittest documentation](https://docs.python.org/3/library/unittest.html#module-unittest) to understand how to write unit tests.
|
||||
- All unit tests can be run on CPUs, so you must mock the device-related functions on the host.
|
||||
- Example: [tests/ut/test_ascend_config.py](https://github.com/vllm-project/vllm-ascend/blob/main/tests/ut/test_ascend_config.py).
|
||||
- You can run the unit tests using `pytest`:
|
||||
|
||||
@@ -161,12 +177,12 @@ TORCH_DEVICE_BACKEND_AUTOLOAD=0 pytest -sv tests/ut
|
||||
|
||||
::::
|
||||
|
||||
::::{tab-item} Single card
|
||||
::::{tab-item} Single-card
|
||||
:sync: single
|
||||
|
||||
```bash
|
||||
cd /vllm-workspace/vllm-ascend/
|
||||
# Run all single card the tests
|
||||
# Run all single-card tests
|
||||
pytest -sv tests/ut
|
||||
|
||||
# Run single test
|
||||
@@ -175,12 +191,12 @@ pytest -sv tests/ut/test_ascend_config.py
|
||||
|
||||
::::
|
||||
|
||||
::::{tab-item} Multi cards test
|
||||
::::{tab-item} Multi-card
|
||||
:sync: multi
|
||||
|
||||
```bash
|
||||
cd /vllm-workspace/vllm-ascend/
|
||||
# Run all single card the tests
|
||||
# Run all multi-card tests
|
||||
pytest -sv tests/ut
|
||||
|
||||
# Run single test
|
||||
@@ -193,8 +209,63 @@ pytest -sv tests/ut/test_ascend_config.py
|
||||
|
||||
### E2E test
|
||||
|
||||
Although vllm-ascend CI provide [e2e test](https://github.com/vllm-project/vllm-ascend/blob/main/.github/workflows/vllm_ascend_test.yaml) on Ascend CI, you can run it
|
||||
locally.
|
||||
Although vllm-ascend CI provides E2E tests on Ascend CI (for example,
|
||||
[schedule_nightly_test_a2.yaml](https://github.com/vllm-project/vllm-ascend/blob/main/.github/workflows/schedule_nightly_test_a2.yaml), [schedule_nightly_test_a3.yaml](https://github.com/vllm-project/vllm-ascend/blob/main/.github/workflows/schedule_nightly_test_a3.yaml), [pr_test.yaml](https://github.com/vllm-project/vllm-ascend/blob/main/.github/workflows/pr_test.yaml)), you can run them locally.
|
||||
|
||||
#### PR-triggered E2E test
|
||||
|
||||
You can run tests with `pytest` as well. Typical examples:
|
||||
:::::{tab-set}
|
||||
:sync-group: e2e
|
||||
|
||||
::::{tab-item} Local (CPU)
|
||||
:sync: cpu
|
||||
|
||||
You can't run the E2E test on CPUs.
|
||||
::::
|
||||
|
||||
::::{tab-item} Single-card
|
||||
:selected:
|
||||
:sync: single
|
||||
|
||||
```bash
|
||||
cd /vllm-workspace/vllm-ascend/
|
||||
# Run all single-card tests
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/pull_request/one_card/
|
||||
|
||||
# Run a certain test script
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/pull_request/one_card/test_camem.py
|
||||
|
||||
# Run a certain case in test script
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/pull_request/one_card/test_camem.py::test_end_to_end
|
||||
```
|
||||
|
||||
::::
|
||||
|
||||
::::{tab-item} Multi-card
|
||||
:sync: multi
|
||||
|
||||
```bash
|
||||
cd /vllm-workspace/vllm-ascend/
|
||||
# Run all multi-card tests
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/pull_request/two_card/
|
||||
|
||||
# Run a certain test script
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/pull_request/two_card/test_qwen3_moe_eplb.py
|
||||
|
||||
# Run a certain case in test script
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/pull_request/two_card/test_qwen3_moe_eplb.py::test_qwen3_moe_w8a8_distributed_tp2_ep_dynamic_eplb
|
||||
```
|
||||
|
||||
::::
|
||||
|
||||
:::::
|
||||
|
||||
This will reproduce the E2E test behavior.
|
||||
|
||||
#### Nightly-triggered E2E test
|
||||
|
||||
You can run tests with `pytest` as well. Typical examples:
|
||||
|
||||
:::::{tab-set}
|
||||
:sync-group: e2e
|
||||
@@ -202,84 +273,102 @@ locally.
|
||||
::::{tab-item} Local (CPU)
|
||||
:sync: cpu
|
||||
|
||||
You can't run e2e test on CPU.
|
||||
You can't run the E2E test on CPUs.
|
||||
::::
|
||||
|
||||
::::{tab-item} Single card
|
||||
::::{tab-item} Single-card
|
||||
:selected:
|
||||
:sync: single
|
||||
|
||||
```bash
|
||||
cd /vllm-workspace/vllm-ascend/
|
||||
# Run all single card the tests
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/singlecard/
|
||||
|
||||
# Run a certain test script
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/singlecard/test_offline_inference.py
|
||||
|
||||
# Run a certain case in test script
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/singlecard/test_offline_inference.py::test_models
|
||||
# run all single-card op tests
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/nightly/single_node/ops/singlecard_ops/
|
||||
```
|
||||
|
||||
::::
|
||||
|
||||
::::{tab-item} Multi cards test
|
||||
::::{tab-item} Multi-card
|
||||
:sync: multi
|
||||
|
||||
```bash
|
||||
cd /vllm-workspace/vllm-ascend/
|
||||
# Run all single card the tests
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/multicard/
|
||||
# run all multi-card op tests on A2
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/nightly/single_node/ops/multicard_ops_a2/
|
||||
|
||||
# Run a certain test script
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/multicard/test_dynamic_npugraph_batchsize.py
|
||||
|
||||
# Run a certain case in test script
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/multicard/test_offline_inference.py::test_models
|
||||
# run all multi-card op tests on A3
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/nightly/single_node/ops/multicard_ops_a3/
|
||||
```
|
||||
|
||||
::::
|
||||
|
||||
:::::
|
||||
|
||||
This will reproduce e2e test: [vllm_ascend_test.yaml](https://github.com/vllm-project/vllm-ascend/blob/main/.github/workflows/vllm_ascend_test.yaml).
|
||||
For running nightly single-node model test cases locally, refer to the following example.
|
||||
|
||||
#### E2E test example:
|
||||
```bash
|
||||
export CONFIG_YAML_PATH=Qwen3-32B.yaml
|
||||
VLLM_USE_MODELSCOPE=true pytest -sv tests/e2e/nightly/single_node/models/scripts/test_single_node.py
|
||||
```
|
||||
|
||||
- Offline test example: [`tests/e2e/singlecard/test_offline_inference.py`](https://github.com/vllm-project/vllm-ascend/blob/main/tests/e2e/singlecard/test_offline_inference.py)
|
||||
- Online test examples: [`tests/e2e/singlecard/test_prompt_embedding.py`](https://github.com/vllm-project/vllm-ascend/blob/main/tests/e2e/singlecard/test_prompt_embedding.py)
|
||||
- Correctness test example: [`tests/e2e/singlecard/test_aclgraph.py`](https://github.com/vllm-project/vllm-ascend/blob/main/tests/e2e/singlecard/test_aclgraph.py)
|
||||
- Reduced Layer model test example: [test_torchair_graph_mode.py - DeepSeek-V3-Pruning](https://github.com/vllm-project/vllm-ascend/blob/20767a043cccb3764214930d4695e53941de87ec/tests/e2e/multicard/test_torchair_graph_mode.py#L48)
|
||||
For running nightly multi-node model test cases locally, refer to the `Running Locally` section in [Multi Node Test](./multi_node_test.md).
|
||||
|
||||
The CI resource is limited, you might need to reduce layer number of the model, below is an example of how to generate a reduced layer model:
|
||||
1. Fork the original model repo in modelscope, we need all the files in the repo except for weights.
|
||||
2. Set `num_hidden_layers` to the expected number of layers, e.g., `{"num_hidden_layers": 2,}`
|
||||
3. Copy the following python script as `generate_random_weight.py`. Set the relevant parameters `MODEL_LOCAL_PATH`, `DIST_DTYPE` and `DIST_MODEL_PATH` as needed:
|
||||
#### E2E test examples
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoConfig
|
||||
from modeling_deepseek import DeepseekV3ForCausalLM
|
||||
from modelscope import snapshot_download
|
||||
- Offline test example: [`tests/e2e/pull_request/one_card/test_camem.py`](https://github.com/vllm-project/vllm-ascend/blob/main/tests/e2e/pull_request/one_card/test_camem.py)
|
||||
|
||||
MODEL_LOCAL_PATH = "~/.cache/modelscope/models/vllm-ascend/DeepSeek-V3-Pruning"
|
||||
DIST_DTYPE = torch.bfloat16
|
||||
DIST_MODEL_PATH = "./random_deepseek_v3_with_2_hidden_layer"
|
||||
The CI resource is limited, and you might need to reduce the number of layers of a model. Below is an example of how to generate a reduced layer model:
|
||||
|
||||
config = AutoConfig.from_pretrained(MODEL_LOCAL_PATH, trust_remote_code=True)
|
||||
model = DeepseekV3ForCausalLM(config)
|
||||
model = model.to(DIST_DTYPE)
|
||||
model.save_pretrained(DIST_MODEL_PATH)
|
||||
```
|
||||
1. Fork the original model repo in modelscope. All the files in the repo except for weights are required.
|
||||
2. Set `num_hidden_layers` to the expected number of layers, e.g., `{"num_hidden_layers": 2,}`
|
||||
3. Copy the following python script as `generate_random_weight.py`. Set the relevant parameters `MODEL_LOCAL_PATH`, `DIST_DTYPE` and `DIST_MODEL_PATH` as needed:
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoConfig
|
||||
from modeling_deepseek import DeepseekV3ForCausalLM
|
||||
from modelscope import snapshot_download
|
||||
|
||||
MODEL_LOCAL_PATH = "~/.cache/modelscope/models/vllm-ascend/DeepSeek-V3-Pruning"
|
||||
DIST_DTYPE = torch.bfloat16
|
||||
DIST_MODEL_PATH = "./random_deepseek_v3_with_2_hidden_layer"
|
||||
|
||||
config = AutoConfig.from_pretrained(MODEL_LOCAL_PATH, trust_remote_code=True)
|
||||
model = DeepseekV3ForCausalLM(config)
|
||||
model = model.to(DIST_DTYPE)
|
||||
model.save_pretrained(DIST_MODEL_PATH)
|
||||
```
|
||||
|
||||
### Run doctest
|
||||
|
||||
vllm-ascend provides a `vllm-ascend/tests/e2e/run_doctests.sh` command to run all doctests in the doc files.
|
||||
The doctest is a good way to make sure the docs are up to date and the examples are executable, you can run it locally as follows:
|
||||
The doctest is a good way to make sure docs stay current and examples remain executable, which can be run locally as follows:
|
||||
|
||||
```bash
|
||||
# Run doctest
|
||||
/vllm-workspace/vllm-ascend/tests/e2e/run_doctests.sh
|
||||
```
|
||||
|
||||
This will reproduce the same environment as the CI: [vllm_ascend_doctest.yaml](https://github.com/vllm-project/vllm-ascend/blob/main/.github/workflows/vllm_ascend_doctest.yaml).
|
||||
This will reproduce the same environment as the CI. See [labeled_doctest.yaml](https://github.com/vllm-project/vllm-ascend/blob/main/.github/workflows/labeled_doctest.yaml).
|
||||
|
||||
### Run docs link check
|
||||
|
||||
You can validate external links in the Sphinx docs locally with:
|
||||
|
||||
```bash
|
||||
make -C docs linkcheck SPHINXOPTS="-W --keep-going"
|
||||
```
|
||||
|
||||
To check links in a specific Markdown file, pass the file to `sphinx-build`.
|
||||
For example, to check only `docs/source/user_guide/release_notes.md`:
|
||||
|
||||
```bash
|
||||
cd docs
|
||||
sphinx-build -b linkcheck -W --keep-going \
|
||||
source _build/linkcheck source/user_guide/release_notes.md
|
||||
```
|
||||
|
||||
The detailed report will be written to:
|
||||
|
||||
- `docs/_build/linkcheck/output.txt`
|
||||
- `docs/_build/linkcheck/output.json`
|
||||
|
||||
@@ -5,6 +5,6 @@
|
||||
:maxdepth: 1
|
||||
using_evalscope
|
||||
using_lm_eval
|
||||
using_ais_bench
|
||||
using_opencompass
|
||||
accuracy_report/index
|
||||
:::
|
||||
|
||||
333
docs/source/developer_guide/evaluation/using_ais_bench.md
Normal file
333
docs/source/developer_guide/evaluation/using_ais_bench.md
Normal file
@@ -0,0 +1,333 @@
|
||||
# Using AISBench
|
||||
|
||||
This document guides you to conduct accuracy testing using [AISBench](https://github.com/AISBench/benchmark/tree/master). AISBench provides accuracy and performance evaluation for many datasets.
|
||||
|
||||
## Online Server
|
||||
|
||||
### 1. Start the vLLM server
|
||||
|
||||
You can run docker container to start the vLLM server on a single NPU:
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Update DEVICE according to your device (/dev/davinci[0-7])
|
||||
export DEVICE=/dev/davinci7
|
||||
# Update the vllm-ascend image
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--shm-size=1g \
|
||||
--device $DEVICE \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
--device /dev/hisi_hdc \
|
||||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||||
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /root/.cache:/root/.cache \
|
||||
-p 8000:8000 \
|
||||
-e VLLM_USE_MODELSCOPE=True \
|
||||
-e PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256 \
|
||||
-it $IMAGE \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
Run the vLLM server in the docker.
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 35000 &
|
||||
```
|
||||
|
||||
:::{note}
|
||||
`--max-model-len` should be greater than `35000`, this will be suitable for most datasets. Otherwise the accuracy evaluation may be affected.
|
||||
:::
|
||||
|
||||
The vLLM server is started successfully, if you see logs as below:
|
||||
|
||||
```shell
|
||||
INFO: Started server process [9446]
|
||||
INFO: Waiting for application startup.
|
||||
INFO: Application startup complete.
|
||||
```
|
||||
|
||||
### 2. Run different datasets using AISBench
|
||||
|
||||
#### Install AISBench
|
||||
|
||||
Refer to [AISBench](https://github.com/AISBench/benchmark/tree/master) for details.
|
||||
Install AISBench from source.
|
||||
|
||||
```shell
|
||||
git clone https://github.com/AISBench/benchmark.git
|
||||
cd benchmark/
|
||||
pip3 install -e ./ --use-pep517
|
||||
```
|
||||
|
||||
Install extra AISBench dependencies.
|
||||
|
||||
```shell
|
||||
pip3 install -r requirements/api.txt
|
||||
pip3 install -r requirements/extra.txt
|
||||
```
|
||||
|
||||
Run `ais_bench -h` to check the installation.
|
||||
|
||||
#### Download Dataset
|
||||
|
||||
You can choose one or multiple datasets to execute accuracy evaluation.
|
||||
|
||||
1. `C-Eval` dataset.
|
||||
|
||||
Take `C-Eval` dataset as an example. You can refer to [Datasets](https://github.com/AISBench/benchmark/tree/master/ais_bench/benchmark/configs/datasets) for more datasets. Each dataset has a `README.md` with detailed download and installation instructions.
|
||||
|
||||
Download dataset and install it to specific path.
|
||||
|
||||
```shell
|
||||
cd ais_bench/datasets
|
||||
mkdir ceval/
|
||||
mkdir ceval/formal_ceval
|
||||
cd ceval/formal_ceval
|
||||
wget https://www.modelscope.cn/datasets/opencompass/ceval-exam/resolve/master/ceval-exam.zip
|
||||
unzip ceval-exam.zip
|
||||
rm ceval-exam.zip
|
||||
```
|
||||
|
||||
2. `MMLU` dataset.
|
||||
|
||||
```shell
|
||||
cd ais_bench/datasets
|
||||
wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/mmlu.zip
|
||||
unzip mmlu.zip
|
||||
rm mmlu.zip
|
||||
```
|
||||
|
||||
3. `GPQA` dataset.
|
||||
|
||||
```shell
|
||||
cd ais_bench/datasets
|
||||
wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/gpqa.zip
|
||||
unzip gpqa.zip
|
||||
rm gpqa.zip
|
||||
```
|
||||
|
||||
4. `MATH` dataset.
|
||||
|
||||
```shell
|
||||
cd ais_bench/datasets
|
||||
wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/math.zip
|
||||
unzip math.zip
|
||||
rm math.zip
|
||||
```
|
||||
|
||||
5. `LiveCodeBench` dataset.
|
||||
|
||||
```shell
|
||||
cd ais_bench/datasets
|
||||
git lfs install
|
||||
git clone https://huggingface.co/datasets/livecodebench/code_generation_lite
|
||||
```
|
||||
|
||||
6. `AIME 2024` dataset.
|
||||
|
||||
```shell
|
||||
cd ais_bench/datasets
|
||||
mkdir aime/
|
||||
cd aime/
|
||||
wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/aime.zip
|
||||
unzip aime.zip
|
||||
rm aime.zip
|
||||
```
|
||||
|
||||
7. `GSM8K` dataset.
|
||||
|
||||
```shell
|
||||
cd ais_bench/datasets
|
||||
wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/gsm8k.zip
|
||||
unzip gsm8k.zip
|
||||
rm gsm8k.zip
|
||||
```
|
||||
|
||||
#### Configuration
|
||||
|
||||
Update the file `benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`.
|
||||
There are several arguments that you should update according to your environment.
|
||||
|
||||
- `attr`: Identifier for the inference backend type, fixed as `service` (serving-based inference) or `local` (local model).
|
||||
- `type`: Used to select different backend API types.
|
||||
- `abbr`: Unique identifier for a local task, used to distinguish between multiple tasks.
|
||||
- `path`: Update to your model weight path.
|
||||
- `model`: Update to your model name in vLLM.
|
||||
- `host_ip` and `host_port`: Update to your vLLM server ip and port.
|
||||
- `max_out_len`: Note `max_out_len` + LLM input length should be less than `max_model_len` (config in your vllm server), `32768` will be suitable for most datasets.
|
||||
- `batch_size`: Update according to your dataset.
|
||||
- `temperature`: Update inference argument.
|
||||
|
||||
```python
|
||||
from ais_bench.benchmark.models import VLLMCustomAPIChat
|
||||
from ais_bench.benchmark.utils.model_postprocessors import extract_non_reasoning_content
|
||||
|
||||
models = [
|
||||
dict(
|
||||
attr="service",
|
||||
type=VLLMCustomAPIChat,
|
||||
abbr='vllm-api-general-chat',
|
||||
path="xxxx",
|
||||
model="xxxx",
|
||||
request_rate = 0,
|
||||
retry = 2,
|
||||
host_ip = "localhost",
|
||||
host_port = 8000,
|
||||
max_out_len = xxx,
|
||||
batch_size = xxx,
|
||||
trust_remote_code=False,
|
||||
generation_kwargs = dict(
|
||||
temperature = 0.6,
|
||||
top_k = 10,
|
||||
top_p = 0.95,
|
||||
seed = None,
|
||||
repetition_penalty = 1.03,
|
||||
),
|
||||
pred_postprocessor=dict(type=extract_non_reasoning_content)
|
||||
)
|
||||
]
|
||||
|
||||
```
|
||||
|
||||
#### Execute Accuracy Evaluation
|
||||
|
||||
Run the following code to execute different accuracy evaluation.
|
||||
|
||||
```shell
|
||||
# run C-Eval dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
|
||||
|
||||
# run MMLU dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets mmlu_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
|
||||
|
||||
# run GPQA dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets gpqa_gen_0_shot_str.py --mode all --dump-eval-details --merge-ds
|
||||
|
||||
# run MATH-500 dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets math500_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
|
||||
|
||||
# run LiveCodeBench dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets livecodebench_code_generate_lite_gen_0_shot_chat.py --mode all --dump-eval-details --merge-ds
|
||||
|
||||
# run AIME 2024 dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets aime2024_gen_0_shot_chat_prompt.py --mode all --dump-eval-details --merge-ds
|
||||
|
||||
# run GSM8K dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
|
||||
|
||||
```
|
||||
|
||||
After each dataset execution, you can get the result from saved files such as `outputs/default/20250628_151326`, there is an example as follows:
|
||||
|
||||
```shell
|
||||
20250628_151326/
|
||||
├── configs # Combined configuration file for model tasks, dataset tasks, and result presentation tasks
|
||||
│ └── 20250628_151326_29317.py
|
||||
├── logs # Execution logs; if --debug is added to the command, no intermediate logs are saved to disk (all are printed directly to the screen)
|
||||
│ ├── eval
|
||||
│ │ └── vllm-api-general-chat
|
||||
│ │ └── demo_gsm8k.out # Logs of the accuracy evaluation process based on inference results in the predictions/ folder
|
||||
│ └── infer
|
||||
│ └── vllm-api-general-chat
|
||||
│ └── demo_gsm8k.out # Logs of the inference process
|
||||
├── predictions
|
||||
│ └── vllm-api-general-chat
|
||||
│ └── demo_gsm8k.json # Inference results (all outputs returned by the inference service)
|
||||
├── results
|
||||
│ └── vllm-api-general-chat
|
||||
│ └── demo_gsm8k.json # Raw scores calculated from the accuracy evaluation
|
||||
└── summary
|
||||
├── summary_20250628_151326.csv # Final accuracy scores (in table format)
|
||||
├── summary_20250628_151326.md # Final accuracy scores (in Markdown format)
|
||||
└── summary_20250628_151326.txt # Final accuracy scores (in text format)
|
||||
```
|
||||
|
||||
#### Execute Performance Evaluation
|
||||
|
||||
Text-only benchmarks:
|
||||
|
||||
```shell
|
||||
# run C-Eval dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt.py --summarizer default_perf --mode perf
|
||||
|
||||
# run MMLU dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets mmlu_gen_0_shot_cot_chat_prompt.py --summarizer default_perf --mode perf
|
||||
|
||||
# run GPQA dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets gpqa_gen_0_shot_str.py --summarizer default_perf --mode perf
|
||||
|
||||
# run MATH-500 dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets math500_gen_0_shot_cot_chat_prompt.py --summarizer default_perf --mode perf
|
||||
|
||||
# run LiveCodeBench dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets livecodebench_code_generate_lite_gen_0_shot_chat.py --summarizer default_perf --mode perf
|
||||
|
||||
# run AIME 2024 dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets aime2024_gen_0_shot_chat_prompt.py --summarizer default_perf --mode perf
|
||||
|
||||
# run GSM8K dataset
|
||||
ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_str_perf.py --summarizer default_perf --mode perf
|
||||
```
|
||||
|
||||
Multi-modal benchmarks (text + images):
|
||||
|
||||
```shell
|
||||
# run textvqa dataset
|
||||
ais_bench --models vllm_api_stream_chat --datasets textvqa_gen_base64 --summarizer default_perf --mode perf
|
||||
```
|
||||
|
||||
After execution, you can get the result from saved files, there is an example as follows:
|
||||
|
||||
```shell
|
||||
20251031_070226/
|
||||
|-- configs # Combined configuration file for model tasks, dataset tasks, and result presentation tasks
|
||||
| `-- 20251031_070226_122485.py
|
||||
|-- logs
|
||||
| `-- performances
|
||||
| `-- vllm-api-general-chat
|
||||
| `-- cevaldataset.out # Logs of the performance evaluation process
|
||||
`-- performances
|
||||
`-- vllm-api-general-chat
|
||||
|-- cevaldataset.csv # Final performance results (in table format)
|
||||
|-- cevaldataset.json # Final performance results (in json format)
|
||||
|-- cevaldataset_details.h5 # Final performance results in details
|
||||
|-- cevaldataset_details.json # Final performance results in details
|
||||
|-- cevaldataset_plot.html # Final performance results (in html format)
|
||||
`-- cevaldataset_rps_distribution_plot_with_actual_rps.html # Final performance results (in html format)
|
||||
```
|
||||
|
||||
### 3. Troubleshooting
|
||||
|
||||
#### Invalid Image Path Error
|
||||
|
||||
If you download the TextVQA dataset following the AISBench documentation:
|
||||
|
||||
```bash
|
||||
cd ais_bench/datasets
|
||||
git lfs install
|
||||
git clone https://huggingface.co/datasets/maoxx241/textvqa_subset
|
||||
mv textvqa_subset/ textvqa/
|
||||
mkdir textvqa/textvqa_json/
|
||||
mv textvqa/*.json textvqa/textvqa_json/
|
||||
mv textvqa/*.jsonl textvqa/textvqa_json/
|
||||
```
|
||||
|
||||
you may encounter the following error:
|
||||
|
||||
```bash
|
||||
AISBench - ERROR - /vllm-workspace/benchmark/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: HTTP status 400. Server response: {"error":{"message":"1 validation error for ChatCompletionContentPartImageParam\nimage_url\n Input should be a valid dictionary [type=dict_type, input_value='data/textvqa/train_images/b2ae0f96dfbea5d8.jpg', input_type=str]\n For further information visit https://errors.pydantic.dev/2.12/v/dict_type None","type":"BadRequestError","param":null,"code":400}}
|
||||
```
|
||||
|
||||
You need to manually replace the dataset image paths with absolute paths, changing `/path/to/benchmark/ais_bench/datasets/textvqa/train_images/` to the actual absolute directory where the images are stored:
|
||||
|
||||
```bash
|
||||
cd ais_bench/datasets/textvqa/textvqa_json
|
||||
sed -i 's#data/textvqa/train_images/#/path/to/benchmark/ais_bench/datasets/textvqa/train_images/#g' textvqa_val.json
|
||||
```
|
||||
@@ -1,8 +1,8 @@
|
||||
# Using EvalScope
|
||||
|
||||
This document will guide you have model inference stress testing and accuracy testing using [EvalScope](https://github.com/modelscope/evalscope).
|
||||
This document will guide you through model inference stress testing and accuracy testing using [EvalScope](https://github.com/modelscope/evalscope).
|
||||
|
||||
## 1. Online serving
|
||||
## 1. Online server
|
||||
|
||||
You can run docker container to start the vLLM server on a single NPU:
|
||||
|
||||
@@ -13,6 +13,7 @@ export DEVICE=/dev/davinci7
|
||||
# Update the vllm-ascend image
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
docker run --rm \
|
||||
--shm-size=1g \
|
||||
--name vllm-ascend \
|
||||
--device $DEVICE \
|
||||
--device /dev/davinci_manager \
|
||||
@@ -31,30 +32,30 @@ docker run --rm \
|
||||
vllm serve Qwen/Qwen2.5-7B-Instruct --max_model_len 26240
|
||||
```
|
||||
|
||||
If your service start successfully, you can see the info shown below:
|
||||
If the vLLM server is started successfully, you can see information shown below:
|
||||
|
||||
```
|
||||
```shell
|
||||
INFO: Started server process [6873]
|
||||
INFO: Waiting for application startup.
|
||||
INFO: Application startup complete.
|
||||
```
|
||||
|
||||
Once your server is started, you can query the model with input prompts in new terminal:
|
||||
Once your server is started, you can query the model with input prompts in a new terminal:
|
||||
|
||||
```
|
||||
```shell
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen2.5-7B-Instruct",
|
||||
"prompt": "The future of AI is",
|
||||
"max_tokens": 7,
|
||||
"max_completion_tokens": 7,
|
||||
"temperature": 0
|
||||
}'
|
||||
```
|
||||
|
||||
## 2. Install EvalScope using pip
|
||||
|
||||
You can install EvalScope by using:
|
||||
You can install EvalScope as follows:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv-evalscope
|
||||
@@ -62,21 +63,21 @@ source .venv-evalscope/bin/activate
|
||||
pip install gradio plotly evalscope
|
||||
```
|
||||
|
||||
## 3. Run gsm8k accuracy test using EvalScope
|
||||
## 3. Run GSM8K using EvalScope for accuracy testing
|
||||
|
||||
You can `evalscope eval` run gsm8k accuracy test:
|
||||
You can use `evalscope eval` to run GSM8K (a grade-school math benchmark dataset) for accuracy testing:
|
||||
|
||||
```
|
||||
```shell
|
||||
evalscope eval \
|
||||
--model Qwen/Qwen2.5-7B-Instruct \
|
||||
--api-url http://localhost:8000/v1 \
|
||||
--api-key EMPTY \
|
||||
--eval-type service \
|
||||
--eval-type server \
|
||||
--datasets gsm8k \
|
||||
--limit 10
|
||||
```
|
||||
|
||||
After 1-2 mins, the output is as shown below:
|
||||
After 1 to 2 minutes, the output is shown below:
|
||||
|
||||
```shell
|
||||
+---------------------+-----------+-----------------+----------+-------+---------+---------+
|
||||
@@ -86,7 +87,7 @@ After 1-2 mins, the output is as shown below:
|
||||
+---------------------+-----------+-----------------+----------+-------+---------+---------+
|
||||
```
|
||||
|
||||
See more detail in: [EvalScope doc - Model API Service Evaluation](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#model-api-service-evaluation).
|
||||
See more details in [EvalScope doc - Model API Service Evaluation](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#model-api-service-evaluation).
|
||||
|
||||
## 4. Run model inference stress testing using EvalScope
|
||||
|
||||
@@ -98,9 +99,9 @@ pip install evalscope[perf] -U
|
||||
|
||||
### Basic usage
|
||||
|
||||
You can use `evalscope perf` run perf test:
|
||||
You can use `evalscope perf` to run perf testing:
|
||||
|
||||
```
|
||||
```shell
|
||||
evalscope perf \
|
||||
--url "http://localhost:8000/v1/chat/completions" \
|
||||
--parallel 5 \
|
||||
@@ -113,7 +114,7 @@ evalscope perf \
|
||||
|
||||
### Output results
|
||||
|
||||
After 1-2 mins, the output is as shown below:
|
||||
After 1 to 2 minutes, the output is shown below:
|
||||
|
||||
```shell
|
||||
Benchmarking summary:
|
||||
@@ -172,4 +173,4 @@ Percentile results:
|
||||
+------------+----------+---------+-------------+--------------+---------------+----------------------+
|
||||
```
|
||||
|
||||
See more detail in: [EvalScope doc - Model Inference Stress Testing](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#basic-usage).
|
||||
See more detail in [EvalScope doc - Model Inference Stress Testing](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#basic-usage).
|
||||
|
||||
@@ -1,9 +1,12 @@
|
||||
# Using lm-eval
|
||||
This document will guide you have a accuracy testing using [lm-eval][1].
|
||||
|
||||
This document guides you to conduct accuracy testing using [lm-eval][1].
|
||||
|
||||
## Online Server
|
||||
### 1. start the vLLM server
|
||||
You can run docker container to start the vLLM server on a single NPU:
|
||||
|
||||
### 1. Start the vLLM server
|
||||
|
||||
You can run a docker container to start the vLLM server on a single NPU:
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
@@ -13,6 +16,7 @@ export DEVICE=/dev/davinci7
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--shm-size=1g \
|
||||
--device $DEVICE \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
@@ -31,46 +35,54 @@ docker run --rm \
|
||||
vllm serve Qwen/Qwen2.5-0.5B-Instruct --max_model_len 4096 &
|
||||
```
|
||||
|
||||
Started the vLLM server successfully,if you see log as below:
|
||||
The vLLM server is started successfully, if you see logs as below:
|
||||
|
||||
```
|
||||
```shell
|
||||
INFO: Started server process [9446]
|
||||
INFO: Waiting for application startup.
|
||||
INFO: Application startup complete.
|
||||
```
|
||||
|
||||
### 2. Run gsm8k accuracy test using lm-eval
|
||||
### 2. Run GSM8K using the vLLM server (curl) and then run lm-eval for accuracy testing
|
||||
|
||||
You can query result with input prompts:
|
||||
You can query the result with input prompts:
|
||||
|
||||
```shell
|
||||
PROMPT='<|im_start|>system
|
||||
You are a professional accountant. Answer questions using accounting knowledge, output only the option letter (A/B/C/D).<|im_end|>
|
||||
<|im_start|>user
|
||||
Question: A company'"'"'s balance sheet as of December 31, 2023 shows:
|
||||
Current assets: Cash and equivalents 5 million yuan, Accounts receivable 8 million yuan, Inventory 6 million yuan
|
||||
Non-current assets: Net fixed assets 12 million yuan
|
||||
Current liabilities: Short-term loans 4 million yuan, Accounts payable 3 million yuan
|
||||
Non-current liabilities: Long-term loans 9 million yuan
|
||||
Owner'"'"'s equity: Paid-in capital 10 million yuan, Retained earnings ?
|
||||
Requirement: Calculate the company'"'"'s Asset-Liability Ratio and Current Ratio (round to two decimal places).
|
||||
Options:
|
||||
A. Asset-Liability Ratio=58.33%, Current Ratio=1.90
|
||||
B. Asset-Liability Ratio=62.50%, Current Ratio=2.17
|
||||
C. Asset-Liability Ratio=65.22%, Current Ratio=1.75
|
||||
D. Asset-Liability Ratio=68.00%, Current Ratio=2.50<|im_end|>
|
||||
<|im_start|>assistant
|
||||
'
|
||||
|
||||
```
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"prompt": "'"<|im_start|>system\nYou are a professional accountant. Answer questions using accounting knowledge, output only the option letter (A/B/C/D).<|im_end|>\n"\
|
||||
"<|im_start|>user\nQuestion: A company's balance sheet as of December 31, 2023 shows:\n"\
|
||||
" Current assets: Cash and equivalents 5 million yuan, Accounts receivable 8 million yuan, Inventory 6 million yuan\n"\
|
||||
" Non-current assets: Net fixed assets 12 million yuan\n"\
|
||||
" Current liabilities: Short-term loans 4 million yuan, Accounts payable 3 million yuan\n"\
|
||||
" Non-current liabilities: Long-term loans 9 million yuan\n"\
|
||||
" Owner's equity: Paid-in capital 10 million yuan, Retained earnings ?\n"\
|
||||
"Requirement: Calculate the company's Asset-Liability Ratio and Current Ratio (round to two decimal places).\n"\
|
||||
"Options:\n"\
|
||||
"A. Asset-Liability Ratio=58.33%, Current Ratio=1.90\n"\
|
||||
"B. Asset-Liability Ratio=62.50%, Current Ratio=2.17\n"\
|
||||
"C. Asset-Liability Ratio=65.22%, Current Ratio=1.75\n"\
|
||||
"D. Asset-Liability Ratio=68.00%, Current Ratio=2.50<|im_end|>\n"\
|
||||
"<|im_start|>assistant\n"'",
|
||||
"max_tokens": 1,
|
||||
"temperature": 0,
|
||||
"stop": ["<|im_end|>"]
|
||||
}' | python3 -m json.tool
|
||||
-d "$(jq -n \
|
||||
--arg model "Qwen/Qwen2.5-0.5B-Instruct" \
|
||||
--arg prompt "$PROMPT" \
|
||||
'{
|
||||
model: $model,
|
||||
prompt: $prompt,
|
||||
max_completion_tokens: 1,
|
||||
temperature: 0,
|
||||
stop: ["<|im_end|>"]
|
||||
}')" | python3 -m json.tool
|
||||
```
|
||||
|
||||
The output format matches the following:
|
||||
|
||||
```
|
||||
```json
|
||||
{
|
||||
"id": "cmpl-2f678e8bdf5a4b209a3f2c1fa5832e25",
|
||||
"object": "text_completion",
|
||||
@@ -98,16 +110,24 @@ The output format matches the following:
|
||||
}
|
||||
```
|
||||
|
||||
Install lm-eval in the container.
|
||||
Install lm-eval in the container:
|
||||
|
||||
```bash
|
||||
export HF_ENDPOINT="https://hf-mirror.com"
|
||||
export USE_MODELSCOPE_HUB=0
|
||||
pip install lm-eval[api]
|
||||
```
|
||||
|
||||
:::{note}
|
||||
The Docker container is launched with `VLLM_USE_MODELSCOPE=True`, which may
|
||||
cause lm-eval to download datasets from ModelScope instead of HuggingFace.
|
||||
Setting `USE_MODELSCOPE_HUB=0` disables this behavior so that lm-eval can
|
||||
fetch datasets from HuggingFace correctly.
|
||||
:::
|
||||
|
||||
Run the following command:
|
||||
|
||||
```
|
||||
```shell
|
||||
# Only test gsm8k dataset in this demo
|
||||
lm_eval \
|
||||
--model local-completions \
|
||||
@@ -116,19 +136,20 @@ lm_eval \
|
||||
--output_path ./
|
||||
```
|
||||
|
||||
After 30 mins, the output is as shown below:
|
||||
After 30 minutes, the output is as shown below:
|
||||
|
||||
```
|
||||
The markdown format results is as below:
|
||||
```shell
|
||||
The results in Markdown format are as follows:
|
||||
|
||||
Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|
||||
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|
||||
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|
||||
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3215|± |0.0129|
|
||||
| | |strict-match | 5|exact_match|↑ |0.2077|± |0.0112|
|
||||
|gsm8k| 3|strict-match | 5|exact_match|↑ |0.2077|± |0.0112|
|
||||
|
||||
```
|
||||
|
||||
## Offline Server
|
||||
|
||||
### 1. Run docker container
|
||||
|
||||
You can run docker container on a single NPU:
|
||||
@@ -141,6 +162,7 @@ export DEVICE=/dev/davinci7
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--shm-size=1g \
|
||||
--device $DEVICE \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
@@ -158,17 +180,26 @@ docker run --rm \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
### 2. Run gsm8k accuracy test using lm-eval
|
||||
Install lm-eval in the container.
|
||||
### 2. Run GSM8K using lm-eval for accuracy testing
|
||||
|
||||
Install lm-eval in the container:
|
||||
|
||||
```bash
|
||||
export HF_ENDPOINT="https://hf-mirror.com"
|
||||
export USE_MODELSCOPE_HUB=0
|
||||
pip install lm-eval
|
||||
```
|
||||
|
||||
:::{note}
|
||||
The Docker container is launched with `VLLM_USE_MODELSCOPE=True`, which may
|
||||
cause lm-eval to download datasets from ModelScope instead of HuggingFace.
|
||||
Setting `USE_MODELSCOPE_HUB=0` disables this behavior so that lm-eval can
|
||||
fetch datasets from HuggingFace correctly.
|
||||
:::
|
||||
|
||||
Run the following command:
|
||||
|
||||
```
|
||||
```shell
|
||||
# Only test gsm8k dataset in this demo
|
||||
lm_eval \
|
||||
--model vllm \
|
||||
@@ -177,21 +208,21 @@ lm_eval \
|
||||
--batch_size auto
|
||||
```
|
||||
|
||||
After 1-2 mins, the output is as shown below:
|
||||
After 1 to 2 minutes, the output is shown below:
|
||||
|
||||
```
|
||||
The markdown format results is as below:
|
||||
```shell
|
||||
The markdown format results are as below:
|
||||
|
||||
Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|
||||
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|
||||
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|
||||
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3412|± |0.0131|
|
||||
| | |strict-match | 5|exact_match|↑ |0.3139|± |0.0128|
|
||||
|gsm8k| 3|strict-match | 5|exact_match|↑ |0.3139|± |0.0128|
|
||||
|
||||
```
|
||||
|
||||
## Use offline Datasets
|
||||
## Use Offline Datasets
|
||||
|
||||
Take gsm8k(single dataset) and mmlu(multi-subject dataset) as examples, and you can see more from [here][2].
|
||||
Take GSM8K (single dataset) and MMLU (multi-subject dataset) as examples, and you can see more from [using-local-datasets][2].
|
||||
|
||||
```bash
|
||||
# set HF_DATASETS_OFFLINE when using offline datasets
|
||||
@@ -205,7 +236,7 @@ cd lm_eval/tasks/gsm8k
|
||||
cd lm_eval/tasks/mmlu/default
|
||||
```
|
||||
|
||||
set [gsm8k.yaml][3] as follows:
|
||||
Set [gsm8k.yaml][3] as follows:
|
||||
|
||||
```yaml
|
||||
tag:
|
||||
@@ -230,7 +261,7 @@ training_split: train
|
||||
fewshot_split: train
|
||||
test_split: test
|
||||
doc_to_text: 'Q: {{question}}
|
||||
A(Please follow the summarize the result at the end with the format of "The answer is xxx", where xx is the result.):'
|
||||
A(Please follow the summarized result at the end with the format of "The answer is xxx", where xx is the result.):'
|
||||
doc_to_target: "{{answer}}" #" {{answer.split('### ')[-1].rstrip()}}"
|
||||
metric_list:
|
||||
- metric: exact_match
|
||||
@@ -268,7 +299,7 @@ metadata:
|
||||
version: 3.0
|
||||
```
|
||||
|
||||
set [_default_template_yaml][4] as follows:
|
||||
Set [_default_template_yaml][4] as follows:
|
||||
|
||||
```yaml
|
||||
# set dataset_path according to the downloaded dataset
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
# Using OpenCompass
|
||||
This document will guide you have a accuracy testing using [OpenCompass](https://github.com/open-compass/opencompass).
|
||||
|
||||
## 1. Online Serving
|
||||
This document guides you to conduct accuracy testing using [OpenCompass](https://github.com/open-compass/opencompass).
|
||||
|
||||
You can run docker container to start the vLLM server on a single NPU:
|
||||
## 1. Online Server
|
||||
|
||||
You can run a docker container to start the vLLM server on a single NPU:
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
@@ -13,6 +14,7 @@ export DEVICE=/dev/davinci7
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--shm-size=1g \
|
||||
--device $DEVICE \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
@@ -30,29 +32,30 @@ docker run --rm \
|
||||
vllm serve Qwen/Qwen2.5-7B-Instruct --max_model_len 26240
|
||||
```
|
||||
|
||||
If your service start successfully, you can see the info shown below:
|
||||
The vLLM server is started successfully, if you see information as below:
|
||||
|
||||
```
|
||||
```shell
|
||||
INFO: Started server process [6873]
|
||||
INFO: Waiting for application startup.
|
||||
INFO: Application startup complete.
|
||||
```
|
||||
|
||||
Once your server is started, you can query the model with input prompts in new terminal:
|
||||
Once your server is started, you can query the model with input prompts in a new terminal.
|
||||
|
||||
```
|
||||
```shell
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen2.5-7B-Instruct",
|
||||
"prompt": "The future of AI is",
|
||||
"max_tokens": 7,
|
||||
"max_completion_tokens": 7,
|
||||
"temperature": 0
|
||||
}'
|
||||
```
|
||||
|
||||
## 2. Run ceval accuracy test using OpenCompass
|
||||
Install OpenCompass and configure the environment variables in the container.
|
||||
## 2. Run C-Eval (a Chinese language model evaluation benchmark) using OpenCompass for accuracy testing
|
||||
|
||||
Install OpenCompass and configure the environment variables in the container:
|
||||
|
||||
```bash
|
||||
# Pin Python 3.10 due to:
|
||||
@@ -64,7 +67,7 @@ export DATASET_SOURCE=ModelScope
|
||||
git clone https://github.com/open-compass/opencompass.git
|
||||
```
|
||||
|
||||
Add `opencompass/configs/eval_vllm_ascend_demo.py` with the following content:
|
||||
Add the following content to `opencompass/configs/eval_vllm_ascend_demo.py`:
|
||||
|
||||
```python
|
||||
from mmengine.config import read_base
|
||||
@@ -106,14 +109,14 @@ models = [
|
||||
|
||||
Run the following command:
|
||||
|
||||
```
|
||||
```shell
|
||||
python3 run.py opencompass/configs/eval_vllm_ascend_demo.py --debug
|
||||
```
|
||||
|
||||
After 1-2 mins, the output is as shown below:
|
||||
After 1 to 2 minutes, the output is shown below:
|
||||
|
||||
```
|
||||
The markdown format results is as below:
|
||||
```shell
|
||||
The markdown format results are as below:
|
||||
|
||||
| dataset | version | metric | mode | Qwen2.5-7B-Instruct-vLLM-API |
|
||||
|----- | ----- | ----- | ----- | -----|
|
||||
|
||||
10
docs/source/developer_guide/performance_and_debug/index.md
Normal file
10
docs/source/developer_guide/performance_and_debug/index.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# Performance and Debug
|
||||
|
||||
::::{toctree}
|
||||
:caption: Performance and Debug
|
||||
:maxdepth: 1
|
||||
performance_benchmark
|
||||
optimization_and_tuning
|
||||
service_profiling_guide
|
||||
msprobe_guide
|
||||
::::
|
||||
@@ -0,0 +1,450 @@
|
||||
# MSProbe Debugging Guide
|
||||
|
||||
During inference or training runs we often encounter accuracy anomalies such as outputs drifting away from the expectation, unstable numerical behavior (NaN/Inf), or predictions that no longer match the labels. To pinpoint the root cause we have to monitor and capture intermediate data produced while the model executes—feature maps, weights, activations, and layer outputs. By capturing key tensors at specific stages, logging I/O pairs for the core layers, and retaining contextual metadata (prompts, tensor dtypes, hardware configuration, etc.), we can systematically trace where the accuracy degradation or numerical error started. This guide describes the end-to-end workflow for diagnosing accuracy issues for AI models (with a focus on vllm-ascend services): preparation, data capture, and analysis & verification.
|
||||
|
||||
For more details, see [Ascend/msprobe](https://gitcode.com/Ascend/msprobe).
|
||||
|
||||
## 0. Background Concepts
|
||||
|
||||
`msprobe` supports three accuracy levels:
|
||||
|
||||
- **L0**: dumps tensors at the module level and generates `construct.json` so that visualization tools can rebuild the network structure. A model or submodule handle must be passed in.
|
||||
- **L1**: collects operator-level statistics only, which is suitable for lightweight troubleshooting.
|
||||
- **mix**: captures both structural information and operator statistics, which is useful when you need both graph reconstruction and numerical comparisons.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
### 1.1 Install `msprobe`
|
||||
|
||||
Install msprobe with pip:
|
||||
|
||||
```bash
|
||||
pip install mindstudio-probe
|
||||
```
|
||||
|
||||
### 1.2 Graph mode dump (optional)
|
||||
|
||||
If you need to dump cudagraph graphs, you need to install from source code:
|
||||
|
||||
1. Install `aclgraph_dump` from source code:
|
||||
|
||||
```bash
|
||||
git clone https://gitcode.com/Ascend/msprobe.git
|
||||
cd msprobe
|
||||
python3 setup.py bdist_wheel --include-mod=aclgraph_dump --no-check
|
||||
pip install dist/*.whl
|
||||
```
|
||||
|
||||
## 2. Collecting Data with `msprobe`
|
||||
|
||||
We generally follow a coarse-to-fine strategy when capturing data. First, identify the token where the issue shows up, and then decide which range needs to be sampled around that token. The typical workflow is described below.
|
||||
|
||||
### 2.1 Prepare the dump configuration content
|
||||
|
||||
Prepare configuration content that can be parsed by `PrecisionDebugger`. You can use either of the following ways:
|
||||
|
||||
- Pass the config object directly through `--additional-config.dump_config`.
|
||||
- Pass a config file path through `--additional-config.dump_config_path`.
|
||||
|
||||
Common fields are:
|
||||
|
||||
| Field | Description | Required | Eager Mode | Graph Mode |
|
||||
|:-----------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|:-------------:|:-------------:|
|
||||
| `task` | Type of dump task. Common PyTorch values include `"statistics"` and `"tensor"`. A statistics task collects tensor statistics (mean, variance, max, min, etc.) while a tensor task captures arbitrary tensors. | Yes | ✅ | ✅ |
|
||||
| `dump_path` | Directory where dump results are stored. When omitted, `msprobe` uses its default path. | No | ✅ | ✅ |
|
||||
| `rank` | Ranks to sample. An empty list collects every rank. For single-card tasks, you must set this field to `[]`. | No | ✅ | ✅ |
|
||||
| `step` | Token iteration(s) to sample. An empty list means every iteration. | No | ✅ | ❌ |
|
||||
| `level` | Dump level string (`"L0"`, `"L1"`, or `"mix"`). `L0` targets `nn.Module`, `L1` targets `torch.api`, and `mix` collects both. | Yes | ✅ | ✅ |
|
||||
| `async_dump`| Whether to enable asynchronous dump (supported for PyTorch `statistics`/`tensor` tasks). Defaults to `false`. | No | ✅ | ❌ |
|
||||
| `scope` | Module range to sample. An empty list collects every module. | No | ✅ | ❌ |
|
||||
| `dump_enable` | Dynamic switch for enabling/disabling dump in `PrecisionDebugger` during one running training/inference job. This allows turning dump on or off on demand in the same job. | No | ✅ | ❌ |
|
||||
| `list` | Operator range to sample. An empty list collects every operator. | No | ✅ | ✅ |
|
||||
|
||||
To restrict the operators that are captured, configure the `list` block:
|
||||
|
||||
- `scope` (list[str]): In PyTorch PyNative scenarios this field restricts the dump range. Provide two module or API names that follow the tool's naming convention to lock a range; only data between the two names will be dumped. Examples:
|
||||
|
||||
```json
|
||||
"scope": ["Module.conv1.Conv2d.forward.0", "Module.fc2.Linear.forward.0"]
|
||||
"scope": ["Cell.conv1.Conv2d.forward.0", "Cell.fc2.Dense.forward.0"]
|
||||
"scope": ["Tensor.add.0.forward", "Functional.square.2.forward"]
|
||||
```
|
||||
|
||||
The `level` setting determines what can be provided—modules when `level=L0`, APIs when `level=L1`, and either modules or APIs when `level=mix`.
|
||||
|
||||
- `list` (list[str]): Custom operator list. Options include:
|
||||
- Supply the full names of specific APIs in PyTorch pynative scenarios to only dump those APIs. Example: `"list": ["Tensor.permute.1.forward", "Tensor.transpose.2.forward", "Torch.relu.3.forward"]`.
|
||||
- When `level=mix`, you can provide module names so that the dump expands to everything produced while the module is running. Example: `"list": ["Module.module.language_model.encoder.layers.0.mlp.ParallelMlp.forward.0"]`.
|
||||
- Provide a substring such as `"list": ["relu"]` to dump every API whose name contains the substring. When `level=mix`, modules whose names contain the substring are also expanded.
|
||||
|
||||
Example configuration:
|
||||
eager mode:
|
||||
|
||||
```json
|
||||
{
|
||||
"task": "statistics",
|
||||
"dump_path": "/home/data_dump",
|
||||
"rank": [],
|
||||
"step": [],
|
||||
"level": "L1",
|
||||
"async_dump": false,
|
||||
|
||||
"statistics": {
|
||||
"scope": [],
|
||||
"list": [],
|
||||
"tensor_list": [],
|
||||
"data_mode": ["all"],
|
||||
"summary_mode": "statistics"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Graph mode:
|
||||
|
||||
```json
|
||||
{
|
||||
"task": "statistics",
|
||||
"level": "L1",
|
||||
"dump_path": "/home/data_dump",
|
||||
"statistics": {
|
||||
"list": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 3. Enable `msprobe` in vllm-ascend
|
||||
|
||||
1. Start vLLM and pass the dump config content through `--additional-config`:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-0.5B-Instruct \
|
||||
--dtype bfloat16 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--additional-config '{
|
||||
"dump_config": {
|
||||
"task": "statistics",
|
||||
"level": "L1",
|
||||
"dump_path": "/data/msprobe_dump",
|
||||
"statistics": {
|
||||
"list": []
|
||||
}
|
||||
}
|
||||
}' &
|
||||
```
|
||||
|
||||
Compatibility mode (legacy) is still supported:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-0.5B-Instruct \
|
||||
--dtype bfloat16 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--additional-config '{"dump_config_path": "/data/msprobe_config.json"}' &
|
||||
```
|
||||
|
||||
## 4. Send requests and collect dumps
|
||||
|
||||
1. Send inference requests as usual, for example:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"prompt": "Explain gravity in one sentence.",
|
||||
"max_completion_tokens": 32,
|
||||
"temperature": 0
|
||||
}' | python -m json.tool
|
||||
```
|
||||
|
||||
2. Each request drives the sequence `msprobe: start -> forward -> stop -> step`. The runner invokes `step()` on every code path, so you always get a complete dataset even if inference returns early.
|
||||
|
||||
3. Dump files are written into `dump_path`. They usually contain:
|
||||
- Tensor files grouped by operator/module.
|
||||
- `dump.json`, which records metadata such as dtype, shape, min/max, and `requires_grad`.
|
||||
- `construct.json`, which is generated when `level` is `L0` or `mix` (required for visualization).
|
||||
|
||||
Example directory layout:
|
||||
eager mode:
|
||||
|
||||
```text
|
||||
├── dump_path
|
||||
│ ├── step0
|
||||
│ │ ├── rank0
|
||||
│ │ │ ├── dump_tensor_data
|
||||
│ │ │ │ ├── Tensor.permute.1.forward.pt # Format: {api_type}.{api_name}.{call_count}.forward.{input/output}.{arg_index}.
|
||||
│ │ │ │ │ # arg_index is the nth input or output of the API. If an input is a list, keep numbering with decimals (e.g., 1.1 is the first element of the first argument).
|
||||
│ │ │ │ ├── Module.conv1.Conv2d.forward.0.input.0.pt # Format: {Module}.{module_name}.{class_name}.forward.{call_count}.{input/output}.{arg_index}.
|
||||
│ │ │ │ └── Module.conv1.Conv2d.forward.0.parameters.bias.pt # Module parameter data: {Module}.{module_name}.{class_name}.forward.{call_count}.parameters.{parameter_name}.
|
||||
│ │ │ │ # When the `model` argument passed to dump is a List[torch.nn.Module] or Tuple[torch.nn.Module], module-level data names also include the index inside the list ({Module}.{index}.*), e.g., Module.0.conv1.Conv2d.forward.0.input.0.pt.
|
||||
│ │ │ ├── dump.json
|
||||
│ │ │ ├── stack.json
|
||||
│ │ │ ├── dump_error_info.log
|
||||
│ │ │ └── construct.json
|
||||
│ │ ├── rank1
|
||||
│ │ │ ├── dump_tensor_data
|
||||
│ │ │ │ └── ...
|
||||
│ │ │ ├── dump.json
|
||||
│ │ │ ├── stack.json
|
||||
│ │ │ ├── dump_error_info.log
|
||||
│ │ │ └── construct.json
|
||||
│ │ ├── ...
|
||||
│ │ │
|
||||
│ │ └── rank7
|
||||
│ ├── step1
|
||||
│ │ ├── ...
|
||||
│ ├── step2
|
||||
```
|
||||
|
||||
- `rank`: Device ID. Each card writes its data to the corresponding `rank{ID}` directory. In non-distributed scenarios the directory is simply named `rank`.
|
||||
- `dump_tensor_data`: Tensor payloads that were collected.
|
||||
- `dump.json`: Statistics for the forward data of each API or module, including names, dtype, shape, max, min, mean, L2 norm (square root of the L2 variance), and CRC-32 when `summary_mode="md5"`. See [dump.json file description](#dumpjson-file-description) for details.
|
||||
- `dump_error_info.log`: Present only when the dump tool encountered an error and records the failure log.
|
||||
- `stack.json`: Call stacks for APIs/modules.
|
||||
- `construct.json`: Hierarchical structure description. Empty when `level=L1`.
|
||||
|
||||
graph mode:
|
||||
|
||||
```text
|
||||
L0_dump
|
||||
├── step0
|
||||
│ └── rank0
|
||||
│ └── dump.json
|
||||
├── step1
|
||||
│ └── rank0
|
||||
│ └── dump.json
|
||||
├── step2
|
||||
│ └── rank0
|
||||
│ └── dump.json
|
||||
├── step3
|
||||
│ └── rank0
|
||||
│ └── dump.json
|
||||
├── step4
|
||||
│ └── rank0
|
||||
│ └── dump.json
|
||||
└── step5
|
||||
└── rank0
|
||||
└── dump.json
|
||||
```
|
||||
|
||||
- `dump.json`: See [dump.json file description](#dumpjson-file-description) for details.
|
||||
|
||||
## 5. Analyze the results
|
||||
|
||||
### 5.1 Prerequisites
|
||||
|
||||
You typically need two dump datasets: one from the "problem side" (the run that exposes the accuracy or numerical error) and another from the "benchmark side" (a good baseline). These datasets do not have to be identical—they can come from different branches, framework versions, or even alternative implementations (operator substitutions, different graph-optimization switches, etc.). As long as they use the same or similar inputs, hardware topology, and sampling points (step/token), `msprobe` can compare them and locate the divergent nodes. If you cannot find a perfectly clean benchmark, start by capturing the problem-side data, craft the smallest reproducible case by hand, and perform a self-comparison. Below we assume the problem dump is `problem_dump` and the benchmark dump is `bench_dump`.
|
||||
|
||||
### 5.2 Visualization
|
||||
|
||||
Use `msprobe graph_visualize` to build or compare graphs, then open the generated `*.vis.db` file(s) with TensorBoard (`tb_graph_ascend` plugin).
|
||||
|
||||
1. Ensure dump data is visualization-ready:
|
||||
- Dump level must be `L0` or `mix` so `construct.json` is non-empty.
|
||||
- Each rank directory should contain `dump.json`, `stack.json`, and `construct.json`.
|
||||
|
||||
2. Choose command mode:
|
||||
- Single-graph build:
|
||||
|
||||
```bash
|
||||
msprobe graph_visualize -tp <target_path> -o <output_path>
|
||||
```
|
||||
|
||||
- Graph comparison:
|
||||
|
||||
```bash
|
||||
msprobe graph_visualize -tp <target_path> -gp <golden_path> -o <output_path>
|
||||
```
|
||||
|
||||
- Common optional flags:
|
||||
- `-oc` / `--overflow_check`: enable overflow marking
|
||||
- `-fm` / `--fuzzy_match`: enable fuzzy matching for node mapping
|
||||
- `-lm` / `--layer_mapping [mapping.yaml]`: cross-framework/layer mapping compare
|
||||
- `-tensor_log`: print per-node compare log (tensor dump scenarios)
|
||||
- `-progress_log`: print detailed progress log
|
||||
|
||||
3. Path granularity is auto-detected by `graph_visualize`:
|
||||
- Single-rank: `.../step0/rank0`
|
||||
- Multi-rank (batch): `.../step0`
|
||||
- Multi-step (batch): dump root path containing `step*`
|
||||
|
||||
4. Output files:
|
||||
- Single-graph build: `build_{timestamp}.vis.db`
|
||||
- Graph comparison: `compare_{timestamp}.vis.db`
|
||||
|
||||
5. Launch TensorBoard with the output directory:
|
||||
|
||||
```bash
|
||||
tensorboard --logdir <output_path> --bind_all --port <optional_port>
|
||||
```
|
||||
|
||||
6. In the visualization UI, inspect structure and numeric differences:
|
||||
- Switch rank/step to locate unstable nodes quickly.
|
||||
- Use search/filter to focus on target ops/modules.
|
||||
- For compare mode, prioritize highlighted high-difference nodes and trace surrounding I/O/parameters.
|
||||
|
||||
## 6. Troubleshooting
|
||||
|
||||
- `RuntimeError: Please enforce eager mode`: Restart vLLM and add the `--enforce-eager` flag.
|
||||
- No dump files: Confirm that the JSON path is correct and every node has write permission. In distributed scenarios set `keep_all_ranks` so that every rank writes its own dump.
|
||||
- Dumps are too large: Start with a `statistics` task to locate abnormal tensors, then narrow the scope with `scope`/`list`/`tensor_list`, `filters`, `token_range`, etc.
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### dump.json file description
|
||||
|
||||
#### L0 level
|
||||
|
||||
An L0 `dump.json` contains forward I/O for modules together with parameters. Using PyTorch's `Conv2d` as an example, the network code looks like:
|
||||
|
||||
`output = self.conv2(input) # self.conv2 = torch.nn.Conv2d(64, 128, 5, padding=2, bias=True)`
|
||||
|
||||
`dump.json` contains the following entries:
|
||||
|
||||
- `Module.conv2.Conv2d.forward.0`: Forward data of the module. `input_args` represents positional inputs, `input_kwargs` represents keyword inputs, `output` stores forward outputs, and `parameters` stores weights/biases.
|
||||
|
||||
**Note**: When the `model` parameter passed to the dump API is `List[torch.nn.Module]` or `Tuple[torch.nn.Module]`, module-level names include the index inside the list (`{Module}.{index}.*`). Example: `Module.0.conv1.Conv2d.forward.0`.
|
||||
|
||||
```json
|
||||
{
|
||||
"task": "tensor",
|
||||
"level": "L0",
|
||||
"framework": "pytorch",
|
||||
"dump_data_dir": "/dump/path",
|
||||
"data": {
|
||||
"Module.conv2.Conv2d.forward.0": {
|
||||
"input_args": [
|
||||
{
|
||||
"type": "torch.Tensor",
|
||||
"dtype": "torch.float32",
|
||||
"shape": [
|
||||
8,
|
||||
16,
|
||||
14,
|
||||
14
|
||||
],
|
||||
"Max": 1.638758659362793,
|
||||
"Min": 0.0,
|
||||
"Mean": 0.2544615864753723,
|
||||
"Norm": 70.50277709960938,
|
||||
"requires_grad": true,
|
||||
"data_name": "Module.conv2.Conv2d.forward.0.input.0.pt"
|
||||
}
|
||||
],
|
||||
"input_kwargs": {},
|
||||
"output": [
|
||||
{
|
||||
"type": "torch.Tensor",
|
||||
"dtype": "torch.float32",
|
||||
"shape": [
|
||||
8,
|
||||
32,
|
||||
10,
|
||||
10
|
||||
],
|
||||
"Max": 1.6815717220306396,
|
||||
"Min": -1.5120246410369873,
|
||||
"Mean": -0.025344856083393097,
|
||||
"Norm": 149.65576171875,
|
||||
"requires_grad": true,
|
||||
"data_name": "Module.conv2.Conv2d.forward.0.output.0.pt"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"weight": {
|
||||
"type": "torch.Tensor",
|
||||
"dtype": "torch.float32",
|
||||
"shape": [
|
||||
32,
|
||||
16,
|
||||
5,
|
||||
5
|
||||
],
|
||||
"Max": 0.05992485210299492,
|
||||
"Min": -0.05999220535159111,
|
||||
"Mean": -0.0006165213999338448,
|
||||
"Norm": 3.421217441558838,
|
||||
"requires_grad": true,
|
||||
"data_name": "Module.conv2.Conv2d.forward.0.parameters.weight.pt"
|
||||
},
|
||||
"bias": {
|
||||
"type": "torch.Tensor",
|
||||
"dtype": "torch.float32",
|
||||
"shape": [
|
||||
32
|
||||
],
|
||||
"Max": 0.05744686722755432,
|
||||
"Min": -0.04894155263900757,
|
||||
"Mean": 0.006410328671336174,
|
||||
"Norm": 0.17263513803482056,
|
||||
"requires_grad": true,
|
||||
"data_name": "Module.conv2.Conv2d.forward.0.parameters.bias.pt"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### L1 level
|
||||
|
||||
An L1 `dump.json` records forward I/O for APIs. Using PyTorch's `relu` function as an example (`output = torch.nn.functional.relu(input)`), the file contains:
|
||||
|
||||
- `Functional.relu.0.forward`: Forward data of the API. `input_args` are positional inputs, `input_kwargs` are keyword inputs, and `output` stores the forward outputs.
|
||||
|
||||
```json
|
||||
{
|
||||
"task": "tensor",
|
||||
"level": "L1",
|
||||
"framework": "pytorch",
|
||||
"dump_data_dir":"/dump/path",
|
||||
"data": {
|
||||
"Functional.relu.0.forward": {
|
||||
"input_args": [
|
||||
{
|
||||
"type": "torch.Tensor",
|
||||
"dtype": "torch.float32",
|
||||
"shape": [
|
||||
32,
|
||||
16,
|
||||
28,
|
||||
28
|
||||
],
|
||||
"Max": 1.3864083290100098,
|
||||
"Min": -1.3364859819412231,
|
||||
"Mean": 0.03711778670549393,
|
||||
"Norm": 236.20692443847656,
|
||||
"requires_grad": true,
|
||||
"data_name": "Functional.relu.0.forward.input.0.pt"
|
||||
}
|
||||
],
|
||||
"input_kwargs": {},
|
||||
"output": [
|
||||
{
|
||||
"type": "torch.Tensor",
|
||||
"dtype": "torch.float32",
|
||||
"shape": [
|
||||
32,
|
||||
16,
|
||||
28,
|
||||
28
|
||||
],
|
||||
"Max": 1.3864083290100098,
|
||||
"Min": 0.0,
|
||||
"Mean": 0.16849493980407715,
|
||||
"Norm": 175.23345947265625,
|
||||
"requires_grad": true,
|
||||
"data_name": "Functional.relu.0.forward.output.0.pt"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### mix level
|
||||
|
||||
A `mix` dump.json contains both L0 and L1 level data; the file format is the same as the examples above.
|
||||
@@ -0,0 +1,248 @@
|
||||
# Optimization and Tuning
|
||||
|
||||
This guide aims to help users improve vLLM Ascend performance at the system level. It includes OS configuration, library optimization, deployment guide, and so on. Any feedback is welcome.
|
||||
|
||||
## Preparation
|
||||
|
||||
### 1.Run the container
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Update DEVICE according to your device (/dev/davinci[0-7])
|
||||
export DEVICE=/dev/davinci0
|
||||
# Update the cann base image
|
||||
export IMAGE=m.daocloud.io/quay.io/ascend/cann:|cann_image_tag|
|
||||
docker run --rm \
|
||||
--name performance-test \
|
||||
--shm-size=1g \
|
||||
--device $DEVICE \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
--device /dev/hisi_hdc \
|
||||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||||
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /root/.cache:/root/.cache \
|
||||
-it $IMAGE bash
|
||||
```
|
||||
|
||||
### 2.Configure your environment
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Configure the mirror
|
||||
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy main restricted universe multiverse" > /etc/apt/sources.list && \
|
||||
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy main restricted universe multiverse" >> /etc/apt/sources.list && \
|
||||
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
|
||||
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
|
||||
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
|
||||
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
|
||||
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list && \
|
||||
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list
|
||||
|
||||
# Install os packages
|
||||
apt update && apt install wget gcc g++ libnuma-dev git vim -y
|
||||
```
|
||||
|
||||
### 3.Install vLLM and vLLM Ascend
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Install necessary dependencies
|
||||
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
pip install modelscope pandas datasets gevent sacrebleu rouge_score pybind11 pytest
|
||||
|
||||
# Configure this var to speed up model download
|
||||
export VLLM_USE_MODELSCOPE=True
|
||||
```
|
||||
|
||||
Please follow the [Installation Guide](https://docs.vllm.ai/projects/ascend/en/latest/installation.html) to make sure vLLM and vLLM Ascend are installed correctly.
|
||||
|
||||
:::{note}
|
||||
Make sure your vLLM and vLLM Ascend are installed after your Python configuration is completed, because these packages will build binary files using python in current environment. If you install vLLM and vLLM Ascend before completing [Configure your environment](#2configure-your-environment), the binary files will not use the optimized python.
|
||||
:::
|
||||
|
||||
## Optimizations
|
||||
|
||||
### 1. Memory Allocator Optimization
|
||||
|
||||
#### 1.1. jemalloc
|
||||
|
||||
**jemalloc** is a memory allocator that improves performance for multi-threaded scenarios and can reduce memory fragmentation. jemalloc uses a local thread memory manager to allocate variables, which can avoid lock contention between threads and can hugely optimize performance.
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Install jemalloc
|
||||
sudo apt update
|
||||
sudo apt install libjemalloc2
|
||||
|
||||
# Configure jemalloc
|
||||
export LD_PRELOAD=/usr/lib/"$(uname -i)"-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
|
||||
```
|
||||
|
||||
#### 1.2. Tcmalloc
|
||||
|
||||
**TCMalloc (Thread Caching Malloc)** is a universal memory allocator that improves overall performance while ensuring low latency by introducing a multi-level cache structure, reducing lock contention and optimizing large object processing flow. Find more [details](https://www.hiascend.com/document/detail/zh/Pytorch/700/ptmoddevg/trainingmigrguide/performance_tuning_0068.html).
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Install tcmalloc
|
||||
sudo apt update
|
||||
sudo apt install libgoogle-perftools4 libgoogle-perftools-dev
|
||||
|
||||
# Get the location of libtcmalloc.so*
|
||||
find /usr -name libtcmalloc.so*
|
||||
|
||||
# Make the priority of tcmalloc higher
|
||||
# The <path> is the location of libtcmalloc.so we get from the upper command
|
||||
# Example: "$LD_PRELOAD:/usr/lib/aarch64-linux-gnu/libtcmalloc.so"
|
||||
export LD_PRELOAD="$LD_PRELOAD:<path>"
|
||||
|
||||
# Verify your configuration
|
||||
# The path of libtcmalloc.so will be contained in the result if your configuration is valid
|
||||
ldd `which python`
|
||||
```
|
||||
|
||||
### 2. `torch_npu` Optimization
|
||||
|
||||
Some performance tuning features in `torch_npu` are controlled by environment variables. Some features and their related environment variables are shown below.
|
||||
|
||||
Memory optimization:
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Upper limit of memory block splitting allowed (MB): Setting this parameter can prevent large memory blocks from being split.
|
||||
export PYTORCH_NPU_ALLOC_CONF="max_split_size_mb:250"
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# When operators on the communication stream have dependencies, they all need to be ended before being released for reuse. The logic of multi-stream reuse is to release the memory on the communication stream in advance so that the computing stream can be reused.
|
||||
export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"
|
||||
```
|
||||
|
||||
Scheduling optimization:
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Optimize operator delivery queue. This will affect the memory peak value, and may degrade if the memory is tight.
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
|
||||
# This will greatly improve the CPU bottleneck model and ensure the same performance for the NPU bottleneck model.
|
||||
export CPU_AFFINITY_CONF=1
|
||||
```
|
||||
|
||||
### 3. CANN Optimization
|
||||
|
||||
#### 3.1. HCCL Optimization
|
||||
|
||||
There are some performance tuning features in HCCL, which are controlled by environment variables.
|
||||
|
||||
You can configure HCCL to use "AIV" mode to optimize performance by setting the environment variable shown below. In "AIV" mode, the communication is scheduled by AI vector core directly with RoCE, instead of being scheduled by AI CPU.
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
export HCCL_OP_EXPANSION_MODE="AIV"
|
||||
```
|
||||
|
||||
Plus, there are more features for performance optimization in specific scenarios, which are shown below.
|
||||
|
||||
- `HCCL_INTRA_ROCE_ENABLE`: Use RDMA link instead of SDMA link between two 8Ps as the mesh interconnect link. Find more [details](https://www.hiascend.com/document/detail/zh/Pytorch/600/ptmoddevg/trainingmigrguide/performance_tuning_0044.html).
|
||||
- `HCCL_RDMA_TC`: Use this var to configure traffic class of RDMA NIC. Find more [details](https://www.hiascend.com/document/detail/zh/Pytorch/600/ptmoddevg/trainingmigrguide/performance_tuning_0045.html).
|
||||
- `HCCL_RDMA_SL`: Use this var to configure service level of RDMA NIC. Find more [details](https://www.hiascend.com/document/detail/zh/Pytorch/600/ptmoddevg/trainingmigrguide/performance_tuning_0046.html).
|
||||
- `HCCL_BUFFSIZE`: Use this var to control the cache size for sharing data between two NPUs. Find more [details](https://www.hiascend.com/document/detail/zh/Pytorch/600/ptmoddevg/trainingmigrguide/performance_tuning_0047.html).
|
||||
|
||||
### 4. Kernel Optimization
|
||||
|
||||
This section describes operating system–level optimizations applied on the host machine (bare metal or Kubernetes node) to improve performance stability, latency, and throughput for inference workloads.
|
||||
|
||||
:::{note}
|
||||
These settings must be applied on the host OS and with root privileges. Not inside containers.
|
||||
:::
|
||||
|
||||
#### 4.1 Set CPU Frequency Governor to `performance`
|
||||
|
||||
Set CPU Frequency Governor to `performance`
|
||||
|
||||
```shell
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
```
|
||||
|
||||
Purpose
|
||||
|
||||
- Forces all CPU cores to run under the `performance` governor
|
||||
- Disables dynamic frequency scaling (e.g., `ondemand`, `powersave`)
|
||||
|
||||
Benefits
|
||||
|
||||
- Keeps CPU cores at maximum frequency
|
||||
- Reduces latency jitter
|
||||
- Improves predictability for inference workloads
|
||||
|
||||
#### 4.2 Disable Swap Usage
|
||||
|
||||
```shell
|
||||
sysctl -w vm.swappiness=0
|
||||
```
|
||||
|
||||
Purpose
|
||||
|
||||
- Minimizes the kernel’s tendency to swap memory pages to disk
|
||||
|
||||
Benefits
|
||||
|
||||
- Prevents severe latency spikes caused by swapping
|
||||
- Improves stability for large in-memory models
|
||||
|
||||
Notes
|
||||
|
||||
- For inference workloads, swap can introduce second-level latency
|
||||
- Recommended values are `0` or `1`
|
||||
|
||||
#### 4.3 Disable Automatic NUMA Balancing
|
||||
|
||||
```shell
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
```
|
||||
|
||||
Purpose
|
||||
|
||||
- Disables the kernel’s automatic NUMA page migration mechanism
|
||||
|
||||
Benefits
|
||||
|
||||
- Prevents background memory page migrations
|
||||
- Reduces unpredictable memory access latency
|
||||
- Improves performance stability on NUMA systems
|
||||
|
||||
Recommended For
|
||||
|
||||
- Multi-socket servers
|
||||
- Ascend / NPU deployments with explicit NUMA binding
|
||||
- Systems with manually managed CPU and memory affinity
|
||||
|
||||
#### 4.4 Increase Scheduler Migration Cost
|
||||
|
||||
```shell
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
```
|
||||
|
||||
Purpose
|
||||
|
||||
- Increases the cost for the scheduler to migrate tasks between CPU cores
|
||||
|
||||
Benefits
|
||||
|
||||
- Reduces frequent thread migration
|
||||
- Improves CPU cache locality
|
||||
- Lowers latency jitter for inference workloads
|
||||
|
||||
Parameter Details
|
||||
|
||||
- Unit: nanoseconds (ns)
|
||||
- Typical recommended range: 50000–100000
|
||||
- Higher values encourage threads to stay on the same CPU core
|
||||
@@ -0,0 +1,249 @@
|
||||
# Performance Benchmark
|
||||
|
||||
This document details the benchmark methodology for vllm-ascend, aimed at evaluating the performance under a variety of workloads. To maintain alignment with vLLM, we use the [benchmark](https://github.com/vllm-project/vllm/tree/main/benchmarks) script provided by the vllm project.
|
||||
|
||||
**Benchmark Coverage**: We measure offline E2E latency and throughput, and fixed-QPS online serving benchmarks. For more details, see [vllm-ascend benchmark scripts](https://github.com/vllm-project/vllm-ascend/tree/main/benchmarks).
|
||||
|
||||
**Legend Description**:
|
||||
|
||||
- ✅ = Supported
|
||||
- 🟡 = Partial / Work in progress
|
||||
- 🚧 = Under development
|
||||
|
||||
## 1. Run docker container
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Update DEVICE according to your device (/dev/davinci[0-7])
|
||||
export DEVICE=/dev/davinci7
|
||||
export IMAGE=m.daocloud.io/quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--shm-size=1g \
|
||||
--device $DEVICE \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
--device /dev/hisi_hdc \
|
||||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||||
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /root/.cache:/root/.cache \
|
||||
-p 8000:8000 \
|
||||
-e VLLM_USE_MODELSCOPE=True \
|
||||
-it $IMAGE \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
## 2. Install dependencies
|
||||
|
||||
```bash
|
||||
cd /workspace/vllm-ascend
|
||||
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
|
||||
pip install -r benchmarks/requirements-bench.txt
|
||||
```
|
||||
|
||||
## 3. Run basic benchmarks
|
||||
|
||||
This section introduces how to perform performance testing using the benchmark suite built into vLLM.
|
||||
|
||||
### 3.1 Dataset
|
||||
|
||||
VLLM supports a variety of [datasets](https://github.com/vllm-project/vllm/blob/main/vllm/benchmarks/datasets/datasets.py).
|
||||
|
||||
<style>
|
||||
th {
|
||||
min-width: 0 !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
| Dataset | Online | Offline | Data Path |
|
||||
|---------|--------|---------|-----------|
|
||||
| ShareGPT | ✅ | ✅ | `wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json` |
|
||||
| ShareGPT4V (Image) | ✅ | ✅ | `wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json`<br>Note that the images need to be downloaded separately. For example, to download COCO's 2017 Train images:<br>`wget http://images.cocodataset.org/zips/train2017.zip` |
|
||||
| ShareGPT4Video (Video) | ✅ | ✅ | `git clone https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video` |
|
||||
| BurstGPT | ✅ | ✅ | `wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv` |
|
||||
| Sonnet (deprecated) | ✅ | ✅ | Local file: `benchmarks/sonnet.txt` |
|
||||
| Random | ✅ | ✅ | `synthetic` |
|
||||
| RandomMultiModal (Image/Video) | 🟡 | 🚧 | `synthetic` |
|
||||
| RandomForReranking | ✅ | ✅ | `synthetic` |
|
||||
| Prefix Repetition | ✅ | ✅ | `synthetic` |
|
||||
| HuggingFace-VisionArena | ✅ | ✅ | `lmarena-ai/VisionArena-Chat` |
|
||||
| HuggingFace-MMVU | ✅ | ✅ | `yale-nlp/MMVU` |
|
||||
| HuggingFace-InstructCoder | ✅ | ✅ | `likaixin/InstructCoder` |
|
||||
| HuggingFace-AIMO | ✅ | ✅ | `AI-MO/aimo-validation-aime`, `AI-MO/NuminaMath-1.5`, `AI-MO/NuminaMath-CoT` |
|
||||
| HuggingFace-Other | ✅ | ✅ | `lmms-lab/LLaVA-OneVision-Data`, `Aeala/ShareGPT_Vicuna_unfiltered` |
|
||||
| HuggingFace-MTBench | ✅ | ✅ | `philschmid/mt-bench` |
|
||||
| HuggingFace-Blazedit | ✅ | ✅ | `vdaita/edit_5k_char`, `vdaita/edit_10k_char` |
|
||||
| Spec Bench | ✅ | ✅ | `wget https://raw.githubusercontent.com/hemingkx/Spec-Bench/refs/heads/main/data/spec_bench/question.jsonl` |
|
||||
| Custom | ✅ | ✅ | Local file: `data.jsonl` |
|
||||
|
||||
:::{note}
|
||||
The datasets mentioned above are all links to datasets on huggingface.
|
||||
The dataset's `dataset-name` should be set to `hf`.
|
||||
For local `dataset-path`, please set `hf-name` to its Hugging Face ID like
|
||||
|
||||
```bash
|
||||
--dataset-path /datasets/VisionArena-Chat/ --hf-name lmarena-ai/VisionArena-Chat
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
### 3.2 Run basic benchmark
|
||||
|
||||
#### 3.2.1 Online serving
|
||||
|
||||
First start serving your model:
|
||||
|
||||
```bash
|
||||
export VLLM_USE_MODELSCOPE=True
|
||||
vllm serve Qwen/Qwen3-8B
|
||||
```
|
||||
|
||||
Then run the benchmarking script:
|
||||
|
||||
```bash
|
||||
# download dataset
|
||||
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
|
||||
export VLLM_USE_MODELSCOPE=True
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--model Qwen/Qwen3-8B \
|
||||
--endpoint /v1/completions \
|
||||
--dataset-name sharegpt \
|
||||
--dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
|
||||
--num-prompts 10
|
||||
```
|
||||
|
||||
If successful, you will see the following output:
|
||||
|
||||
```shell
|
||||
============ Serving Benchmark Result ============
|
||||
Successful requests: 10
|
||||
Failed requests: 0
|
||||
Benchmark duration (s): 19.92
|
||||
Total input tokens: 1374
|
||||
Total generated tokens: 2663
|
||||
Request throughput (req/s): 0.50
|
||||
Output token throughput (tok/s): 133.67
|
||||
Peak output token throughput (tok/s): 312.00
|
||||
Peak concurrent requests: 10.00
|
||||
Total Token throughput (tok/s): 202.64
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 127.10
|
||||
Median TTFT (ms): 136.29
|
||||
P99 TTFT (ms): 137.83
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 25.85
|
||||
Median TPOT (ms): 25.78
|
||||
P99 TPOT (ms): 26.64
|
||||
---------------Inter-token Latency----------------
|
||||
Mean ITL (ms): 25.78
|
||||
Median ITL (ms): 25.74
|
||||
P99 ITL (ms): 28.85
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 3.2.2 Offline Throughput Benchmark
|
||||
|
||||
```bash
|
||||
export VLLM_USE_MODELSCOPE=True
|
||||
vllm bench throughput \
|
||||
--model Qwen/Qwen3-8B \
|
||||
--dataset-name random \
|
||||
--input-len 128 \
|
||||
--output-len 128
|
||||
```
|
||||
|
||||
If successful, you will see the following output
|
||||
|
||||
```shell
|
||||
Processed prompts: 100%|█| 10/10 [00:03<00:00, 2.74it/s, est. speed input: 351.02 toks/s, output: 351.02 toks/s]
|
||||
Throughput: 2.73 requests/s, 699.93 total tokens/s, 349.97 output tokens/s
|
||||
Total num prompt tokens: 1280
|
||||
Total num output tokens: 1280
|
||||
```
|
||||
|
||||
#### 3.2.3 Multi-Modal Benchmark
|
||||
|
||||
```shell
|
||||
export VLLM_USE_MODELSCOPE=True
|
||||
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--dtype bfloat16 \
|
||||
--limit-mm-per-prompt '{"image": 1}' \
|
||||
--allowed-local-media-path /path/to/sharegpt4v/images
|
||||
```
|
||||
|
||||
```shell
|
||||
export HF_ENDPOINT="https://hf-mirror.com"
|
||||
vllm bench serve --model Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--backend "openai-chat" \
|
||||
--dataset-name hf \
|
||||
--hf-split train \
|
||||
--endpoint "/v1/chat/completions" \
|
||||
--dataset-path "lmarena-ai/vision-arena-bench-v0.1" \
|
||||
--num-prompts 10 \
|
||||
--no-stream
|
||||
```
|
||||
|
||||
```shell
|
||||
============ Serving Benchmark Result ============
|
||||
Successful requests: 10
|
||||
Failed requests: 0
|
||||
Benchmark duration (s): 4.89
|
||||
Total input tokens: 7191
|
||||
Total generated tokens: 951
|
||||
Request throughput (req/s): 2.05
|
||||
Output token throughput (tok/s): 194.63
|
||||
Peak output token throughput (tok/s): 290.00
|
||||
Peak concurrent requests: 10.00
|
||||
Total Token throughput (tok/s): 1666.35
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 722.22
|
||||
Median TTFT (ms): 589.81
|
||||
P99 TTFT (ms): 1377.02
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 44.13
|
||||
Median TPOT (ms): 34.58
|
||||
P99 TPOT (ms): 124.72
|
||||
---------------Inter-token Latency----------------
|
||||
Mean ITL (ms): 33.14
|
||||
Median ITL (ms): 28.01
|
||||
P99 ITL (ms): 182.28
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 3.2.4 Embedding Benchmark
|
||||
|
||||
```shell
|
||||
vllm serve Qwen/Qwen3-Embedding-8B --trust-remote-code
|
||||
```
|
||||
|
||||
```shell
|
||||
# download dataset
|
||||
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
|
||||
export VLLM_USE_MODELSCOPE=True
|
||||
vllm bench serve \
|
||||
--model Qwen/Qwen3-Embedding-8B \
|
||||
--backend openai-embeddings \
|
||||
--endpoint /v1/embeddings \
|
||||
--dataset-name sharegpt \
|
||||
--num-prompts 10 \
|
||||
--dataset-path <your dataset path>/datasets/ShareGPT_V3_unfiltered_cleaned_split.json
|
||||
```
|
||||
|
||||
```shell
|
||||
============ Serving Benchmark Result ============
|
||||
Successful requests: 10
|
||||
Failed requests: 0
|
||||
Benchmark duration (s): 0.18
|
||||
Total input tokens: 1372
|
||||
Request throughput (req/s): 56.32
|
||||
Total Token throughput (tok/s): 7726.76
|
||||
----------------End-to-end Latency----------------
|
||||
Mean E2EL (ms): 154.06
|
||||
Median E2EL (ms): 165.57
|
||||
P99 E2EL (ms): 166.66
|
||||
==================================================
|
||||
```
|
||||
@@ -0,0 +1,371 @@
|
||||
# Service Profiling Guide
|
||||
|
||||
In an inference service process, it is sometimes necessary to monitor the internal execution flow of the inference service framework to identify performance issues. By collecting start and end timestamps of key processes, identifying key functions or iterations, recording critical events, and gathering various types of information, performance bottlenecks can be quickly located.
|
||||
|
||||
This guide will walk you through the process of collecting performance data from the vLLM-Ascend service framework and operators. It covers the complete workflow from preparation, collection, analysis, to visualization, helping you quickly get started with performance collection tools.
|
||||
|
||||
Two performance collection solutions are provided below: Ascend PyTorch Profiler and MS Service Profiler. You can choose the appropriate tool for performance analysis and troubleshooting based on your actual requirements.
|
||||
|
||||
## Solution Comparison
|
||||
|
||||
| Feature | Ascend PyTorch Profiler | MS Service Profiler |
|
||||
|:-----|:------------------------|:------------------|
|
||||
| Installation Method | Built-in, no additional installation required | Requires building msserviceprofiler from source |
|
||||
| Collection Granularity | PyTorch operator level | Service framework function level |
|
||||
| Control Method | API request control | Configuration file control |
|
||||
| Applicable Scenarios | Model operator performance analysis | Service framework workflow analysis |
|
||||
| Data Format | ascend_pt format | Chrome Tracing + CSV |
|
||||
| Main Advantage | Operator-level performance analysis | Service framework workflow visualization |
|
||||
| Supported Collection Capabilities | PyTorch operator level | PyTorch operator level and Service framework function level |
|
||||
|
||||
## Quick Selection Guide
|
||||
|
||||
- [**Model Operator Performance** → Use Ascend PyTorch Profiler](#ascend-pytorch-profiler)
|
||||
- [**Service Framework Workflow** → Use MS Service Profiler](#ms-service-profiler)
|
||||
|
||||
---
|
||||
|
||||
## Ascend PyTorch Profiler
|
||||
|
||||
### 0. Installation and Configuration
|
||||
|
||||
No additional packages need to be installed; it can be enabled through command-line configuration. Currently, vLLM enables **python stack** by default, which can significantly inflate the collected performance data. If you do not wish to collect **python stack**, you can disable it using `torch_profiler_with_stack=false`.
|
||||
|
||||
### 1. Preparation for Collection
|
||||
|
||||
Start the online service and set the `--profiler-config` parameter to control the path for saving performance files. After the parameter is set, the collection function is enabled.
|
||||
|
||||
```bash
|
||||
VLLM_PROMPT_SEQ_BUCKET_MAX=128
|
||||
VLLM_PROMPT_SEQ_BUCKET_MIN=128
|
||||
python3 -m vllm.entrypoints.openai.api_server \
|
||||
--port 8080 \
|
||||
--model "facebook/opt-125m" \
|
||||
--tensor-parallel-size 1 \
|
||||
--max-num-seqs 128 \
|
||||
--profiler-config '{"profiler": "torch", "torch_profiler_dir": "./vllm_profile", "torch_profiler_with_stack": false}' \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 256
|
||||
```
|
||||
|
||||
> Note:**January 19, 2026: The vLLM mainline has deprecated the VLLM_TORCH_PROFILER_DIR environment variable.**[Related PR](https://github.com/vllm-project/vllm-ascend/pull/5928) When using the vLLM Ascend mainline code to collect profiler data, remember to use the `--profiler-config` (online) parameter or the `profiler_config` (offline) parameter.
|
||||
|
||||
### 2. Start Collection
|
||||
|
||||
Performance collection is controlled by sending API requests. You can start collection after stabilizing the actual business data and collect profiling for a few seconds before stopping; or you can start collection first, then send business requests, and finally stop.
|
||||
|
||||
Send the following request to start the profiling service:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/start_profile
|
||||
```
|
||||
|
||||
Send the following request to stop the profiling service:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/stop_profile
|
||||
```
|
||||
|
||||
### 3. Send Requests
|
||||
|
||||
Send requests according to your actual business data. After sending the requests, stop the profiling service, and the data will be automatically saved to the previously configured path:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "facebook/opt-125m",
|
||||
"prompt": "San Francisco is a",
|
||||
"max_tokens": 7,
|
||||
"temperature": 0
|
||||
}'
|
||||
|
||||
curl -X POST http://localhost:8080/stop_profile
|
||||
```
|
||||
|
||||
### 4. Analyze Data
|
||||
|
||||
Navigate to the `./vllm_profile` directory and locate the generated `*ascend_pt` folder. This folder needs to be analyzed before profiling data can be examined.
|
||||
|
||||
```python
|
||||
from torch_npu.profiler.profiler import analyse
|
||||
analyse("./vllm_profile/localhost.localdomain_*_ascend_pt/")
|
||||
```
|
||||
|
||||
### 5. View Results
|
||||
|
||||
After analysis, the `*ascend_pt` directory will contain many files, with the main analysis focus being the `ASCEND_PROFILER_OUTPUT` folder. This directory will include the following files:
|
||||
|
||||
- `analysis.db`: Performance data in database format
|
||||
|
||||
- `api_statistic.csv`: API call statistics
|
||||
|
||||
- `ascend_pytorch_profiler_0.db`: Performance data in database format
|
||||
|
||||
- `kernel_details.csv`: Kernel-level related data
|
||||
|
||||
- `operator_details.csv`: Operator-level related data
|
||||
|
||||
- `op_statistic.csv`: Operator utilization data
|
||||
|
||||
- `step_trace_time.csv`: Scheduling data
|
||||
|
||||
- `trace_view.json`: Chrome tracing format data, can be opened with [MindStudio Insight](https://www.hiascend.com/document/detail/zh/mindstudio/81RC1/GUI_baseddevelopmenttool/msascendinsightug/Insight_userguide_0002.html)
|
||||
|
||||
[↑ Back to Top](#service-profiling-guide)
|
||||
|
||||
---
|
||||
|
||||
## MS Service Profiler
|
||||
|
||||
### 0. Build from Source and Upgrade
|
||||
|
||||
The `msserviceprofiler` tool is pre-installed with the CANN Toolkit package. Use the following commands to install or upgrade from source.
|
||||
|
||||
```bash
|
||||
git clone https://gitcode.com/Ascend/msserviceprofiler.git
|
||||
cd msserviceprofiler
|
||||
bash scripts/build_and_upgrade.sh
|
||||
```
|
||||
|
||||
### 1. Preparation
|
||||
|
||||
Before starting the service, set the environment variable `SERVICE_PROF_CONFIG_PATH` to point to the profiling configuration file, and set the environment variable `PROFILING_SYMBOLS_PATH` to specify the YAML configuration file for the symbols that need to be imported. After that, start the vLLM service according to your deployment method.
|
||||
|
||||
```bash
|
||||
cd ${path_to_store_profiling_files}
|
||||
# Set environment variable
|
||||
export SERVICE_PROF_CONFIG_PATH=ms_service_profiler_config.json
|
||||
export PROFILING_SYMBOLS_PATH=service_profiling_symbols.yaml
|
||||
|
||||
# Start vLLM service
|
||||
vllm serve Qwen/Qwen2.5-0.5B-Instruct &
|
||||
```
|
||||
|
||||
The file `ms_service_profiler_config.json` is the profiling configuration. If it does not exist at the specified path, a default configuration will be generated automatically. If needed, you can customize it in advance according to the instructions in the `Profiling Configuration File` section below.
|
||||
|
||||
`service_profiling_symbols.yaml` is the configuration file containing the profiling points to be imported. You can choose **not** to set the `PROFILING_SYMBOLS_PATH` environment variable, in which case the default configuration file will be used. If the file does not exist at the path you specified, likewise, the system will generate a configuration file at your specified path for future configuration. You can customize it according to the instructions in the `Symbols Configuration File` section below.
|
||||
|
||||
### 2. Enable Profiling
|
||||
|
||||
To enable the performance data collection switch, change the `enable` field from `0` to `1` in the configuration file `ms_service_profiler_config.json`. This can be accomplished by executing the following sed command:
|
||||
|
||||
```bash
|
||||
sed -i 's/"enable":\s*0/"enable": 1/' ./ms_service_profiler_config.json
|
||||
```
|
||||
|
||||
### 3. Send Requests
|
||||
|
||||
Choose a request-sending method that suits your actual profiling needs:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"prompt": "Beijing is a",
|
||||
"max_tokens": 5,
|
||||
"temperature": 0
|
||||
}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
### 4. Parse Data
|
||||
|
||||
```bash
|
||||
# xxxx-xxxx is the directory automatically created based on vLLM startup time
|
||||
cd /root/.ms_server_profiler/xxxx-xxxx
|
||||
|
||||
# parse data
|
||||
msserviceprofiler parse --input-path=./ --output-path output
|
||||
```
|
||||
|
||||
### 5. View Results
|
||||
|
||||
After parsing, the `output` directory will contain:
|
||||
|
||||
- `chrome_tracing.json`: Chrome tracing format data, which can be opened in [MindStudio Insight](https://www.hiascend.com/document/detail/zh/mindstudio/830/GUI_baseddevelopmenttool/msascendinsightug/Insight_userguide_0002.html?framework=mindspore).
|
||||
- `profiler.db`: Performance data in database format.
|
||||
- `request.csv`: Request-related data.
|
||||
- `kvcache.csv`: KV Cache-related data.
|
||||
- `batch.csv`: Batch scheduling-related data.
|
||||
|
||||
---
|
||||
|
||||
### 6. Appendix related to MS Service Profiler
|
||||
|
||||
(profiling-configuration-file)=
|
||||
|
||||
#### 6.1 Profiling Configuration File
|
||||
|
||||
The profiling configuration file controls profiling parameters and behavior.
|
||||
|
||||
##### File Format
|
||||
|
||||
The configuration is in JSON format. Main parameters:
|
||||
|
||||
| Parameter | Description | Required |
|
||||
|:------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----:|
|
||||
| enable | Switch for profiling: <br />0: disable<br />1: enable<br />Default: 0 | Yes |
|
||||
| prof_dir | Directory to store collected performance data. <br />Default: `${HOME}/.ms_server_profiler` | No |
|
||||
| profiler_level | Data collection level. Default is "INFO" (normal level). | No |
|
||||
| acl_task_time | Switch to collect operator dispatch latency and execution latency. Values: <br />0: off. Default; 0 or any invalid value means off.<br />1: on. When enabled, calls `aclprofCreateConfig` with `ACL_PROF_TASK_TIME_L0`.<br />2: on. MSPTI-based dump. When enabled, set before starting the service: `export LD_PRELOAD={INSTALL_DIR}/lib64/libmspti.so`, where `{INSTALL_DIR}` is the CANN installation root (e.g. `/usr/local/Ascend/cann` for a typical root install).<br />3: on. Torch Profiler–based dump. | No |
|
||||
| acl_prof_task_time_level | Profiling level and duration. Values: <br />L0: collect operator dispatch and execution latency only; lower overhead (no operator basic info).<br />L1: collect AscendCL interface performance (host–device and inter-device sync/async memory copy latencies), plus operator dispatch, execution, and basic info for comprehensive analysis.<br />`{time}`: optional duration segment; integer 1–999, unit seconds.<br />If unset, defaults to L0 until program exit; invalid values fall back to defaults.<br />Level and duration can be combined, e.g., `"acl_prof_task_time_level": "L1;10"`.<br />**Note:** When Torch Profiler is used (`acl_task_time` set to `3`), `{time}` duration is not supported. | No |
|
||||
| timelimit | Profiling duration for the service. The process stops automatically after this time. Range: integer 0–7200, unit: seconds. Default 0 means unlimited. Recommend at least 120s; shorter runs may lack data for parsed outputs and trigger warnings. | No |
|
||||
| domain | Limit profiling to the specified domains to reduce data volume. String, separated by semicolons, case-sensitive, e.g., "Request; KVCache".<br />Empty means all available domains.<br />Available domains: Request, KVCache, ModelExecute, BatchSchedule, Communication.<br />Note: If the selected domains are incomplete, analysis output may show warnings due to missing data. See [Reference Table 1](https://www.hiascend.com/document/detail/zh/canncommercial/850/devaids/Profiling/mindieprofiling_0010.html). | No |
|
||||
| torch_prof_stack | Collect operator call stacks (framework and CPU operators). Values: `false` (default, off), `true` (on). Requires `acl_task_time` set to `3`. **Note:** Enabling this configuration introduces additional performance overhead. | No |
|
||||
| torch_prof_step_num | Torch Profiler step limit. Integer ≥ 0. Default `0` means collect all steps.<br />Requires `acl_task_time` set to `3`. | No |
|
||||
| profiler_step_num | Step limit for operator and service framework profiling. Integer ≥ 0.<br />`0` or invalid values stop the entire service profiling process.<br />The number of steps actually recorded depends on `modelRunnerExec` events. | No |
|
||||
|
||||
##### Example Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"enable": 1,
|
||||
"prof_dir": "./vllm_prof",
|
||||
"acl_task_time": 0,
|
||||
"acl_prof_task_time_level": ""
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
(symbols-configuration-file)=
|
||||
|
||||
#### 6.2 Symbols Configuration File
|
||||
|
||||
The symbols configuration file defines which functions/methods to profile and supports flexible configuration with custom attribute collection.
|
||||
|
||||
##### File Name and Loading
|
||||
|
||||
- Default load path: `~/.config/vllm_ascend/service_profiling_symbols.MAJOR.MINOR.PATCH.yaml` (According to the installed version of vLLM)
|
||||
|
||||
If you need to customize the profiling points, it is highly recommended to copy a symbol configuration file to your working directory and point to it with the `PROFILING_SYMBOLS_PATH` environment variable.
|
||||
|
||||
##### Configuration file updates
|
||||
|
||||
After you change profiling symbols, restart the vLLM service so the updated configuration file is loaded.
|
||||
|
||||
##### Field Descriptions
|
||||
|
||||
| Field | Description | Example |
|
||||
|:-----:|:-----|:-----|
|
||||
| symbol | Python import path + attribute chain | `"vllm.v1.core.kv_cache_manager:KVCacheManager.free"` |
|
||||
| handler | Handler type | `"timer"` (default) or `"pkg.mod:func"` (custom) |
|
||||
| domain | Domain tag | `"KVCache"`, `"ModelExecute"` |
|
||||
| name | Event name | `"EngineCoreExecute"` |
|
||||
| min_version | Minimum supported vLLM version | `"0.9.1"` |
|
||||
| max_version | Maximum supported vLLM version | `"0.11.0"` |
|
||||
| attributes | Custom attribute collection | Only supported for `"timer"` handler. See the section below |
|
||||
|
||||
##### Configuration Examples
|
||||
|
||||
- Example 1: Custom handler
|
||||
|
||||
```yaml
|
||||
- symbol: vllm.v1.core.kv_cache_manager:KVCacheManager.free
|
||||
handler: ms_service_profiler.patcher.config.custom_handler_example.kvcache_manager_free_example_handler
|
||||
domain: Example
|
||||
name: example_custom
|
||||
```
|
||||
|
||||
- Example 2: Default timer
|
||||
|
||||
```yaml
|
||||
- symbol: vllm.v1.engine.core:EngineCore.execute_model
|
||||
domain: ModelExecute
|
||||
name: EngineCoreExecute
|
||||
```
|
||||
|
||||
- Example 3: Version constraint
|
||||
|
||||
```yaml
|
||||
- symbol: vllm.v1.executor.abstract:Executor.execute_model
|
||||
min_version: "0.9.1"
|
||||
# No handler specified -> default timer
|
||||
```
|
||||
|
||||
##### Custom Attribute Collection
|
||||
|
||||
The `attributes` field supports flexible custom attribute collection and allows operations and transformations on function arguments and return values.
|
||||
|
||||
###### Basic Syntax
|
||||
|
||||
- Argument access: use the parameter name directly, e.g., `input_ids`
|
||||
- Return value access: use the `return` keyword
|
||||
- Pipeline operations: use `|` to chain multiple operations
|
||||
- Attribute access: use `attr` to access object attributes
|
||||
|
||||
###### Example
|
||||
|
||||
```yaml
|
||||
- symbol: vllm_ascend.worker.model_runner_v1:NPUModelRunner.execute_model
|
||||
name: ModelRunnerExecuteModel
|
||||
domain: ModelExecute
|
||||
attributes:
|
||||
- name: device
|
||||
expr: args[0] | attr device | str
|
||||
- name: dp
|
||||
expr: args[0] | attr dp_rank | str
|
||||
- name: batch_size
|
||||
expr: args[0] | attr input_batch | attr _req_ids | len
|
||||
```
|
||||
|
||||
###### Expression Notes
|
||||
|
||||
1. `len(input_ids)`: get the length of parameter `input_ids`.
|
||||
2. `len(return) | str`: get the length of the return value and convert to string (equivalent to `str(len(return))`).
|
||||
3. `return[0] | attr input_ids | len`: get the length of the `input_ids` attribute of the first element in the return value.
|
||||
|
||||
###### Supported Expression Types
|
||||
|
||||
- Basic operations: `len()`, `str()`, `int()`, `float()`
|
||||
- Index access: `return[0]`, `return['key']`
|
||||
- Attribute access: `return | attr attr_name`
|
||||
- Pipeline composition: chain operations with `|`
|
||||
|
||||
###### Advanced Examples
|
||||
|
||||
```yaml
|
||||
attributes:
|
||||
# Get tensor shape
|
||||
- name: tensor_shape
|
||||
expr: input_tensor | attr shape | str
|
||||
|
||||
# Get specific value from a dict
|
||||
- name: batch_size
|
||||
expr: kwargs['batch_size']
|
||||
|
||||
# Conditional expression (requires custom handler support)
|
||||
- name: is_training_mode
|
||||
expr: training | bool
|
||||
|
||||
# Complex data processing
|
||||
- name: processed_data_len
|
||||
expr: data | attr items | len | str
|
||||
```
|
||||
|
||||
##### Custom Handler
|
||||
|
||||
When `handler` specifies a custom function, it must match the following signature:
|
||||
|
||||
```python
|
||||
def custom_handler(original_func, this, *args, **kwargs):
|
||||
"""
|
||||
Custom handler
|
||||
|
||||
Args:
|
||||
original_func: the original function object
|
||||
this: the bound object (for methods)
|
||||
*args: positional arguments
|
||||
**kwargs: keyword arguments
|
||||
|
||||
Returns:
|
||||
processing result
|
||||
"""
|
||||
# Custom logic
|
||||
pass
|
||||
```
|
||||
|
||||
If the custom handler fails to import, the system will automatically fall back to the default timer mode.
|
||||
|
||||
[↑ Back to Top](#service-profiling-guide)
|
||||
@@ -2,41 +2,51 @@
|
||||
|
||||
## Version Specific FAQs
|
||||
|
||||
- [[v0.9.1] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/2643)
|
||||
- [[v0.11.0rc1] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/3222)
|
||||
- [[v0.23.0] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/12916)
|
||||
- [[v0.23.0rc1] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/12238)
|
||||
- [[v0.22.1rc1] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/10593)
|
||||
- [[v0.21.0rc1] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/9970)
|
||||
- [[v0.20.2rc1] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/9586)
|
||||
- [[v0.19.1rc1] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/8819)
|
||||
- [[v0.18.0] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/8238)
|
||||
|
||||
## General FAQs
|
||||
|
||||
### 1. What devices are currently supported?
|
||||
|
||||
Currently, **ONLY** Atlas A2 series(Ascend-cann-kernels-910b)，Atlas A3 series(Atlas-A3-cann-kernels) and Atlas 300I(Ascend-cann-kernels-310p) series are supported:
|
||||
Currently, **ONLY** Atlas A2 series (Ascend-cann-kernels-910b), Atlas A3 series (Atlas-A3-cann-kernels) and Atlas 300I (Ascend-cann-kernels-310p) series are supported:
|
||||
|
||||
- Atlas A2 Training series (Atlas 800T A2, Atlas 900 A2 PoD, Atlas 200T A2 Box16, Atlas 300T A2)
|
||||
- Atlas 800I A2 Inference series (Atlas 800I A2)
|
||||
- Atlas A3 Training series (Atlas 800T A3, Atlas 900 A3 SuperPoD, Atlas 9000 A3 SuperPoD)
|
||||
- Atlas 800I A3 Inference series (Atlas 800I A3)
|
||||
- [Experimental] Atlas 300I Inference series (Atlas 300I Duo)
|
||||
- [Experimental] Atlas 300I Inference series (Atlas 300I Duo).
|
||||
- [Experimental] Currently for 310I Duo the stable version is vllm-ascend v0.10.0rc1.
|
||||
|
||||
Below series are NOT supported yet:
|
||||
|
||||
- Atlas 200I A2 (Ascend-cann-kernels-310b) unplanned yet
|
||||
- Ascend 910, Ascend 910 Pro B (Ascend-cann-kernels-910) unplanned yet
|
||||
|
||||
From a technical view, vllm-ascend support would be possible if the torch-npu is supported. Otherwise, we have to implement it by using custom ops. We are also welcome to join us to improve together.
|
||||
From a technical view, vllm-ascend supports devices if TorchNPU is supported. Otherwise, we have to implement it by using custom ops. We also welcome you to join us to improve together.
|
||||
|
||||
### 2. How to get our docker containers?
|
||||
|
||||
You can get our containers at `Quay.io`, e.g., [<u>vllm-ascend</u>](https://quay.io/repository/ascend/vllm-ascend?tab=tags) and [<u>cann</u>](https://quay.io/repository/ascend/cann?tab=tags).
|
||||
|
||||
If you are in China, you can use `daocloud` to accelerate your downloading:
|
||||
If you are in China, you can use `daocloud` or some other mirror sites to accelerate your downloading:
|
||||
|
||||
```bash
|
||||
# Replace with tag you want to pull
|
||||
TAG=v0.7.3rc2
|
||||
TAG=v0.9.1
|
||||
docker pull m.daocloud.io/quay.io/ascend/vllm-ascend:$TAG
|
||||
# or
|
||||
docker pull quay.nju.edu.cn/ascend/vllm-ascend:$TAG
|
||||
```
|
||||
|
||||
#### Load Docker Images for offline environment
|
||||
If you want to use container image for offline environments (no internet connection), you need to download container image in a environment with internet access:
|
||||
|
||||
If you want to use container image for offline environments (no internet connection), you need to download container image in an environment with internet access:
|
||||
|
||||
**Exporting Docker images:**
|
||||
|
||||
@@ -62,9 +72,9 @@ docker load -i vllm-ascend-$TAG.tar.gz
|
||||
docker images | grep vllm-ascend
|
||||
```
|
||||
|
||||
### 3. What models does vllm-ascend supports?
|
||||
### 3. What models does vllm-ascend support?
|
||||
|
||||
Find more details [<u>here</u>](https://vllm-ascend.readthedocs.io/en/latest/user_guide/support_matrix/supported_models.html).
|
||||
Find more details [<u>here</u>](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/support_matrix/supported_models.html).
|
||||
|
||||
### 4. How to get in touch with our community?
|
||||
|
||||
@@ -72,23 +82,24 @@ There are many channels that you can communicate with our community developers /
|
||||
|
||||
- Submit a GitHub [<u>issue</u>](https://github.com/vllm-project/vllm-ascend/issues?page=1).
|
||||
- Join our [<u>weekly meeting</u>](https://docs.google.com/document/d/1hCSzRTMZhIB8vRq1_qOOjx4c9uYUxvdQvDsMV2JcSrw/edit?tab=t.0#heading=h.911qu8j8h35z) and share your ideas.
|
||||
- Join our [<u>WeChat</u>](https://github.com/vllm-project/vllm-ascend/issues/227) group and ask your quenstions.
|
||||
- Join our [<u>WeChat</u>](https://github.com/vllm-project/vllm-ascend/issues/227) group and ask your questions.
|
||||
- Join our ascend channel in [<u>vLLM forums</u>](https://discuss.vllm.ai/c/hardware-support/vllm-ascend-support/6) and publish your topics.
|
||||
|
||||
### 5. What features does vllm-ascend V1 supports?
|
||||
### 5. What features does vllm-ascend V1 support?
|
||||
|
||||
Find more details [<u>here</u>](https://vllm-ascend.readthedocs.io/en/latest/user_guide/support_matrix/supported_features.html).
|
||||
Find more details [<u>here</u>](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/support_matrix/supported_features.html).
|
||||
|
||||
### 6. How to solve the problem of "Failed to infer device type" or "libatb.so: cannot open shared object file"?
|
||||
|
||||
Basically, the reason is that the NPU environment is not configured correctly. You can:
|
||||
|
||||
1. try `source /usr/local/Ascend/nnal/atb/set_env.sh` to enable NNAL package.
|
||||
2. try `source /usr/local/Ascend/ascend-toolkit/set_env.sh` to enable CANN package.
|
||||
3. try `npu-smi info` to check whether the NPU is working.
|
||||
|
||||
If all above steps are not working, you can try the following code with python to check whether there is any error:
|
||||
If the above steps are not working, you can try the following code in Python to check whether there are any errors:
|
||||
|
||||
```
|
||||
```python
|
||||
import torch
|
||||
import torch_npu
|
||||
import vllm
|
||||
@@ -96,121 +107,206 @@ import vllm
|
||||
|
||||
If all above steps are not working, feel free to submit a GitHub issue.
|
||||
|
||||
### 7. How does vllm-ascend perform?
|
||||
### 7. How does vllm-ascend work with vLLM?
|
||||
|
||||
Currently, only some models are improved. Such as `Qwen2.5 VL`, `Qwen3`, `Deepseek V3`. Others are not good enough. From 0.9.0rc2, Qwen and Deepseek works with graph mode to play a good performance. What's more, you can install `mindie-turbo` with `vllm-ascend v0.7.3` to speed up the inference as well.
|
||||
`vllm-ascend` is a hardware plugin for vLLM. Stable releases usually align with the same vLLM version, while RC releases may use the corresponding vLLM final release version. For example, `vllm-ascend` `v0.18.0rc1` matches vLLM `v0.18.0`. For the main branch, we ensure that `vllm-ascend` and `vllm` are compatible at every commit.
|
||||
|
||||
### 8. How vllm-ascend work with vllm?
|
||||
vllm-ascend is a plugin for vllm. Basically, the version of vllm-ascend is the same as the version of vllm. For example, if you use vllm 0.7.3, you should use vllm-ascend 0.7.3 as well. For main branch, we will make sure `vllm-ascend` and `vllm` are compatible by each commit.
|
||||
### 8. Does vllm-ascend support Prefill-Decode (PD) Disaggregation feature?
|
||||
|
||||
### 9. Does vllm-ascend support Prefill Disaggregation feature?
|
||||
Yes, vllm-ascend supports Prefill-Decode Disaggregation feature with Mooncake backend. See the [official tutorial](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_disaggregation_mooncake_multi_node.html) for example.
|
||||
|
||||
Currently, only 1P1D is supported on V0 Engine. For V1 Engine or NPND support, We will make it stable and supported by vllm-ascend in the future.
|
||||
### 9. Does vllm-ascend support quantization method?
|
||||
|
||||
### 10. Does vllm-ascend support quantization method?
|
||||
Currently, w8a8, w4a8, and w4a4 quantization methods are already supported by vllm-ascend.
|
||||
|
||||
Currently, w8a8 quantization is already supported by vllm-ascend originally on v0.8.4rc2 or higher, If you're using vllm 0.7.3 version, w8a8 quantization is supporeted with the integration of vllm-ascend and mindie-turbo, please use `pip install vllm-ascend[mindie-turbo]`.
|
||||
### 10. How is vllm-ascend tested?
|
||||
|
||||
### 11. How to run w8a8 DeepSeek model?
|
||||
vllm-ascend is tested in three aspects: functions, performance, and accuracy.
|
||||
|
||||
Please following the [inferencing tutorail](https://vllm-ascend.readthedocs.io/en/latest/tutorials/multi_node.html) and replace model to DeepSeek.
|
||||
- **Functional test**: We added CI, including part of vllm's native unit tests and vllm-ascend's own unit tests. In vllm-ascend's tests, we test basic functionalities, popular model availability, and [supported features](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/support_matrix/supported_features.html) through E2E test.
|
||||
|
||||
### 12. There is no output in log when loading models using vllm-ascend, How to solve it?
|
||||
- **Performance test**: We provide [benchmark](https://github.com/vllm-project/vllm-ascend/tree/main/benchmarks) tools for E2E performance benchmark, which can be easily re-run locally. We will publish a perf website to show the performance test results for each pull request.
|
||||
|
||||
If you're using vllm 0.7.3 version, this is a known progress bar display issue in VLLM, which has been resolved in [this PR](https://github.com/vllm-project/vllm/pull/12428), please cherry-pick it locally by yourself. Otherwise, please fill up an issue.
|
||||
- **Accuracy test**: We are working on adding accuracy test to the CI as well.
|
||||
|
||||
### 13. How vllm-ascend is tested
|
||||
- **Nightly test**: we'll run full test every night to make sure the code is working.
|
||||
|
||||
vllm-ascend is tested by functional test, performance test and accuracy test.
|
||||
For each release, we'll publish the performance test and accuracy test report in the future.
|
||||
|
||||
- **Functional test**: we added CI, includes portion of vllm's native unit tests and vllm-ascend's own unit tests，on vllm-ascend's test, we test basic functionality、popular models availability and [supported features](https://vllm-ascend.readthedocs.io/en/latest/user_guide/support_matrix/supported_features.html) via e2e test
|
||||
### 11. How to fix the error "InvalidVersion" when using vllm-ascend?
|
||||
|
||||
- **Performance test**: we provide [benchmark](https://github.com/vllm-project/vllm-ascend/tree/main/benchmarks) tools for end-to-end performance benchmark which can easily to re-route locally, we'll publish a perf website to show the performance test results for each pull request
|
||||
The problem is usually caused by the installation of a development or editable version of the vLLM package. In this case, we provide the environment variable `VLLM_VERSION` to let users specify the version of vLLM package to use. Please set the environment variable `VLLM_VERSION` to the version of the vLLM package you have installed. The format of `VLLM_VERSION` should be `X.Y.Z`.
|
||||
|
||||
- **Accuracy test**: we're working on adding accuracy test to CI as well.
|
||||
### 12. How to handle the out-of-memory issue?
|
||||
|
||||
Finnall, for each release, we'll publish the performance test and accuracy test report in the future.
|
||||
OOM errors typically occur when the model exceeds the memory capacity of a single NPU. For general guidance, you can refer to [vLLM OOM troubleshooting documentation](https://docs.vllm.ai/en/latest/usage/troubleshooting/#out-of-memory).
|
||||
|
||||
### 14. How to fix the error "InvalidVersion" when using vllm-ascend?
|
||||
It's usually because you have installed an dev/editable version of vLLM package. In this case, we provide the env variable `VLLM_VERSION` to let users specify the version of vLLM package to use. Please set the env variable `VLLM_VERSION` to the version of vLLM package you have installed. The format of `VLLM_VERSION` should be `X.Y.Z`.
|
||||
In scenarios where NPUs have limited high bandwidth memory (on-chip memory) capacity, dynamic memory allocation/deallocation during inference can exacerbate memory fragmentation, leading to OOM. To address this:
|
||||
|
||||
### 15. How to handle Out Of Memory?
|
||||
OOM errors typically occur when the model exceeds the memory capacity of a single NPU. For general guidance, you can refer to [vLLM's OOM troubleshooting documentation](https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#out-of-memory).
|
||||
- **Limit `--max-model-len`**: It can save the on-chip memory usage for KV cache initialization step.
|
||||
|
||||
In scenarios where NPUs have limited HBM (High Bandwidth Memory) capacity, dynamic memory allocation/deallocation during inference can exacerbate memory fragmentation, leading to OOM. To address this:
|
||||
- **Adjust `--gpu-memory-utilization`**: If unspecified, the default value is `0.9`. You can decrease this value to reserve more memory to reduce fragmentation risks. See details in: [vLLM - Inference and Serving - Engine Arguments](https://docs.vllm.ai/en/latest/cli/serve/#-gpu-memory-utilization).
|
||||
|
||||
- **Adjust `--gpu-memory-utilization`**: If unspecified, will use the default value of `0.9`. You can decrease this param to reserve more memory to reduce fragmentation risks. See more note in: [vLLM - Inference and Serving - Engine Arguments](https://docs.vllm.ai/en/latest/serving/engine_args.html#vllm.engine.arg_utils-_engine_args_parser-cacheconfig).
|
||||
- **Configure `PYTORCH_NPU_ALLOC_CONF`**: Set this environment variable to optimize NPU memory management. For example, you can use `export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True` to enable virtual memory feature to mitigate memory fragmentation caused by frequent dynamic memory size adjustments during runtime. See details in [PYTORCH_NPU_ALLOC_CONF](https://www.hiascend.com/document/detail/zh/Pytorch/700/comref/Envvariables/Envir_012.html).
|
||||
|
||||
- **Configure `PYTORCH_NPU_ALLOC_CONF`**: Set this environment variable to optimize NPU memory management. For example, you can `export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True` to enable virtual memory feature to mitigate memory fragmentation caused by frequent dynamic memory size adjustments during runtime, see more note in: [PYTORCH_NPU_ALLOC_CONF](https://www.hiascend.com/document/detail/zh/Pytorch/700/comref/Envvariables/Envir_012.html).
|
||||
### 13. Failed to enable NPU graph mode when running DeepSeek
|
||||
|
||||
### 16. Failed to enable NPU graph mode when running DeepSeek?
|
||||
You may encounter the following error if running DeepSeek with NPU graph mode enabled. The allowed number of queries per kv when enabling both MLA and Graph mode only support {32, 64, 128}, **Thus this is not supported for DeepSeek-V2-Lite**, as it only has 16 attention heads. The NPU graph mode support on DeepSeek-V2-Lite will be done in the future.
|
||||
Enabling NPU graph mode for DeepSeek may trigger an error. This is because when both MLA (Multi-Head Latent Attention) and NPU graph mode are active, the number of queries per KV head must be 32, 64, or 128. However, DeepSeek-V2-Lite has only 16 attention heads, which results in 16 queries per KV—a value outside the supported range. Support for NPU graph mode on DeepSeek-V2-Lite will be added in a future update.
|
||||
|
||||
And if you're using DeepSeek-V3 or DeepSeek-R1, please make sure after the tensor parallel split, num_heads / num_kv_heads in {32, 64, 128}.
|
||||
And if you're using DeepSeek-V3 or DeepSeek-R1, please make sure after the tensor parallel split, `num_heads`/`num_kv_heads` is {32, 64, 128}.
|
||||
|
||||
```bash
|
||||
[rank0]: RuntimeError: EZ9999: Inner Error!
|
||||
[rank0]: EZ9999: [PID: 62938] 2025-05-27-06:52:12.455.807 numHeads / numKvHeads = 8, MLA only support {32, 64, 128}.[FUNC:CheckMlaAttrs][FILE:incre_flash_attention_tiling_check.cc][LINE:1218]
|
||||
```
|
||||
|
||||
### 17. Failed to reinstall vllm-ascend from source after uninstalling vllm-ascend?
|
||||
You may encounter the problem of C compilation failure when reinstalling vllm-ascend from source using pip. If the installation fails, it is recommended to use `python setup.py install` to install, or use `python setup.py clean` to clear the cache.
|
||||
### 14. Failed to reinstall vllm-ascend from source after uninstalling vllm-ascend
|
||||
|
||||
### 18. How to generate determinitic results when using vllm-ascend?
|
||||
There are several factors that affect output certainty:
|
||||
You may encounter the problem of C/C++ compilation failure when reinstalling vllm-ascend from source using pip. If the installation fails, use `python setup.py install` (recommended) to install, or use `python setup.py clean` to clear the cache.
|
||||
|
||||
1. Sampler Method: using **Greedy sample** by setting `temperature=0` in `SamplingParams`, e.g.:
|
||||
### 15. How to generate deterministic results when using vllm-ascend?
|
||||
|
||||
```python
|
||||
from vllm import LLM, SamplingParams
|
||||
There are several factors that affect output determinism:
|
||||
|
||||
prompts = [
|
||||
"Hello, my name is",
|
||||
"The president of the United States is",
|
||||
"The capital of France is",
|
||||
"The future of AI is",
|
||||
]
|
||||
1. Sampler method: using **greedy sampling** by setting `temperature=0` in `SamplingParams`, e.g.:
|
||||
|
||||
# Create a sampling params object.
|
||||
sampling_params = SamplingParams(temperature=0)
|
||||
# Create an LLM.
|
||||
llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct")
|
||||
```python
|
||||
from vllm import LLM, SamplingParams
|
||||
|
||||
# Generate texts from the prompts.
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for output in outputs:
|
||||
prompt = output.prompt
|
||||
generated_text = output.outputs[0].text
|
||||
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
|
||||
prompts = [
|
||||
"Hello, my name is",
|
||||
"The president of the United States is",
|
||||
"The capital of France is",
|
||||
"The future of AI is",
|
||||
]
|
||||
|
||||
# Create a sampling params object.
|
||||
sampling_params = SamplingParams(temperature=0)
|
||||
# Create an LLM.
|
||||
llm = LLM(model="Qwen/Qwen3-0.6B")
|
||||
|
||||
# Generate texts from the prompts.
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for output in outputs:
|
||||
prompt = output.prompt
|
||||
generated_text = output.outputs[0].text
|
||||
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
|
||||
```
|
||||
|
||||
2. Set the following environment parameters:
|
||||
|
||||
```bash
|
||||
export LCCL_DETERMINISTIC=1
|
||||
export HCCL_DETERMINISTIC=true
|
||||
export ATB_MATMUL_SHUFFLE_K_ENABLE=0
|
||||
export ATB_LLM_LCOC_ENABLE=0
|
||||
```
|
||||
|
||||
### 16. How to fix the error "ImportError: Please install vllm[audio] for audio support" for the multi-modal models?
|
||||
|
||||
Some multi-modal models requires the `librosa` package to be installed, you need to install the `qwen-omni-utils` package to ensure all dependencies are met, for Qwen-omni, run `pip install qwen-omni-utils`.
|
||||
This package will install `librosa` and its related dependencies, resolving the `ImportError: No module named 'librosa'` issue and ensuring that the audio processing functionality works correctly.
|
||||
|
||||
### 17. How to troubleshoot and resolve size capture failures resulting from stream resource exhaustion, and what are the underlying causes?
|
||||
|
||||
```text
|
||||
capture_begin:../torch_npu/csrc/core/npu/NPUGraph.cpp:230 NPU function error: c10_npu::acl::AclmdlRICaptureBegin(capture_stream_, capture_mode), error code is 207008
|
||||
[Error]: Stream resources are insufficient.
|
||||
[PID: ...] Insufficient_Stream_Resources(EL0009): The stream resources are insufficient.
|
||||
```
|
||||
|
||||
2. Set the following enveriments parameters:
|
||||
|
||||
```bash
|
||||
export LCCL_DETERMINISTIC=1
|
||||
export HCCL_DETERMINISTIC=true
|
||||
export ATB_MATMUL_SHUFFLE_K_ENABLE=0
|
||||
export ATB_LLM_LCOC_ENABLE=0
|
||||
```
|
||||
|
||||
### 19. How to fix the error "ImportError: Please install vllm[audio] for audio support" for Qwen2.5-Omni model？
|
||||
The `Qwen2.5-Omni` model requires the `librosa` package to be installed, you need to install the `qwen-omni-utils` package to ensure all dependencies are met `pip install qwen-omni-utils`,
|
||||
this package will install `librosa` and its related dependencies, resolving the `ImportError: No module named 'librosa'` issue and ensuring audio processing functionality works correctly.
|
||||
|
||||
### 20. How to troubleshoot and resolve size capture failures resulting from stream resource exhaustion, and what are the underlying causes?
|
||||
|
||||
```
|
||||
error example in detail:
|
||||
ERROR 09-26 10:48:07 [model_runner_v1.py:3029] ACLgraph sizes capture fail: RuntimeError:
|
||||
ERROR 09-26 10:48:07 [model_runner_v1.py:3029] ACLgraph has insufficient available streams to capture the configured number of sizes.Please verify both the availability of adequate streams and the appropriateness of the configured size count.
|
||||
```
|
||||
When vLLM Ascend recognizes this capture-time stream-resource signature in the error text, it re-raises the error with targeted guidance for ACL graph sizing and mitigation.
|
||||
|
||||
Recommended mitigation strategies:
|
||||
1. Manually configure the compilation_config parameter with a reduced size set: '{"cudagraph_capture_sizes":[size1, size2, size3, ...]}'.
|
||||
2. Employ ACLgraph's full graph mode as an alternative to the piece-wise approach.
|
||||
|
||||
1. Upgrade to a newer HDK/CANN stack if one is available for your environment. Recent releases improve ACL graph capacity, so older workarounds may no longer be necessary.
|
||||
2. Manually reduce the configured graph sizes, for example: '{"cudagraph_capture_sizes":[size1, size2, size3, ...]}', or lower `max_cudagraph_capture_size`.
|
||||
3. If your workload is mostly uniform decode, try ACLGraph's `FULL` or `FULL_DECODE_ONLY` mode instead of the `PIECEWISE`.
|
||||
4. If you use `PIECEWISE` or `FULL_AND_PIECEWISE` and still hit this failure after upgrading, set `cudagraph_capture_sizes` manually according to your real workload and reduce the configured coverage.
|
||||
5. If you are debugging a startup failure, temporarily disable graph mode (`cudagraph_mode="NONE"` / `enforce_eager=True`) to confirm the issue is capture-related.
|
||||
|
||||
Root cause analysis:
|
||||
The current stream requirement calculation for size captures only accounts for measurable factors including: data parallel size, tensor parallel size, expert parallel configuration, piece graph count, multistream overlap shared expert settings, and HCCL communication mode (AIV/AICPU). However, numerous unquantifiable elements - such as operator characteristics and specific hardware features - consume additional streams outside of this calculation framework, resulting in stream resource exhaustion during size capture operations.
|
||||
ACL graph capture can still fail when the runtime resources required by the selected graph sizes exceed what the current software/hardware stack can provide. This is most visible in `PIECEWISE` scenarios because the number of captured graphs scales with model depth and capture-size coverage. vLLM Ascend no longer auto-shrinks the PIECEWISE capture-size set locally, so the practical mitigations are to upgrade the HDK/CANN stack or reduce the configured graph sizes explicitly. The runtime guidance is intentionally narrow: it is only added when capture fails with the confirmed stream-resource signature above.
|
||||
|
||||
### 21. Installing vllm-ascend will overwrite the existing torch-npu package？
|
||||
Installing vllm-ascend will overwrite the existing torch-npu package. If you need to install a specific version of torch-npu, you can manually install the specified version of torch-npu after installing vllm-ascend.
|
||||
### 18. How to install custom version of torch_npu?
|
||||
|
||||
TorchNPU will be overridden when installing vllm-ascend. If you need to install a specific version of TorchNPU, you can manually install the specified version of TorchNPU after vllm-ascend is installed.
|
||||
|
||||
### 19. On certain systems (e.g., Kylin OS), `docker pull` may fail with an `invalid tar header` error
|
||||
|
||||
On certain operating systems, such as Kylin OS, you may encounter an `invalid tar header` error during the `docker pull` process:
|
||||
|
||||
```text
|
||||
failed to register layer: ApplyLayer exit status 1 stdout: stderr: archive/tar: invalid tar header
|
||||
```
|
||||
|
||||
This is often due to system compatibility issues. You can resolve this by using an offline loading method with a second machine.
|
||||
|
||||
1. On a separate host machine (e.g., a standard Ubuntu server), pull the image for the target ARM64 architecture and package it into a `.tar` file.
|
||||
|
||||
```bash
|
||||
export IMAGE_TAG=v0.10.0rc1-310p
|
||||
export IMAGE_NAME="quay.io/ascend/vllm-ascend:${IMAGE_TAG}"
|
||||
# If in China region, uncomment to use a mirror:
|
||||
# export IMAGE_NAME="m.daocloud.io/quay.io/ascend/vllm-ascend:${IMAGE_TAG}"
|
||||
|
||||
# Pull the image for the ARM64 platform and save it
|
||||
docker pull --platform linux/arm64 "${IMAGE_NAME}"
|
||||
docker save -o "vllm_ascend_${IMAGE_TAG}.tar" "${IMAGE_NAME}"
|
||||
```
|
||||
|
||||
2. Transfer the image archive
|
||||
|
||||
Copy the `vllm_ascend_<tag>.tar` file (where `<tag>` is the image tag you used) to your target machine
|
||||
|
||||
### 20. Why am I getting an error when executing the script to start a Docker container? The error message is: "operation not permitted"
|
||||
|
||||
When using `--shm-size`, you may need to add the `--privileged=true` flag to your `docker run` command to grant the container necessary permissions. Please be aware that using `--privileged=true` grants the container extensive privileges on the host system, which can be a security risk. Only use this option if you understand the implications and trust the container's source.
|
||||
|
||||
### 21. How to set `SOC_VERSION` when building from source on a CPU-only machine?
|
||||
|
||||
When building from source (e.g. `pip install -e .`), the build may try to infer the target chip via `npu-smi`. If `npu-smi` is not available (common in CPU-only build environments), you must set `SOC_VERSION` manually before installation.
|
||||
|
||||
You can use the defaults from `Dockerfile*` as a reference. For example:
|
||||
|
||||
```bash
|
||||
# Atlas A2
|
||||
export SOC_VERSION="ascend910b1"
|
||||
|
||||
# Atlas A3
|
||||
export SOC_VERSION="ascend910_9391"
|
||||
|
||||
# Atlas 300I
|
||||
export SOC_VERSION="ascend310p1"
|
||||
|
||||
# Ascend 950 Products
|
||||
export SOC_VERSION="<value starting with ascend950>"
|
||||
```
|
||||
|
||||
### 22. Why does TPOT increase drastically as concurrency grows?
|
||||
|
||||
When testing a vLLM server, one may find that TPOT increases as concurrency increases (for example, TPOT increases by 0.5 ~ 1ms when concurrency increases by 4). This phenomenon is normal in most cases. However, sometimes TPOT may increase dramatically (10 to 100ms for example) as concurrency grows. This is possibly caused by [**PREEMPTION**](https://docs.vllm.ai/en/latest/configuration/optimization/#preemption) in vLLM.
|
||||
Generally, when your server hits KV cache limits, vLLM tries to free KV cache of requests to ensure sufficient space for other requests, which is called preemption in vLLM. When a request is preempted, the default behavior is to recompute the KV cache of this request again in the future, which is why the performance might drop significantly. There are several ways to verify this:
|
||||
|
||||
- vLLM usually logs stats on your server. You might see metrics like `GPU KV cache usage: 99.0%,`. When reaching 100%, it triggers preemption.
|
||||
- When launching a vLLM server, you will see logs like `GPU KV cache size: 66340 tokens` and `Maximum concurrency for 16,384 tokens per request: 4.05`. These are estimated KV cache capacity for a single DP group. You can adjust the overall request traffic according to this.
|
||||
|
||||
Preemption cannot be avoided completely since KV cache usage always has a limit. But there are methods to reduce the chances of preemption. As is suggested in [**PREEMPTION**](https://docs.vllm.ai/en/latest/configuration/optimization/#preemption), the core strategy is to increase available KV cache. For example, one can increase `--gpu-memory-utilization` or decrease `--max-num-seqs` && `--max-num-batched-tokens`.
|
||||
|
||||
### 23. How do I choose between single-node and multi-node deployment?
|
||||
|
||||
Single-node deployment is recommended when the model fits within the memory of a single node's NPUs. For models like Qwen3-32B (BF16), which requires 4 × 64GB cards, multi-NPU within a single node (TP) is sufficient. Multi-node deployment is only needed when the total NPU count exceeds a single node's capacity.
|
||||
|
||||
### 24. What quantization method should I use?
|
||||
|
||||
- **BF16**: Best accuracy, highest memory footprint. Use for accuracy-critical applications or when memory is sufficient.
|
||||
- **W8A8**: Good balance of accuracy and memory reduction. Use for large models (e.g., 32B) on memory-constrained hardware.
|
||||
- **W4A8/W4A4**: Maximum memory reduction. Suitable for deploying larger models on smaller hardware configurations, with some accuracy trade-off.
|
||||
|
||||
### 25. When should I enable FlashComm_v1?
|
||||
|
||||
Enable FlashComm_v1 (`VLLM_ASCEND_ENABLE_FLASHCOMM1=1`) when using Tensor Parallelism (TP ≥ 2) with high concurrency. It is threshold-protected and will not activate in low-concurrency scenarios where it could degrade performance.
|
||||
|
||||
### 26. What is the difference between FIA and PA operators for attention?
|
||||
|
||||
FIA (Flash Attention) is the default attention operator in vLLM-Ascend. In some batch-size settings (particularly medium concurrency), FIA may exhibit suboptimal performance. The PA (Page Attention) operator can be manually enabled via `pa_shape_list` in `--additional-config`. When the runtime batch size matches a value in `pa_shape_list`, the framework switches to PA. This is a temporary tuning knob — future FIA optimizations will make this parameter obsolete.
|
||||
|
||||
@@ -21,11 +21,11 @@
|
||||
</p>
|
||||
:::
|
||||
|
||||
vLLM Ascend plugin (vllm-ascend) is a community maintained hardware plugin for running vLLM on the Ascend NPU.
|
||||
vLLM Ascend plugin (vllm-ascend) is a community-maintained hardware plugin for running vLLM on the Ascend NPU.
|
||||
|
||||
This plugin is the recommended approach for supporting the Ascend backend within the vLLM community. It adheres to the principles outlined in the [[RFC]: Hardware pluggable](https://github.com/vllm-project/vllm/issues/11162), providing a hardware-pluggable interface that decouples the integration of the Ascend NPU with vLLM.
|
||||
|
||||
By using vLLM Ascend plugin, popular open-source models, including Transformer-like, Mixture-of-Expert, Embedding, Multi-modal LLMs can run seamlessly on the Ascend NPU.
|
||||
By using vLLM Ascend plugin, popular open-source models, including Transformer-like, Mixture-of-Experts, Embedding, Multi-modal LLMs can run seamlessly on the Ascend NPU.
|
||||
|
||||
## Documentation
|
||||
|
||||
@@ -35,7 +35,8 @@ By using vLLM Ascend plugin, popular open-source models, including Transformer-l
|
||||
:maxdepth: 1
|
||||
quick_start
|
||||
installation
|
||||
tutorials/index.md
|
||||
tutorials/models/index
|
||||
tutorials/features/index
|
||||
faqs
|
||||
:::
|
||||
|
||||
@@ -46,6 +47,7 @@ faqs
|
||||
user_guide/support_matrix/index
|
||||
user_guide/configuration/index
|
||||
user_guide/feature_guide/index
|
||||
user_guide/deployment_guide/index
|
||||
user_guide/release_notes
|
||||
:::
|
||||
|
||||
@@ -54,10 +56,9 @@ user_guide/release_notes
|
||||
:caption: Developer Guide
|
||||
:maxdepth: 1
|
||||
developer_guide/contribution/index
|
||||
developer_guide/feature_guide/index
|
||||
developer_guide/Design_Documents/index
|
||||
developer_guide/evaluation/index
|
||||
developer_guide/performance/index
|
||||
developer_guide/modeling/index
|
||||
developer_guide/performance_and_debug/index
|
||||
:::
|
||||
|
||||
% How to involve vLLM Ascend
|
||||
@@ -66,6 +67,8 @@ developer_guide/modeling/index
|
||||
:maxdepth: 1
|
||||
community/governance
|
||||
community/contributors
|
||||
community/issue-workflow-guidelines
|
||||
community/slash-commands
|
||||
community/versioning_policy
|
||||
community/user_stories/index
|
||||
:::
|
||||
|
||||
@@ -4,25 +4,50 @@ This document describes how to install vllm-ascend manually.
|
||||
|
||||
## Requirements
|
||||
|
||||
:::::{tab-set}
|
||||
::::{tab-item} Atlas A2/A3/950DT inference products
|
||||
|
||||
- OS: Linux
|
||||
- Python: >= 3.9, < 3.12
|
||||
- A hardware with Ascend NPU. It's usually the Atlas 800 A2 series.
|
||||
- Python: >= 3.10, < 3.13
|
||||
- Hardware with Ascend NPUs. It's usually the Atlas 800 A2 series.
|
||||
- Atlas 300I DUO.
|
||||
- Software:
|
||||
|
||||
| Software | Supported version | Note |
|
||||
|---------------|----------------------------------|-------------------------------------------|
|
||||
| Ascend HDK | Refer to [here](https://www.hiascend.com/document/detail/zh/canncommercial/82RC1/releasenote/releasenote_0000.html) | Required for CANN |
|
||||
| CANN | >= 8.2.RC1 | Required for vllm-ascend and torch-npu |
|
||||
| torch-npu | >= 2.7.1.dev20250724 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
|
||||
| torch | >= 2.7.1 | Required for torch-npu and vllm |
|
||||
| Ascend HDK | Refer to the [CANN 9.1.0 Release Notes](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910/softwareinst/releasenote/9.1.0/release-notes.md) | Required for CANN |
|
||||
| CANN | == 9.1.0 | Required for vllm-ascend and TorchNPU |
|
||||
| TorchNPU | == 2.10.0.post4 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
|
||||
| torch | == 2.10.0 | Required for TorchNPU and vllm, No need to install manually, it will be auto installed in below steps |
|
||||
| NNAL | == 9.1.0 | Required for libatb.so, enables advanced tensor operations |
|
||||
|
||||
You have 2 way to install:
|
||||
- **Using pip**: first prepare env manually or via CANN image, then install `vllm-ascend` using pip.
|
||||
```{note}
|
||||
Atlas 300I DUO uses its platform-specific CANN 9.1.0 package; refer to the 310P table below for its requirements.
|
||||
```
|
||||
|
||||
::::
|
||||
::::{tab-item} Atlas 300I DUO
|
||||
|
||||
| Software | Supported version | Note |
|
||||
|---------------|----------------------------------|-------------------------------------------|
|
||||
| Ascend HDK | Refer to the [CANN 9.1.0 Release Notes](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910/softwareinst/releasenote/9.1.0/release-notes.md) | Required for CANN |
|
||||
| CANN | == 9.1.0 | Required for vllm-ascend and TorchNPU |
|
||||
| TorchNPU | == 2.10.0.post4 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps |
|
||||
| torch | == 2.10.0 | Required for TorchNPU and vllm, No need to install manually, it will be auto installed in below steps |
|
||||
| NNAL | == 9.1.0 | Required for libatb.so, enables advanced tensor operations |
|
||||
| triton / triton-ascend | Not supported | Uninstalled in `Dockerfile.310p` |
|
||||
|
||||
::::
|
||||
:::::
|
||||
|
||||
There are two installation methods:
|
||||
|
||||
- **Using pip**: first prepare the environment manually or via a CANN image, then install `vllm-ascend` using pip.
|
||||
- **Using docker**: use the `vllm-ascend` pre-built docker image directly.
|
||||
|
||||
## Configure a new environment
|
||||
## Configure Ascend CANN environment
|
||||
|
||||
Before installing, you need to make sure firmware/driver and CANN are installed correctly, refer to [link](https://ascend.github.io/docs/sources/ascend/quick_install.html) for more details.
|
||||
Before installation, you need to make sure firmware/driver, and CANN are installed correctly, refer to [CANN Installation](https://www.hiascend.com/cann/download?versionId=735&ids=d806%2Ch0501%2Ch0601%2Ch0702) for more details.
|
||||
|
||||
### Configure hardware environment
|
||||
|
||||
@@ -32,7 +57,7 @@ To verify that the Ascend NPU firmware and driver were correctly installed, run:
|
||||
npu-smi info
|
||||
```
|
||||
|
||||
Refer to [Ascend Environment Setup Guide](https://ascend.github.io/docs/sources/ascend/quick_install.html) for more details.
|
||||
Refer to [CANN Installation](https://www.hiascend.com/cann/download?versionId=735&ids=d806%2Ch0501%2Ch0601%2Ch0702) for more details.
|
||||
|
||||
### Configure software environment
|
||||
|
||||
@@ -45,6 +70,10 @@ Refer to [Ascend Environment Setup Guide](https://ascend.github.io/docs/sources/
|
||||
|
||||
The easiest way to prepare your software environment is using CANN image directly:
|
||||
|
||||
```{note}
|
||||
The CANN prebuilt image includes NNAL (Ascend Neural Network Acceleration Library), which provides libatb.so for advanced tensor operations. No additional installation is required when using the prebuilt image.
|
||||
```
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Update DEVICE according to your device (/dev/davinci[0-7])
|
||||
@@ -53,6 +82,7 @@ export DEVICE=/dev/davinci7
|
||||
export IMAGE=quay.io/ascend/cann:|cann_image_tag|
|
||||
docker run --rm \
|
||||
--name vllm-ascend-env \
|
||||
--shm-size=1g \
|
||||
--device $DEVICE \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
@@ -70,30 +100,31 @@ docker run --rm \
|
||||
:animate: fade-in-slide-down
|
||||
You can also install CANN manually:
|
||||
|
||||
```{warning}
|
||||
If you encounter "libatb.so not found" errors during runtime, please ensure NNAL is properly installed as shown in the manual installation steps below.
|
||||
```
|
||||
|
||||
```bash
|
||||
# Create a virtual environment
|
||||
# Create a virtual environment.
|
||||
python -m venv vllm-ascend-env
|
||||
source vllm-ascend-env/bin/activate
|
||||
|
||||
# Install required python packages.
|
||||
pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple attrs 'numpy<2.0.0' decorator sympy cffi pyyaml pathlib2 psutil protobuf scipy requests absl-py wheel typing_extensions
|
||||
# Install required Python packages.
|
||||
python -m pip install --upgrade pip
|
||||
pip3 install attrs numpy decorator sympy cffi pyyaml pathlib2 psutil protobuf scipy requests absl-py wheel typing_extensions
|
||||
|
||||
# Download and install the CANN package.
|
||||
wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.2.RC1/Ascend-cann-toolkit_8.2.RC1_linux-"$(uname -i)".run
|
||||
chmod +x ./Ascend-cann-toolkit_8.2.RC1_linux-"$(uname -i)".run
|
||||
./Ascend-cann-toolkit_8.2.RC1_linux-"$(uname -i)".run --full
|
||||
# https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Milan-ASL/Milan-ASL%20V100R001C22B800TP052/Ascend-cann-kernels-910b_8.2.rc1_linux-aarch64.run
|
||||
wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.1.0/Ascend-cann-toolkit_9.1.0_linux-"$(uname -i)".run
|
||||
chmod +x ./Ascend-cann-toolkit_9.1.0_linux-"$(uname -i)".run
|
||||
./Ascend-cann-toolkit_9.1.0_linux-"$(uname -i)".run --full
|
||||
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.2.RC1/Ascend-cann-kernels-910b_8.2.RC1_linux-"$(uname -i)".run
|
||||
chmod +x ./Ascend-cann-kernels-910b_8.2.RC1_linux-"$(uname -i)".run
|
||||
./Ascend-cann-kernels-910b_8.2.RC1_linux-"$(uname -i)".run --install
|
||||
wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.1.0/Ascend-cann-910b-ops_9.1.0_linux-"$(uname -i)".run
|
||||
chmod +x ./Ascend-cann-910b-ops_9.1.0_linux-"$(uname -i)".run
|
||||
./Ascend-cann-910b-ops_9.1.0_linux-"$(uname -i)".run --install
|
||||
|
||||
wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.2.RC1/Ascend-cann-nnal_8.2.RC1_linux-"$(uname -i)".run
|
||||
chmod +x ./Ascend-cann-nnal_8.2.RC1_linux-"$(uname -i)".run
|
||||
./Ascend-cann-nnal_8.2.RC1_linux-"$(uname -i)".run --install
|
||||
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.1.0/Ascend-cann-nnal_9.1.0_linux-"$(uname -i)".run
|
||||
chmod +x ./Ascend-cann-nnal_9.1.0_linux-"$(uname -i)".run
|
||||
./Ascend-cann-nnal_9.1.0_linux-"$(uname -i)".run --install
|
||||
```
|
||||
|
||||
:::
|
||||
@@ -102,22 +133,15 @@ source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
::::{tab-item} Before using docker
|
||||
:sync: docker
|
||||
No more extra step if you are using `vllm-ascend` prebuilt docker image.
|
||||
No extra steps are needed if you are using the `vllm-ascend` prebuilt Docker image.
|
||||
::::
|
||||
:::::
|
||||
|
||||
Once it's done, you can start to set up `vllm` and `vllm-ascend`.
|
||||
Once this is done, you can start to set up `vllm` and `vllm-ascend`.
|
||||
|
||||
## Setup vllm and vllm-ascend
|
||||
## Set up using Python
|
||||
|
||||
:::::{tab-set}
|
||||
:sync-group: install
|
||||
|
||||
::::{tab-item} Using pip
|
||||
:selected:
|
||||
:sync: pip
|
||||
|
||||
First install system dependencies and config pip mirror:
|
||||
First, install system dependencies and configure the pip mirror:
|
||||
|
||||
```bash
|
||||
# Using apt-get with mirror
|
||||
@@ -125,63 +149,150 @@ sed -i 's|ports.ubuntu.com|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list
|
||||
apt-get update -y && apt-get install -y gcc g++ cmake libnuma-dev wget git curl jq
|
||||
# Or using yum
|
||||
# yum update -y && yum install -y gcc g++ cmake numactl-devel wget git curl jq
|
||||
# Config pip mirror
|
||||
# Config pip mirror,only versions 0.11.0 and earlier are supported, if using a version later than 0.11.0, do not execute this command
|
||||
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
|
||||
```
|
||||
|
||||
**[Optional]** Then config the extra-index of `pip` if you are working on a x86 machine or using torch-npu dev version:
|
||||
**[Optional]** Then configure the extra-index of `pip` if you are working on an x86 machine or using TorchNPU dev version:
|
||||
|
||||
```bash
|
||||
# For torch-npu dev version or x86 machine
|
||||
pip config set global.extra-index-url "https://download.pytorch.org/whl/cpu/ https://mirrors.huaweicloud.com/ascend/repos/pypi"
|
||||
# For TorchNPU dev version or x86 machine
|
||||
pip config set global.extra-index-url "https://download.pytorch.org/whl/cpu/"
|
||||
```
|
||||
|
||||
Then you can install `vllm` and `vllm-ascend` from **pre-built wheel**:
|
||||
Then you can install `vllm` and `vllm-ascend` from a **pre-built wheel** using one of the following methods:
|
||||
|
||||
:::::{tab-set}
|
||||
:sync-group: install-method
|
||||
|
||||
::::{tab-item} Original installation
|
||||
:sync: original
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
|
||||
# Install vllm-project/vllm from pypi
|
||||
# Install vllm-project/vllm. The newest supported version is |vllm_version|.
|
||||
pip install vllm==|pip_vllm_version|
|
||||
|
||||
# Install vllm-project/vllm-ascend from pypi.
|
||||
pip install vllm-ascend==|pip_vllm_ascend_version|
|
||||
```
|
||||
# Install vllm-project/vllm-ascend.
|
||||
pip install \
|
||||
--extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi/variant \
|
||||
--extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi \
|
||||
vllm-ascend==|pip_vllm_ascend_version|
|
||||
|
||||
:::{dropdown} Click here to see "Build from source code"
|
||||
or build from **source code**:
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
|
||||
# Install vLLM
|
||||
git clone --depth 1 --branch |vllm_version| https://github.com/vllm-project/vllm
|
||||
cd vllm
|
||||
VLLM_TARGET_DEVICE=empty pip install -v -e .
|
||||
cd ..
|
||||
|
||||
# Install vLLM Ascend
|
||||
git clone --depth 1 --branch |vllm_ascend_version| https://github.com/vllm-project/vllm-ascend.git
|
||||
cd vllm-ascend
|
||||
pip install -v -e .
|
||||
cd ..
|
||||
```
|
||||
|
||||
vllm-ascend will build custom ops by default. If you don't want to build it, set `COMPILE_CUSTOM_KERNELS=0` environment to disable it.
|
||||
:::
|
||||
|
||||
```{note}
|
||||
If you are building from v0.7.3-dev and intend to use sleep mode feature, you should set `COMPILE_CUSTOM_KERNELS=1` manually.
|
||||
To build custom ops, gcc/g++ higher than 8 and c++ 17 or higher is required. If you're using `pip install -e .` and encounter a torch-npu version conflict, please install with `pip install --no-build-isolation -e .` to build on system env.
|
||||
If you encounter other problems during compiling, it is probably because unexpected compiler is being used, you may export `CXX_COMPILER` and `C_COMPILER` in env to specify your g++ and gcc locations before compiling.
|
||||
```
|
||||
|
||||
::::
|
||||
|
||||
::::{tab-item} Using docker
|
||||
:sync: docker
|
||||
::::{tab-item} uv-wheelnext installation
|
||||
:sync: uv-wheelnext
|
||||
|
||||
You can just pull the **prebuilt image** and run it with bash.
|
||||
The `uv-wheelnext` installation downloads only the delta on top of vllm, resulting in a smaller download size. First install `uv-wheelnext` to support incremental wheels:
|
||||
|
||||
```bash
|
||||
# install uv-wheelnext
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sed 's/verify_checksum "$_file"/true/' | INSTALLER_DOWNLOAD_URL=https://wheelnext.astral.sh sh
|
||||
source $HOME/.local/bin/env
|
||||
```
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
|
||||
# Install vllm-project/vllm. The newest supported version is |vllm_version|.
|
||||
pip install vllm==|pip_vllm_version|
|
||||
|
||||
# Install vllm-project/vllm-ascend from wheelnext index.
|
||||
uv pip install --system \
|
||||
--extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi/variant \
|
||||
--extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi \
|
||||
--index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple \
|
||||
vllm-ascend==|pip_vllm_ascend_version|
|
||||
|
||||
```
|
||||
|
||||
```{note}
|
||||
If you encounter errors during `uv pip install` (e.g., corrupted cache or stale package data), try clearing the uv cache first and then re-run the install command:
|
||||
|
||||
uv cache clean
|
||||
|
||||
```
|
||||
|
||||
::::
|
||||
:::::
|
||||
|
||||
:::{dropdown} Click here to see "Build from source code"
|
||||
or build from **source code**:
|
||||
|
||||
```{note}
|
||||
To install `triton-ascend`, run:
|
||||
|
||||
pip install triton-ascend==3.2.2 --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi
|
||||
|
||||
If you are installing via `uv`, make sure to install `triton-ascend` **last**, after all other packages have been installed, to avoid dependency resolution conflicts.
|
||||
```
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
|
||||
# Install vLLM.
|
||||
git clone --depth 1 --branch |vllm_version| https://github.com/vllm-project/vllm
|
||||
cd vllm
|
||||
VLLM_TARGET_DEVICE=empty pip install -e .
|
||||
cd ..
|
||||
|
||||
# Install vLLM Ascend.
|
||||
git clone --depth 1 --branch |vllm_ascend_version| https://github.com/vllm-project/vllm-ascend.git
|
||||
cd vllm-ascend
|
||||
git submodule update --init --recursive
|
||||
pip install -e .
|
||||
cd ..
|
||||
```
|
||||
|
||||
If you are building custom operators for Atlas A3, you should run `git submodule update --init --recursive` manually, or ensure your environment has internet access.
|
||||
:::
|
||||
|
||||
:::{note}
|
||||
Atlas 300I DUO does not support `triton` or `triton-ascend`. Source installations can pull these packages as dependencies; remove them before running on Atlas 300I DUO:
|
||||
|
||||
```bash
|
||||
pip uninstall -y triton-ascend triton
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
```{note}
|
||||
To build custom operators, gcc/g++ higher than 8 and C++17 or higher are required. If you are using `pip install -e .` and encounter a TorchNPU version conflict, please install with `pip install --no-build-isolation -e .` to build on system env.
|
||||
If you encounter other problems during compiling, it is probably because an unexpected compiler is being used, you may export `CXX_COMPILER` and `C_COMPILER` in the environment to specify your g++ and gcc locations before compiling.
|
||||
|
||||
If you are building in a CPU-only environment where `npu-smi` is unavailable, you need to set `SOC_VERSION` before `pip install -e .` so the build can target the correct chip. You can refer to `Dockerfile*` defaults, for example:
|
||||
|
||||
- Atlas A2: `export SOC_VERSION=ascend910b1`
|
||||
- Atlas A3: `export SOC_VERSION=ascend910_9391`
|
||||
- Atlas 300I DUO: `export SOC_VERSION=ascend310p1`
|
||||
- Atlas 950DT: `export SOC_VERSION=ascend950dt_9582`
|
||||
```
|
||||
|
||||
```{note}
|
||||
To enable the batch invariance feature, set `VLLM_BATCH_INVARIANT=1` before building vllm-ascend to install the batch invariance custom operator library during the installation process.
|
||||
For usage guidance on the batch invariance feature, see <https://github.com/vllm-project/vllm-ascend/blob/main/docs/source/user_guide/feature_guide/batch_invariance.md>
|
||||
```
|
||||
|
||||
## Set up using Docker
|
||||
|
||||
`vllm-ascend` offers Docker images for deployment. You can just pull the **prebuilt image** from the image repository [ascend/vllm-ascend](https://quay.io/repository/ascend/vllm-ascend?tab=tags) and run it with bash.
|
||||
|
||||
Supported images as following.
|
||||
|
||||
| image name | Hardware | OS |
|
||||
| - | - | - |
|
||||
| vllm-ascend:{{ vllm_ascend_version }} | Atlas A2 | Ubuntu |
|
||||
| vllm-ascend:{{ vllm_ascend_version }}-openeuler | Atlas A2 | openEuler |
|
||||
| vllm-ascend:{{ vllm_ascend_version }}-a3 | Atlas A3 | Ubuntu |
|
||||
| vllm-ascend:{{ vllm_ascend_version }}-a3-openeuler | Atlas A3 | openEuler |
|
||||
| vllm-ascend:{{ vllm_ascend_version }}-310p | Atlas 300I DUO | Ubuntu |
|
||||
| vllm-ascend:{{ vllm_ascend_version }}-310p-openeuler | Atlas 300I DUO | openEuler |
|
||||
| vllm-ascend:{{ vllm_ascend_version }}-a5 | Atlas 950DT | Ubuntu |
|
||||
| vllm-ascend:{{ vllm_ascend_version }}-a5-openeuler | Atlas 950DT | openEuler |
|
||||
|
||||
:::{dropdown} Click here to see "Build from Dockerfile"
|
||||
or build IMAGE from **source code**:
|
||||
@@ -194,15 +305,56 @@ docker build -t vllm-ascend-dev-image:latest -f ./Dockerfile .
|
||||
|
||||
:::
|
||||
|
||||
:::::{tab-set}
|
||||
::::{tab-item} A2/A3
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
|
||||
# Update DEVICE according to your device (/dev/davinci[0-7])
|
||||
export DEVICE=/dev/davinci7
|
||||
# Update the vllm-ascend image
|
||||
# Update --device according to your device (Atlas A2: /dev/davinci[0-7] Atlas A3:/dev/davinci[0-15] Atlas 950DT: /dev/davinci[0-7]).
|
||||
# Update the vllm-ascend image according to your environment.
|
||||
# Note you should download the weight to /root/.cache in advance.
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
docker run --rm \
|
||||
--name vllm-ascend-env \
|
||||
--shm-size=1g \
|
||||
--net=host \
|
||||
--device /dev/davinci0 \
|
||||
--device /dev/davinci1 \
|
||||
--device /dev/davinci2 \
|
||||
--device /dev/davinci3 \
|
||||
--device /dev/davinci4 \
|
||||
--device /dev/davinci5 \
|
||||
--device /dev/davinci6 \
|
||||
--device /dev/davinci7 \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
--device /dev/hisi_hdc \
|
||||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||||
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
|
||||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||||
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /root/.cache:/root/.cache \
|
||||
-it $IMAGE bash
|
||||
```
|
||||
|
||||
::::
|
||||
|
||||
::::{tab-item} Atlas 300I DUO
|
||||
|
||||
Adjust `/dev/davinci0` to the NPU you want to use.
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
|
||||
export DEVICE=/dev/davinci0
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p
|
||||
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--shm-size=1g \
|
||||
--device $DEVICE \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
@@ -213,14 +365,58 @@ docker run --rm \
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /root/.cache:/root/.cache \
|
||||
-p 8000:8000 \
|
||||
-it $IMAGE bash
|
||||
```
|
||||
|
||||
The default workdir is `/workspace`, vLLM and vLLM Ascend code are placed in `/vllm-workspace` and installed in [development mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html)(`pip install -e`) to help developer immediately take place changes without requiring a new installation.
|
||||
::::
|
||||
|
||||
::::{tab-item} Atlas 200I Pro
|
||||
|
||||
Atlas 200I Pro requires additional device nodes, driver libraries, and configuration files so that `npu-smi` and other driver commands work inside the container. Adjust `/dev/davinci0` to the NPU you want to use.
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p
|
||||
|
||||
docker run --rm \
|
||||
--privileged \
|
||||
--name vllm-ascend \
|
||||
--shm-size=10g \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci_manager \
|
||||
--device=/dev/ascend_manager \
|
||||
--device=/dev/user_config \
|
||||
-v /etc/sys_version.conf:/etc/sys_version.conf \
|
||||
-v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
|
||||
-v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
|
||||
-v /var/dmp_daemon:/var/dmp_daemon \
|
||||
-v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
|
||||
-v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
|
||||
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
|
||||
-v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
|
||||
-v /usr/lib/aarch64-linux-gnu/libyaml-0.so.2:/usr/lib64/libyaml-0.so.2 \
|
||||
-v /etc/slog.conf:/etc/slog.conf \
|
||||
-v /var/slogd:/var/slogd \
|
||||
-v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
|
||||
-v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
|
||||
-v /root/.cache:/root/.cache \
|
||||
-p 8000:8000 \
|
||||
-it $IMAGE bash
|
||||
```
|
||||
|
||||
For openEuler, keep the same command structure and make the following substitutions:
|
||||
|
||||
- Set `IMAGE` to `quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p-openeuler`.
|
||||
- Add `-v /usr/lib64/libsemanage.so.2:/usr/lib64/libsemanage.so.2`.
|
||||
- Replace the `libyaml` mount with `-v /usr/lib64/libyaml-0.so.2.0.9:/usr/lib64/libyaml-0.so.2`.
|
||||
|
||||
::::
|
||||
:::::
|
||||
|
||||
The default workdir is `/workspace`, vLLM and vLLM Ascend code are placed in `/vllm-workspace` and installed in [development mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html) (`pip install -e`) to help developers immediately make changes without requiring a new installation.
|
||||
|
||||
## Extra information
|
||||
|
||||
### Verify installation
|
||||
@@ -240,7 +436,7 @@ prompts = [
|
||||
# Create a sampling params object.
|
||||
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
|
||||
# Create an LLM.
|
||||
llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct")
|
||||
llm = LLM(model="Qwen/Qwen3-0.6B")
|
||||
|
||||
# Generate texts from the prompts.
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
@@ -253,31 +449,317 @@ for output in outputs:
|
||||
Then run:
|
||||
|
||||
```bash
|
||||
# Try `export VLLM_USE_MODELSCOPE=true` and `pip install modelscope`
|
||||
# to speed up download if huggingface is not reachable.
|
||||
python example.py
|
||||
```
|
||||
|
||||
The output will be like:
|
||||
If you encounter a connection error with Hugging Face (e.g., `We couldn't connect to 'https://huggingface.co' to load the files, and couldn't find them in the cached files.`), run the following commands to use ModelScope as an alternative:
|
||||
|
||||
```bash
|
||||
INFO 02-18 08:49:58 __init__.py:28] Available plugins for group vllm.platform_plugins:
|
||||
INFO 02-18 08:49:58 __init__.py:30] name=ascend, value=vllm_ascend:register
|
||||
INFO 02-18 08:49:58 __init__.py:32] all available plugins for group vllm.platform_plugins will be loaded.
|
||||
INFO 02-18 08:49:58 __init__.py:34] set environment variable VLLM_PLUGINS to control which plugins to load.
|
||||
INFO 02-18 08:49:58 __init__.py:42] plugin ascend loaded.
|
||||
INFO 02-18 08:49:58 __init__.py:174] Platform plugin ascend is activated
|
||||
INFO 02-18 08:50:12 config.py:526] This model supports multiple tasks: {'embed', 'classify', 'generate', 'score', 'reward'}. Defaulting to 'generate'.
|
||||
INFO 02-18 08:50:12 llm_engine.py:232] Initializing a V0 LLM engine (v0.7.1) with config: model='./Qwen2.5-0.5B-Instruct', speculative_config=None, tokenizer='./Qwen2.5-0.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=npu, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=./Qwen2.5-0.5B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False,
|
||||
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
|
||||
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 5.86it/s]
|
||||
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 5.85it/s]
|
||||
INFO 02-18 08:50:24 executor_base.py:108] # CPU blocks: 35064, # CPU blocks: 2730
|
||||
INFO 02-18 08:50:24 executor_base.py:113] Maximum concurrency for 32768 tokens per request: 136.97x
|
||||
INFO 02-18 08:50:25 llm_engine.py:429] init engine (profile, create kv cache, warmup model) took 3.87 seconds
|
||||
Processed prompts: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 8.46it/s, est. speed input: 46.55 toks/s, output: 135.41 toks/s]
|
||||
Prompt: 'Hello, my name is', Generated text: " Shinji, a teenage boy from New York City. I'm a computer science"
|
||||
Prompt: 'The president of the United States is', Generated text: ' a very important person. When he or she is elected, many people think that'
|
||||
Prompt: 'The capital of France is', Generated text: ' Paris. The oldest part of the city is Saint-Germain-des-Pr'
|
||||
Prompt: 'The future of AI is', Generated text: ' not bright\n\nThere is no doubt that the evolution of AI will have a huge'
|
||||
export VLLM_USE_MODELSCOPE=True
|
||||
pip install modelscope
|
||||
python example.py
|
||||
```
|
||||
|
||||
```{note}
|
||||
If you encounter custom-op security verification errors while running inference on Atlas 950DT, refer to [Pooling enables UB and UBoE for 950DT and 950PR](https://gitcode.com/Ascend/memcache/wiki/%E6%B1%A0%E5%8C%96%E4%BD%BF%E8%83%BD950DT%E5%92%8C950PR%E7%9A%84UB%E5%92%8CUBoE.md) and run the following commands:
|
||||
|
||||
> Each NPU will prompt for confirmation when running the first command. You must manually enter `Y` for all of them.
|
||||
|
||||
```bash
|
||||
for i in {0..7}; do npu-smi set -t custom-op-secverify-enable -i $i -d 1; done;
|
||||
for i in {0..7}; do npu-smi set -t custom-op-secverify-mode -i $i -d 0; done;
|
||||
```
|
||||
|
||||
This section shows ascend platform is successfully detected in vllm:
|
||||
|
||||
```bash
|
||||
INFO 05-27 11:40:38 [__init__.py:44] Available plugins for group vllm.platform_plugins:
|
||||
INFO 05-27 11:40:38 [__init__.py:46] - ascend -> vllm_ascend:register
|
||||
INFO 05-27 11:40:38 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
|
||||
INFO 05-27 11:40:38 [__init__.py:238] Platform plugin ascend is activated
|
||||
```
|
||||
|
||||
This section shows the final output:
|
||||
|
||||
```bash
|
||||
Prompt: 'Hello, my name is', Generated text: ' Lucy and I am an 8 year old who loves to draw and write stories'
|
||||
Prompt: 'The president of the United States is', Generated text: " a key leader in the federal government, and the president's role in the executive"
|
||||
Prompt: 'The capital of France is', Generated text: ' a city. What is the capital of France? The capital of France is Paris'
|
||||
Prompt: 'The future of AI is', Generated text: ' a topic that is being discussed in various contexts. In the business world, AI'
|
||||
```
|
||||
|
||||
This section shows process exits after offline inference, and does not affect actual inference:
|
||||
|
||||
```bash
|
||||
(EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1201] Shutdown initiated (timeout=0)
|
||||
(EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1224] Shutdown complete
|
||||
ERROR 05-12 11:36:01 [core_client.py:704] Engine core proc EngineCore died unexpectedly, shutting down client.
|
||||
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
|
||||
```
|
||||
|
||||
## Multi-node Deployment
|
||||
|
||||
### Verify Multi-Node Communication
|
||||
|
||||
First, check physical layer connectivity, then verify each node, and finally verify the inter-node connectivity.
|
||||
|
||||
#### Physical Layer Requirements
|
||||
|
||||
- The physical machines must be located on the same LAN, with network connectivity.
|
||||
- All NPUs are connected with optical modules, and the connection status must be normal.
|
||||
|
||||
#### Each Node Verification
|
||||
|
||||
Execute the following commands on each node in sequence. The results must all be `success` and the status must be `UP`:
|
||||
|
||||
:::::{tab-set}
|
||||
:sync-group: multi-node
|
||||
|
||||
::::{tab-item} A2 series
|
||||
:sync: A2
|
||||
|
||||
```bash
|
||||
# Check the remote switch ports
|
||||
for i in {0..7}; do hccn_tool -i $i -lldp -g | grep Ifname; done
|
||||
# Get the link status of the Ethernet ports (UP or DOWN)
|
||||
for i in {0..7}; do hccn_tool -i $i -link -g ; done
|
||||
# Check the network health status
|
||||
for i in {0..7}; do hccn_tool -i $i -net_health -g ; done
|
||||
# View the network detected IP configuration
|
||||
for i in {0..7}; do hccn_tool -i $i -netdetect -g ; done
|
||||
# View gateway configuration
|
||||
for i in {0..7}; do hccn_tool -i $i -gateway -g ; done
|
||||
# View NPU network configuration
|
||||
cat /etc/hccn.conf
|
||||
```
|
||||
|
||||
::::
|
||||
::::{tab-item} A3 series
|
||||
:sync: A3
|
||||
|
||||
```bash
|
||||
# Check the remote switch ports
|
||||
for i in {0..15}; do hccn_tool -i $i -lldp -g | grep Ifname; done
|
||||
# Get the link status of the Ethernet ports (UP or DOWN)
|
||||
for i in {0..15}; do hccn_tool -i $i -link -g ; done
|
||||
# Check the network health status
|
||||
for i in {0..15}; do hccn_tool -i $i -net_health -g ; done
|
||||
# View the network detected IP configuration
|
||||
for i in {0..15}; do hccn_tool -i $i -netdetect -g ; done
|
||||
# View gateway configuration
|
||||
for i in {0..15}; do hccn_tool -i $i -gateway -g ; done
|
||||
# View NPU network configuration
|
||||
cat /etc/hccn.conf
|
||||
```
|
||||
|
||||
::::
|
||||
::::{tab-item} 950DT series
|
||||
:sync: 950DT
|
||||
|
||||
```bash
|
||||
# Check the remote switch ports
|
||||
for i in {0..7}; do hccn_tool -i $i -lldp -g | grep Ifname; done
|
||||
# Get the link status of the Ethernet ports (UP or DOWN)
|
||||
for i in {0..7}; do hccn_tool -i $i -link -g ; done
|
||||
# Check the network health status
|
||||
for i in {0..7}; do hccn_tool -i $i -net_health -g ; done
|
||||
# View the network detected IP configuration
|
||||
for i in {0..7}; do hccn_tool -i $i -netdetect -g ; done
|
||||
# View gateway configuration
|
||||
for i in {0..7}; do hccn_tool -i $i -gateway -g ; done
|
||||
# View NPU network configuration
|
||||
cat /etc/hccn.conf
|
||||
```
|
||||
|
||||
::::
|
||||
:::::
|
||||
|
||||
#### Interconnect Verification
|
||||
|
||||
##### 1. Get NPU IP Addresses
|
||||
|
||||
:::::{tab-set}
|
||||
:sync-group: multi-node
|
||||
|
||||
::::{tab-item} A2 series
|
||||
:sync: A2
|
||||
|
||||
```bash
|
||||
for i in {0..7}; do hccn_tool -i $i -ip -g | grep ipaddr; done
|
||||
```
|
||||
|
||||
::::
|
||||
::::{tab-item} A3 series
|
||||
:sync: A3
|
||||
|
||||
```bash
|
||||
for i in {0..15}; do hccn_tool -i $i -ip -g | grep ipaddr; done
|
||||
```
|
||||
|
||||
::::
|
||||
::::{tab-item} 950DT series
|
||||
:sync: 950DT
|
||||
|
||||
```bash
|
||||
for i in {0..7}; do hccn_tool -i $i -ip -g | grep ipaddr; done
|
||||
```
|
||||
|
||||
::::
|
||||
:::::
|
||||
|
||||
##### 2. Cross-Node PING Test
|
||||
|
||||
```bash
|
||||
# Execute on the target node (replace with actual IP)
|
||||
hccn_tool -i 0 -ping -g address x.x.x.x
|
||||
```
|
||||
|
||||
### Atlas 950 Series Server Pre-check
|
||||
|
||||
This pre-check applies only to Atlas 950 series servers. Other server series can skip it.
|
||||
|
||||
- **Prepare HiXLEP configuration paths**
|
||||
|
||||
When deploying an inference service on Atlas 950 series servers, verify on each server that `/lib/route.conf`, `/etc/hccl_rootinfo.json`, and the `/etc/hixlep` directory (which describes the UB link topology) exist and are configured correctly. If any of them are missing or incorrect, follow the [HiXLEP configuration file generation guide](https://gitcode.com/cann/hixl/wiki/A5%20LocalCommRes%E9%85%8D%E7%BD%AE%E6%8C%87%E5%8D%97.md) to generate the required content. When generating `/etc/hixlep`, use the "D2D scenario".
|
||||
|
||||
### Run Container In Each Node
|
||||
|
||||
Using vLLM-ascend official container is more efficient to run multi-node environment.
|
||||
|
||||
Run the following command to start the container in each node (You should download the weight to /root/.cache in advance):
|
||||
|
||||
:::::{tab-set}
|
||||
:sync-group: multi-node
|
||||
|
||||
::::{tab-item} A2 series
|
||||
:sync: A2
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Update the vllm-ascend image
|
||||
# openEuler:
|
||||
# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-openeuler
|
||||
# Ubuntu:
|
||||
# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|
|
||||
|
||||
# Run the container using the defined variables
|
||||
# Note if you are running bridge network with docker, Please expose available ports
|
||||
# for multiple nodes communication in advance
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--net=host \
|
||||
--shm-size=1g \
|
||||
--device /dev/davinci0 \
|
||||
--device /dev/davinci1 \
|
||||
--device /dev/davinci2 \
|
||||
--device /dev/davinci3 \
|
||||
--device /dev/davinci4 \
|
||||
--device /dev/davinci5 \
|
||||
--device /dev/davinci6 \
|
||||
--device /dev/davinci7 \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
--device /dev/hisi_hdc \
|
||||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||||
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
|
||||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||||
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /root/.cache:/root/.cache \
|
||||
-it $IMAGE bash
|
||||
```
|
||||
|
||||
::::
|
||||
::::{tab-item} A3 series
|
||||
:sync: A3
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Update the vllm-ascend image
|
||||
# openEuler:
|
||||
# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3-openeuler
|
||||
# Ubuntu:
|
||||
# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3
|
||||
|
||||
# Run the container using the defined variables
|
||||
# Note if you are running bridge network with docker, Please expose available ports
|
||||
# for multiple nodes communication in advance
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--net=host \
|
||||
--shm-size=1g \
|
||||
--device /dev/davinci0 \
|
||||
--device /dev/davinci1 \
|
||||
--device /dev/davinci2 \
|
||||
--device /dev/davinci3 \
|
||||
--device /dev/davinci4 \
|
||||
--device /dev/davinci5 \
|
||||
--device /dev/davinci6 \
|
||||
--device /dev/davinci7 \
|
||||
--device /dev/davinci8 \
|
||||
--device /dev/davinci9 \
|
||||
--device /dev/davinci10 \
|
||||
--device /dev/davinci11 \
|
||||
--device /dev/davinci12 \
|
||||
--device /dev/davinci13 \
|
||||
--device /dev/davinci14 \
|
||||
--device /dev/davinci15 \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
--device /dev/hisi_hdc \
|
||||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||||
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
|
||||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||||
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /root/.cache:/root/.cache \
|
||||
-it $IMAGE bash
|
||||
```
|
||||
|
||||
::::
|
||||
::::{tab-item} 950DT series
|
||||
:sync: 950DT
|
||||
|
||||
```{code-block} bash
|
||||
:substitutions:
|
||||
# Update the vllm-ascend image
|
||||
# openEuler:
|
||||
# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a5-openeuler
|
||||
# Ubuntu:
|
||||
# export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a5
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a5
|
||||
|
||||
# Run the container using the defined variables
|
||||
# Note if you are running bridge network with docker, Please expose available ports
|
||||
# for multiple nodes communication in advance
|
||||
docker run --rm \
|
||||
--name vllm-ascend \
|
||||
--net=host \
|
||||
--shm-size=1g \
|
||||
--device /dev/davinci0 \
|
||||
--device /dev/davinci1 \
|
||||
--device /dev/davinci2 \
|
||||
--device /dev/davinci3 \
|
||||
--device /dev/davinci4 \
|
||||
--device /dev/davinci5 \
|
||||
--device /dev/davinci6 \
|
||||
--device /dev/davinci7 \
|
||||
--device /dev/davinci_manager \
|
||||
--device /dev/devmm_svm \
|
||||
--device /dev/hisi_hdc \
|
||||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||||
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
|
||||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||||
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /root/.cache:/root/.cache \
|
||||
-it $IMAGE bash
|
||||
```
|
||||
|
||||
::::
|
||||
:::::
|
||||
|
||||
53
docs/source/llms.txt
Normal file
53
docs/source/llms.txt
Normal file
@@ -0,0 +1,53 @@
|
||||
# vLLM Ascend Plugin
|
||||
> vLLM Ascend is a community-maintained hardware plugin for running vLLM on Huawei Ascend NPUs.
|
||||
|
||||
Each section links to canonical HTML pages. If a public Markdown twin is available and reachable, it may be used. (e.g., latest/installation.html --> latest/_sources/installation.md).
|
||||
|
||||
Official repository: https://github.com/vllm-project/vllm-ascend
|
||||
Documentation: https://docs.vllm.ai/projects/ascend/en/latest/index.html
|
||||
|
||||
## Getting Started
|
||||
- [Installation](https://docs.vllm.ai/projects/ascend/en/latest/installation.html): environment preparation and installation methods.
|
||||
- [Quick Start](https://docs.vllm.ai/projects/ascend/en/latest/quick_start.html): first successful serve and inference flow.
|
||||
- [Model Tutorials](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/index.html): model-specific guides.
|
||||
- [Feature Tutorials](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/index.html): feature-focused walkthroughs.
|
||||
- [Hardware Tutorials](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/hardwares/index.html): hardware-specific setup and usage.
|
||||
- [FAQs](https://docs.vllm.ai/projects/ascend/en/latest/faqs.html): common issues and troubleshooting.
|
||||
|
||||
## Requirements
|
||||
- Hardware: Atlas 800I A2 Inference, Atlas A2 Training, Atlas 800I A3 Inference, Atlas A3 Training, Atlas 300I Duo (Experimental).
|
||||
- For software and version requirements, follow the official Installation and Release Notes pages.
|
||||
|
||||
## Features and Models
|
||||
- [Supported Models Matrix](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/support_matrix/supported_models.html): source of truth for support status (`✅`/`🔵`/`❌`), hardware scope, and feature coverage.
|
||||
- [Supported Features Matrix](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/support_matrix/supported_features.html): feature availability per hardware and model type.
|
||||
|
||||
## Features and Configuration
|
||||
- [Additional Configuration](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/configuration/additional_config.html): Ascend-specific knobs and runtime options.
|
||||
- [Quantization Guide](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/quantization.html): supported quantization flows and usage.
|
||||
- [Graph Mode Guide](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/graph_mode.html): graph execution modes and constraints.
|
||||
- [Feature Guide Index](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/index.html): all feature guides.
|
||||
- [Deployment Guide Index](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/deployment_guide/index.html): deployment patterns and operations.
|
||||
- [Performance Benchmark Guide](https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/performance_and_debug/performance_benchmark.html): benchmarking methodology and commands.
|
||||
|
||||
## Release and Versioning
|
||||
- [Release Notes](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/release_notes.html): user-facing changes by version.
|
||||
- [Versioning Policy](https://docs.vllm.ai/projects/ascend/en/latest/community/versioning_policy.html): branch and release policy.
|
||||
|
||||
## Developer Guide
|
||||
- [Contribution Guide](https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/index.html): setup and contribution workflow.
|
||||
- [Developer Design Documents](https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/Design_Documents/index.html): developer-side feature docs.
|
||||
- [Evaluation Guide](https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/evaluation/index.html): evaluation workflow.
|
||||
- [Performance and Debug Index](https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/performance_and_debug/index.html): profiling and debugging docs.
|
||||
|
||||
## Community and Support
|
||||
- [GitHub Issues](https://github.com/vllm-project/vllm-ascend/issues): bug reports and feature requests.
|
||||
- [vLLM Discuss Forum (Ascend)](https://discuss.vllm.ai/c/hardware-support/vllm-ascend-support): usage Q&A.
|
||||
- [Slack `#SIG-Ascend`](https://slack.vllm.ai): community channel.
|
||||
- [Weekly Meeting](https://tinyurl.com/vllm-ascend-meeting): Wednesday 15:00-16:00 (UTC+8).
|
||||
- [Governance](https://docs.vllm.ai/projects/ascend/en/latest/community/governance.html): project governance.
|
||||
- [Contributors](https://docs.vllm.ai/projects/ascend/en/latest/community/contributors.html): contributor roster and acknowledgements.
|
||||
- [User Stories](https://docs.vllm.ai/projects/ascend/en/latest/community/user_stories/index.html): community usage cases.
|
||||
|
||||
## License
|
||||
- [Apache License 2.0](https://github.com/vllm-project/vllm-ascend/blob/main/LICENSE).
|
||||
File diff suppressed because it is too large
Load Diff
@@ -2,14 +2,14 @@
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2025.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-07-18 09:01+0800\n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
@@ -18,187 +18,182 @@ msgstr ""
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.17.0\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../community/governance.md:1
|
||||
#: ../../source/community/governance.md:1
|
||||
msgid "Governance"
|
||||
msgstr "治理"
|
||||
|
||||
#: ../../community/governance.md:3
|
||||
#: ../../source/community/governance.md:3
|
||||
msgid "Mission"
|
||||
msgstr "使命"
|
||||
|
||||
#: ../../community/governance.md:4
|
||||
#: ../../source/community/governance.md:5
|
||||
msgid ""
|
||||
"As a vital component of vLLM, the vLLM Ascend project is dedicated to "
|
||||
"providing an easy, fast, and cheap LLM Serving for Everyone on Ascend NPU, "
|
||||
"providing an easy, fast, and cheap LLM Serving for everyone on Ascend NPUs "
|
||||
"and to actively contribute to the enrichment of vLLM."
|
||||
msgstr ""
|
||||
"作为 vLLM 的重要组成部分，vLLM Ascend 项目致力于为所有人在 Ascend NPU 上提供简单、快速且低成本的大语言模型服务，并积极促进"
|
||||
" vLLM 的丰富发展。"
|
||||
"作为vLLM的重要组成部分，vLLM "
|
||||
"Ascend项目致力于在昇腾NPU上为所有人提供简单、快速且低成本的LLM服务，并积极为vLLM的丰富发展做出贡献。"
|
||||
|
||||
#: ../../community/governance.md:6
|
||||
#: ../../source/community/governance.md:7
|
||||
msgid "Principles"
|
||||
msgstr "原则"
|
||||
|
||||
#: ../../community/governance.md:7
|
||||
#: ../../source/community/governance.md:9
|
||||
msgid ""
|
||||
"vLLM Ascend follows the vLLM community's code of conduct：[vLLM - CODE OF "
|
||||
"vLLM Ascend follows the vLLM community's code of conduct: [vLLM - CODE OF "
|
||||
"CONDUCT](https://github.com/vllm-project/vllm/blob/main/CODE_OF_CONDUCT.md)"
|
||||
msgstr ""
|
||||
"vLLM Ascend 遵循 vLLM 社区的行为准则：[vLLM - 行为准则](https://github.com/vllm-"
|
||||
"project/vllm/blob/main/CODE_OF_CONDUCT.md)"
|
||||
|
||||
#: ../../community/governance.md:9
|
||||
#: ../../source/community/governance.md:11
|
||||
msgid "Governance - Mechanics"
|
||||
msgstr "治理 - 机制"
|
||||
msgstr "治理机制"
|
||||
|
||||
#: ../../community/governance.md:10
|
||||
#: ../../source/community/governance.md:13
|
||||
msgid ""
|
||||
"vLLM Ascend is an open-source project under the vLLM community, where the "
|
||||
"authority to appoint roles is ultimately determined by the vLLM community. "
|
||||
"It adopts a hierarchical technical governance structure."
|
||||
msgstr "vLLM Ascend 是 vLLM 社区下的一个开源项目，其角色任命权最终由 vLLM 社区决定。它采用分层的技术治理结构。"
|
||||
msgstr "vLLM Ascend 是 vLLM 社区下的一个开源项目，角色任命的权力最终由 vLLM 社区决定。它采用分层技术治理结构。"
|
||||
|
||||
#: ../../community/governance.md:12
|
||||
#: ../../source/community/governance.md:15
|
||||
msgid "Contributor:"
|
||||
msgstr "贡献者："
|
||||
|
||||
#: ../../community/governance.md:14
|
||||
#: ../../source/community/governance.md:17
|
||||
msgid ""
|
||||
"**Responsibility:** Help new contributors on boarding, handle and respond to"
|
||||
" community questions, review RFCs, code"
|
||||
msgstr "**职责：** 帮助新贡献者加入，处理和回复社区问题，审查RFC和代码"
|
||||
"**Responsibility:** Help new contributors with onboarding, handle and "
|
||||
"respond to community questions, review RFCs and code."
|
||||
msgstr "**职责：** 帮助新贡献者入门，处理和回复社区问题，审查 RFC 和代码。"
|
||||
|
||||
#: ../../community/governance.md:16
|
||||
#: ../../source/community/governance.md:19
|
||||
msgid ""
|
||||
"**Requirements:** Complete at least 1 contribution. Contributor is someone "
|
||||
"who consistently and actively participates in a project, included but not "
|
||||
"**Requirements:** Complete at least 1 contribution. A contributor is someone"
|
||||
" who consistently and actively participates in a project, including but not "
|
||||
"limited to issue/review/commits/community involvement."
|
||||
msgstr "**要求：** 完成至少1次贡献。贡献者是指持续且积极参与项目的人，包括但不限于问题、评审、提交和社区参与。"
|
||||
msgstr "**要求：** 完成至少 1 次贡献。贡献者是持续积极参与项目的人员，包括但不限于问题/审查/提交/社区参与。"
|
||||
|
||||
#: ../../community/governance.md:18
|
||||
#: ../../source/community/governance.md:21
|
||||
msgid ""
|
||||
"Contributors will be empowered [vllm-project/vllm-"
|
||||
"ascend](https://github.com/vllm-project/vllm-ascend) Github repo `Triage` "
|
||||
"permissions (`Can read and clone this repository. Can also manage issues and"
|
||||
" pull requests`) to help community developers collaborate more efficiently."
|
||||
"Contributors are granted the `Triage` role in the [vllm-project/vllm-"
|
||||
"ascend](https://github.com/vllm-project/vllm-ascend) GitHub repository, "
|
||||
"which allows them to read and clone the repo, manage issues and PRs, and "
|
||||
"collaborate efficiently with community developers."
|
||||
msgstr ""
|
||||
"贡献者将被赋予 [vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-"
|
||||
"ascend) Github 仓库的 `Triage` 权限（`可读取和克隆此仓库。还可以管理问题和拉取请求`），以帮助社区开发者更加高效地协作。"
|
||||
"贡献者将被授予[vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-"
|
||||
"ascend) GitHub仓库中的`Triage`角色，该角色允许他们读取和克隆仓库、管理问题和PR，并与社区开发者高效协作。"
|
||||
|
||||
#: ../../community/governance.md:20
|
||||
#: ../../source/community/governance.md:23
|
||||
msgid "Maintainer:"
|
||||
msgstr "维护者："
|
||||
|
||||
#: ../../community/governance.md:22
|
||||
#: ../../source/community/governance.md:25
|
||||
msgid ""
|
||||
"**Responsibility:** Develop the project's vision and mission. Maintainers "
|
||||
"are responsible for driving the technical direction of the entire project "
|
||||
"and ensuring its overall success, possessing code merge permissions. They "
|
||||
"formulate the roadmap, review contributions from community members, "
|
||||
"continuously contribute code, and actively engage in community activities "
|
||||
"(such as regular meetings/events)."
|
||||
"are responsible for shaping the technical direction of the project and "
|
||||
"ensuring its long-term success. With code merge permissions, they lead "
|
||||
"roadmap planning, review community contributions, make ongoing code "
|
||||
"improvements, and actively participate in community engagement—such as "
|
||||
"regular meetings and events."
|
||||
msgstr ""
|
||||
"**责任：** "
|
||||
"制定项目的愿景和使命。维护者负责引领整个项目的技术方向并确保其整体成功，拥有代码合并权限。他们制定路线图，审核社区成员的贡献，持续贡献代码，并积极参与社区活动（如定期会议/活动）。"
|
||||
"**职责：** "
|
||||
"制定项目的愿景和使命。维护者负责塑造项目的技术方向并确保其长期成功。拥有代码合并权限，主导路线图规划，审查社区贡献，持续改进代码，并积极参与社区互动——例如定期会议和活动。"
|
||||
|
||||
#: ../../community/governance.md:24
|
||||
#: ../../source/community/governance.md:27
|
||||
msgid ""
|
||||
"**Requirements:** Deep understanding of vLLM and vLLM Ascend codebases, "
|
||||
"with a commitment to sustained code contributions. Competency in "
|
||||
"design/development/PR review workflows."
|
||||
msgstr ""
|
||||
"**要求：** 深入理解 vLLM 和 vLLM Ascend 代码库，并承诺持续贡献代码。具备 设计/开发/PR 审核流程 的能力。"
|
||||
"**Requirements:** Deep understanding of vLLM and vLLM Ascend code bases,"
|
||||
" with a commitment to sustained code contributions and competency in "
|
||||
"design, development, and PR review workflows."
|
||||
msgstr "**要求：** 深入理解 vLLM 和 vLLM Ascend 代码库，致力于持续贡献代码，并具备设计、开发和 PR 审查工作流的能力。"
|
||||
|
||||
#: ../../community/governance.md:25
|
||||
#: ../../source/community/governance.md:29
|
||||
msgid ""
|
||||
"**Review Quality:** Actively participate in community code reviews, "
|
||||
"**Review quality:** Actively participate in community code reviews, "
|
||||
"ensuring high-quality code integration."
|
||||
msgstr "**评审质量：** 积极参与社区代码评审，确保高质量的代码集成。"
|
||||
msgstr "**审查质量：** 积极参与社区代码审查，确保高质量的代码集成。"
|
||||
|
||||
#: ../../community/governance.md:26
|
||||
#: ../../source/community/governance.md:30
|
||||
msgid ""
|
||||
"**Quality Contribution:** Successfully develop and deliver at least one "
|
||||
"**Quality contribution:** Successfully develop and deliver at least one "
|
||||
"major feature while maintaining consistent high-quality contributions."
|
||||
msgstr "**质量贡献：** 成功开发并交付至少一个主要功能，同时持续保持高质量的贡献。"
|
||||
msgstr "**高质量贡献：** 成功开发和交付至少一个主要功能，同时保持持续的高质量贡献。"
|
||||
|
||||
#: ../../community/governance.md:27
|
||||
#: ../../source/community/governance.md:31
|
||||
msgid ""
|
||||
"**Community Involvement:** Actively address issues, respond to forum "
|
||||
"**Community involvement:** Actively address issues, respond to forum "
|
||||
"inquiries, participate in discussions, and engage in community-driven tasks."
|
||||
msgstr "**社区参与：** 积极解决问题，回复论坛询问，参与讨论，并参与社区驱动的任务。"
|
||||
msgstr "**社区参与：** 积极处理问题，回复论坛咨询，参与讨论，并投身于社区驱动的任务。"
|
||||
|
||||
#: ../../community/governance.md:29
|
||||
#: ../../source/community/governance.md:33
|
||||
msgid ""
|
||||
"Requires approval from existing Maintainers. The vLLM community has the "
|
||||
"final decision-making authority."
|
||||
msgstr "需要现有维护者的批准。vLLM社区拥有最终决策权。"
|
||||
|
||||
#: ../../community/governance.md:31
|
||||
msgid ""
|
||||
"Maintainer will be empowered [vllm-project/vllm-"
|
||||
"ascend](https://github.com/vllm-project/vllm-ascend) Github repo write "
|
||||
"permissions (`Can read, clone, and push to this repository. Can also manage "
|
||||
"issues and pull requests`)."
|
||||
"The approval from existing Maintainers is required. The vLLM community has "
|
||||
"the final decision-making authority. Maintainers will be granted write "
|
||||
"access to the [vllm-project/vllm-ascend](https://github.com/vllm-"
|
||||
"project/vllm-ascend) GitHub repo. This includes permission to read, clone, "
|
||||
"and push to the repository, as well as manage issues and pull requests."
|
||||
msgstr ""
|
||||
"维护者将被授予 [vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-"
|
||||
"ascend) Github 仓库的写入权限（`可以读取、克隆和推送到此仓库。还可以管理问题和拉取请求`）。"
|
||||
"需要获得现有维护者的批准。vLLM 社区拥有最终决策权。维护者将被授予对 [vllm-project/vllm-"
|
||||
"ascend](https://github.com/vllm-project/vllm-ascend) GitHub "
|
||||
"仓库的写入权限。这包括读取、克隆和推送仓库的权限，以及管理问题和拉取请求的权限。"
|
||||
|
||||
#: ../../community/governance.md:33
|
||||
#: ../../source/community/governance.md:36
|
||||
msgid "Nominating and Removing Maintainers"
|
||||
msgstr "提名和移除维护者"
|
||||
|
||||
#: ../../community/governance.md:35
|
||||
#: ../../source/community/governance.md:38
|
||||
msgid "The Principles"
|
||||
msgstr "原则"
|
||||
|
||||
#: ../../community/governance.md:37
|
||||
#: ../../source/community/governance.md:40
|
||||
msgid ""
|
||||
"Membership in vLLM Ascend is given to individuals on merit basis after they "
|
||||
"demonstrated strong expertise of the vLLM / vLLM Ascend through "
|
||||
"contributions, reviews and discussions."
|
||||
"Membership in vLLM Ascend is given to individuals on a merit basis after "
|
||||
"they demonstrate their strong expertise in vLLM/vLLM Ascend through "
|
||||
"contributions, reviews, and discussions."
|
||||
msgstr ""
|
||||
"vLLM Ascend 的成员资格是基于个人能力授予的，只有在通过贡献、评审和讨论展示出对 vLLM / vLLM Ascend "
|
||||
"的深厚专业知识后，才可获得。"
|
||||
"vLLM Ascend 的成员资格基于个人贡献，在通过贡献、审查和讨论展示其在 vLLM/vLLM Ascend 方面的深厚专业知识后授予。"
|
||||
|
||||
#: ../../community/governance.md:39
|
||||
#: ../../source/community/governance.md:42
|
||||
msgid ""
|
||||
"For membership in the maintainer group the individual has to demonstrate "
|
||||
"strong and continued alignment with the overall vLLM / vLLM Ascend "
|
||||
"principles."
|
||||
msgstr "要成为维护者组成员，个人必须表现出与 vLLM / vLLM Ascend 总体原则的高度一致并持续支持。"
|
||||
"For membership in the maintainer group, individuals have to demonstrate "
|
||||
"strong and continued alignment with the overall vLLM/vLLM Ascend principles."
|
||||
msgstr "要成为维护者组成员，个人必须展示其与 vLLM/vLLM Ascend 整体原则的持续高度一致性。"
|
||||
|
||||
#: ../../community/governance.md:41
|
||||
#: ../../source/community/governance.md:44
|
||||
msgid ""
|
||||
"Light criteria of moving module maintenance to ‘emeritus’ status if they "
|
||||
"don’t actively participate over long periods of time."
|
||||
msgstr "如果模块维护人员在长时间内没有积极参与，可根据较宽松的标准将其维护状态转为“荣誉”状态。"
|
||||
"Maintainers who have been inactive for a long time may be transitioned to "
|
||||
"**emeritus** status under lenient criteria."
|
||||
msgstr "长期不活跃的维护者可能会在宽松标准下转为**荣誉**状态。"
|
||||
|
||||
#: ../../community/governance.md:43
|
||||
#: ../../source/community/governance.md:46
|
||||
msgid "The membership is for an individual, not a company."
|
||||
msgstr "该会员资格属于个人，而非公司。"
|
||||
msgstr "成员资格属于个人，而非公司。"
|
||||
|
||||
#: ../../community/governance.md:45
|
||||
#: ../../source/community/governance.md:48
|
||||
msgid "Nomination and Removal"
|
||||
msgstr "提名与罢免"
|
||||
msgstr "提名与移除"
|
||||
|
||||
#: ../../community/governance.md:47
|
||||
#: ../../source/community/governance.md:50
|
||||
msgid ""
|
||||
"Nomination: Anyone can nominate someone to become a maintainer (include "
|
||||
"self-nominate). All existing maintainers are responsible for evaluating the "
|
||||
"nomination. The nominator should provide nominee's info around the strength "
|
||||
"of the candidate to be a maintainer, include but not limited to review "
|
||||
"quality, quality contribution, community involvement."
|
||||
"Nomination: Anyone can nominate a candidate to become a maintainer, "
|
||||
"including self-nominations. All existing maintainers are responsible for "
|
||||
"reviewing and evaluating each nomination. The nominator should provide "
|
||||
"relevant information about the nominee's qualifications—such as review "
|
||||
"quality, quality contribution, and community involvement—among other "
|
||||
"strengths."
|
||||
msgstr ""
|
||||
"提名：任何人都可以提名他人成为维护者（包括自荐）。所有现有维护者都有责任评估提名。提名人应提供被提名人成为维护者的相关优势信息，包括但不限于评审质量、优质贡献、社区参与等。"
|
||||
"提名：任何人都可以提名候选人成为维护者，包括自我提名。所有现有维护者负责审查和评估每项提名。提名人应提供关于被提名人资格的相关信息——例如审查质量、高质量贡献和社区参与度等优势。"
|
||||
|
||||
#: ../../community/governance.md:48
|
||||
#: ../../source/community/governance.md:51
|
||||
msgid ""
|
||||
"Removal: Anyone can nominate a person to be removed from maintainer position"
|
||||
" (include self-nominate). All existing maintainers are responsible for "
|
||||
"evaluating the nomination. The nominator should provide nominee's info, "
|
||||
"include but not limited to lack of activity, conflict with the overall "
|
||||
"direction and other information that makes them unfit to be a maintainer."
|
||||
"Removal: Anyone may nominate an individual for removal from the maintainer "
|
||||
"role, including self-nominations. All current maintainers are responsible "
|
||||
"for reviewing and evaluating such nominations. The nominator should provide "
|
||||
"relevant information about the nominee—such as prolonged inactivity, "
|
||||
"misalignment with the project's overall direction, or other factors that may"
|
||||
" render them unsuitable for the maintainer position."
|
||||
msgstr ""
|
||||
"移除：任何人都可以提名某人被移出维护者职位（包括自荐）。所有现有维护者都有责任评估该提名。提名者应提供被提名人的相关信息，包括但不限于缺乏活动、与整体方向冲突以及使其不适合作为维护者的其他信息。"
|
||||
"移除：任何人都可以提名将某人从维护者角色中移除，包括自我提名。所有当前维护者负责审查和评估此类提名。提名人应提供关于被提名人的相关信息——例如长期不活跃、与项目整体方向不一致，或其他可能使其不适合担任维护者职务的因素。"
|
||||
|
||||
@@ -0,0 +1,373 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:1
|
||||
msgid "Issue Workflow Guidelines"
|
||||
msgstr "问题工作流指南"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:3
|
||||
msgid ""
|
||||
"This document defines the standard lifecycle for "
|
||||
"[Issues](https://github.com/vllm-project/vllm-ascend/issues) in the vLLM "
|
||||
"Ascend project — from creation through triage, active handling, and final "
|
||||
"closure. It establishes consistent label usage, owner assignment, and "
|
||||
"communication expectations to ensure smooth collaboration between "
|
||||
"contributors and maintainers."
|
||||
msgstr ""
|
||||
"本文档定义了 vLLM Ascend 项目中 [Issues](https://github.com/vllm-project/vllm-"
|
||||
"ascend/issues) "
|
||||
"的标准生命周期——从创建、分类、积极处理到最终关闭。它确立了统一的标签使用、负责人分配和沟通期望，以确保贡献者与维护者之间的顺畅协作。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:5
|
||||
msgid "1. Label Categories"
|
||||
msgstr "1.标签类别"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:7
|
||||
msgid "1.1 Status Labels"
|
||||
msgstr "1.1状态标签"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:9
|
||||
msgid "These labels track where an issue stands in the workflow."
|
||||
msgstr "这些标签用于追踪问题在工作流中的当前状态。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Label"
|
||||
msgstr "标签"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Description"
|
||||
msgstr "描述"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`triage review`"
|
||||
msgstr "`triage review`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid ""
|
||||
"Newly filed or unseen issue awaiting initial assessment by a maintainer"
|
||||
msgstr "新提交或未查看的问题，等待维护者进行初步评估"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`triaged`"
|
||||
msgstr "`triaged`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Assessment complete; type, priority, and module have been determined"
|
||||
msgstr "评估完成；已确定类型、优先级和所属模块"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`wait-feedback`"
|
||||
msgstr "`wait-feedback`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid ""
|
||||
"Blocked on an external dependency or awaiting a response before work can "
|
||||
"proceed"
|
||||
msgstr "因外部依赖或等待回复而阻塞，无法继续推进"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`resolved`"
|
||||
msgstr "`resolved`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid ""
|
||||
"Issue has been closed — either via a merged PR, or through non-code "
|
||||
"resolution (e.g., answered question, configuration guidance)"
|
||||
msgstr "问题已关闭——通过合并的 PR 或非代码方式解决（例如，已回答的问题、配置指导）"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`stale`"
|
||||
msgstr "`stale`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid ""
|
||||
"No activity for an extended period; parties have been notified and the issue"
|
||||
" will be auto-closed if there is no response"
|
||||
msgstr "长时间无活动；已通知相关方，若无回复将自动关闭"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`duplicated`"
|
||||
msgstr "`duplicated`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "A duplicate of an existing open issue or merged PR"
|
||||
msgstr "与现有开放问题或已合并 PR 重复"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`invalid`"
|
||||
msgstr "`invalid`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "The issue report is invalid, unclear, or not reproducible"
|
||||
msgstr "问题报告无效、不清晰或无法复现"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`wontfix`"
|
||||
msgstr "`wontfix`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid ""
|
||||
"The issue will not be addressed; as this issue is out of scope, unviable, or"
|
||||
" intentionally ignored for the foreseeable future"
|
||||
msgstr "该问题将不予处理；因其超出范围、不可行或在可预见的未来被有意忽略"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:22
|
||||
msgid "1.2 Type Labels"
|
||||
msgstr "1.2类型标签"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:24
|
||||
msgid "These labels describe the nature of the issue."
|
||||
msgstr "这些标签描述问题的性质。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`feature request`"
|
||||
msgstr "`feature request`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Request for new functionality"
|
||||
msgstr "请求新增功能"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`RFC`"
|
||||
msgstr "`RFC`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid ""
|
||||
"Request for Comments — significant architectural or design change requiring "
|
||||
"community discussion"
|
||||
msgstr "征求意见——需要社区讨论的重大架构或设计变更"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`new model`"
|
||||
msgstr "`new model`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Request to add support for a new model on Ascend NPU"
|
||||
msgstr "请求在 Ascend NPU 上新增对某个模型的支持"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`usage`"
|
||||
msgstr "`usage`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "A usage question; no code change may be required"
|
||||
msgstr "使用问题；可能不需要修改代码"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`question`"
|
||||
msgstr "`question`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "A general question; no code change may be required"
|
||||
msgstr "一般性问题；可能不需要修改代码"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`documentation`"
|
||||
msgstr "`documentation`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Improvements or corrections to documentation"
|
||||
msgstr "对文档的改进或修正"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`installation`"
|
||||
msgstr "`installation`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Issues related to setup and deployment"
|
||||
msgstr "与安装和部署相关的问题"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`performance`"
|
||||
msgstr "`performance`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Performance regression, bottleneck, or optimization request"
|
||||
msgstr "性能回退、瓶颈或优化请求"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`bug`"
|
||||
msgstr "`bug`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Something is not working correctly or behaves unexpectedly"
|
||||
msgstr "某些功能无法正常工作或行为异常"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:38
|
||||
msgid "1.3 Priority Labels (Optional)"
|
||||
msgstr "1.3优先级标签（可选）"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`high`"
|
||||
msgstr "`high`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "High priority; should be resolved in the current or next cycle"
|
||||
msgstr "高优先级；应在当前或下一个周期内解决"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`medium`"
|
||||
msgstr "`medium`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Normal priority; handled in the regular development flow"
|
||||
msgstr "普通优先级；在常规开发流程中处理"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`low`"
|
||||
msgstr "`low`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Low priority; edge case or minor issue that can be deferred"
|
||||
msgstr "低优先级；可以推迟处理的边缘情况或小问题"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:46
|
||||
msgid "1.4 Contribution Labels (Optional)"
|
||||
msgstr "1.4贡献标签（可选）"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`good first issue`"
|
||||
msgstr "`good first issue`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "A well-scoped, low-complexity task suitable for new contributors"
|
||||
msgstr "一个范围明确、复杂度低的任务，适合新贡献者"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "`help wanted`"
|
||||
msgstr "`help wanted`"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md
|
||||
msgid "Community contributions are welcome and encouraged"
|
||||
msgstr "欢迎并鼓励社区贡献"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:53
|
||||
msgid "2. Workflow"
|
||||
msgstr "2.工作流"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:55
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:55
|
||||
msgid "issue_label_workflow"
|
||||
msgstr "issue_label_workflow"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:57
|
||||
msgid "Phase 1 — First Response"
|
||||
msgstr "阶段1 — 首次响应"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:59
|
||||
msgid "When an issue is first picked up by the on-call maintainer:"
|
||||
msgstr "当值班维护者首次接手问题时："
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:61
|
||||
msgid ""
|
||||
"Apply `triaged` to signal that the issue can be classified and add the "
|
||||
"relevant module label so the issue can be routed to the appropriate module "
|
||||
"maintainer for detailed triage."
|
||||
msgstr "应用 `triaged` 标签表示该问题可以被分类，并添加相关的模块标签，以便将问题路由到相应的模块维护者进行详细分类。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:62
|
||||
msgid ""
|
||||
"Apply `triage review` to signal that the issue requires more review and "
|
||||
"specific analysis before classification."
|
||||
msgstr "应用 `triage review` 标签表示该问题在分类之前需要更多的审查和具体分析。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:64
|
||||
msgid "Phase 2 — Triage and Analysis"
|
||||
msgstr "阶段2 — 分类与分析"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:66
|
||||
msgid "After a thorough review of the issue content:"
|
||||
msgstr "在彻底审查问题内容后："
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:68
|
||||
msgid ""
|
||||
"Verify and apply the appropriate **issue type** label (`bug`, `feature "
|
||||
"request`, `RFC`, `question`, `documentation`, `installation`, `performance`,"
|
||||
" `new model`, etc.)."
|
||||
msgstr ""
|
||||
"确认并应用适当的**问题类型**标签（`bug`、`feature "
|
||||
"request`、`RFC`、`question`、`documentation`、`installation`、`performance`、`new "
|
||||
"model` 等）。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:69
|
||||
msgid "Handle terminal states:"
|
||||
msgstr "处理终态："
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:70
|
||||
msgid ""
|
||||
"For duplicates, apply the `duplicated` label, provide an explanation and a "
|
||||
"link to the existing issue or PR. If there are no further questions, close "
|
||||
"the issue."
|
||||
msgstr "对于重复问题，应用 `duplicated` 标签，提供解释和指向现有问题或 PR 的链接。如果没有其他问题，则关闭该问题。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:71
|
||||
msgid ""
|
||||
"For invalid reports, provide an explanation, apply the `invalid` and "
|
||||
"`resolved` label, and close the issue. The issue creator can comment or "
|
||||
"request to reopen if they have further questions."
|
||||
msgstr ""
|
||||
"对于无效报告，提供解释，应用 `invalid` 和 `resolved` 标签，然后关闭问题。问题创建者如有其他问题可以评论或请求重新打开。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:72
|
||||
msgid "Optionally apply a **priority** label (`high`, `medium`, or `low`)."
|
||||
msgstr "可选地应用**优先级**标签（`high`、`medium` 或 `low`）。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:73
|
||||
msgid ""
|
||||
"If community contributions are welcome, apply `help wanted`. For well-scoped"
|
||||
" beginner-friendly tasks, also apply `good first issue`."
|
||||
msgstr "如果欢迎社区贡献，应用 `help wanted`。对于范围明确、适合初学者的任务，同时应用 `good first issue`。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:74
|
||||
msgid ""
|
||||
"Assign the issue owner and replace `triage review` with `triaged` to "
|
||||
"indicate that triage is complete."
|
||||
msgstr "指定问题负责人，并将 `triage review` 替换为 `triaged`，表示分类已完成。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:76
|
||||
msgid "Phase 3 — Closure"
|
||||
msgstr "阶段3 — 关闭"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:78
|
||||
msgid "After triage, the issue moves into tracking and implementation:"
|
||||
msgstr "分类完成后，问题进入跟踪和实施阶段："
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:80
|
||||
msgid ""
|
||||
"Keep the issue in progress until it is resolved through a merged PR or "
|
||||
"another confirmed resolution path."
|
||||
msgstr "保持问题处于进行中状态，直到通过合并的 PR 或其他确认的解决路径解决。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:81
|
||||
msgid ""
|
||||
"Once the issue is resolved, apply `resolved` and close it, ideally with a "
|
||||
"reference to the merged PR or a short explanation of the resolution."
|
||||
msgstr "问题解决后，应用 `resolved` 标签并关闭，最好附上合并的 PR 引用或简要的解决方案说明。"
|
||||
|
||||
#: ../../source/community/issue-workflow-guidelines.md:82
|
||||
msgid ""
|
||||
"If the issue remains inactive for an extended period, apply `stale` as the "
|
||||
"final state before auto-closure."
|
||||
msgstr "如果问题长时间处于非活动状态，在自动关闭前应用 `stale` 作为最终状态。"
|
||||
366
docs/source/locale/zh_CN/LC_MESSAGES/community/slash-commands.po
Normal file
366
docs/source/locale/zh_CN/LC_MESSAGES/community/slash-commands.po
Normal file
@@ -0,0 +1,366 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/community/slash-commands.md:1
|
||||
msgid "Slash Commands"
|
||||
msgstr "斜杠命令"
|
||||
|
||||
#: ../../source/community/slash-commands.md:3
|
||||
msgid ""
|
||||
"vLLM Ascend supports slash commands in pull request comments to trigger CI "
|
||||
"workflows. See the [Permission](#permission) section for who can trigger "
|
||||
"each command."
|
||||
msgstr ""
|
||||
"vLLM Ascend 支持在拉取请求评论中使用斜杠命令触发 CI 工作流。有关谁可以触发每个命令，请参阅[权限](#permission)部分。"
|
||||
|
||||
#: ../../source/community/slash-commands.md:5
|
||||
msgid "Available Commands"
|
||||
msgstr "可用命令"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
#: ../../source/community/slash-commands.md:7
|
||||
msgid "`/e2e`"
|
||||
msgstr "`/e2e`"
|
||||
|
||||
#: ../../source/community/slash-commands.md:9
|
||||
msgid ""
|
||||
"Run specific E2E tests under `tests/e2e/pull_request/`. Tests are "
|
||||
"automatically routed to the appropriate NPU runner based on the test path."
|
||||
msgstr "运行 `tests/e2e/pull_request/` 下的特定 E2E 测试。测试会根据路径自动路由到相应的 NPU 运行器。"
|
||||
|
||||
#: ../../source/community/slash-commands.md:11
|
||||
#: ../../source/community/slash-commands.md:61
|
||||
#: ../../source/community/slash-commands.md:95
|
||||
#: ../../source/community/slash-commands.md:133
|
||||
msgid "**Examples:**"
|
||||
msgstr "**示例：**"
|
||||
|
||||
#: ../../source/community/slash-commands.md:24
|
||||
msgid "**Routing rules** (matched in order):"
|
||||
msgstr "**路由规则**（按顺序匹配）："
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Test path contains"
|
||||
msgstr "测试路径包含"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Runner"
|
||||
msgstr "运行器"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "`four_card/_310p`"
|
||||
msgstr "`four_card/_310p`"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "310P 4-card"
|
||||
msgstr "310P 4卡"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "`_310p` (under `one_card`/`two_card`)"
|
||||
msgstr "`_310p`（位于 `one_card`/`two_card` 下）"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "310P single card"
|
||||
msgstr "310P 单卡"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "`four_card`"
|
||||
msgstr "`four_card`"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "A3 4-card"
|
||||
msgstr "A3 4卡"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "`two_card`"
|
||||
msgstr "`two_card`"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "A3 2-card"
|
||||
msgstr "A3 2卡"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Others (e.g. `one_card`)"
|
||||
msgstr "其他（例如 `one_card`）"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "A2 single card"
|
||||
msgstr "A2 单卡"
|
||||
|
||||
#: ../../source/community/slash-commands.md:34
|
||||
msgid ""
|
||||
"Only test paths under `tests/e2e/pull_request/` are supported. Tests in "
|
||||
"`tests/e2e/nightly/`, `tests/e2e/models/`, or `tests/e2e/doctests/` are not "
|
||||
"accepted by `/e2e`. Use `/nightly` for nightly tests."
|
||||
msgstr ""
|
||||
"仅支持 `tests/e2e/pull_request/` 下的测试路径。`/e2e` 不接受 "
|
||||
"`tests/e2e/nightly/`、`tests/e2e/models/` 或 `tests/e2e/doctests/` "
|
||||
"中的测试。夜间测试请使用 `/nightly`。"
|
||||
|
||||
#: ../../source/community/slash-commands.md:36
|
||||
msgid ""
|
||||
"Tests are run against both the community vLLM version and the latest "
|
||||
"release."
|
||||
msgstr "测试会针对社区版 vLLM 和最新发布版本同时运行。"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
#: ../../source/community/slash-commands.md:38
|
||||
msgid "`/nightly`"
|
||||
msgstr "`/nightly`"
|
||||
|
||||
#: ../../source/community/slash-commands.md:40
|
||||
msgid ""
|
||||
"Trigger specific nightly test cases on A2 and A3. Supports only PR comments."
|
||||
" Test case names correspond to the `test_config.name` entries defined in "
|
||||
"`schedule_nightly_test_a2.yaml` and `schedule_nightly_test_a3.yaml`."
|
||||
msgstr ""
|
||||
"在 A2 和 A3 上触发特定的夜间测试用例。仅支持 PR 评论。测试用例名称对应于 `schedule_nightly_test_a2.yaml` 和"
|
||||
" `schedule_nightly_test_a3.yaml` 中定义的 `test_config.name` 条目。"
|
||||
|
||||
#: ../../source/community/slash-commands.md:42
|
||||
#: ../../source/community/slash-commands.md:89
|
||||
#: ../../source/community/slash-commands.md:113
|
||||
msgid "**Usage:**"
|
||||
msgstr "**用法：**"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Syntax"
|
||||
msgstr "语法"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
#: ../../source/community/slash-commands.md:146
|
||||
msgid "Scope"
|
||||
msgstr "范围"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "`/nightly <test_cases>`"
|
||||
msgstr "`/nightly <test_cases>`"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Runs on `main` branch"
|
||||
msgstr "在 `main` 分支上运行"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "`/nightly <test_cases> --branch <branch>`"
|
||||
msgstr "`/nightly <test_cases> --branch <branch>`"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Runs on the specified branch"
|
||||
msgstr "在指定分支上运行"
|
||||
|
||||
#: ../../source/community/slash-commands.md:49
|
||||
msgid ""
|
||||
"Use `--branch <name>` to specify a target branch. Without `--branch`, all "
|
||||
"arguments are treated as test cases (separated by commas or spaces) and the "
|
||||
"branch defaults to `main`."
|
||||
msgstr ""
|
||||
"使用 `--branch <name>` 指定目标分支。如果不使用 `--branch`，所有参数都被视为测试用例（以逗号或空格分隔），分支默认为 "
|
||||
"`main`。"
|
||||
|
||||
#: ../../source/community/slash-commands.md:51
|
||||
msgid ""
|
||||
"**Note**: When commenting on a PR, the tests run on the PR branch "
|
||||
"automatically in the triggered workflow; the `--branch` flag is primarily "
|
||||
"used in issue comments."
|
||||
msgstr "**注意**：在 PR 上评论时，测试会在触发的工作流中自动在 PR 分支上运行；`--branch` 标志主要用于 issue 评论。"
|
||||
|
||||
#: ../../source/community/slash-commands.md:53
|
||||
msgid "**Common test case names (A2):**"
|
||||
msgstr "**常见测试用例名称（A2）：**"
|
||||
|
||||
#: ../../source/community/slash-commands.md:55
|
||||
msgid ""
|
||||
"`test_custom_op`, `test_custom_op_multi_card`, `qwen3-vl-32b-instruct-w8a8`,"
|
||||
" `qwen3-32b-int8`, `MiniMax-M2.5-w8a8-QuaRot-A2`, `Qwen3.5-27B-w8a8-A2`, "
|
||||
"`Qwen3.5-397B-A17B-w4a8-mtp`, `accuracy-group`"
|
||||
msgstr ""
|
||||
"`test_custom_op`, `test_custom_op_multi_card`, `qwen3-vl-32b-instruct-w8a8`,"
|
||||
" `qwen3-32b-int8`, `MiniMax-M2.5-w8a8-QuaRot-A2`, `Qwen3.5-27B-w8a8-A2`, "
|
||||
"`Qwen3.5-397B-A17B-w4a8-mtp`, `accuracy-group`"
|
||||
|
||||
#: ../../source/community/slash-commands.md:57
|
||||
msgid "**Common test case names (A3):**"
|
||||
msgstr "**常见测试用例名称（A3）：**"
|
||||
|
||||
#: ../../source/community/slash-commands.md:59
|
||||
msgid ""
|
||||
"`multi-node-deepseek-v3.2-W8A8-EP`, `mtpx-deepseek-r1-0528-w8a8`, "
|
||||
"`deepseek-r1-0528-w8a8`, `kimi-k2-thinking`, "
|
||||
"`qwen3-vl-235b-a22b-instruct-w8a8`, `custom-multi-ops`, ..."
|
||||
msgstr ""
|
||||
"`multi-node-deepseek-v3.2-W8A8-EP`, `mtpx-deepseek-r1-0528-w8a8`, "
|
||||
"`deepseek-r1-0528-w8a8`, `kimi-k2-thinking`, "
|
||||
"`qwen3-vl-235b-a22b-instruct-w8a8`, `custom-multi-ops`, ..."
|
||||
|
||||
#: ../../source/community/slash-commands.md:83
|
||||
msgid ""
|
||||
"This triggers `workflow_dispatch` on both `schedule_nightly_test_a2.yaml` "
|
||||
"and `schedule_nightly_test_a3.yaml`."
|
||||
msgstr ""
|
||||
"这会在 `schedule_nightly_test_a2.yaml` 和 `schedule_nightly_test_a3.yaml` 上触发 "
|
||||
"`workflow_dispatch`。"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
#: ../../source/community/slash-commands.md:85
|
||||
msgid "`/cherry-pick`"
|
||||
msgstr "`/cherry-pick`"
|
||||
|
||||
#: ../../source/community/slash-commands.md:87
|
||||
msgid ""
|
||||
"Cherry-pick a PR's commits onto a specified target branch and create a new "
|
||||
"PR. This is useful for backporting fixes to release branches."
|
||||
msgstr "将 PR 的提交拣选到指定的目标分支并创建一个新 PR。这对于将修复反向移植到发布分支非常有用。"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Description"
|
||||
msgstr "描述"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "`/cherry-pick <target_branch>`"
|
||||
msgstr "`/cherry-pick <target_branch>`"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Cherry-pick onto the specified branch"
|
||||
msgstr "拣选到指定分支"
|
||||
|
||||
#: ../../source/community/slash-commands.md:105
|
||||
msgid ""
|
||||
"A new PR will be created with the title format `[Cherry-pick] "
|
||||
"<original_title> (from #<PR_NUMBER>)` and a body linking back to the "
|
||||
"original PR."
|
||||
msgstr ""
|
||||
"将创建一个新 PR，标题格式为 `[Cherry-pick] <original_title> (from "
|
||||
"#<PR_NUMBER>)`，正文包含指向原始 PR 的链接。"
|
||||
|
||||
#: ../../source/community/slash-commands.md:107
|
||||
msgid ""
|
||||
"If the cherry-pick encounters merge conflicts, the command will report the "
|
||||
"failure and the cherry-pick must be done manually."
|
||||
msgstr "如果拣选遇到合并冲突，命令将报告失败，必须手动进行拣选。"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
#: ../../source/community/slash-commands.md:109
|
||||
msgid "`/revert`"
|
||||
msgstr "`/revert`"
|
||||
|
||||
#: ../../source/community/slash-commands.md:111
|
||||
msgid ""
|
||||
"Revert a merged PR by creating a new PR that reverses its changes. The "
|
||||
"revert targets the same base branch the original PR was merged into."
|
||||
msgstr "通过创建一个撤销更改的新 PR 来还原已合并的 PR。还原目标为原始 PR 合并到的同一基础分支。"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Revert this PR (no arguments needed)"
|
||||
msgstr "还原此 PR（无需参数）"
|
||||
|
||||
#: ../../source/community/slash-commands.md:119
|
||||
msgid "**Example:**"
|
||||
msgstr "**示例：**"
|
||||
|
||||
#: ../../source/community/slash-commands.md:125
|
||||
msgid ""
|
||||
"A new PR will be created with the title format `[Revert] Revert "
|
||||
"\"original_title\" (#PR_NUMBER)` and a body linking back to the original PR "
|
||||
"and its merge commit."
|
||||
msgstr ""
|
||||
"将创建一个新 PR，标题格式为 `[Revert] Revert \"original_title\" (#PR_NUMBER)`，正文包含指向原始 "
|
||||
"PR 及其合并提交的链接。"
|
||||
|
||||
#: ../../source/community/slash-commands.md:127
|
||||
msgid ""
|
||||
"Only merged PRs can be reverted. If the revert encounters merge conflicts "
|
||||
"(e.g., because the base branch has diverged significantly), the command will"
|
||||
" report the failure and the revert must be done manually."
|
||||
msgstr "只能还原已合并的 PR。如果还原遇到合并冲突（例如，因为基础分支已严重偏离），命令将报告失败，必须手动进行还原。"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
#: ../../source/community/slash-commands.md:129
|
||||
msgid "`/rerun`"
|
||||
msgstr "`/rerun`"
|
||||
|
||||
#: ../../source/community/slash-commands.md:131
|
||||
msgid ""
|
||||
"Re-run all failed workflow runs on the current PR commit. Useful when CI "
|
||||
"jobs failed due to infrastructure issues."
|
||||
msgstr "在当前 PR 提交上重新运行所有失败的工作流。当 CI 作业因基础设施问题失败时非常有用。"
|
||||
|
||||
#: ../../source/community/slash-commands.md:140
|
||||
msgid "Behavior"
|
||||
msgstr "行为"
|
||||
|
||||
#: ../../source/community/slash-commands.md:142
|
||||
msgid ""
|
||||
"When you comment a slash command, a 👀 reaction is added to your comment to "
|
||||
"indicate it has been received"
|
||||
msgstr "当您评论斜杠命令时，您的评论会添加一个 👀 反应，表示已收到"
|
||||
|
||||
#: ../../source/community/slash-commands.md:143
|
||||
msgid "The corresponding CI workflow is triggered asynchronously"
|
||||
msgstr "相应的 CI 工作流会被异步触发"
|
||||
|
||||
#: ../../source/community/slash-commands.md:144
|
||||
msgid "Upon completion, a 🎉 reaction and a summary comment are added"
|
||||
msgstr "完成后，会添加一个 🎉 反应和一条摘要评论"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Command"
|
||||
msgstr "命令"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "PR comments"
|
||||
msgstr "PR 评论"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Issue comments"
|
||||
msgstr "Issue 评论"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "✅"
|
||||
msgstr "✅"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "❌"
|
||||
msgstr "❌"
|
||||
|
||||
#: ../../source/community/slash-commands.md:156
|
||||
msgid "Permission"
|
||||
msgstr "权限"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Who can trigger"
|
||||
msgstr "谁可以触发"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "PR author, or users with triage+ permission on the repository"
|
||||
msgstr "PR 作者，或对仓库具有 triage+ 权限的用户"
|
||||
|
||||
#: ../../source/community/slash-commands.md
|
||||
msgid "Users with triage+ permission on the repository only"
|
||||
msgstr "仅限对仓库具有 triage+ 权限的用户"
|
||||
|
||||
#: ../../source/community/slash-commands.md:166
|
||||
#, python-brace-format
|
||||
msgid ""
|
||||
"Permission is verified via the GitHub API "
|
||||
"(`repos/{owner}/{repo}/collaborators/{user}/permission`)."
|
||||
msgstr ""
|
||||
"权限通过 GitHub API (`repos/{owner}/{repo}/collaborators/{user}/permission`) 验证。"
|
||||
@@ -2,14 +2,14 @@
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2025.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-07-18 09:01+0800\n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
@@ -18,86 +18,83 @@ msgstr ""
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.17.0\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../community/user_stories/index.md:15
|
||||
#: ../../source/community/user_stories/index.md:15
|
||||
msgid "More details"
|
||||
msgstr "更多细节"
|
||||
msgstr "更多详情"
|
||||
|
||||
#: ../../community/user_stories/index.md:1
|
||||
#: ../../source/community/user_stories/index.md:1
|
||||
msgid "User Stories"
|
||||
msgstr "用户故事"
|
||||
msgstr "用户案例"
|
||||
|
||||
#: ../../community/user_stories/index.md:3
|
||||
#: ../../source/community/user_stories/index.md:3
|
||||
msgid ""
|
||||
"Read case studies on how users and developers solves real, everyday problems"
|
||||
" with vLLM Ascend"
|
||||
msgstr "阅读案例研究，了解用户和开发者如何使用 vLLM Ascend 解决实际日常问题。"
|
||||
"Read case studies on how users and developers solve real, everyday problems "
|
||||
"with vLLM Ascend"
|
||||
msgstr "阅读用户和开发者如何利用 vLLM Ascend 解决实际日常问题的案例研究"
|
||||
|
||||
#: ../../community/user_stories/index.md:5
|
||||
#: ../../source/community/user_stories/index.md:5
|
||||
msgid ""
|
||||
"[LLaMA-Factory](./llamafactory.md) is an easy-to-use and efficient platform "
|
||||
"for training and fine-tuning large language models, it supports vLLM Ascend "
|
||||
"for training and fine-tuning large language models. It supports vLLM Ascend "
|
||||
"to speed up inference since [LLaMA-"
|
||||
"Factory#7739](https://github.com/hiyouga/LLaMA-Factory/pull/7739), gain 2x "
|
||||
"performance enhancement of inference."
|
||||
"Factory#7739](https://github.com/hiyouga/LlamaFactory/pull/7739), gaining 2x"
|
||||
" performance enhancement in inference."
|
||||
msgstr ""
|
||||
"[LLaMA-Factory](./llamafactory.md) 是一个易于使用且高效的大语言模型训练与微调平台，自 [LLaMA-"
|
||||
"Factory#7739](https://github.com/hiyouga/LLaMA-Factory/pull/7739) 起支持 vLLM "
|
||||
"Ascend 加速推理，推理性能提升 2 倍。"
|
||||
"[LLaMA-Factory](./llamafactory.md) 是一个易于使用且高效的大语言模型训练与微调平台。自 [LLaMA-"
|
||||
"Factory#7739](https://github.com/hiyouga/LlamaFactory/pull/7739) 起，它支持 vLLM "
|
||||
"Ascend 加速推理，在推理性能上获得了 2 倍的提升。"
|
||||
|
||||
#: ../../community/user_stories/index.md:7
|
||||
#: ../../source/community/user_stories/index.md:7
|
||||
msgid ""
|
||||
"[Huggingface/trl](https://github.com/huggingface/trl) is a cutting-edge "
|
||||
"library designed for post-training foundation models using advanced "
|
||||
"techniques like SFT, PPO and DPO, it uses vLLM Ascend since "
|
||||
"techniques like SFT, PPO and DPO. It uses vLLM Ascend since "
|
||||
"[v0.17.0](https://github.com/huggingface/trl/releases/tag/v0.17.0) to "
|
||||
"support RLHF on Ascend NPU."
|
||||
"support RLHF on Ascend NPUs."
|
||||
msgstr ""
|
||||
"[Huggingface/trl](https://github.com/huggingface/trl) 是一个前沿的库，专为使用 SFT、PPO 和"
|
||||
" DPO 等先进技术对基础模型进行后训练而设计。从 "
|
||||
"[v0.17.0](https://github.com/huggingface/trl/releases/tag/v0.17.0) 版本开始，该库利用"
|
||||
" vLLM Ascend 来支持在 Ascend NPU 上进行 RLHF。"
|
||||
"[Huggingface/trl](https://github.com/huggingface/trl) 是一个前沿的库，旨在使用 SFT、PPO 和"
|
||||
" DPO 等先进技术对基础模型进行后训练。自 "
|
||||
"[v0.17.0](https://github.com/huggingface/trl/releases/tag/v0.17.0) 起，它使用 "
|
||||
"vLLM Ascend 来支持在 Ascend NPU 上进行 RLHF。"
|
||||
|
||||
#: ../../community/user_stories/index.md:9
|
||||
#: ../../source/community/user_stories/index.md:9
|
||||
msgid ""
|
||||
"[MindIE Turbo](https://pypi.org/project/mindie-turbo) is an LLM inference "
|
||||
"engine acceleration plug-in library developed by Huawei on Ascend hardware, "
|
||||
"which includes self-developed large language model optimization algorithms "
|
||||
"and optimizations related to the inference engine framework. It supports "
|
||||
"vLLM Ascend since "
|
||||
"engine acceleration plugin library developed by Huawei on Ascend hardware, "
|
||||
"which includes self-developed LLM optimization algorithms and optimizations "
|
||||
"related to the inference engine framework. It supports vLLM Ascend since "
|
||||
"[2.0rc1](https://www.hiascend.com/document/detail/zh/mindie/20RC1/AcceleratePlugin/turbodev/mindie-"
|
||||
"turbo-0001.html)."
|
||||
msgstr ""
|
||||
"[MindIE Turbo](https://pypi.org/project/mindie-turbo) "
|
||||
"是华为在昇腾硬件上开发的一款用于加速LLM推理引擎的插件库，包含自主研发的大语言模型优化算法及与推理引擎框架相关的优化。从 "
|
||||
"[MindIE Turbo](https://pypi.org/project/mindie-turbo) 是华为在 Ascend 硬件上开发的 LLM"
|
||||
" 推理引擎加速插件库，包含自研的 LLM 优化算法以及与推理引擎框架相关的优化。自 "
|
||||
"[2.0rc1](https://www.hiascend.com/document/detail/zh/mindie/20RC1/AcceleratePlugin/turbodev/mindie-"
|
||||
"turbo-0001.html) 起，支持 vLLM Ascend。"
|
||||
"turbo-0001.html) 起，它支持 vLLM Ascend。"
|
||||
|
||||
#: ../../community/user_stories/index.md:11
|
||||
#: ../../source/community/user_stories/index.md:11
|
||||
msgid ""
|
||||
"[GPUStack](https://github.com/gpustack/gpustack) is an open-source GPU "
|
||||
"cluster manager for running AI models. It supports vLLM Ascend since "
|
||||
"[v0.6.2](https://github.com/gpustack/gpustack/releases/tag/v0.6.2), see more"
|
||||
" GPUStack performance evaluation info on "
|
||||
"[link](https://mp.weixin.qq.com/s/pkytJVjcH9_OnffnsFGaew)."
|
||||
"[v0.6.2](https://github.com/gpustack/gpustack/releases/tag/v0.6.2). See more"
|
||||
" GPUStack performance evaluation information at [this "
|
||||
"link](https://mp.weixin.qq.com/s/pkytJVjcH9_OnffnsFGaew)."
|
||||
msgstr ""
|
||||
"[GPUStack](https://github.com/gpustack/gpustack) 是一个开源的 GPU 集群管理器，用于运行 AI "
|
||||
"模型。从 [v0.6.2](https://github.com/gpustack/gpustack/releases/tag/v0.6.2) "
|
||||
"版本开始支持 vLLM Ascend，更多 GPUStack 性能评测信息见 "
|
||||
"[链接](https://mp.weixin.qq.com/s/pkytJVjcH9_OnffnsFGaew)。"
|
||||
"[GPUStack](https://github.com/gpustack/gpustack) 是一个用于运行 AI 模型的开源 GPU "
|
||||
"集群管理器。自 [v0.6.2](https://github.com/gpustack/gpustack/releases/tag/v0.6.2) "
|
||||
"起，它支持 vLLM Ascend。更多 GPUStack "
|
||||
"性能评估信息请参见[此链接](https://mp.weixin.qq.com/s/pkytJVjcH9_OnffnsFGaew)。"
|
||||
|
||||
#: ../../community/user_stories/index.md:13
|
||||
#: ../../source/community/user_stories/index.md:13
|
||||
msgid ""
|
||||
"[verl](https://github.com/volcengine/verl) is a flexible, efficient and "
|
||||
"production-ready RL training library for large language models (LLMs), uses "
|
||||
"vLLM Ascend since "
|
||||
"[v0.4.0](https://github.com/volcengine/verl/releases/tag/v0.4.0), see more "
|
||||
"info on [verl x Ascend "
|
||||
"Quickstart](https://verl.readthedocs.io/en/latest/ascend_tutorial/ascend_quick_start.html)."
|
||||
"[verl](https://github.com/verl-project/verl) is a flexible, efficient, and "
|
||||
"production-ready RL training library for LLMs. It uses vLLM Ascend since "
|
||||
"[v0.4.0](https://github.com/verl-project/verl/releases/tag/v0.4.0). See more"
|
||||
" information on [Ascend "
|
||||
"Quickstart](https://verl.readthedocs.io/en/latest/ascend_tutorial/get_start/quick_start.html)."
|
||||
msgstr ""
|
||||
"[verl](https://github.com/volcengine/verl) "
|
||||
"是一个灵活、高效且可用于生产环境的大型语言模型（LLM）强化学习训练库，自 "
|
||||
"[v0.4.0](https://github.com/volcengine/verl/releases/tag/v0.4.0) 起支持 vLLM "
|
||||
"Ascend，更多信息请参见 [verl x Ascend "
|
||||
"快速上手](https://verl.readthedocs.io/en/latest/ascend_tutorial/ascend_quick_start.html)。"
|
||||
"[verl](https://github.com/verl-project/verl) 是一个灵活、高效且可用于生产环境的 LLM 强化学习训练库。自"
|
||||
" [v0.4.0](https://github.com/verl-project/verl/releases/tag/v0.4.0) 起，它使用 "
|
||||
"vLLM Ascend。更多信息请参见 [Ascend "
|
||||
"快速入门](https://verl.readthedocs.io/en/latest/ascend_tutorial/get_start/quick_start.html)。"
|
||||
|
||||
@@ -2,14 +2,14 @@
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2025.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-07-18 09:01+0800\n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
@@ -18,70 +18,66 @@ msgstr ""
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.17.0\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../community/user_stories/llamafactory.md:1
|
||||
#: ../../source/community/user_stories/llamafactory.md:1
|
||||
msgid "LLaMA-Factory"
|
||||
msgstr "LLaMA-Factory"
|
||||
|
||||
#: ../../community/user_stories/llamafactory.md:3
|
||||
msgid "**About / Introduction**"
|
||||
msgstr "**关于 / 介绍**"
|
||||
#: ../../source/community/user_stories/llamafactory.md:3
|
||||
msgid "Introduction"
|
||||
msgstr "简介"
|
||||
|
||||
#: ../../community/user_stories/llamafactory.md:5
|
||||
#: ../../source/community/user_stories/llamafactory.md:5
|
||||
msgid ""
|
||||
"[LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) is an easy-to-use "
|
||||
"[LLaMA-Factory](https://github.com/hiyouga/LlamaFactory) is an easy-to-use "
|
||||
"and efficient platform for training and fine-tuning large language models. "
|
||||
"With LLaMA-Factory, you can fine-tune hundreds of pre-trained models locally"
|
||||
" without writing any code."
|
||||
msgstr ""
|
||||
"[LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) "
|
||||
"是一个易于使用且高效的平台，用于训练和微调大型语言模型。有了 LLaMA-Factory，你可以在本地对数百个预训练模型进行微调，无需编写任何代码。"
|
||||
"[LLaMA-Factory](https://github.com/hiyouga/LlamaFactory) "
|
||||
"是一个易于使用且高效的平台，用于训练和微调大型语言模型。借助 LLaMA-Factory，您无需编写任何代码即可在本地微调数百个预训练模型。"
|
||||
|
||||
#: ../../community/user_stories/llamafactory.md:7
|
||||
#: ../../source/community/user_stories/llamafactory.md:7
|
||||
msgid ""
|
||||
"LLaMA-Facotory users need to evaluate and inference the model after fine-"
|
||||
"tuning the model."
|
||||
msgstr "LLaMA-Facotory 用户需要在对模型进行微调后对模型进行评估和推理。"
|
||||
"LLaMA-Factory users need to evaluate the model and perform inference after "
|
||||
"fine-tuning."
|
||||
msgstr "LLaMA-Factory 用户在微调后需要对模型进行评估并执行推理。"
|
||||
|
||||
#: ../../community/user_stories/llamafactory.md:9
|
||||
msgid "**The Business Challenge**"
|
||||
msgstr "**业务挑战**"
|
||||
#: ../../source/community/user_stories/llamafactory.md:9
|
||||
msgid "Business challenge"
|
||||
msgstr "业务挑战"
|
||||
|
||||
#: ../../community/user_stories/llamafactory.md:11
|
||||
#: ../../source/community/user_stories/llamafactory.md:11
|
||||
msgid ""
|
||||
"LLaMA-Factory used transformers to perform inference on Ascend NPU, but the "
|
||||
"speed was slow."
|
||||
msgstr "LLaMA-Factory 使用 transformers 在 Ascend NPU 上进行推理，但速度较慢。"
|
||||
"LLaMA-Factory uses Transformers to perform inference on Ascend NPUs, but the"
|
||||
" speed is slow."
|
||||
msgstr "LLaMA-Factory 使用 Transformers 在 Ascend NPU 上执行推理，但速度较慢。"
|
||||
|
||||
#: ../../community/user_stories/llamafactory.md:13
|
||||
msgid "**Solving Challenges and Benefits with vLLM Ascend**"
|
||||
msgstr "**通过 vLLM Ascend 解决挑战与收益**"
|
||||
#: ../../source/community/user_stories/llamafactory.md:13
|
||||
msgid "Benefits with vLLM Ascend"
|
||||
msgstr "vLLM Ascend 的优势"
|
||||
|
||||
#: ../../community/user_stories/llamafactory.md:15
|
||||
#: ../../source/community/user_stories/llamafactory.md:15
|
||||
msgid ""
|
||||
"With the joint efforts of LLaMA-Factory and vLLM Ascend ([LLaMA-"
|
||||
"Factory#7739](https://github.com/hiyouga/LLaMA-Factory/pull/7739)), the "
|
||||
"performance of LLaMA-Factory in the model inference stage has been "
|
||||
"significantly improved. According to the test results, the inference speed "
|
||||
"of LLaMA-Factory has been increased to 2x compared to the transformers "
|
||||
"version."
|
||||
"Factory#7739](https://github.com/hiyouga/LlamaFactory/pull/7739)), LLaMA-"
|
||||
"Factory has achieved significant performance gains during model inference. "
|
||||
"Benchmark results show that its inference speed is now up to 2× faster "
|
||||
"compared to the Transformers implementation."
|
||||
msgstr ""
|
||||
"在 LLaMA-Factory 和 vLLM Ascend 的共同努力下（参见 [LLaMA-"
|
||||
"Factory#7739](https://github.com/hiyouga/LLaMA-Factory/pull/7739)），LLaMA-"
|
||||
"Factory 在模型推理阶段的性能得到了显著提升。根据测试结果，LLaMA-Factory 的推理速度相比 transformers 版本提升到了 2"
|
||||
" 倍。"
|
||||
"在 LLaMA-Factory 和 vLLM Ascend 的共同努力下（[LLaMA-"
|
||||
"Factory#7739](https://github.com/hiyouga/LlamaFactory/pull/7739)），LLaMA-"
|
||||
"Factory 在模型推理过程中实现了显著的性能提升。基准测试结果显示，其推理速度相比 Transformers 实现现在最高可提升 2 倍。"
|
||||
|
||||
#: ../../community/user_stories/llamafactory.md:17
|
||||
msgid "**Learn more**"
|
||||
msgstr "**了解更多**"
|
||||
#: ../../source/community/user_stories/llamafactory.md:17
|
||||
msgid "Learn more"
|
||||
msgstr "了解更多"
|
||||
|
||||
#: ../../community/user_stories/llamafactory.md:19
|
||||
#: ../../source/community/user_stories/llamafactory.md:19
|
||||
msgid ""
|
||||
"See more about LLaMA-Factory and how it uses vLLM Ascend for inference on "
|
||||
"the Ascend NPU in the following documentation: [LLaMA-Factory Ascend NPU "
|
||||
"Inference](https://llamafactory.readthedocs.io/en/latest/advanced/npu_inference.html)."
|
||||
"For more details about LLaMA-Factory, please refer to the [official "
|
||||
"documentation](https://llamafactory.readthedocs.io/en/latest/index.html)."
|
||||
msgstr ""
|
||||
"在以下文档中查看更多关于 LLaMA-Factory 以及其如何在 Ascend NPU 上使用 vLLM Ascend 进行推理的信息：[LLaMA-"
|
||||
"Factory Ascend NPU "
|
||||
"推理](https://llamafactory.readthedocs.io/en/latest/advanced/npu_inference.html)。"
|
||||
"有关 LLaMA-Factory "
|
||||
"的更多详细信息，请参阅[官方文档](https://llamafactory.readthedocs.io/en/latest/index.html)。"
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,545 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:1
|
||||
msgid "ACL Graph"
|
||||
msgstr "ACL 图"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:3
|
||||
msgid "Overview"
|
||||
msgstr "概述"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:5
|
||||
msgid ""
|
||||
"ACL Graph is the Ascend realization of vLLM static graph execution. Upstream"
|
||||
" vLLM and PyTorch documents already describe the generic graph model, "
|
||||
"including `CUDAGraphMode`, runtime dispatch, batch descriptors, bucketing "
|
||||
"and padding, and the definitions of full graph and piecewise graph. This "
|
||||
"document focuses on what is specific to Ascend in `vllm-ascend`: the "
|
||||
"platform integration points, the extra constraints introduced by ACL graph "
|
||||
"capture, and the mechanisms used to keep attention parameters correct during"
|
||||
" replay."
|
||||
msgstr ""
|
||||
"ACL 图是 vLLM 静态图执行的昇腾实现。上游 vLLM 和 PyTorch 文档已描述了通用图模型，包括 "
|
||||
"`CUDAGraphMode`、运行时调度、批处理描述符、分桶与填充，以及全图和分段图的定义。本文档聚焦于 `vllm-ascend` "
|
||||
"中昇腾特有的内容：平台集成点、ACL 图捕获引入的额外约束，以及在重放期间保持注意力参数正确的机制。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:7
|
||||
msgid ""
|
||||
"On Ascend, the design goal is the same as upstream static graph execution: "
|
||||
"reduce host launch overhead for small and medium runtime shapes. The "
|
||||
"implementation boundary is different. vLLM provides the generic dispatch "
|
||||
"path, while `vllm-ascend` supplies the platform wrapper, capture-size "
|
||||
"trimming, and attention-specific update logic needed by ACL graph replay."
|
||||
msgstr ""
|
||||
"在昇腾上，设计目标与上游静态图执行相同：减少中小运行时形状的主机启动开销。实现边界不同。vLLM 提供通用调度路径，而 `vllm-ascend` 提供"
|
||||
" ACL 图重放所需的平台包装器、捕获大小修剪和注意力特定更新逻辑。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:9
|
||||
msgid "Prerequisites and References"
|
||||
msgstr "前提条件与参考资料"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:11
|
||||
msgid ""
|
||||
"Upstream vLLM design doc for generic graph concepts: [CUDA "
|
||||
"Graphs](https://docs.vllm.ai/en/latest/design/cuda_graphs/)."
|
||||
msgstr ""
|
||||
"上游 vLLM 通用图概念设计文档：[CUDA "
|
||||
"Graphs](https://docs.vllm.ai/en/latest/design/cuda_graphs/)。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:12
|
||||
msgid ""
|
||||
"PyTorch graph documentation for generic capture and replay semantics: "
|
||||
"[Accelerating PyTorch with CUDA "
|
||||
"Graphs](https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/)."
|
||||
msgstr ""
|
||||
"PyTorch 通用捕获与重放语义文档：[Accelerating PyTorch with CUDA "
|
||||
"Graphs](https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/)。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:13
|
||||
msgid ""
|
||||
"Ascend user guide for operational enablement: [Graph Mode "
|
||||
"Guide](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/graph_mode.html)."
|
||||
msgstr ""
|
||||
"昇腾操作启用用户指南：[Graph Mode "
|
||||
"Guide](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/graph_mode.html)。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:14
|
||||
msgid ""
|
||||
"Existing repo design note: [ACL "
|
||||
"Graph](https://docs.vllm.ai/projects/ascend/zh-"
|
||||
"cn/latest/developer_guide/Design_Documents/ACL_Graph.html)"
|
||||
msgstr ""
|
||||
"现有仓库设计说明：[ACL Graph](https://docs.vllm.ai/projects/ascend/zh-"
|
||||
"cn/latest/developer_guide/Design_Documents/ACL_Graph.html)"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:16
|
||||
msgid ""
|
||||
"This document intentionally does not re-explain upstream topics such as "
|
||||
"graph mode selection, dispatcher behavior, batch descriptor construction, "
|
||||
"capture bucketing, padding policy, or the generic meaning of full versus "
|
||||
"piecewise execution."
|
||||
msgstr "本文档有意不重新解释上游主题，如图模式选择、调度器行为、批处理描述符构建、捕获分桶、填充策略，或全图与分段执行的通用含义。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:18
|
||||
msgid "How ACL Graph Fits into vLLM"
|
||||
msgstr "ACL 图如何融入 vLLM"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:20
|
||||
msgid ""
|
||||
"vLLM owns the generic static graph flow. On Ascend, "
|
||||
"`NPUPlatform.get_static_graph_wrapper_cls()` returns "
|
||||
"`vllm_ascend.compilation.acl_graph.ACLGraphWrapper`, which is the platform-"
|
||||
"specific wrapper used when vLLM enables static graph mode."
|
||||
msgstr ""
|
||||
"vLLM 拥有通用静态图流程。在昇腾上，`NPUPlatform.get_static_graph_wrapper_cls()` 返回 "
|
||||
"`vllm_ascend.compilation.acl_graph.ACLGraphWrapper`，这是 vLLM "
|
||||
"启用静态图模式时使用的平台特定包装器。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:22
|
||||
msgid "`ACLGraphWrapper` is responsible for:"
|
||||
msgstr "`ACLGraphWrapper` 负责："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:24
|
||||
msgid ""
|
||||
"reading the runtime mode and `batch_descriptor` from the forward context,"
|
||||
msgstr "从前向上下文中读取运行时模式和 `batch_descriptor`，"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:25
|
||||
msgid ""
|
||||
"deciding whether to run eagerly, capture a new ACL graph, or replay a cached"
|
||||
" ACL graph,"
|
||||
msgstr "决定是立即执行、捕获新的 ACL 图，还是重放缓存的 ACL 图，"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:26
|
||||
msgid "caching graph entries per batch descriptor,"
|
||||
msgstr "按批处理描述符缓存图条目，"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:27
|
||||
msgid ""
|
||||
"preserving the graph pool and replay bookkeeping needed by the Ascend "
|
||||
"backend."
|
||||
msgstr "保留昇腾后端所需的图池和重放记账信息。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:29
|
||||
msgid ""
|
||||
"The wrapper does not define the upstream dispatch policy. It assumes the "
|
||||
"runtime mode and batch descriptor have already been chosen correctly by "
|
||||
"vLLM, then applies Ascend capture or replay to that concrete runtime shape."
|
||||
msgstr "该包装器不定义上游调度策略。它假设运行时模式和批处理描述符已由 vLLM 正确选择，然后对该具体运行时形状应用昇腾捕获或重放。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:31
|
||||
msgid "Capture Sizes and Bucketing"
|
||||
msgstr "捕获大小与分桶"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:33
|
||||
msgid ""
|
||||
"vLLM graph replay requires stable runtime shapes, so vLLM does not try to "
|
||||
"capture every possible batch shape. Instead, it prepares a finite set of "
|
||||
"capture sizes and dispatches a runtime batch to the nearest supported size. "
|
||||
"If the runtime batch is larger than the largest configured capture size, "
|
||||
"graph mode is skipped and execution falls back to eager mode."
|
||||
msgstr ""
|
||||
"vLLM 图重放需要稳定的运行时形状，因此 vLLM "
|
||||
"不会尝试捕获所有可能的批处理形状。相反，它准备一组有限的捕获大小，并将运行时批处理调度到最近的支持大小。如果运行时批处理大于配置的最大捕获大小，则跳过图模式并回退到立即执行模式。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:35
|
||||
msgid "By default, vLLM builds capture sizes as:"
|
||||
msgstr "默认情况下，vLLM 构建捕获大小为："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:37
|
||||
msgid "`1`, `2`, `4`"
|
||||
msgstr "`1`, `2`, `4`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:38
|
||||
msgid "multiples of `8` from `8` up to `255`"
|
||||
msgstr "从 `8` 到 `255` 的 `8` 的倍数"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:39
|
||||
msgid "multiples of `16` from `256` up to `max_cudagraph_capture_size`"
|
||||
msgstr "从 `256` 到 `max_cudagraph_capture_size` 的 `16` 的倍数"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:41
|
||||
msgid "Conceptually, the default list looks like:"
|
||||
msgstr "概念上，默认列表如下："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:47
|
||||
msgid ""
|
||||
"The smaller step at small batch sizes reduces padding overhead where latency"
|
||||
" is most sensitive, while the larger step at bigger sizes keeps the number "
|
||||
"of captured graphs under control."
|
||||
msgstr "小批量时步长较小，可减少延迟最敏感区域的填充开销；大批量时步长较大，可控制捕获图的数量。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:49
|
||||
msgid ""
|
||||
"On Ascend, this generic upstream bucketing strategy is still the starting "
|
||||
"point, but the final capture sizes may be reduced further by platform-"
|
||||
"specific constraints:"
|
||||
msgstr "在昇腾上，此通用上游分桶策略仍是起点，但最终捕获大小可能因平台特定约束而进一步减少："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:51
|
||||
msgid "sequence-parallel filtering may remove unsupported sizes,"
|
||||
msgstr "序列并行过滤可能移除不支持的大小，"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:52
|
||||
msgid ""
|
||||
"runtime resource limits may still prevent some configured sizes from being "
|
||||
"captured,"
|
||||
msgstr "运行时资源限制仍可能阻止某些配置大小被捕获，"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:53
|
||||
msgid "some runtime modes may be normalized before capture begins."
|
||||
msgstr "某些运行时模式可能在捕获开始前被规范化。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:55
|
||||
msgid "Ascend-Specific Design Constraints"
|
||||
msgstr "昇腾特定设计约束"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:57
|
||||
msgid "Capture breadth is still constrained by runtime resources"
|
||||
msgstr "捕获广度仍受运行时资源限制"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:59
|
||||
msgid ""
|
||||
"Unlike CUDA Graph on CUDA devices, ACL graph capture on Ascend can still "
|
||||
"fail when the selected graph sizes consume more runtime resources than the "
|
||||
"current backend can supply. Piecewise mode is the most sensitive case "
|
||||
"because it captures many subgraphs and the total capture cost scales with "
|
||||
"model depth and configured size coverage."
|
||||
msgstr ""
|
||||
"与 CUDA 设备上的 CUDA Graph 不同，昇腾上的 ACL "
|
||||
"图捕获在所选图大小消耗的运行时资源超过当前后端可提供的能力时仍可能失败。分段模式是最敏感的情况，因为它捕获许多子图，且总捕获成本随模型深度和配置大小覆盖范围而扩展。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:61
|
||||
msgid ""
|
||||
"Older versions of vLLM Ascend applied a local `update_aclgraph_sizes()` "
|
||||
"heuristic to shrink the PIECEWISE capture-size set before final capture. "
|
||||
"That heuristic has been removed. The current implementation keeps upstream "
|
||||
"sizing and dispatch behavior intact, then intercepts the confirmed capture-"
|
||||
"time stream-resource signature in `vllm_ascend/compilation/acl_graph.py` and"
|
||||
" re-raises it with clearer mitigation guidance."
|
||||
msgstr ""
|
||||
"旧版 vLLM Ascend 应用了本地 `update_aclgraph_sizes()` 启发式方法，在最终捕获前缩小PIECEWISE "
|
||||
"捕获大小集。该启发式方法已被移除。当前实现保持上游大小和调度行为不变，然后在 "
|
||||
"`vllm_ascend/compilation/acl_graph.py` 中拦截确认的捕获时流资源签名，并重新抛出更清晰的缓解指导。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:63
|
||||
msgid ""
|
||||
"In practice, this means users should treat `cudagraph_capture_sizes` and "
|
||||
"`max_cudagraph_capture_size` as the primary tuning levers when capture "
|
||||
"fails. Newer HDK/CANN combinations can materially improve ACL graph "
|
||||
"capacity, while communication-heavy configurations may still require a "
|
||||
"smaller configured size set."
|
||||
msgstr ""
|
||||
"在实践中，这意味着当捕获失败时，用户应将 `cudagraph_capture_sizes` 和 "
|
||||
"`max_cudagraph_capture_size` 作为主要调优杠杆。更新的 HDK/CANN 组合可以显著改善ACL "
|
||||
"图容量，而通信密集型配置可能仍需要更小的配置大小集。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:65
|
||||
msgid "Platform mode normalization is stricter than generic upstream behavior"
|
||||
msgstr "平台模式规范化比通用上游行为更严格"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:67
|
||||
msgid ""
|
||||
"Ascend currently narrows some generic upstream modes in "
|
||||
"`vllm_ascend.platform.NPUPlatform.check_and_update_config()`."
|
||||
msgstr ""
|
||||
"昇腾目前在 `vllm_ascend.platform.NPUPlatform.check_and_update_config()` "
|
||||
"中收窄了一些通用上游模式。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:69
|
||||
msgid "Encoder-decoder models are forced to `PIECEWISE`."
|
||||
msgstr "编码器-解码器模型被强制使用 `PIECEWISE`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:70
|
||||
msgid "`use_inductor` is disabled for ACL graph paths."
|
||||
msgstr "`use_inductor` 在 ACL 图路径上被禁用。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:71
|
||||
msgid "`ASCEND_LAUNCH_BLOCKING=1` is rejected when ACL graph is enabled."
|
||||
msgstr "启用 ACL 图时，`ASCEND_LAUNCH_BLOCKING=1` 被拒绝。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:72
|
||||
msgid ""
|
||||
"Xlite graph mode can disable ACL graph full mode or fall back to "
|
||||
"`FULL_DECODE_ONLY`, depending on configuration."
|
||||
msgstr "Xlite 图模式可以禁用 ACL 图全模式，或根据配置回退到 `FULL_DECODE_ONLY`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:74
|
||||
msgid ""
|
||||
"These checks document the subset of upstream graph behavior that the current"
|
||||
" Ascend backend can execute safely. Some of them are long-term platform "
|
||||
"constraints, while others are clearly transitional in the current "
|
||||
"implementation."
|
||||
msgstr "这些检查记录了当前 Ascend 后端能够安全执行的上游图行为子集。其中一些是长期平台约束，而另一些在当前实现中显然是过渡性的。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:76
|
||||
msgid "Key Ascend-Specific Mechanisms"
|
||||
msgstr "关键 Ascend 特有机制"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:78
|
||||
msgid "Host-side attention parameter update for full graph replay"
|
||||
msgstr "用于全图重放的主机端注意力参数更新"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:80
|
||||
msgid ""
|
||||
"Full graph replay on Ascend has an extra problem that upstream generic "
|
||||
"documentation does not cover in detail: some attention operators need "
|
||||
"runtime metadata updates even when the overall graph is static. The Ascend "
|
||||
"implementation handles this by separating graph capture from host-side task "
|
||||
"parameter updates."
|
||||
msgstr ""
|
||||
"Ascend 上的全图重放有一个上游通用文档未详细涵盖的额外问题：即使整体图是静态的，某些注意力算子也需要运行时元数据更新。Ascend "
|
||||
"实现通过将图捕获与主机端任务参数更新分离来处理此问题。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:82
|
||||
msgid "The flow is:"
|
||||
msgstr "流程如下："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:84
|
||||
msgid ""
|
||||
"During capture, attention backends record per-graph task handles, events, "
|
||||
"workspaces, and weak references to the tensors or metadata that must be "
|
||||
"refreshed."
|
||||
msgstr "在捕获期间，注意力后端记录每个图的任务句柄、事件、工作空间，以及需要刷新的张量或元数据的弱引用。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:85
|
||||
msgid ""
|
||||
"Before replay, `update_full_graph_params()` calls the backend specific "
|
||||
"`update_graph_params()` implementation."
|
||||
msgstr ""
|
||||
"在重放之前，`update_full_graph_params()` 调用后端特定的 `update_graph_params()` 实现。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:86
|
||||
msgid ""
|
||||
"That backend runs parameter refresh on an update stream with "
|
||||
"`torch.npu.graph_task_update_begin(...)` and "
|
||||
"`torch.npu.graph_task_update_end(...)` around the underlying attention "
|
||||
"operator launch."
|
||||
msgstr ""
|
||||
"该后端在更新流上运行参数刷新，在底层注意力算子启动前后分别调用 `torch.npu.graph_task_update_begin(...)` 和 "
|
||||
"`torch.npu.graph_task_update_end(...)`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:87
|
||||
msgid ""
|
||||
"`torch.npu.ExternalEvent` objects are used to enforce ordering between the "
|
||||
"host-side update stream and the replay stream."
|
||||
msgstr "`torch.npu.ExternalEvent` 对象用于强制主机端更新流与重放流之间的顺序。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:89
|
||||
msgid "This mechanism is implemented in attention backends such as:"
|
||||
msgstr "该机制在以下注意力后端中实现："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:91
|
||||
msgid "`vllm_ascend/attention/attention_v1.py`"
|
||||
msgstr "`vllm_ascend/attention/attention_v1.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:92
|
||||
msgid "`vllm_ascend/attention/mla_v1.py`"
|
||||
msgstr "`vllm_ascend/attention/mla_v1.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:93
|
||||
msgid "`vllm_ascend/attention/context_parallel/attention_cp.py`"
|
||||
msgstr "`vllm_ascend/attention/context_parallel/attention_cp.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:94
|
||||
msgid "`vllm_ascend/attention/context_parallel/mla_cp.py`"
|
||||
msgstr "`vllm_ascend/attention/context_parallel/mla_cp.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:96
|
||||
msgid ""
|
||||
"The important design point is that Ascend full graph support depends on "
|
||||
"backend-provided `update_graph_params()` hooks. Without that hook, capture "
|
||||
"alone is not enough to replay the correct attention state."
|
||||
msgstr ""
|
||||
"重要的设计点是，Ascend 全图支持依赖于后端提供的 `update_graph_params()` "
|
||||
"钩子。没有该钩子，仅靠捕获不足以重放正确的注意力状态。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:98
|
||||
msgid "Replay ordering and synchronization"
|
||||
msgstr "重放顺序与同步"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:100
|
||||
msgid ""
|
||||
"`ACLGraphWrapper` synchronizes the current stream before replay in the "
|
||||
"common path to ensure that host-side parameter updates stay aligned with the"
|
||||
" graph execution that will consume them. This is especially relevant in "
|
||||
"asynchronous scheduling or multi-threaded execution."
|
||||
msgstr ""
|
||||
"`ACLGraphWrapper` "
|
||||
"在通用路径上的重放之前同步当前流，以确保主机端参数更新与将要消费它们的图执行保持对齐。这在异步调度或多线程执行中尤其重要。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:102
|
||||
msgid ""
|
||||
"If ordering is not preserved, the parameter update for iteration *i* can be "
|
||||
"observed by the replay of iteration *i-1*, or the replay of iteration *i* "
|
||||
"can start before its own parameter update has completed. In practice, this "
|
||||
"means the attention operator may run with mismatched runtime metadata, which"
|
||||
" can cause incorrect results, precision issues, or even hangs. The code "
|
||||
"keeps a narrower path for the main full-graph eagle case, but the general "
|
||||
"design assumption is the same: replay must not overtake pending parameter "
|
||||
"update work."
|
||||
msgstr ""
|
||||
"如果顺序未被保持，迭代 *i* 的参数更新可能被迭代 *i-1* 的重放观察到，或者迭代 *i* "
|
||||
"的重放可能在其自身参数更新完成之前开始。实际上，这意味着注意力算子可能以不匹配的运行时元数据运行，从而导致结果错误、精度问题甚至挂起。代码为主全图 "
|
||||
"eagle 场景保留了一条更窄的路径，但总体设计假设相同：重放不得超越待处理的参数更新工作。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:104
|
||||
msgid "Full vs Piecewise on Ascend"
|
||||
msgstr "Ascend 上的全图与分段图"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:106
|
||||
msgid ""
|
||||
"Upstream docs already define full graph and piecewise graph semantically. On"
|
||||
" Ascend, the practical difference is driven by backend support and resource "
|
||||
"cost."
|
||||
msgstr "上游文档已在语义上定义了全图和分段图。在 Ascend 上，实际差异由后端支持和资源成本驱动。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:108
|
||||
msgid "Piecewise mode"
|
||||
msgstr "分段模式"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:110
|
||||
msgid ""
|
||||
"Piecewise mode is the conservative path. It relies on the generic vLLM split"
|
||||
" execution strategy, then applies ACL graph capture to the non-attention "
|
||||
"segments selected by the compilation path. On Ascend, this mode is currently"
|
||||
" the more widely supported option, but it is also the most sensitive to "
|
||||
"stream pressure because the number of captured graphs scales with model "
|
||||
"depth."
|
||||
msgstr ""
|
||||
"分段模式是保守路径。它依赖于通用的 vLLM 拆分执行策略，然后将 ACL 图捕获应用于编译路径选择的非注意力段。在 Ascend "
|
||||
"上，此模式目前是更广泛支持的选项，但它对流压力也最敏感，因为捕获的图数量随模型深度扩展。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:112
|
||||
msgid "Full graph mode"
|
||||
msgstr "全图模式"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:114
|
||||
msgid ""
|
||||
"Full graph mode is the more performance-oriented path when the attention "
|
||||
"backend can support runtime parameter patching through "
|
||||
"`update_graph_params()`. On Ascend, full graph support is tied to those "
|
||||
"attention-specific update hooks, workspace caching, and replay ordering "
|
||||
"guarantees."
|
||||
msgstr ""
|
||||
"当注意力后端能够通过 `update_graph_params()` 支持运行时参数修补时，全图模式是更注重性能的路径。在 Ascend "
|
||||
"上，全图支持与那些注意力特定的更新钩子、工作空间缓存和重放顺序保证相关联。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:116
|
||||
msgid "Diagnostics and Operational Notes"
|
||||
msgstr "诊断与操作说明"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:118
|
||||
msgid ""
|
||||
"The simplest way to confirm that graph mode is active is to enable cudagraph"
|
||||
" metrics and keep log stats enabled. In CLI usage, use `--cudagraph-metrics`"
|
||||
" and do not pass `--disable-log-stats`. In Python usage, set "
|
||||
"`cudagraph_metrics=True` and `disable_log_stats=False`. Then inspect the "
|
||||
"emitted metrics and logs."
|
||||
msgstr ""
|
||||
"确认图模式已激活的最简单方法是启用 cudagraph 指标并保持日志统计启用。在 CLI 使用中，使用 `--cudagraph-metrics` "
|
||||
"且不要传递 `--disable-log-stats`。在 Python 使用中，设置 `cudagraph_metrics=True` 和 "
|
||||
"`disable_log_stats=False`。然后检查输出的指标和日志。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:119
|
||||
msgid ""
|
||||
"Profiling can also confirm whether replay is happening, and developers can "
|
||||
"add temporary prints before replay when debugging locally, but those are "
|
||||
"secondary methods and are not expanded here."
|
||||
msgstr "性能分析也可以确认是否正在发生重放，开发者在本地调试时可以在重放前添加临时打印，但这些是次要方法，此处不展开说明。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:120
|
||||
msgid ""
|
||||
"Capture-size selection primarily follows upstream configuration and dispatch"
|
||||
" behavior; only the confirmed stream-resource capture failure is rewritten "
|
||||
"with user-facing guidance at runtime."
|
||||
msgstr "捕获大小选择主要遵循上游配置和分发行为；仅在确认流资源捕获失败时，才会在运行时重写为用户可见的指导信息。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:121
|
||||
msgid ""
|
||||
"In debug mode, `ACLGraphWrapper` asserts that replay uses the same tensor "
|
||||
"addresses recorded during capture."
|
||||
msgstr "在调试模式下，`ACLGraphWrapper` 断言重放使用捕获期间记录的相同张量地址。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:122
|
||||
msgid ""
|
||||
"`ASCEND_LAUNCH_BLOCKING=1` is incompatible with ACL graph enablement in the "
|
||||
"current implementation."
|
||||
msgstr "在当前实现中，`ASCEND_LAUNCH_BLOCKING=1` 与 ACL 图启用不兼容。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:123
|
||||
msgid ""
|
||||
"For debugging inside graph execution, the repo also provides graph-aware "
|
||||
"print helpers in `vllm_ascend.utils`, but those are developer diagnostics "
|
||||
"rather than part of the execution design."
|
||||
msgstr ""
|
||||
"为了在图执行中进行调试，该仓库还在 `vllm_ascend.utils` "
|
||||
"中提供了图感知的打印辅助工具，但这些属于开发者诊断功能，而非执行设计的一部分。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:125
|
||||
msgid "Related Files"
|
||||
msgstr "相关文件"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:127
|
||||
msgid ""
|
||||
"`vllm_ascend/platform.py`, mode normalization, platform hooks, and static "
|
||||
"graph wrapper selection."
|
||||
msgstr "`vllm_ascend/platform.py`，模式归一化、平台钩子以及静态图包装器选择。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:128
|
||||
msgid ""
|
||||
"`vllm_ascend/compilation/acl_graph.py`, ACL graph wrapper, capture and "
|
||||
"replay cache, graph parameter containers, and full graph update dispatch."
|
||||
msgstr ""
|
||||
"`vllm_ascend/compilation/acl_graph.py`，ACL 图包装器、捕获与重放缓存、图参数容器以及全图更新分发。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:129
|
||||
msgid ""
|
||||
"`vllm_ascend/compilation/acl_graph.py`, runtime ACL graph capture, replay, "
|
||||
"and capture-failure guidance."
|
||||
msgstr "`vllm_ascend/compilation/acl_graph.py`，运行时 ACL 图捕获、重放以及捕获失败引导。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:130
|
||||
msgid ""
|
||||
"`vllm_ascend/attention/attention_v1.py`, full graph attention parameter "
|
||||
"capture and update logic."
|
||||
msgstr "`vllm_ascend/attention/attention_v1.py`，全图注意力参数捕获与更新逻辑。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:131
|
||||
msgid ""
|
||||
"`vllm_ascend/attention/mla_v1.py`, MLA (Multi-Head Latent Attention) "
|
||||
"specific full graph parameter capture and update logic."
|
||||
msgstr "`vllm_ascend/attention/mla_v1.py`，MLA（多头潜在注意力）特定的全图参数捕获与更新逻辑。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:132
|
||||
msgid ""
|
||||
"`vllm_ascend/attention/context_parallel/attention_cp.py`, context parallel "
|
||||
"attention update path."
|
||||
msgstr ""
|
||||
"`vllm_ascend/attention/context_parallel/attention_cp.py`，上下文并行注意力更新路径。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ACL_Graph.md:133
|
||||
msgid ""
|
||||
"`vllm_ascend/attention/context_parallel/mla_cp.py`, context parallel MLA "
|
||||
"update path."
|
||||
msgstr "`vllm_ascend/attention/context_parallel/mla_cp.py`，上下文并行 MLA 更新路径。"
|
||||
@@ -0,0 +1,340 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:1
|
||||
msgid "KV Cache Pool"
|
||||
msgstr "KV缓存池"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:3
|
||||
msgid "Why KV Cache Pool?"
|
||||
msgstr "为什么需要KV缓存池？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:5
|
||||
msgid ""
|
||||
"Prefix caching is an important feature in LLM inference that can reduce "
|
||||
"prefill computation time drastically."
|
||||
msgstr "前缀缓存是LLM推理中的一个重要特性，可以大幅减少预填充计算时间。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:7
|
||||
msgid ""
|
||||
"However, the performance gain from prefix caching is highly dependent on the"
|
||||
" cache hit rate, while the cache hit rate can be limited if one only uses "
|
||||
"on-chip memory for KV cache storage."
|
||||
msgstr "然而，前缀缓存的性能提升高度依赖于缓存命中率，而如果仅使用片上内存存储KV缓存，缓存命中率可能会受到限制。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:9
|
||||
msgid ""
|
||||
"Hence, KV Cache Pool is proposed to utilize various types of storage "
|
||||
"including on-chip memory, DRAM, and SSD, making a pool for KV Cache storage "
|
||||
"while making the prefix of requests visible across all nodes, increasing the"
|
||||
" cache hit rate for all requests."
|
||||
msgstr ""
|
||||
"因此，我们提出了KV缓存池，利用包括片上内存、DRAM和SSD在内的多种存储类型，构建KV缓存存储池，同时使请求前缀在所有节点间可见，从而提高所有请求的缓存命中率。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:11
|
||||
msgid ""
|
||||
"vLLM Ascend currently supports [MooncakeStore](https://github.com/kvcache-"
|
||||
"ai/Mooncake), one of the most recognized KV Cache storage engines."
|
||||
msgstr ""
|
||||
"vLLM Ascend目前支持[MooncakeStore](https://github.com/kvcache-"
|
||||
"ai/Mooncake)，这是最受认可的KV缓存存储引擎之一。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:13
|
||||
msgid ""
|
||||
"While one can utilize MooncakeStore in vLLM V1 engine by setting it as a "
|
||||
"remote backend of LMCache with GPU (see "
|
||||
"[Tutorial](https://github.com/LMCache/LMCache/blob/dev/examples/kv_cache_reuse/remote_backends/mooncakestore/README.md)),"
|
||||
" we find it would be better to integrate a connector that directly supports "
|
||||
"MooncakeStore and can utilize the data transfer strategy that best fits "
|
||||
"Huawei NPU hardware."
|
||||
msgstr ""
|
||||
"虽然可以通过将MooncakeStore设置为LMCache的远程后端（使用GPU）在vLLM "
|
||||
"V1引擎中使用它（参见[教程](https://github.com/LMCache/LMCache/blob/dev/examples/kv_cache_reuse/remote_backends/mooncakestore/README.md)），但我们发现集成一个直接支持MooncakeStore并能利用最适合华为NPU硬件的数据传输策略的连接器会更好。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:15
|
||||
msgid ""
|
||||
"Hence, we propose to integrate MooncakeStore with a brand new "
|
||||
"**MooncakeStoreConnectorV1**, which is indeed largely inspired by "
|
||||
"**LMCacheConnectorV1** (see the [How is MooncakeStoreConnectorV1 "
|
||||
"Implemented?](#how-is-mooncakestoreconnectorv1-implemented) section)."
|
||||
msgstr ""
|
||||
"因此，我们建议将MooncakeStore与全新的**MooncakeStoreConnectorV1**集成，该连接器在很大程度上受到了**LMCacheConnectorV1**的启发（参见[如何实现MooncakeStoreConnectorV1？](#how-"
|
||||
"is-mooncakestoreconnectorv1-implemented)部分）。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:17
|
||||
msgid "Usage"
|
||||
msgstr "使用方法"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:19
|
||||
msgid ""
|
||||
"vLLM Ascend currently supports MooncakeStore for KV Cache Pool. To enable "
|
||||
"MooncakeStore, one needs to configure `kv-transfer-config` and choose "
|
||||
"`MooncakeStoreConnector` as the KV Connector."
|
||||
msgstr ""
|
||||
"vLLM Ascend目前支持MooncakeStore用于KV缓存池。要启用MooncakeStore，需要配置`kv-transfer-"
|
||||
"config`并选择`MooncakeStoreConnector`作为KV连接器。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:21
|
||||
msgid ""
|
||||
"For step-by-step deployment and configuration, please refer to the [KV Pool "
|
||||
"User "
|
||||
"Guide](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/kv_pool.html)."
|
||||
msgstr ""
|
||||
"有关分步部署和配置，请参考[KV池用户指南](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/kv_pool.html)。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:23
|
||||
msgid "How it works?"
|
||||
msgstr "工作原理"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:25
|
||||
msgid ""
|
||||
"The KV Cache Pool integrates multiple memory tiers (on-chip memory, DRAM, "
|
||||
"SSD, etc.) through a connector-based architecture."
|
||||
msgstr "KV缓存池通过基于连接器的架构集成了多个内存层级（片上内存、DRAM、SSD等）。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:27
|
||||
msgid ""
|
||||
"Each connector implements a unified interface for storing, retrieving, and "
|
||||
"transferring KV blocks between tiers, depending on access frequency and "
|
||||
"hardware bandwidth."
|
||||
msgstr "每个连接器实现了统一的接口，用于在不同层级之间存储、检索和传输KV块，具体取决于访问频率和硬件带宽。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:29
|
||||
msgid ""
|
||||
"When combined with vLLM's Prefix Caching mechanism, the pool enables "
|
||||
"efficient caching both locally (in on-chip memory) and globally (via "
|
||||
"Mooncake), ensuring that frequently used prefixes remain hot while less "
|
||||
"frequently accessed KV data can spill over to lower-cost memory."
|
||||
msgstr ""
|
||||
"当与vLLM的前缀缓存机制结合使用时，该池实现了本地（片上内存）和全局（通过Mooncake）的高效缓存，确保常用前缀保持热状态，而较少访问的KV数据可以溢出到成本更低的内存。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:31
|
||||
msgid "1. Combining KV Cache Pool with on-chip memory Prefix Caching"
|
||||
msgstr "1.将KV缓存池与片上内存前缀缓存结合"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:33
|
||||
msgid ""
|
||||
"Prefix Caching with on-chip memory is already supported by the vLLM V1 "
|
||||
"Engine. By introducing KV Connector V1, users can seamlessly combine on-chip"
|
||||
" memory-based Prefix Caching with Mooncake-backed KV Pool."
|
||||
msgstr ""
|
||||
"vLLM V1引擎已支持基于片上内存的前缀缓存。通过引入KV连接器V1，用户可以无缝地将基于片上内存的前缀缓存与Mooncake支持的KV池结合起来。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:36
|
||||
msgid ""
|
||||
"The user can enable both features simply by enabling Prefix Caching, which "
|
||||
"is enabled by default in vLLM V1 unless the `--no-enable-prefix-caching` "
|
||||
"flag is set, and setting up the KV Connector for KV Pool (e.g., the "
|
||||
"MooncakeStoreConnector)."
|
||||
msgstr ""
|
||||
"用户只需启用前缀缓存（vLLM V1中默认启用，除非设置了`--no-enable-prefix-"
|
||||
"caching`标志）并为KV池设置KV连接器（例如MooncakeStoreConnector），即可同时启用这两个功能。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:38
|
||||
msgid "**Workflow**:"
|
||||
msgstr "**工作流程**："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:40
|
||||
msgid "The engine first checks for prefix hits in the on-chip memory cache."
|
||||
msgstr "引擎首先检查片上内存缓存中的前缀命中。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:42
|
||||
msgid ""
|
||||
"After getting the number of hit tokens on on-chip memory, it queries the KV "
|
||||
"Pool via the connector. If there are additional hits in the KV Pool, we get "
|
||||
"the **additional blocks only** from the KV Pool, and get the rest of the "
|
||||
"blocks directly from on-chip memory to minimize the data transfer latency."
|
||||
msgstr ""
|
||||
"在获取片上内存的命中token数量后，它通过连接器查询KV池。如果KV池中有额外的命中，我们仅从KV池获取**额外的块**，其余块直接从片上内存获取，以最小化数据传输延迟。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:44
|
||||
msgid ""
|
||||
"After the KV Caches in the KV Pool are loaded into on-chip memory, the "
|
||||
"remaining process is the same as Prefix Caching in on-chip memory."
|
||||
msgstr "KV池中的KV缓存加载到片上内存后，剩余过程与片上内存中的前缀缓存相同。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:46
|
||||
msgid "2. Combining KV Cache Pool with Mooncake PD Disaggregation"
|
||||
msgstr "2.将KV缓存池与Mooncake PD分离结合"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:48
|
||||
msgid ""
|
||||
"When used together with Mooncake PD (Prefill-Decode) Disaggregation, the KV "
|
||||
"Cache Pool can further decouple prefill and decode stages across devices or "
|
||||
"nodes."
|
||||
msgstr "当与Mooncake PD（预填充-解码）分离一起使用时，KV缓存池可以进一步跨设备或节点解耦预填充和解码阶段。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:50
|
||||
msgid ""
|
||||
"Currently, we only perform put and get operations of KV Pool for **Prefill "
|
||||
"Nodes**, and Decode Nodes get their KV Cache from Mooncake P2P KV Connector,"
|
||||
" i.e., MooncakeConnector."
|
||||
msgstr ""
|
||||
"目前，我们仅对**预填充节点**执行KV池的put和get操作，解码节点则从Mooncake P2P "
|
||||
"KV连接器（即MooncakeConnector）获取其KV缓存。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:52
|
||||
msgid ""
|
||||
"The key benefit of doing this is that we can keep the gain in performance by"
|
||||
" computing less with Prefix Caching from on-chip memory and KV Pool for "
|
||||
"Prefill Nodes, while not sacrificing the data transfer efficiency between "
|
||||
"Prefill and Decode nodes with P2P KV Connector that transfers KV Caches "
|
||||
"between NPU devices directly."
|
||||
msgstr ""
|
||||
"这样做的主要好处是，我们可以通过片上内存和KV池的前缀缓存减少预填充节点的计算量，从而保持性能提升，同时不牺牲预填充节点和解码节点之间的数据传输效率，因为P2P"
|
||||
" KV连接器直接在NPU设备之间传输KV缓存。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:54
|
||||
msgid ""
|
||||
"To enable this feature, we need to set up both Mooncake Connector and "
|
||||
"MooncakeStore Connector with a Multi Connector, which is a KV Connector "
|
||||
"class provided by vLLM that can call multiple KV Connectors in a specific "
|
||||
"order."
|
||||
msgstr ""
|
||||
"要启用此功能，我们需要使用Multi Connector同时设置Mooncake Connector和MooncakeStore "
|
||||
"Connector，Multi Connector是vLLM提供的一个KV连接器类，可以按特定顺序调用多个KV连接器。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:56
|
||||
msgid ""
|
||||
"For details, please also refer to the [Mooncake connector deployment "
|
||||
"guide](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/examples/disaggregated_prefill_v1/mooncake_connector_deployment_guide.md)."
|
||||
msgstr ""
|
||||
"有关详细信息，请同时参考[Mooncake连接器部署指南](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/examples/disaggregated_prefill_v1/mooncake_connector_deployment_guide.md)。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:58
|
||||
msgid "How is MooncakeStoreConnectorV1 Implemented?"
|
||||
msgstr "MooncakeStoreConnectorV1是如何实现的？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:60
|
||||
msgid ""
|
||||
"**MooncakeStoreConnectorV1** inherits the KV Connector V1 class in vLLM V1: "
|
||||
"through implementing the required methods defined in the KV connector V1 "
|
||||
"base class, one can integrate a third-party KV cache transfer/storage "
|
||||
"backend into the vLLM framework."
|
||||
msgstr ""
|
||||
"**MooncakeStoreConnectorV1**继承了vLLM "
|
||||
"V1中的KV连接器V1类：通过实现KV连接器V1基类中定义的必要方法，可以将第三方KV缓存传输/存储后端集成到vLLM框架中。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:62
|
||||
msgid ""
|
||||
"MooncakeStoreConnectorV1 is also largely inspired by LMCacheConnectorV1 in "
|
||||
"terms of the `Lookup Engine`/`Lookup Client` design for looking up KV cache "
|
||||
"keys, and the `ChunkedTokenDatabase` class for processing tokens into "
|
||||
"prefix-aware hashes as well as other hashing related designs. On top of "
|
||||
"this, we have also added our own design including `KVTransferThread` that "
|
||||
"allows async `get` and `put` of KV caches with multi-threading, and NPU-"
|
||||
"related data transfer optimization such as removing the `LocalBuffer` in "
|
||||
"LMCache to remove redundant data transfer."
|
||||
msgstr ""
|
||||
"MooncakeStoreConnectorV1 在很大程度上也借鉴了 LMCacheConnectorV1 的 `Lookup "
|
||||
"Engine`/`Lookup Client` 设计（用于查找 KV 缓存键），以及 `ChunkedTokenDatabase` "
|
||||
"类（用于将令牌处理为前缀感知哈希）和其他哈希相关设计。在此基础上，我们还添加了自己的设计，包括允许通过多线程异步 `get` 和 `put` KV "
|
||||
"缓存的 `KVTransferThread`，以及 NPU 相关的数据传输优化，例如移除 LMCache 中的 `LocalBuffer` "
|
||||
"以消除冗余数据传输。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:64
|
||||
msgid ""
|
||||
"The KV Connector methods that need to be implemented can be categorized into"
|
||||
" scheduler-side methods that are called in V1 scheduler and worker-side "
|
||||
"methods that are called in V1 worker, namely:"
|
||||
msgstr "需要实现的 KV Connector 方法可分为调度器端方法（在 V1 调度器中调用）和工作器端方法（在 V1 工作器中调用），具体如下："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:66
|
||||
msgid "KV Connector Scheduler-Side Methods"
|
||||
msgstr "KV Connector 调度器端方法"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:68
|
||||
msgid ""
|
||||
"`get_num_new_matched_tokens`: Get prefix cache hit in number of tokens "
|
||||
"through looking up into the KV pool. `update_states_after_alloc`: Update "
|
||||
"KVConnector state after temporary buffer alloc. `build_connector_meta`: "
|
||||
"Attach the connector metadata to the request object. `request_finished`: "
|
||||
"Once a request is finished, determine whether request blocks should be freed"
|
||||
" now or will be sent asynchronously and freed later."
|
||||
msgstr ""
|
||||
"`get_num_new_matched_tokens`：通过查找 KV "
|
||||
"池获取前缀缓存命中（以令牌数计）。`update_states_after_alloc`：在临时缓冲区分配后更新 KVConnector "
|
||||
"状态。`build_connector_meta`：将连接器元数据附加到请求对象。`request_finished`：请求完成后，判断请求块是应立即释放，还是将异步发送并稍后释放。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:73
|
||||
msgid "Connector Worker-Side Methods"
|
||||
msgstr "Connector 工作器端方法"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:75
|
||||
msgid ""
|
||||
"`register_kv_caches`: Register KV cache buffers needed for KV cache "
|
||||
"transfer. `start_load_kv`: Perform KV cache load operation that transfers KV"
|
||||
" cache from storage to device. `wait_for_layer_load`: Optional; Wait for "
|
||||
"layer load in layerwise + async KV load scenario. `save_kv_layer`: Optional;"
|
||||
" Do layerwise KV cache put into KV Pool. `wait_for_save`: Wait for KV Save "
|
||||
"to finish if async KV cache save/put. `get_finished`: Get request that "
|
||||
"finished KV transfer, `done_sending` if `put` finished, `done_receiving` if "
|
||||
"`get` finished."
|
||||
msgstr ""
|
||||
"`register_kv_caches`：注册 KV 缓存传输所需的 KV 缓存缓冲区。`start_load_kv`：执行 KV 缓存加载操作，将 "
|
||||
"KV 缓存从存储传输到设备。`wait_for_layer_load`：可选；在逐层 + 异步 KV "
|
||||
"加载场景中等待层加载。`save_kv_layer`：可选；执行逐层 KV 缓存放入 KV 池。`wait_for_save`：如果异步保存/放入 KV"
|
||||
" 缓存，则等待 KV 保存完成。`get_finished`：获取完成 KV 传输的请求，如果 `put` 完成则为 `done_sending`，如果"
|
||||
" `get` 完成则为 `done_receiving`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:82
|
||||
msgid "DFX"
|
||||
msgstr "DFX"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:84
|
||||
msgid ""
|
||||
"When looking up a key in KV Pool, if we cannot find the key, there is no "
|
||||
"Cache Hit for this specific block; we return no hit for this block and do "
|
||||
"not look up further blocks for the current request."
|
||||
msgstr "在 KV 池中查找键时，如果找不到该键，则此特定块没有缓存命中；我们返回此块未命中，并且不再为当前请求查找后续块。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:85
|
||||
msgid ""
|
||||
"Similarly, when we are trying to put a block into KV Pool and it fails, we "
|
||||
"do not put further blocks (subject to change)."
|
||||
msgstr "类似地，当尝试将块放入 KV 池但失败时，我们不会继续放入后续块（可能更改）。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:87
|
||||
msgid "Limitations"
|
||||
msgstr "限制"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:89
|
||||
msgid ""
|
||||
"Currently, MooncakeStore for vLLM Ascend only supports DRAM as the storage "
|
||||
"for KV Cache Pool."
|
||||
msgstr "目前，用于vLLM Ascend的MooncakeStore仅支持DRAM作为KV缓存池的存储。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/KV_Cache_Pool_Guide.md:91
|
||||
msgid ""
|
||||
"For now, if we successfully looked up a key and found it exists, but failed "
|
||||
"to get it when calling KV Pool's get function, we just output a log "
|
||||
"indicating the get operation failed and keep going; hence, the accuracy of "
|
||||
"that specific request may be affected. We will handle this situation by "
|
||||
"falling back the request and re-compute everything assuming there's no "
|
||||
"prefix cache hit (or even better, revert only one block and keep using the "
|
||||
"Prefix Caches before that)."
|
||||
msgstr ""
|
||||
"目前，如果我们成功查找键并发现其存在，但在调用 KV 池的 get 函数时获取失败，我们仅输出一条日志指示 get "
|
||||
"操作失败并继续执行；因此，该特定请求的准确性可能会受到影响。我们将通过回退请求并假设没有前缀缓存命中来重新计算所有内容（或者更好的做法是，仅回退一个块并继续使用之前的"
|
||||
" Prefix Cache）来处理这种情况。"
|
||||
@@ -0,0 +1,672 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:1
|
||||
msgid "Prepare inputs for model forwarding"
|
||||
msgstr "准备模型前向传播的输入"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:3
|
||||
msgid "Purpose"
|
||||
msgstr "目的"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:5
|
||||
msgid "Information required to perform model forward pass:"
|
||||
msgstr "执行模型前向传播所需的信息："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:7
|
||||
msgid "the inputs"
|
||||
msgstr "输入"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:8
|
||||
msgid "the corresponding attention metadata of the inputs"
|
||||
msgstr "输入对应的注意力元数据"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:10
|
||||
msgid ""
|
||||
"The following diagram shows what we should prepare for model inference."
|
||||
msgstr "下图展示了模型推理需要准备的内容。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:20
|
||||
msgid ""
|
||||
"Therefore, as long as we have these two pieces of information mentioned "
|
||||
"above, we can perform the model's forward propagation."
|
||||
msgstr "因此，只要拥有上述两部分信息，即可执行模型的前向传播。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:22
|
||||
msgid ""
|
||||
"This document will explain **how we obtain the inputs and their "
|
||||
"corresponding attention metadata**."
|
||||
msgstr "本文档将说明**如何获取输入及其对应的注意力元数据**。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:24
|
||||
msgid "Overview"
|
||||
msgstr "概述"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:26
|
||||
msgid "1. Obtain inputs"
|
||||
msgstr "1.获取输入"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:28
|
||||
msgid "The workflow of obtaining inputs:"
|
||||
msgstr "获取输入的工作流程："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:30
|
||||
msgid ""
|
||||
"Get `token positions`: relative position of each token within its request "
|
||||
"sequence."
|
||||
msgstr "获取`token positions`：每个token在其请求序列中的相对位置。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:32
|
||||
msgid "Get `token indices`: index of each scheduled token in the token table."
|
||||
msgstr "获取`token indices`：每个已调度token在token表中的索引。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:34
|
||||
msgid ""
|
||||
"Get `Token IDs`: using token indices to retrieve the Token IDs from **token "
|
||||
"id table**."
|
||||
msgstr "获取`Token IDs`：使用token索引从**token id table**中检索Token ID。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:36
|
||||
msgid ""
|
||||
"At last, these `Token IDs` are required to be fed into a model, and "
|
||||
"`positions` should also be sent into the model to create `RoPE` (Rotary "
|
||||
"positional embedding). Both of them are the inputs of the model."
|
||||
msgstr "最后，这些`Token IDs`需要输入模型，`positions`也应传入模型以生成`RoPE`（旋转位置编码）。这两者都是模型的输入。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:38
|
||||
msgid ""
|
||||
"**Note**: The `Token IDs` are the inputs of a model, so we also call them "
|
||||
"`Input IDs`."
|
||||
msgstr "**注意**：`Token IDs`是模型的输入，因此我们也称之为`Input IDs`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:40
|
||||
msgid "2. Build inputs attention metadata"
|
||||
msgstr "2.构建输入注意力元数据"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:42
|
||||
msgid "A model requires these attention metadata during the forward pass:"
|
||||
msgstr "模型在前向传播过程中需要以下注意力元数据："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:44
|
||||
msgid ""
|
||||
"`query start location`: start and end location of each request corresponding"
|
||||
" to the scheduled tokens."
|
||||
msgstr "`query start location`：每个请求对应已调度token的起始和结束位置。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:45
|
||||
msgid ""
|
||||
"`sequence length`: length of each request including both computed tokens and"
|
||||
" newly scheduled tokens."
|
||||
msgstr "`sequence length`：每个请求的长度，包括已计算token和新调度token。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:46
|
||||
msgid ""
|
||||
"`number of computed tokens`: number of computed tokens for each request."
|
||||
msgstr "`number of computed tokens`：每个请求已计算的token数量。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:47
|
||||
msgid "`number of requests`: number of requests in this batch."
|
||||
msgstr "`number of requests`：当前批次中的请求数量。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:48
|
||||
msgid "`number of tokens`: total number of scheduled tokens in this batch."
|
||||
msgstr "`number of tokens`：当前批次中已调度的token总数。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:49
|
||||
msgid ""
|
||||
"**`block table`**: translates the logical address (within its sequence) of "
|
||||
"each block to its global physical address in the device's memory."
|
||||
msgstr "**`block table`**：将每个块在其序列中的逻辑地址转换为设备内存中的全局物理地址。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:50
|
||||
msgid ""
|
||||
"`max query len`: the longest scheduled tokens length in this request batch."
|
||||
msgstr "`max query len`：当前请求批次中最长的已调度token长度。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:51
|
||||
msgid ""
|
||||
"`slot mapping`: indices of each token that input token will be stored into."
|
||||
msgstr "`slot mapping`：每个输入token将被存储到的槽位索引。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:52
|
||||
msgid ""
|
||||
"`attention mask`: mask matrix applied to attention scores before softmax to "
|
||||
"control which tokens can attend to each other (usually a causal attention)."
|
||||
msgstr "`attention mask`：在softmax之前应用于注意力分数的掩码矩阵，用于控制哪些token可以相互关注（通常是因果注意力）。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:54
|
||||
msgid "Before start"
|
||||
msgstr "开始之前"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:56
|
||||
msgid "There are mainly three types of variables."
|
||||
msgstr "主要有三种类型的变量。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:58
|
||||
msgid ""
|
||||
"token level: represents one attribute corresponding to each scheduled token,"
|
||||
" so the length of this variable is the number of scheduled tokens."
|
||||
msgstr "token级别：表示每个已调度token对应的一个属性，因此该变量的长度等于已调度token的数量。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:59
|
||||
msgid ""
|
||||
"request level: represents one attribute of each scheduled request, whose "
|
||||
"length usually is the number of scheduled requests. (`query start location` "
|
||||
"is a special case, which has one more element.)"
|
||||
msgstr ""
|
||||
"请求级别：表示每个已调度请求的一个属性，其长度通常等于已调度请求的数量。（`query start location`是特殊情况，多一个元素。）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:60
|
||||
msgid "system level:"
|
||||
msgstr "系统级别："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:61
|
||||
msgid ""
|
||||
"**Token IDs table**: stores the token IDs (i.e. the inputs of a model) of "
|
||||
"each request. The shape of this table is `(max num request, max model len)`."
|
||||
" Here, `max num request` is the maximum count of concurrent requests allowed"
|
||||
" in a forward batch and `max model len` is the maximum token count that can "
|
||||
"be handled at one request sequence in this model."
|
||||
msgstr ""
|
||||
"**Token IDs table**：存储每个请求的token ID（即模型的输入）。该表的形状为`(max num request, max "
|
||||
"model len)`。其中，`max num request`是前向批次中允许的最大并发请求数，`max model "
|
||||
"len`是该模型单个请求序列能处理的最大token数。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:62
|
||||
msgid ""
|
||||
"**Block table**: translates the logical address (within its sequence) of "
|
||||
"each block to its global physical address in the device's memory. The shape "
|
||||
"of this table is `(max num request, max model len / block size)`"
|
||||
msgstr ""
|
||||
"**Block table**：将每个块在其序列中的逻辑地址转换为设备内存中的全局物理地址。该表的形状为`(max num request, max "
|
||||
"model len / block size)`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:64
|
||||
msgid ""
|
||||
"**Note**: Both of these two tables come from the `_update_states` method "
|
||||
"before **preparing inputs**. You can take a look if you need more "
|
||||
"inspiration."
|
||||
msgstr "**注意**：这两个表都来自**准备输入**之前的`_update_states`方法。如需更多启发，可以查看该方法。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:66
|
||||
msgid "Tips"
|
||||
msgstr "提示"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:68
|
||||
msgid ""
|
||||
"Simply put, a `token ID` is an **integer** (usually `int32`), which "
|
||||
"represents a token. Example of `Token ID`:"
|
||||
msgstr "简单来说，`token ID`是一个**整数**（通常为`int32`），代表一个token。`Token ID`示例："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:88
|
||||
msgid "Go through details"
|
||||
msgstr "详细说明"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:90
|
||||
msgid "Assumptions:"
|
||||
msgstr "假设条件："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:92
|
||||
msgid "maximum number of tokens that can be scheduled at once: 10"
|
||||
msgstr "单次可调度的最大token数：10"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:93
|
||||
msgid "`block size`: 2"
|
||||
msgstr "`block size`：2"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:94
|
||||
msgid ""
|
||||
"Totally schedule 3 requests. Their prompt lengths are 3, 2, and 8 "
|
||||
"respectively."
|
||||
msgstr "总共调度3个请求。它们的提示长度分别为3、2和8。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:95
|
||||
msgid ""
|
||||
"`max model length`: 12 (the maximum token count that can be handled at one "
|
||||
"request sequence in a model)."
|
||||
msgstr "`max model length`：12（模型单个请求序列能处理的最大token数）。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:97
|
||||
msgid ""
|
||||
"These assumptions are configured at the beginning when starting vLLM. They "
|
||||
"are not fixed, so you can manually set them."
|
||||
msgstr "这些假设条件在启动vLLM时配置。它们并非固定不变，您可以手动设置。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:99
|
||||
msgid "Step 1: All requests in the prefill phase"
|
||||
msgstr "步骤1：所有请求处于预填充阶段"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:101
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:213
|
||||
msgid "Obtain inputs"
|
||||
msgstr "获取输入"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:103
|
||||
#, python-brace-format
|
||||
msgid ""
|
||||
"As the maximum number of tokens that can be scheduled is 10, the scheduled "
|
||||
"tokens of each request can be represented as `{'0': 3, '1': 2, '2': 5}`. "
|
||||
"Note that `request_2` uses chunked prefill, leaving 3 prompt tokens "
|
||||
"unscheduled."
|
||||
msgstr ""
|
||||
"由于可调度的最大token数为10，每个请求的已调度token可表示为`{'0': 3, '1': 2, '2': "
|
||||
"5}`。注意，`request_2`使用了分块预填充，剩余3个提示token未调度。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:105
|
||||
msgid "1. Get token positions"
|
||||
msgstr "1.获取token positions"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:107
|
||||
msgid ""
|
||||
"First, determine which request each token belongs to: tokens 0–2 are "
|
||||
"assigned to **request_0**, tokens 3–4 to **request_1**, and tokens 5–9 to "
|
||||
"**request_2**. To represent this mapping, we use `request indices`, for "
|
||||
"example, `request indices`: `[0, 0, 0, 1, 1, 2, 2, 2, 2, 2]`."
|
||||
msgstr ""
|
||||
"首先，确定每个token所属的请求：token 0–2分配给**request_0**，token 3–4分配给**request_1**，token "
|
||||
"5–9分配给**request_2**。为了表示这种映射关系，我们使用`request indices`，例如`request "
|
||||
"indices`：`[0, 0, 0, 1, 1, 2, 2, 2, 2, 2]`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:109
|
||||
msgid ""
|
||||
"For each request, use **the number of computed tokens** + **the relative "
|
||||
"position of current scheduled tokens** (`request_0: [0 + 0, 0 + 1, 0 + 2]`, "
|
||||
"`request_1: [0 + 0, 0 + 1]`, `request_2: [0 + 0, 0 + 1,..., 0 + 4]`) and "
|
||||
"then concatenate them together (`[0, 1, 2, 0, 1, 0, 1, 2, 3, 4]`)."
|
||||
msgstr ""
|
||||
"对于每个请求，使用**已计算token数** + **当前已调度token的相对位置**（`request_0: [0 + 0, 0 + 1, 0 + "
|
||||
"2]`，`request_1: [0 + 0, 0 + 1]`，`request_2: [0 + 0, 0 + 1,..., 0 + "
|
||||
"4]`），然后将它们拼接在一起（`[0, 1, 2, 0, 1, 0, 1, 2, 3, 4]`）。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:111
|
||||
msgid ""
|
||||
"Note: there is a more efficient way (using `request indices`) to create "
|
||||
"positions in actual code."
|
||||
msgstr "注意：实际代码中有更高效的方法（使用`request indices`）来创建positions。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:113
|
||||
msgid ""
|
||||
"Finally, `token positions` can be obtained as `[0, 1, 2, 0, 1, 0, 1, 2, 3, "
|
||||
"4]`. This variable is **token level**."
|
||||
msgstr ""
|
||||
"最终，`token positions`为`[0, 1, 2, 0, 1, 0, 1, 2, 3, 4]`。该变量属于**token级别**。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:115
|
||||
msgid "2. Get token indices"
|
||||
msgstr "2.获取token indices"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:117
|
||||
msgid ""
|
||||
"The shape of the current **Token IDs table** is `(max num request, max model"
|
||||
" len)`."
|
||||
msgstr "当前 **Token IDs 表** 的形状为 `(max num request, max model len)`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:119
|
||||
msgid ""
|
||||
"Why are these `T_3_5`, `T_3_6`, `T_3_7` in this table without being "
|
||||
"scheduled?"
|
||||
msgstr "为什么 `T_3_5`、`T_3_6`、`T_3_7` 出现在此表中但未被调度？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:121
|
||||
msgid ""
|
||||
"We fill all Token IDs in one request sequence to this table at once, but we "
|
||||
"only retrieve the tokens we scheduled this time. Then we retrieve the "
|
||||
"remaining Token IDs next time."
|
||||
msgstr "我们将一个请求序列中的所有 Token ID 一次性填入此表，但仅检索本次调度的 token。剩余的 Token ID 将在下次检索。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:133
|
||||
msgid "Note that `T_x_x` is an `int32`."
|
||||
msgstr "注意 `T_x_x` 是 `int32` 类型。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:135
|
||||
msgid ""
|
||||
"Let's say `M = max model len`. Then we can use `token positions` together "
|
||||
"with `request indices` of each token to construct `token indices`."
|
||||
msgstr ""
|
||||
"假设 `M = max model len`。然后我们可以使用每个 token 的 `token positions` 和 `request "
|
||||
"indices` 来构建 `token indices`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:137
|
||||
msgid ""
|
||||
"So `token indices` = `[0 + 0 * M, 1 + 0 * M, 2 + 0 * M, 0 + 1 * M, 1 + 1 * "
|
||||
"M, 0 + 2 * M, 1 + 2 * M, 2 + 2 * M, 3 + 2 * M, 4 + 2 * M]` = `[0, 1, 2, 12, "
|
||||
"13, 24, 25, 26, 27, 28]`"
|
||||
msgstr ""
|
||||
"因此 `token indices` = `[0 + 0 * M, 1 + 0 * M, 2 + 0 * M, 0 + 1 * M, 1 + 1 * "
|
||||
"M, 0 + 2 * M, 1 + 2 * M, 2 + 2 * M, 3 + 2 * M, 4 + 2 * M]` = `[0, 1, 2, 12, "
|
||||
"13, 24, 25, 26, 27, 28]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:139
|
||||
msgid "3. Retrieve the Token IDs"
|
||||
msgstr "3.检索 Token ID"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:141
|
||||
msgid ""
|
||||
"We use `token indices` to select out the corresponding `Input IDs` from the "
|
||||
"token table. The pseudocode is as follows:"
|
||||
msgstr "我们使用 `token indices` 从 token 表中选出对应的 `Input IDs`。伪代码如下："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:147
|
||||
msgid "As mentioned before, we refer to these `Token IDs` as `Input IDs`."
|
||||
msgstr "如前所述，我们将这些 `Token IDs` 称为 `Input IDs`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:149
|
||||
msgid ""
|
||||
"`Input IDs` = `[T_0_0, T_0_1, T_0_2, T_1_0, T_1_1, T_2_0, T_2_1, T_3_2, "
|
||||
"T_3_3, T_3_4]`"
|
||||
msgstr ""
|
||||
"`Input IDs` = `[T_0_0, T_0_1, T_0_2, T_1_0, T_1_1, T_2_0, T_2_1, T_3_2, "
|
||||
"T_3_3, T_3_4]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:151
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:237
|
||||
msgid "Build inputs attention metadata"
|
||||
msgstr "构建输入注意力元数据"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:153
|
||||
msgid ""
|
||||
"In the current **Block Table**, we use the first block (i.e. block_0) to "
|
||||
"mark the unused block. The shape of the block is `(max num request, max "
|
||||
"model len / block size)`, where `max model len / block size = 12 / 2 = 6`."
|
||||
msgstr ""
|
||||
"在当前 **Block Table** 中，我们使用第一个 block（即 block_0）来标记未使用的块。该块的形状为 `(max num "
|
||||
"request, max model len / block size)`，其中 `max model len / block size = 12 / "
|
||||
"2 = 6`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:165
|
||||
msgid "The KV cache block in the device memory is like:"
|
||||
msgstr "设备内存中的 KV 缓存块如下："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:171
|
||||
msgid ""
|
||||
"Let's say `K = max model len / block size = 6`, and we can get token `device"
|
||||
" block number`."
|
||||
msgstr ""
|
||||
"假设 `K = max model len / block size = 6`，我们可以得到 token 的 `device block "
|
||||
"number`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:173
|
||||
msgid "The workflow of achieving slot mapping:"
|
||||
msgstr "实现 slot mapping 的工作流程："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:175
|
||||
msgid ""
|
||||
"Get `block table indices` using `K`, `positions` and `request indices`."
|
||||
msgstr "使用 `K`、`positions` 和 `request indices` 获取 `block table indices`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:177
|
||||
msgid ""
|
||||
"Purpose: For each token, it could be used to select `device block number` "
|
||||
"from `block table`."
|
||||
msgstr "目的：对于每个 token，它可用于从 `block table` 中选择 `device block number`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:179
|
||||
msgid "Get `device block number` using `block table indices`."
|
||||
msgstr "使用 `block table indices` 获取 `device block number`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:181
|
||||
msgid ""
|
||||
"Purpose: `device block number` indicates which device block each token "
|
||||
"belongs to."
|
||||
msgstr "目的：`device block number` 指示每个 token 属于哪个设备块。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:183
|
||||
msgid "Get `block offsets` using `positions` and `block size`."
|
||||
msgstr "使用 `positions` 和 `block size` 获取 `block offsets`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:185
|
||||
msgid ""
|
||||
"Purpose: `block offsets` indicates the offsets of each token within a block."
|
||||
msgstr "目的：`block offsets` 指示每个 token 在块内的偏移量。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:187
|
||||
msgid ""
|
||||
"construct `slot mapping` using `device block number` and `block offsets`."
|
||||
msgstr "使用 `device block number` 和 `block offsets` 构建 `slot mapping`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:189
|
||||
msgid ""
|
||||
"Purpose: we can use `slot mapping` to store Token IDs into token slots."
|
||||
msgstr "目的：我们可以使用 `slot mapping` 将 Token ID 存储到 token 槽中。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:191
|
||||
msgid "Details:"
|
||||
msgstr "详细信息："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:193
|
||||
msgid ""
|
||||
"(**Token level**) Use a simple formula to calculate `block table indices`: "
|
||||
"`request indices * K + positions / block size`. So it equals `[0 * 6 + 0 / "
|
||||
"2, 0 * 6 + 1 / 2, 0 * 6 + 2 / 2, 1 * 6 + 0 / 2, 1 * 6 + 1 / 2, 2 * 6 + 0 / "
|
||||
"2, 2 * 6 + 1 / 2, 2 * 6 + 2 / 2, 2 * 6 + 3 / 2, 2 * 6 + 4 / 2] = [0, 0, 1, "
|
||||
"6, 6, 12, 12, 13, 13, 14]`. This could be used to select `device block "
|
||||
"number` from `block table`."
|
||||
msgstr ""
|
||||
"(**Token 级别**) 使用简单公式计算 `block table indices`：`request indices * K + "
|
||||
"positions / block size`。因此等于 `[0 * 6 + 0 / 2, 0 * 6 + 1 / 2, 0 * 6 + 2 / 2, "
|
||||
"1 * 6 + 0 / 2, 1 * 6 + 1 / 2, 2 * 6 + 0 / 2, 2 * 6 + 1 / 2, 2 * 6 + 2 / 2, 2"
|
||||
" * 6 + 3 / 2, 2 * 6 + 4 / 2] = [0, 0, 1, 6, 6, 12, 12, 13, 13, 14]`。这可用于从 "
|
||||
"`block table` 中选择 `device block number`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:194
|
||||
msgid ""
|
||||
"(**Token level**) Use `block table indices` to select out `device block "
|
||||
"number` for each scheduled token. The pseudocode is `block_numbers = "
|
||||
"block_table[block_table_indices]`. So `device block number=[1, 1, 2, 3, 3, "
|
||||
"4, 4, 5, 5, 6]`"
|
||||
msgstr ""
|
||||
"(**Token 级别**) 使用 `block table indices` 为每个调度的 token 选出 `device block "
|
||||
"number`。伪代码为 `block_numbers = block_table[block_table_indices]`。因此 `device "
|
||||
"block number=[1, 1, 2, 3, 3, 4, 4, 5, 5, 6]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:195
|
||||
msgid ""
|
||||
"(**Token level**) `block offsets` could be computed by `block offsets = "
|
||||
"positions % block size = [0, 1, 0, 0, 1, 0, 1, 0, 1, 0]`."
|
||||
msgstr ""
|
||||
"(**Token 级别**) `block offsets` 可通过 `block offsets = positions % block size ="
|
||||
" [0, 1, 0, 0, 1, 0, 1, 0, 1, 0]` 计算。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:196
|
||||
msgid ""
|
||||
"Finally, use `block offsets` and `device block number` to create `slot "
|
||||
"mapping`: `device block number * block size + block_offsets = [2, 3, 4, 6, "
|
||||
"7, 8, 9, 10, 11, 12]`"
|
||||
msgstr ""
|
||||
"最后，使用 `block offsets` 和 `device block number` 创建 `slot mapping`：`device "
|
||||
"block number * block size + block_offsets = [2, 3, 4, 6, 7, 8, 9, 10, 11, "
|
||||
"12]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:198
|
||||
msgid ""
|
||||
"(**Request level**) As we know the scheduled token count is `[3, 2, 5]`:"
|
||||
msgstr "(**请求级别**) 如我们已知，调度的 token 数量为 `[3, 2, 5]`："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:200
|
||||
msgid ""
|
||||
"(**Request level**) Use prefix sum to calculate `query start location`: `[0,"
|
||||
" 3, 5, 10]`."
|
||||
msgstr "(**请求级别**) 使用前缀和计算 `query start location`：`[0, 3, 5, 10]`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:201
|
||||
msgid ""
|
||||
"(**Request level**) All tokens in step 1 are in the prefill stage, and the "
|
||||
"computed tokens count is 0; then `sequence length` = `[3, 2, 5]`."
|
||||
msgstr ""
|
||||
"(**请求级别**) 步骤 1 中的所有 token 均处于 prefill 阶段，已计算的 token 数量为 0；因此 `sequence "
|
||||
"length` = `[3, 2, 5]`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:202
|
||||
msgid ""
|
||||
"(**Request level**) As mentioned above, `number of computed tokens` are all "
|
||||
"0s: `[0, 0, 0]`."
|
||||
msgstr "(**请求级别**) 如上所述，`number of computed tokens` 均为 0：`[0, 0, 0]`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:203
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:272
|
||||
msgid "`number of requests`: `3`"
|
||||
msgstr "`number of requests`：`3`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:204
|
||||
msgid "(**Request level**) `number of tokens`: `[3, 2, 5]`"
|
||||
msgstr "(**请求级别**) `number of tokens`：`[3, 2, 5]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:205
|
||||
msgid "`max query len`: `5`"
|
||||
msgstr "`max query len`：`5`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:206
|
||||
msgid "(**Token level**) `slot mapping`: `[2, 3, 4, 6, 7, 8, 9, 10, 11, 12]`"
|
||||
msgstr "(**Token 级别**) `slot mapping`：`[2, 3, 4, 6, 7, 8, 9, 10, 11, 12]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:207
|
||||
msgid ""
|
||||
"`attention mask`: For all requests that initiate a prefill process, we "
|
||||
"simply create only one mask matrix for reuse across different requests. The "
|
||||
"shape of this mask matrix is `5 * 5`:"
|
||||
msgstr ""
|
||||
"`attention mask`：对于所有发起 prefill 过程的请求，我们仅创建一个掩码矩阵以便在不同请求间复用。该掩码矩阵的形状为 `5 * "
|
||||
"5`："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:209
|
||||
msgid "Step 2: Chunked prefill"
|
||||
msgstr "步骤 2：分块预填充"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:211
|
||||
msgid ""
|
||||
"In Step 2, we no longer provide explanations or perform calculations; "
|
||||
"instead, we directly present the final result."
|
||||
msgstr "在步骤 2 中，我们不再提供解释或进行计算；而是直接给出最终结果。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:215
|
||||
#, python-brace-format
|
||||
msgid "Scheduled token of each request: `{'0': 1, '1': 1, '2': 3}`"
|
||||
msgstr "每个请求的调度 token：`{'0': 1, '1': 1, '2': 3}`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:217
|
||||
msgid "`request indices`: `[0, 1, 2, 2, 2]`"
|
||||
msgstr "`request indices`：`[0, 1, 2, 2, 2]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:218
|
||||
msgid "`token positions`: `[3, 2, 5, 6, 7]`"
|
||||
msgstr "`token positions`：`[3, 2, 5, 6, 7]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:220
|
||||
msgid "Current **Token IDs table**:"
|
||||
msgstr "当前 **Token IDs 表**："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:232
|
||||
msgid ""
|
||||
"**Note**: **T_0_3**, **T_1_2** are new Token IDs of **request_0** and "
|
||||
"**request_1** respectively. They are sampled from the output of the model."
|
||||
msgstr ""
|
||||
"**注意**：**T_0_3** 和 **T_1_2** 分别是 **request_0** 和 **request_1** 的新 Token "
|
||||
"ID。它们是从模型输出中采样得到的。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:234
|
||||
msgid "`token indices`: `[3, 14, 29, 30, 31]`"
|
||||
msgstr "`token indices`：`[3, 14, 29, 30, 31]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:235
|
||||
msgid "`Input IDs`: `[T_0_3, T_1_2, T_3_5, T_3_6, T_3_7]`"
|
||||
msgstr "`Input IDs`：`[T_0_3, T_1_2, T_3_5, T_3_6, T_3_7]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:239
|
||||
msgid ""
|
||||
"We allocate the blocks `7` and `8` to `request_1` and `request_2` "
|
||||
"respectively, as they need more space in device to store KV cache following "
|
||||
"token generation or chunked prefill."
|
||||
msgstr ""
|
||||
"我们将块 `7` 和 `8` 分别分配给 `request_1` 和 `request_2`，因为它们在 token "
|
||||
"生成或分块预填充后需要在设备上占用更多空间来存储 KV 缓存。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:241
|
||||
msgid "Current **Block Table**:"
|
||||
msgstr "当前 **Block Table**："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:253
|
||||
msgid "KV cache block in the device memory:"
|
||||
msgstr "设备内存中的 KV 缓存块："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:259
|
||||
msgid "(**Token level**) `block table indices`: `[1, 7, 14, 15, 15]`"
|
||||
msgstr "(**Token 级别**) `块表索引`: `[1, 7, 14, 15, 15]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:260
|
||||
msgid "(**Token level**) `device block number`: `[2, 7, 6, 8, 8]`"
|
||||
msgstr "(**Token 级别**) `设备块编号`: `[2, 7, 6, 8, 8]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:261
|
||||
msgid "(**Token level**) `block offsets`: `[1, 0, 1, 0, 1]`"
|
||||
msgstr "(**Token 级别**) `块偏移量`: `[1, 0, 1, 0, 1]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:262
|
||||
msgid "(**Token level**) `slot mapping`: `[5, 14, 13, 16, 17]`"
|
||||
msgstr "(**Token 级别**) `槽位映射`: `[5, 14, 13, 16, 17]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:264
|
||||
msgid "Scheduled token count: `[1, 1, 3]`"
|
||||
msgstr "已调度 token 数量: `[1, 1, 3]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:266
|
||||
msgid "`query start location`: `[0, 1, 2, 5]`"
|
||||
msgstr "`查询起始位置`: `[0, 1, 2, 5]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:268
|
||||
msgid "`sequence length`: `[4, 3, 8]`"
|
||||
msgstr "`序列长度`: `[4, 3, 8]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:270
|
||||
msgid "`number of computed tokens`: `[3, 2, 5]`"
|
||||
msgstr "`已计算 token 数量`: `[3, 2, 5]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:274
|
||||
msgid "`max query len`: `3`"
|
||||
msgstr "`最大查询长度`: `3`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:276
|
||||
msgid "`slot mapping`: `[5, 14, 13, 16, 17]`"
|
||||
msgstr "`槽位映射`: `[5, 14, 13, 16, 17]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:278
|
||||
msgid "`attention mask`: `5 * 8`"
|
||||
msgstr "`注意力掩码`: `5 * 8`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:280
|
||||
msgid "Each token has a `1 * 8` vector, and there are 5 scheduled tokens."
|
||||
msgstr "每个 token 有一个 `1 * 8` 的向量，共有 5 个已调度 token。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:282
|
||||
msgid "At last"
|
||||
msgstr "最后"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:284
|
||||
msgid ""
|
||||
"If you understand step 1 and step 2, you will know all the following steps."
|
||||
msgstr "如果你理解了步骤 1 和步骤 2，你就会明白后续所有步骤。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/ModelRunner_prepare_inputs.md:286
|
||||
msgid ""
|
||||
"Hope this document helps you better understand how vLLM prepares inputs for "
|
||||
"model forwarding. If you have any good ideas, you are welcome to contribute "
|
||||
"to us."
|
||||
msgstr "希望本文档能帮助你更好地理解 vLLM 如何为模型前向传播准备输入。如果你有任何好想法，欢迎向我们贡献。"
|
||||
@@ -0,0 +1,89 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:1
|
||||
msgid "Adding a custom aclnn operation"
|
||||
msgstr "添加自定义aclnn算子"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:3
|
||||
msgid ""
|
||||
"This document describes how to add a custom aclnn operation to vllm-ascend."
|
||||
msgstr "本文档介绍如何向vllm-ascend添加自定义aclnn算子。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:5
|
||||
msgid "How custom aclnn operation works in vllm-ascend?"
|
||||
msgstr "自定义aclnn算子在vllm-ascend中的工作原理？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:7
|
||||
msgid ""
|
||||
"Custom aclnn operations are built and installed into "
|
||||
"`vllm_ascend/cann_ops_custom` directory during the build process of vllm-"
|
||||
"ascend. Then the aclnn operators are bound to `torch.ops._C_ascend` module, "
|
||||
"enabling users to invoke them in vllm-ascend python code."
|
||||
msgstr ""
|
||||
"自定义aclnn算子在vllm-"
|
||||
"ascend构建过程中会被编译并安装到`vllm_ascend/cann_ops_custom`目录。随后这些aclnn算子会被绑定到`torch.ops._C_ascend`模块，使用户能够在vllm-"
|
||||
"ascend的Python代码中调用它们。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:9
|
||||
msgid "To enable custom operations, use the following code:"
|
||||
msgstr "要启用自定义算子，请使用以下代码："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:17
|
||||
msgid "How to add a custom aclnn operation?"
|
||||
msgstr "如何添加自定义aclnn算子？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:19
|
||||
msgid "Create a new operation folder under `csrc` directory."
|
||||
msgstr "在`csrc`目录下创建新的算子文件夹。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:20
|
||||
msgid ""
|
||||
"Create `op_host` and `op_kernel` directories for host and kernel source "
|
||||
"code."
|
||||
msgstr "创建`op_host`和`op_kernel`目录，分别存放主机端和内核端源代码。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:21
|
||||
msgid ""
|
||||
"Add build options in `csrc/build_aclnn.sh` for supported SOC. Note that "
|
||||
"multiple ops should be separated with `;`, i.e. "
|
||||
"`CUSTOM_OPS=\"op1;op2;op3\"`."
|
||||
msgstr ""
|
||||
"在`csrc/build_aclnn.sh`中为支持的SOC添加构建选项。注意多个算子之间需用`;`分隔，例如`CUSTOM_OPS=\"op1;op2;op3\"`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:22
|
||||
msgid ""
|
||||
"Bind aclnn operators to torch.ops._C_ascend module in "
|
||||
"`csrc/torch_binding.cpp`."
|
||||
msgstr "在`csrc/torch_binding.cpp`中将aclnn算子绑定到torch.ops._C_ascend模块。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:23
|
||||
msgid ""
|
||||
"Write a meta implementation in `csrc/torch_binding_meta.cpp` for the op to "
|
||||
"be captured into the aclgraph."
|
||||
msgstr "在`csrc/torch_binding_meta.cpp`中为算子编写元实现，使其能够被捕获到aclgraph中。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/add_custom_aclnn_op.md:25
|
||||
msgid ""
|
||||
"After a successful build of vllm-ascend, the custom aclnn operation can be "
|
||||
"invoked in python code."
|
||||
msgstr "成功构建vllm-ascend后，即可在Python代码中调用自定义aclnn算子。"
|
||||
@@ -0,0 +1,585 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:1
|
||||
msgid "Context Parallel (CP)"
|
||||
msgstr "上下文并行（CP）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:3
|
||||
msgid ""
|
||||
"TL;DR: PCP accelerates prefill via sequence splitting. DCP eliminates KV "
|
||||
"cache redundancy."
|
||||
msgstr "摘要：PCP 通过序列拆分加速预填充，DCP 消除 KV 缓存冗余。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:5
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:5
|
||||
msgid "ContextParallel"
|
||||
msgstr "上下文并行"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:7
|
||||
msgid ""
|
||||
"For the main discussions during the development process, please refer to the"
|
||||
" [RFC](https://github.com/vllm-project/vllm/issues/25749) and the relevant "
|
||||
"links referenced by or referencing this RFC."
|
||||
msgstr ""
|
||||
"开发过程中的主要讨论请参考 [RFC](https://github.com/vllm-project/vllm/issues/25749) 以及该 "
|
||||
"RFC 引用或引用的相关链接。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:9
|
||||
msgid "What is CP?"
|
||||
msgstr "什么是 CP？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:11
|
||||
msgid ""
|
||||
"**Context Parallel (CP)** is a strategy for parallelizing computation along "
|
||||
"the sequence dimension across multiple devices."
|
||||
msgstr "**上下文并行（CP）** 是一种沿序列维度在多个设备间并行化计算的策略。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:13
|
||||
msgid ""
|
||||
"**Prefill Context Parallel (PCP)** expands the world size of devices and "
|
||||
"uses dedicated communication domains. Its primary goal is to partition the "
|
||||
"sequence dimension during the prefill phase, enabling different devices to "
|
||||
"compute distinct chunks of the sequence simultaneously. The KV cache is "
|
||||
"sharded along the sequence dimension across devices. This approach impacts "
|
||||
"the computational logic of both the Prefill and Decode stages to varying "
|
||||
"degrees."
|
||||
msgstr ""
|
||||
"**预填充上下文并行（PCP）** 扩展了设备的 world size "
|
||||
"并使用专用通信域。其主要目标是在预填充阶段拆分序列维度，使不同设备能同时计算序列的不同块。KV "
|
||||
"缓存沿序列维度分片存储于各设备。该方法在不同程度上影响预填充和解码阶段的计算逻辑。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:18
|
||||
msgid ""
|
||||
"**Decode Context Parallel (DCP)** reuses the communication domain of Tensor "
|
||||
"Parallelism (TP) and does not require additional devices. Its main objective"
|
||||
" is to eliminate duplicated storage of the KV cache by sharding it along the"
|
||||
" sequence dimension across devices within the TP domain that would otherwise"
|
||||
" hold redundant copies. DCP primarily influences the Decode logic, as well "
|
||||
"as the logic for chunked prefill and cached prefill."
|
||||
msgstr ""
|
||||
"**解码上下文并行（DCP）** 复用张量并行（TP）的通信域，无需额外设备。其主要目标是通过沿序列维度将 KV 缓存分片存储到 TP "
|
||||
"域内原本持有冗余副本的设备上，消除 KV 缓存的重复存储。DCP 主要影响解码逻辑，以及分块预填充和缓存预填充的逻辑。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:22
|
||||
msgid "How to Use CP?"
|
||||
msgstr "如何使用 CP？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:24
|
||||
msgid ""
|
||||
"Please refer to the [context parallel user "
|
||||
"guide](../../user_guide/feature_guide/context_parallel.md) for detailed "
|
||||
"information."
|
||||
msgstr ""
|
||||
"详细信息请参考[上下文并行用户指南](../../user_guide/feature_guide/context_parallel.md)。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:26
|
||||
msgid "How It Works?"
|
||||
msgstr "工作原理"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:28
|
||||
msgid "Device Distribution"
|
||||
msgstr "设备分布"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:30
|
||||
msgid ""
|
||||
"We introduce new communication domains for PCP and reuse TP for DCP, and "
|
||||
"this is the new layout of devices for PCP2, DCP2, and TP4. "
|
||||
""
|
||||
msgstr ""
|
||||
"我们为 PCP 引入新的通信域，并为 DCP 复用 TP，以下是 PCP2、DCP2 和 TP4 "
|
||||
"的新设备布局。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:30
|
||||
msgid "device_world"
|
||||
msgstr "设备世界"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:33
|
||||
msgid "Block Table"
|
||||
msgstr "块表"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:35
|
||||
msgid ""
|
||||
"CP performs sequence sharding on the KV cache storage. To facilitate "
|
||||
"efficient storage and access, tokens are stored in an interleaved manner "
|
||||
"across devices, with the interleaving granularity determined by "
|
||||
"`cp_kv_cache_interleave_size`, whose default value is "
|
||||
"`cp_kv_cache_interleave_size=1`, a.k.a. 'token interleave'."
|
||||
msgstr ""
|
||||
"CP 对 KV 缓存存储进行序列分片。为便于高效存储和访问，令牌以交错方式存储于各设备，交错粒度由 "
|
||||
"`cp_kv_cache_interleave_size` 决定，其默认值为 "
|
||||
"`cp_kv_cache_interleave_size=1`，即“令牌交错”。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:37
|
||||
msgid ""
|
||||
"Given that PCP and DCP behave similarly for KV cache sharding, we refer to "
|
||||
"them collectively as CP. Specifically, `cp_size = pcp_size * dcp_size`, and "
|
||||
"`cp_rank = pcp_rank * dcp_size + dcp_rank`."
|
||||
msgstr ""
|
||||
"鉴于 PCP 和 DCP 在 KV 缓存分片方面行为相似，我们统称为 CP。具体而言，`cp_size = pcp_size * dcp_size`，且"
|
||||
" `cp_rank = pcp_rank * dcp_size + dcp_rank`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:39
|
||||
msgid ""
|
||||
"As illustrated, a virtual block is defined in the block table, where blocks "
|
||||
"within the same CP device group form a virtual block. The virtual block size"
|
||||
" is `virtual_block_size = block_size * cp_size`."
|
||||
msgstr ""
|
||||
"如图所示，块表中定义了一个虚拟块，同一 CP 设备组内的块构成一个虚拟块。虚拟块大小为 `virtual_block_size = block_size"
|
||||
" * cp_size`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:41
|
||||
#, python-format
|
||||
msgid ""
|
||||
"For any token `x`, referencing the following figure, its (virtual) block "
|
||||
"index is `x // virtual_block_size`, and the offset within the virtual block "
|
||||
"is `offset_within_virtual_block = x % virtual_block_size`. The local block "
|
||||
"index is `local_block_index = offset_within_virtual_block // "
|
||||
"cp_kv_cache_interleave_size`, and the device number is `target_rank = "
|
||||
"local_block_index % cp_size`. The offset within the local block is "
|
||||
"`(local_block_index // cp_size) * cp_kv_cache_interleave_size + "
|
||||
"offset_within_virtual_block % cp_kv_cache_interleave_size`."
|
||||
msgstr ""
|
||||
"对于任意令牌 `x`，参考下图，其（虚拟）块索引为 `x // virtual_block_size`，虚拟块内偏移为 "
|
||||
"`offset_within_virtual_block = x % virtual_block_size`。本地块索引为 "
|
||||
"`local_block_index = offset_within_virtual_block // "
|
||||
"cp_kv_cache_interleave_size`，设备编号为 `target_rank = local_block_index % "
|
||||
"cp_size`。本地块内偏移为 `(local_block_index // cp_size) * "
|
||||
"cp_kv_cache_interleave_size + offset_within_virtual_block % "
|
||||
"cp_kv_cache_interleave_size`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:45
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:45
|
||||
msgid "BlockTable"
|
||||
msgstr "块表"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:47
|
||||
msgid ""
|
||||
"Based on the logic above, the `slot_mapping` calculation process is "
|
||||
"adjusted, and the `slot_mapping` values on each device are modified to "
|
||||
"ensure the KV cache is sharded along the sequence dimension and stored "
|
||||
"across different devices as expected."
|
||||
msgstr ""
|
||||
"基于上述逻辑，调整了 `slot_mapping` 计算过程，并修改了各设备上的 `slot_mapping` 值，以确保 KV "
|
||||
"缓存按预期沿序列维度分片并存储于不同设备。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:49
|
||||
#, python-format
|
||||
msgid ""
|
||||
"The current implementation requires that `block_size % "
|
||||
"cp_kv_cache_interleave_size == 0`."
|
||||
msgstr "当前实现要求 `block_size % cp_kv_cache_interleave_size == 0`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:51
|
||||
msgid "Decode Context Parallel (DCP)"
|
||||
msgstr "解码上下文并行（DCP）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:53
|
||||
msgid ""
|
||||
"As mentioned above, the primary function of DCP is to shard the KV cache "
|
||||
"along the sequence dimension for storage. Its impact lies in the logic of "
|
||||
"the decode and chunked prefill phases."
|
||||
msgstr "如上所述，DCP 的主要功能是沿序列维度对 KV 缓存进行分片存储。其影响体现在解码和分块预填充阶段的逻辑中。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:55
|
||||
msgid ""
|
||||
"**Prefill Phase:** As illustrated, during the Chunked Prefill computation,"
|
||||
" two distinct logic implementations are employed for MLA and GQA backends."
|
||||
msgstr "**预填充阶段：** 如图所示，在分块预填充计算中，MLA 和 GQA 后端采用了两种不同的逻辑实现。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:58
|
||||
msgid ""
|
||||
"In the **MLA backend**, a Context KV Cache `all_gather` operation is "
|
||||
"performed to aggregate the full KV values. These are then used for attention"
|
||||
" computation with the Q values of the current chunk. Note that in multi-"
|
||||
"request scenarios, the directly gathered KV results are interleaved across "
|
||||
"requests. The `reorg_kvcache` function is used to reorganize the KV cache, "
|
||||
"ensuring that the KV cache of the same request is stored contiguously."
|
||||
msgstr ""
|
||||
"在 **MLA 后端**中，执行上下文 KV 缓存的 `all_gather` 操作以聚合完整的 KV 值，然后用于与当前块的 Q "
|
||||
"值进行注意力计算。注意，在多请求场景下，直接收集的 KV 结果在请求间是交错的。使用 `reorg_kvcache` 函数重新组织 KV "
|
||||
"缓存，确保同一请求的 KV 缓存连续存储。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:63
|
||||
msgid ""
|
||||
"In the **GQA backend**, an `all_gather` is performed along the head "
|
||||
"dimension for Q. This is because DCP overlaps with the TP communication "
|
||||
"domain, and the Q heads within a DCP group differ. However, they need to "
|
||||
"exchange results with the locally computed KV cache for online Softmax "
|
||||
"updates. To ensure correctness during result updates, the Q values are "
|
||||
"synchronized across the DCP group via head-dimension `all_gather`. During "
|
||||
"the result update process, `cp_lse_ag_out_rs` is invoked to aggregate "
|
||||
"`attn_output` and `attn_lse`, update the results, and perform a reduce-"
|
||||
"scatter operation on the outputs. Alternatively, we can use an all-to-all "
|
||||
"communication to exchange the output and LSE results, followed by direct "
|
||||
"local updates. This approach aligns with the logic adapted for PCP "
|
||||
"compatibility."
|
||||
msgstr ""
|
||||
"在 **GQA 后端**中，对 Q 沿头维度执行 `all_gather`。这是因为 DCP 与 TP 通信域重叠，DCP 组内的 Q "
|
||||
"头不同，但它们需要与本地计算的 KV 缓存交换结果以进行在线 Softmax 更新。为确保结果更新时的正确性，通过头维度的 `all_gather` 在"
|
||||
" DCP 组内同步 Q 值。在结果更新过程中，调用 `cp_lse_ag_out_rs` 聚合 `attn_output` 和 "
|
||||
"`attn_lse`，更新结果，并对输出执行 reduce-scatter 操作。或者，我们可以使用 all-to-all 通信交换输出和 LSE "
|
||||
"结果，然后直接进行本地更新。此方法与为兼容 PCP 而调整的逻辑一致。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:70
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:70
|
||||
msgid "DCP-Prefill"
|
||||
msgstr "DCP-预填充"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:72
|
||||
msgid ""
|
||||
"**Decode Phase:** The logic during the decode phase is consistent with that "
|
||||
"of GQA's chunked prefill: an all-gather operation is first performed along "
|
||||
"the Q head dimension to ensure consistency within the DCP group. After "
|
||||
"computing the results with the local KV cache, the results are updated via "
|
||||
"the `cp_lse_ag_out_rs` function."
|
||||
msgstr ""
|
||||
"**解码阶段：** 解码阶段的逻辑与 GQA 的分块预填充一致：首先沿 Q 头维度执行 all-gather 操作，以确保 DCP "
|
||||
"组内的一致性。使用本地 KV 缓存计算结果后，通过 `cp_lse_ag_out_rs` 函数更新结果。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:76
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:76
|
||||
msgid "DCP-Decode"
|
||||
msgstr "DCP-Decode"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:78
|
||||
msgid "GLM-5.2 SFA DCP Replicated Indexer"
|
||||
msgstr "GLM-5.2 SFA DCP 复制索引器"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:80
|
||||
msgid ""
|
||||
"GLM-5.2 uses Sparse Flash Attention (SFA) with a LightningIndexer. For DCP,"
|
||||
" the indexer needs a full-sequence view to select the same sparse top-k "
|
||||
"blocks as non-DCP SFA, while the much larger SFA KV cache should remain "
|
||||
"sharded to retain DCP's memory benefit. The replicated-indexer path "
|
||||
"provides this split layout:"
|
||||
msgstr ""
|
||||
"GLM-5.2 使用带有 LightningIndexer 的稀疏闪存注意力（SFA）。对于 DCP，索引器需要完整的序列视图，以选择与非 DCP "
|
||||
"SFA 相同的稀疏 top-k 块，而更大的 SFA KV 缓存应保持分片，以保留 DCP 的内存优势。复制索引器路径提供了这种拆分布局："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:86
|
||||
msgid ""
|
||||
"The LightningIndexer cache is replicated on every DCP rank. Index selection"
|
||||
" therefore uses the complete sequence and produces globally consistent "
|
||||
"sparse top-k indices."
|
||||
msgstr "LightningIndexer 缓存在每个 DCP 排名上复制。因此，索引选择使用完整的序列，并生成全局一致的稀疏 top-k 索引。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:89
|
||||
msgid ""
|
||||
"The SFA KV cache remains DCP-local. The global indices from the replicated "
|
||||
"indexer view are remapped to local KV indices before SFA runs."
|
||||
msgstr "SFA KV 缓存保持 DCP 本地。在 SFA 运行之前，来自复制索引器视图的全局索引被重新映射到本地 KV 索引。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:91
|
||||
msgid ""
|
||||
"During prefill or a mixed batch, only the KV blocks referenced by the sparse"
|
||||
" block table are compacted and all-gathered after the current layer has "
|
||||
"written its KV cache. The gathered KV uses a remapped block table for SFA, "
|
||||
"so this path does not all-gather Q and does not need LSE or output post-"
|
||||
"processing."
|
||||
msgstr ""
|
||||
"在预填充或混合批次期间，只有稀疏块表引用的 KV 块在当前层写入其 KV 缓存后被压缩并 all-gather。收集的 KV 使用重新映射的块表进行 "
|
||||
"SFA，因此此路径不会 all-gather Q，也不需要 LSE 或输出后处理。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:96
|
||||
msgid "Decode-only batches retain the DCP SFA Q-gather and result-merge path."
|
||||
msgstr "仅解码批次保留 DCP SFA Q-gather 和结果合并路径。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:98
|
||||
msgid ""
|
||||
"This mode is selected automatically for SFA sparse models when "
|
||||
"`prefill_context_parallel_size=1` and `decode_context_parallel_size>1`. It "
|
||||
"requires `decode_context_parallel_size == tensor_parallel_size`; PCP "
|
||||
"combined with this replicated-indexer path is not supported."
|
||||
msgstr ""
|
||||
"当 `prefill_context_parallel_size=1` 且 `decode_context_parallel_size>1` "
|
||||
"时，此模式会自动为 SFA 稀疏模型选择。它要求 `decode_context_parallel_size == "
|
||||
"tensor_parallel_size`；不支持 PCP 与此复制索引器路径结合使用。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:103
|
||||
msgid ""
|
||||
"For a GLM-5.2 DSA-CP deployment, enable FlashComm1 and DSA CP and keep the "
|
||||
"CP interleave size equal to the KV-cache block size:"
|
||||
msgstr "对于 GLM-5.2 DSA-CP 部署，启用 FlashComm1 和 DSA CP，并保持 CP 交错大小等于 KV 缓存块大小："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:118
|
||||
msgid ""
|
||||
"The replicated indexer increases indexer-cache memory in proportion to the "
|
||||
"DCP world size; the SFA KV cache itself remains sharded. For this SFA CP "
|
||||
"path, `cp_kv_cache_interleave_size` must equal `block_size`. A mismatched "
|
||||
"setting is overridden during configuration validation, but deployments "
|
||||
"should set both values explicitly to avoid relying on that fallback."
|
||||
msgstr ""
|
||||
"复制索引器会按 DCP 世界大小比例增加索引器缓存内存；SFA KV 缓存本身保持分片。对于此 SFA CP "
|
||||
"路径，`cp_kv_cache_interleave_size` 必须等于 "
|
||||
"`block_size`。配置验证期间会覆盖不匹配的设置，但部署应显式设置这两个值，以避免依赖该回退。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:124
|
||||
msgid "Prefill Context Parallel (PCP)"
|
||||
msgstr "预填充上下文并行（PCP）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:126
|
||||
msgid "**Tokens Partition in Head-Tail Style**"
|
||||
msgstr "**令牌的头尾风格分区**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:128
|
||||
msgid ""
|
||||
"PCP requires splitting the input sequence and ensuring balanced "
|
||||
"computational load across devices during the prefill phase. We employ a "
|
||||
"head-tail style for splitting and concatenation: specifically, the sequence "
|
||||
"is first padded to a length of `2*pcp_size`, then divided into `2*pcp_size` "
|
||||
"equal parts. The first part is merged with the last part, the second part "
|
||||
"with the second last part, and so on, thereby assigning computationally "
|
||||
"balanced chunks to each device. Additionally, since allgather aggregation of"
|
||||
" KV or Q results in interleaved chunks from different requests, we compute "
|
||||
"`pcp_allgather_restore_idx` to quickly restore the original order."
|
||||
msgstr ""
|
||||
"PCP 需要拆分输入序列，并确保预填充阶段各设备间的计算负载平衡。我们采用头尾风格进行拆分和拼接：具体来说，序列首先被填充到 `2*pcp_size` "
|
||||
"的长度，然后分成 `2*pcp_size` "
|
||||
"个相等的部分。第一部分与最后一部分合并，第二部分与倒数第二部分合并，依此类推，从而为每个设备分配计算平衡的块。此外，由于 KV 或 Q 的 "
|
||||
"allgather 聚合会导致来自不同请求的块交错，我们计算 `pcp_allgather_restore_idx` 以快速恢复原始顺序。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:133
|
||||
msgid "These logics are implemented in the function `_update_tokens_for_pcp`."
|
||||
msgstr "这些逻辑在函数 `_update_tokens_for_pcp` 中实现。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:135
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:135
|
||||
msgid "PCP-Partition"
|
||||
msgstr "PCP-Partition"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:137
|
||||
msgid "**Prefill Phase:**"
|
||||
msgstr "**预填充阶段：**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:139
|
||||
msgid ""
|
||||
"During the Prefill phase (excluding chunked prefill), we employ an all-"
|
||||
"gather KV approach to address the issue of incomplete sequences on "
|
||||
"individual GPUs. It is important to note that we only aggregate the KV "
|
||||
"values for the current layer at a time, and these are discarded immediately "
|
||||
"after use, avoiding excessive peak memory usage. This method can also be "
|
||||
"directly applied to KV cache storage (since the KV cache partitioning method"
|
||||
" differs from PCP sequence partitioning, it is inevitable that each GPU "
|
||||
"requires a complete copy of the KV values). All attention backends maintain "
|
||||
"consistency in this logic."
|
||||
msgstr ""
|
||||
"在预填充阶段（不包括分块预填充），我们采用 all-gather KV 方法来解决单个 GPU 上序列不完整的问题。需要注意的是，我们每次只聚合当前层的"
|
||||
" KV 值，这些值在使用后立即丢弃，避免了过高的峰值内存使用。此方法也可直接应用于 KV 缓存存储（由于 KV 缓存分区方法与 PCP "
|
||||
"序列分区方法不同，每个 GPU 不可避免地需要完整的 KV 值副本）。所有注意力后端在此逻辑上保持一致。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:144
|
||||
msgid ""
|
||||
"Note: While a Ring Attention approach could also facilitate information "
|
||||
"exchange with lower peak memory and enable computation-communication "
|
||||
"overlap, we prioritized the all-gather KV implementation after evaluating "
|
||||
"that the development complexity was high and the benefits of overlap were "
|
||||
"limited."
|
||||
msgstr ""
|
||||
"注意：虽然环形注意力方法也可以以更低的峰值内存促进信息交换并实现计算-通信重叠，但在评估了开发复杂度高且重叠收益有限后，我们优先考虑了 all-"
|
||||
"gather KV 实现。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:146
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:146
|
||||
msgid "PCP-Prefill"
|
||||
msgstr "PCP-Prefill"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:148
|
||||
msgid "**Decode Phase:**"
|
||||
msgstr "**解码阶段：**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:150
|
||||
msgid ""
|
||||
"During the decode phase, we only need to add an allgather within the PCP "
|
||||
"group after the DCP all-to-all communication exchanges the output and LSE, "
|
||||
"before proceeding with the output update."
|
||||
msgstr ""
|
||||
"在解码阶段，我们只需要在 DCP all-to-all 通信交换输出和 LSE 之后，在 PCP 组内添加一个 allgather，然后再进行输出更新。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:152
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:152
|
||||
msgid "PCP-Decode"
|
||||
msgstr "PCP-Decode"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:154
|
||||
msgid "**Chunked Prefill:**"
|
||||
msgstr "**分块预填充：**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:156
|
||||
msgid ""
|
||||
"Currently, there are three viable approaches for Chunked Prefill "
|
||||
"compatibility: **AllGatherQ**, **AllGatherKV**, and **Ring-Attn**. Since PCP"
|
||||
" performs sequence sharding on both the query sequence and the KV cache, we "
|
||||
"need to ensure that one side has complete information or employ a method "
|
||||
"like Ring-Attn to perform computations sequentially. The advantages and "
|
||||
"disadvantages of Ring-Attn will not be elaborated here."
|
||||
msgstr ""
|
||||
"目前，有三种可行的方法用于分块预填充兼容性：**AllGatherQ**、**AllGatherKV** 和 **Ring-Attn**。由于 PCP "
|
||||
"对查询序列和 KV 缓存都执行序列分片，我们需要确保一侧拥有完整信息，或采用像 Ring-Attn 这样的方法顺序执行计算。Ring-Attn "
|
||||
"的优缺点在此不再赘述。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:160
|
||||
msgid ""
|
||||
"We have implemented the **AllGatherQ** approach in the GQA attention backend"
|
||||
" and the **AllGatherKV** approach in the MLA attention backend. The workflow"
|
||||
" after **AllGatherQ** is identical to the decode phase, while the workflow "
|
||||
"after **AllGatherKV** is the same as the standard prefill phase. For "
|
||||
"details, please refer to the diagram below; specific steps will not be "
|
||||
"repeated."
|
||||
msgstr ""
|
||||
"我们在 GQA 注意力后端实现了 **AllGatherQ** 方法，在 MLA 注意力后端实现了 **AllGatherKV** "
|
||||
"方法。**AllGatherQ** 之后的工作流程与解码阶段相同，而 **AllGatherKV** "
|
||||
"之后的工作流程与标准预填充阶段相同。详情请参考下图，具体步骤不再重复。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:164
|
||||
msgid ""
|
||||
"One important note: **AllGatherKV** may lead to significant peak memory "
|
||||
"usage when the context length becomes excessively long. To mitigate this, we"
|
||||
" adopt a segmented processing strategy. By predefining the maximum amount of"
|
||||
" KV cache processed per round, we sequentially complete the attention "
|
||||
"computation and online softmax updates for each segment."
|
||||
msgstr ""
|
||||
"一个重要提示：**AllGatherKV** 在上下文长度过长时可能导致显著的峰值内存使用。为缓解此问题，我们采用分段处理策略。通过预定义每轮处理的 "
|
||||
"KV 缓存最大量，我们依次完成每个分段的注意力计算和在线 softmax 更新。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:168
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:168
|
||||
msgid "PCP-ChunkedPrefill"
|
||||
msgstr "PCP-分块预填充"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:170
|
||||
msgid "SFA DSA-CP Mixed `o_proj` Path"
|
||||
msgstr "SFA DSA-CP 混合 `o_proj` 路径"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:172
|
||||
msgid ""
|
||||
"SFA DSA-CP mixed execution intentionally reuses the normal TP-sharded "
|
||||
"`o_proj`. This is part of the DSA-CP mixed data path, not a standalone user-"
|
||||
"facing `o_proj` TP switch. The mixed path is used when one instance may "
|
||||
"handle both decode-only and prefill/mixed batches, so `o_proj` must support "
|
||||
"two layouts at runtime:"
|
||||
msgstr ""
|
||||
"SFA DSA-CP 混合执行有意复用了正常的 TP 分片 `o_proj`。这是 DSA-CP 混合数据路径的一部分，而非独立的面向用户的 "
|
||||
"`o_proj` TP 开关。当单个实例可能同时处理仅解码和预填充/混合批次时，使用混合路径，因此 `o_proj` 必须在运行时支持两种布局："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:176
|
||||
msgid ""
|
||||
"**Decode-only batches** keep the decode TP path. SFA outputs are exchanged "
|
||||
"with an all-to-all in the TP group, then the original TP-sharded `o_proj` "
|
||||
"runs normally."
|
||||
msgstr ""
|
||||
"**仅解码批次** 保持解码 TP 路径。SFA 输出通过 TP 组内的 all-to-all 交换，然后原始的 TP 分片 `o_proj` "
|
||||
"正常运行。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:178
|
||||
msgid ""
|
||||
"**Prefill or mixed batches** produce SFA outputs that are not directly "
|
||||
"compatible with the TP-sharded `o_proj` input layout. Before `o_proj` "
|
||||
"forward, each rank all-gathers the TP-sharded `o_proj` weight and all input-"
|
||||
"sharded quantization parameters into temporary full-weight buffers. The "
|
||||
"full-weight `o_proj` forward runs once for that batch, and the module is "
|
||||
"then restored to the TP parameter aliases."
|
||||
msgstr ""
|
||||
"**预填充或混合批次** 产生的 SFA 输出与 TP 分片 `o_proj` 的输入布局不直接兼容。在 `o_proj` 前向传播前，每个 rank "
|
||||
"将 TP 分片的 `o_proj` 权重和所有输入分片的量化参数 all-gather 到临时全权重缓冲区。全权重 `o_proj` "
|
||||
"前向传播对该批次运行一次，然后模块恢复为 TP 参数别名。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:182
|
||||
msgid ""
|
||||
"The storage invariant is that the original TP-sharded `o_proj` parameter "
|
||||
"remains the only persistent source of truth. `o_proj_tp_*` tensors are "
|
||||
"aliases of the original parameter storage. `o_proj_full_*` tensors are "
|
||||
"reusable communication buffers for prefill/mixed full-gather execution only."
|
||||
" They must not become a second persistent copy of the TP weight."
|
||||
msgstr ""
|
||||
"存储不变性是：原始的 TP 分片 `o_proj` 参数仍然是唯一持久化的真实来源。`o_proj_tp_*` "
|
||||
"张量是原始参数存储的别名。`o_proj_full_*` 张量是仅用于预填充/混合全收集执行的可重用通信缓冲区。它们不得成为 TP "
|
||||
"权重的第二个持久化副本。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:187
|
||||
msgid ""
|
||||
"This coupling preserves the existing decode TP behavior, supports "
|
||||
"prefill/mixed DSA-CP batches, and avoids adding an extra configuration path "
|
||||
"whose state can drift from DSA-CP mixed execution."
|
||||
msgstr ""
|
||||
"这种耦合保留了现有的解码 TP 行为，支持预填充/混合 DSA-CP 批次，并避免了添加额外的配置路径，该路径的状态可能与 DSA-CP "
|
||||
"混合执行产生偏差。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:189
|
||||
msgid "Related Files"
|
||||
msgstr "相关文件"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:191
|
||||
msgid "slot_mapping computation: `vllm_ascend/worker/block_table.py`"
|
||||
msgstr "slot_mapping 计算：`vllm_ascend/worker/block_table.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:192
|
||||
msgid ""
|
||||
"sequences splitting and metadata prepare: "
|
||||
"`vllm_ascend/worker/model_runner_v1.py`"
|
||||
msgstr "序列拆分与元数据准备：`vllm_ascend/worker/model_runner_v1.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:193
|
||||
msgid ""
|
||||
"PCP token splitting and metadata generation: "
|
||||
"`vllm_ascend/worker/pcp_utils.py`"
|
||||
msgstr "PCP token 拆分与元数据生成：`vllm_ascend/worker/pcp_utils.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:194
|
||||
msgid "GQA backend: `vllm_ascend/attention/context_parallel/attention_cp.py`"
|
||||
msgstr "GQA 后端：`vllm_ascend/attention/context_parallel/attention_cp.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:195
|
||||
msgid "MLA backend: `vllm_ascend/attention/context_parallel/mla_cp.py`"
|
||||
msgstr "MLA 后端：`vllm_ascend/attention/context_parallel/mla_cp.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:196
|
||||
msgid "DSA backend: `vllm_ascend/attention/context_parallel/dsa_cp.py`"
|
||||
msgstr "DSA 后端：`vllm_ascend/attention/context_parallel/dsa_cp.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/context_parallel.md:197
|
||||
msgid "SFA backend: `vllm_ascend/attention/context_parallel/sfa_cp.py`"
|
||||
msgstr "SFA 后端：`vllm_ascend/attention/context_parallel/sfa_cp.py`"
|
||||
@@ -0,0 +1,924 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:1
|
||||
msgid "CPU Binding"
|
||||
msgstr "CPU 绑定"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:3
|
||||
msgid "Overview"
|
||||
msgstr "概述"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:5
|
||||
msgid ""
|
||||
"CPU binding is an **Ascend-native host-side optimization** for vLLM workers "
|
||||
"on ARM servers. **Starting from vllm-ascend v0.18.0rc1, it is enabled by "
|
||||
"default through `enable_cpu_binding=True`.**"
|
||||
msgstr ""
|
||||
"CPU 绑定是针对 ARM 服务器上 vLLM 工作进程的 **Ascend 原生主机端优化**。**从 vllm-ascend v0.18.0rc1 "
|
||||
"开始，该功能通过 `enable_cpu_binding=True` 默认启用。**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:9
|
||||
msgid ""
|
||||
"The feature does not change model execution logic or numerical results. It "
|
||||
"only controls CPU placement for the worker process, key runtime threads, "
|
||||
"memory pages, and NPU IRQs when the host environment allows it. By keeping "
|
||||
"the main worker, ACL, and release threads on dedicated CPU ranges, it "
|
||||
"**helps reduce context-switch overhead from scheduler preemption on busy "
|
||||
"hosts.**"
|
||||
msgstr ""
|
||||
"该功能不会改变模型执行逻辑或数值结果。它仅在主机环境允许的情况下，控制工作进程、关键运行时线程、内存页和 NPU IRQ 的 CPU "
|
||||
"放置。通过将主工作线程、ACL 和释放线程保持在专用的 CPU 范围内，**有助于减少繁忙主机上因调度器抢占而产生的上下文切换开销。**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:15
|
||||
msgid "Why CPU Binding?"
|
||||
msgstr "为什么需要 CPU 绑定？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:17
|
||||
msgid ""
|
||||
"On multi-socket ARM systems, the Linux scheduler may place worker threads on"
|
||||
" CPUs far from the NPU that the worker drives. This can increase cross-NUMA "
|
||||
"traffic, increase thread preemption, and introduce latency jitter. The "
|
||||
"Ascend backend therefore owns a CPU allocation policy to **reduce cross-NUMA"
|
||||
" traffic, reduce thread preemption, and improve latency stability** instead "
|
||||
"of relying on upstream GPU NUMA binding flags."
|
||||
msgstr ""
|
||||
"在多插槽 ARM 系统上，Linux 调度器可能将工作线程放置在远离其驱动的 NPU 的 CPU 上。这会增加跨 NUMA "
|
||||
"流量、增加线程抢占并引入延迟抖动。因此，Ascend 后端拥有自己的 CPU 分配策略，以**减少跨 NUMA "
|
||||
"流量、减少线程抢占并提高延迟稳定性**，而不是依赖上游的 GPU NUMA 绑定标志。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:24
|
||||
msgid "This is also why upstream NUMA flags are adapted on Ascend:"
|
||||
msgstr "这也是上游 NUMA 标志在 Ascend 上被适配的原因："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:26
|
||||
#, python-brace-format
|
||||
msgid ""
|
||||
"`--numa-bind` is converted to `additional_config={\"enable_cpu_binding\": "
|
||||
"true}`."
|
||||
msgstr "`--numa-bind` 被转换为 `additional_config={\"enable_cpu_binding\": true}`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:27
|
||||
msgid ""
|
||||
"`--numa-bind-nodes` and `--numa-bind-cpus` are ignored because Ascend "
|
||||
"computes CPU pools from NPU topology or global logical NPU IDs."
|
||||
msgstr ""
|
||||
"`--numa-bind-nodes` 和 `--numa-bind-cpus` 被忽略，因为 Ascend 根据 NPU 拓扑或全局逻辑 NPU ID"
|
||||
" 计算 CPU 池。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:29
|
||||
msgid "How It Works?"
|
||||
msgstr "工作原理？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:31
|
||||
msgid "The allocator derives its plan from runtime host state:"
|
||||
msgstr "分配器根据运行时主机状态制定计划："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Input"
|
||||
msgstr "输入"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Source"
|
||||
msgstr "来源"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Purpose"
|
||||
msgstr "用途"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Allowed CPUs"
|
||||
msgstr "允许的 CPU"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`/proc/self/status` `Cpus_allowed_list`"
|
||||
msgstr "`/proc/self/status` `Cpus_allowed_list`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "The only CPUs eligible for binding. Container cpusets are respected."
|
||||
msgstr "唯一有资格绑定的 CPU。会尊重容器的 cpuset 设置。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Logical NPU map"
|
||||
msgstr "逻辑 NPU 映射"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`npu-smi info -m`"
|
||||
msgstr "`npu-smi info -m`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid ""
|
||||
"Maps card/chip IDs to global logical NPU IDs and gives `total_logic_npus`. "
|
||||
"On Ascend 950, `Chip Logic ID` is not reported, so `NPU ID` is used as the "
|
||||
"logical ID."
|
||||
msgstr ""
|
||||
"将卡/芯片 ID 映射到全局逻辑 NPU ID，并提供 `total_logic_npus`。在 Ascend 950 上，不报告 `Chip "
|
||||
"Logic ID`，因此使用 `NPU ID` 作为逻辑 ID。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Running NPUs"
|
||||
msgstr "运行中的 NPU"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`npu-smi info` process table, filtered by `ASCEND_RT_VISIBLE_DEVICES`"
|
||||
msgstr "`npu-smi info` 进程表，按 `ASCEND_RT_VISIBLE_DEVICES` 过滤"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid ""
|
||||
"Identifies the logical NPUs used by this worker process. A2/A3 process rows "
|
||||
"use `NPU Chip`; Ascend 950 process rows use `NPU ID`."
|
||||
msgstr "识别此工作进程使用的逻辑 NPU。A2/A3 进程行使用 `NPU Chip`；Ascend 950 进程行使用 `NPU ID`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Topology affinity"
|
||||
msgstr "拓扑亲和性"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`npu-smi info -t topo`"
|
||||
msgstr "`npu-smi info -t topo`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Provides NPU-to-CPU affinity for `topo_affinity` mode."
|
||||
msgstr "为 `topo_affinity` 模式提供 NPU 到 CPU 的亲和性信息。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "CPU NUMA map"
|
||||
msgstr "CPU NUMA 映射"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`lscpu -e=CPU,NODE`"
|
||||
msgstr "`lscpu -e=CPU,NODE`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Used to extend single-NUMA affinity pools to the next NUMA node."
|
||||
msgstr "用于将单 NUMA 亲和性池扩展到下一个 NUMA 节点。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Thread topology"
|
||||
msgstr "线程拓扑"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`lscpu` `Thread(s) per core`"
|
||||
msgstr "`lscpu` `Thread(s) per core`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid ""
|
||||
"Determines Ascend 950 cluster size: 8 CPUs for 1 thread per core, 16 CPUs "
|
||||
"for 2 threads per core."
|
||||
msgstr "确定 Ascend 950 集群大小：每核 1 线程时为 8 个 CPU，每核 2 线程时为 16 个 CPU。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "UVB polling threads"
|
||||
msgstr "UVB 轮询线程"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`ps -Te`"
|
||||
msgstr "`ps -Te`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid ""
|
||||
"Finds host `uvb_poll_window_thread` threads for Ascend 950 UVB CPU binding. "
|
||||
"Docker containers must use `--pid=host` to see these host threads."
|
||||
msgstr ""
|
||||
"查找主机 `uvb_poll_window_thread` 线程，用于 Ascend 950 UVB CPU 绑定。Docker 容器必须使用 "
|
||||
"`--pid=host` 才能看到这些主机线程。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:43
|
||||
msgid "Strategy Selection"
|
||||
msgstr "策略选择"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:45
|
||||
msgid "The binding strategy is selected by Ascend device type:"
|
||||
msgstr "绑定策略根据 Ascend 设备类型选择："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Device type"
|
||||
msgstr "设备类型"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Strategy"
|
||||
msgstr "策略"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Reason"
|
||||
msgstr "原因"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "A3"
|
||||
msgstr "A3"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`global_slice`"
|
||||
msgstr "`global_slice`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid ""
|
||||
"A3 uses HCCS card-to-card interconnect. Each NPU is nearly equidistant from "
|
||||
"all NUMA nodes, so there is no strong NPU-to-NUMA affinity signal. Global "
|
||||
"logical NPU ID based slicing gives deterministic, non-overlapping CPU pools "
|
||||
"and CPU/NUMA isolation between workers."
|
||||
msgstr ""
|
||||
"A3 使用 HCCS 卡间互联。每个 NPU 到所有 NUMA 节点的距离几乎相等，因此没有强烈的 NPU 到 NUMA 亲和性信号。基于全局逻辑 "
|
||||
"NPU ID 的分片提供了确定性的、不重叠的 CPU 池以及工作进程间的 CPU/NUMA 隔离。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Ascend 950"
|
||||
msgstr "Ascend 950"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`topo_affinity`"
|
||||
msgstr "`topo_affinity`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid ""
|
||||
"Ascend 950 uses NPU-to-CPU affinity from `npu-smi info -t topo` to choose an"
|
||||
" affinity NUMA node, then assigns one CPU cluster from that NUMA node to "
|
||||
"each worker. It also reports process rows by `NPU ID` instead of `NPU Chip`,"
|
||||
" skips IRQ binding, and binds host UVB polling threads."
|
||||
msgstr ""
|
||||
"Ascend 950 使用来自 `npu-smi info -t topo` 的 NPU 到 CPU 亲和性信息来选择亲和 NUMA 节点，然后从该 "
|
||||
"NUMA 节点分配一个 CPU 集群给每个工作进程。它还通过 `NPU ID` 而非 `NPU Chip` 报告进程行，跳过 IRQ 绑定，并绑定主机 "
|
||||
"UVB 轮询线程。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "A2 and Atlas 300 inference products"
|
||||
msgstr "A2 和 Atlas 300 推理产品"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid ""
|
||||
"A2 and Atlas 300 inference products provide NPU-to-CPU affinity information "
|
||||
"through `npu-smi info -t topo`, so they use this topology signal when "
|
||||
"available."
|
||||
msgstr ""
|
||||
"A2 和 Atlas 300 推理产品通过 `npu-smi info -t topo` 提供 NPU 到 CPU "
|
||||
"的亲和性信息，因此它们在可用时使用此拓扑信号。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:53
|
||||
msgid ""
|
||||
"If `topo_affinity` is selected but topo affinity is unavailable, the "
|
||||
"allocator falls back to `global_slice`."
|
||||
msgstr "如果选择了 `topo_affinity` 但拓扑亲和性不可用，分配器将回退到 `global_slice`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:55
|
||||
msgid "CPU Pool Construction"
|
||||
msgstr "CPU 池构建"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:57
|
||||
msgid "global_slice"
|
||||
msgstr "global_slice"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:59
|
||||
msgid ""
|
||||
"`global_slice` is designed for devices without a useful NPU-to-CPU affinity "
|
||||
"signal, including A3. Because A3's **HCCS interconnect makes the distance "
|
||||
"from each NPU to each NUMA node nearly the same**, topology affinity is not "
|
||||
"a useful placement signal. The allocator therefore partitions the sorted "
|
||||
"`allowed_cpus` list by global logical NPU ID."
|
||||
msgstr ""
|
||||
"`global_slice` 专为没有有用 NPU 到 CPU 亲和性信号的设备设计，包括 A3。因为 A3 的 **HCCS 互联使得每个 NPU "
|
||||
"到每个 NUMA 节点的距离几乎相同**，拓扑亲和性不是一个有用的放置信号。因此，分配器根据全局逻辑 NPU ID 对排序后的 "
|
||||
"`allowed_cpus` 列表进行分区。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:65
|
||||
msgid "Determine `total_npus` in this order:"
|
||||
msgstr "按以下顺序确定 `total_npus`："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:66
|
||||
msgid "`total_logic_npus` from `npu-smi info -m`"
|
||||
msgstr "来自 `npu-smi info -m` 的 `total_logic_npus`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:67
|
||||
msgid "number of topo affinity entries"
|
||||
msgstr "拓扑亲和性条目数量"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:68
|
||||
msgid "number of running NPUs"
|
||||
msgstr "运行中的 NPU 数量"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:69
|
||||
msgid "Compute:"
|
||||
msgstr "计算："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:70
|
||||
msgid "`base = len(allowed_cpus) // total_npus`"
|
||||
msgstr "`base = len(allowed_cpus) // total_npus`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:71
|
||||
msgid "`extra = len(allowed_cpus) % total_npus`"
|
||||
msgstr "`extra = len(allowed_cpus) % total_npus`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:72
|
||||
msgid "Each logical NPU gets a deterministic slice:"
|
||||
msgstr "每个逻辑 NPU 获得一个确定性的分片："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:73
|
||||
msgid "NPU IDs `< extra` receive `base + 1` CPUs."
|
||||
msgstr "NPU ID `< extra` 接收 `base + 1` 个 CPU。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:74
|
||||
msgid "Remaining NPU IDs receive `base` CPUs."
|
||||
msgstr "剩余的 NPU ID 接收 `base` 个 CPU。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:75
|
||||
msgid "Only running NPUs are materialized into `npu_cpu_pool`."
|
||||
msgstr "只有运行中的 NPU 会被实例化到 `npu_cpu_pool` 中。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:77
|
||||
msgid ""
|
||||
"This is the key property: two independent worker processes with the same "
|
||||
"cpuset but different visible NPU IDs still get **non-overlapping CPU pools**"
|
||||
" because both processes slice against the same global NPU ID space. With a "
|
||||
"NUMA-aligned cpuset, this also provides **CPU/NUMA isolation between "
|
||||
"workers**, so one worker does not share the same CPU or NUMA slice with "
|
||||
"another worker."
|
||||
msgstr ""
|
||||
"这是关键属性：两个独立的工作进程，即使具有相同的 cpuset 但不同的可见 NPU ID，仍然会获得**不重叠的 CPU "
|
||||
"池**，因为两个进程都针对相同的全局 NPU ID 空间进行分片。使用 NUMA 对齐的 cpuset，这还提供了**工作进程间的 CPU/NUMA "
|
||||
"隔离**，因此一个工作进程不会与另一个工作进程共享相同的 CPU 或 NUMA 分片。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:83
|
||||
msgid ""
|
||||
"`global_slice` requires enough CPUs for the selected device's role split:"
|
||||
msgstr "`global_slice`需要足够的CPU以满足所选设备的角色拆分："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:85
|
||||
msgid ""
|
||||
"Devices with IRQ binding require `base >= 5`: 2 CPUs for SQ/CQ IRQ binding, "
|
||||
"at least 1 CPU for the main worker, 1 CPU for ACL thread, and 1 CPU for "
|
||||
"release thread."
|
||||
msgstr ""
|
||||
"支持IRQ绑定的设备要求`base >= 5`：2个CPU用于SQ/CQ "
|
||||
"IRQ绑定，至少1个CPU用于主工作线程，1个CPU用于ACL线程，1个CPU用于释放线程。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:89
|
||||
msgid "topo_affinity"
|
||||
msgstr "topo_affinity"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:91
|
||||
msgid ""
|
||||
"`topo_affinity` is designed for A2, Atlas 300 inference products, Ascend "
|
||||
"950, and other non-A3 device types. A2 and Atlas 300 inference products "
|
||||
"expose **meaningful NPU-to-CPU affinity information**, so the allocator "
|
||||
"starts from NPU topology affinity when it is available and then avoids "
|
||||
"overlap for shared affinity groups."
|
||||
msgstr ""
|
||||
"`topo_affinity`专为A2、Atlas 300推理产品、Ascend 950及其他非A3设备类型设计。A2和Atlas "
|
||||
"300推理产品暴露**有意义的NPU到CPU亲和性信息**，因此分配器在可用时从NPU拓扑亲和性开始，然后避免共享亲和性组的重叠。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:97
|
||||
msgid "Build candidate NPUs from all logical NPUs:"
|
||||
msgstr "从所有逻辑NPU构建候选NPU："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:98
|
||||
msgid "always include running NPUs"
|
||||
msgstr "始终包含正在运行的NPU"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:99
|
||||
msgid ""
|
||||
"include non-running NPUs only when their affinity overlaps this process's "
|
||||
"allowed cpuset"
|
||||
msgstr "仅当非运行NPU的亲和性与当前进程的允许cpuset重叠时才包含它们"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:100
|
||||
msgid "For each candidate NPU, intersect topo affinity with `allowed_cpus`."
|
||||
msgstr "对于每个候选NPU，将拓扑亲和性与`allowed_cpus`取交集。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:101
|
||||
msgid ""
|
||||
"If the intersection is empty for a candidate, binding fails for this rank."
|
||||
msgstr "如果某个候选NPU的交集为空，则该rank的绑定失败。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:102
|
||||
msgid ""
|
||||
"If the affinity CPUs are all on one NUMA node, extend the pool with CPUs "
|
||||
"from the next NUMA node, constrained by `allowed_cpus`."
|
||||
msgstr "如果亲和性CPU都在同一个NUMA节点上，则用下一个NUMA节点的CPU扩展池，受`allowed_cpus`约束。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:103
|
||||
msgid ""
|
||||
"Group NPUs with identical extended pools and split each shared pool evenly "
|
||||
"across that group."
|
||||
msgstr "将具有相同扩展池的NPU分组，并在该组内平均分配每个共享池。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:104
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:117
|
||||
msgid "Keep only running NPUs in the final `npu_cpu_pool`."
|
||||
msgstr "在最终的`npu_cpu_pool`中仅保留正在运行的NPU。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:106
|
||||
msgid ""
|
||||
"The non-running candidate step is intentional. It prevents two independent "
|
||||
"single-card workers from selecting the same CPU range when their visible "
|
||||
"NPUs share the same topology affinity."
|
||||
msgstr "包含非运行候选NPU的步骤是有意为之。它防止两个独立的单卡工作线程在其可见NPU共享相同拓扑亲和性时选择相同的CPU范围。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:110
|
||||
msgid "For Ascend 950, topology affinity is used differently:"
|
||||
msgstr "对于Ascend 950，拓扑亲和性的使用方式不同："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:112
|
||||
msgid ""
|
||||
"Bind all visible host `uvb_poll_window_thread` threads to NUMA0 CPUs except "
|
||||
"CPU0, constrained by `allowed_cpus`. Docker containers must use `--pid=host`"
|
||||
" to make these host threads visible."
|
||||
msgstr ""
|
||||
"将所有可见的主机`uvb_poll_window_thread`线程绑定到NUMA0上除CPU0之外的CPU，受`allowed_cpus`约束。Docker容器必须使用`--pid=host`才能使这些主机线程可见。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:113
|
||||
msgid "Use topo affinity to identify each NPU's single affinity NUMA node."
|
||||
msgstr "使用拓扑亲和性识别每个NPU的单一亲和性NUMA节点。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:114
|
||||
msgid ""
|
||||
"Parse `Thread(s) per core` from `lscpu` and set cluster size to 8 CPUs when "
|
||||
"it is 1, or 16 CPUs when it is 2."
|
||||
msgstr "从`lscpu`解析`Thread(s) per core`，当值为1时将集群大小设为8个CPU，当值为2时设为16个CPU。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:115
|
||||
msgid ""
|
||||
"Split each affinity NUMA's sorted allowed CPU list into contiguous clusters."
|
||||
msgstr "将每个亲和性NUMA的已排序允许CPU列表分割成连续的集群。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:116
|
||||
msgid ""
|
||||
"Assign clusters by sorted logical NPU ID, including hidden NPUs that share "
|
||||
"the same affinity NUMA."
|
||||
msgstr "按排序后的逻辑NPU ID分配集群，包括共享相同亲和性NUMA的隐藏NPU。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:119
|
||||
msgid ""
|
||||
"If Ascend 950 topo affinity is missing, spans multiple NUMA nodes, has too "
|
||||
"few clusters, or reports an unsupported `Thread(s) per core`, worker CPU "
|
||||
"binding is skipped without raising to the worker process."
|
||||
msgstr ""
|
||||
"如果Ascend 950的拓扑亲和性缺失、跨越多个NUMA节点、集群数量过少或报告不支持的`Thread(s) per "
|
||||
"core`，则跳过工作线程CPU绑定，不会向工作进程抛出异常。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:123
|
||||
msgid "Role Split"
|
||||
msgstr "角色拆分"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:125
|
||||
msgid "After a CPU pool is built, the allocator splits it by role:"
|
||||
msgstr "构建CPU池后，分配器按角色对其进行拆分："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:127
|
||||
msgid "For devices with IRQ binding:"
|
||||
msgstr "对于支持IRQ绑定的设备："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Role"
|
||||
msgstr "角色"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "CPUs"
|
||||
msgstr "CPU"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "SQ/CQ IRQ"
|
||||
msgstr "SQ/CQ IRQ"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`pool[0]`, `pool[1]`"
|
||||
msgstr "`pool[0]`, `pool[1]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Main worker process and subthreads"
|
||||
msgstr "主工作进程及子线程"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`pool[2:-2]`"
|
||||
msgstr "`pool[2:-2]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "ACL thread"
|
||||
msgstr "ACL线程"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`pool[-2]`"
|
||||
msgstr "`pool[-2]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Release thread"
|
||||
msgstr "释放线程"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "`pool[-1]`"
|
||||
msgstr "`pool[-1]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:136
|
||||
msgid "For Ascend 950:"
|
||||
msgstr "对于Ascend 950："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "the whole assigned cluster"
|
||||
msgstr "整个分配的集群"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "not separately pinned"
|
||||
msgstr "不单独固定"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:144
|
||||
msgid ""
|
||||
"If a final pool has fewer CPUs than the selected role split requires, "
|
||||
"binding fails for this rank and the worker logs a warning from the caller. "
|
||||
"The minimum is 5 CPUs per NPU for devices with IRQ binding. Ascend 950 "
|
||||
"requires one full cluster per worker."
|
||||
msgstr ""
|
||||
"如果最终池中的CPU数量少于所选角色拆分所需的数量，则该rank的绑定失败，工作线程会从调用方记录警告。对于支持IRQ绑定的设备，每个NPU至少需要5个CPU。Ascend"
|
||||
" 950每个工作线程需要一个完整的集群。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:149
|
||||
msgid "Conditional Host Tuning"
|
||||
msgstr "条件性主机调优"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:151
|
||||
msgid ""
|
||||
"After CPU affinity is applied, CPU binding can also apply two host-side "
|
||||
"tuning steps when the environment supports them:"
|
||||
msgstr "应用CPU亲和性后，当环境支持时，CPU绑定还可以应用两个主机端调优步骤："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:154
|
||||
msgid ""
|
||||
"Memory migration uses `migratepages` to move the worker process's existing "
|
||||
"pages to the selected NUMA node. This keeps the worker closer to the memory "
|
||||
"it reads and reduces remote-NUMA memory read latency."
|
||||
msgstr ""
|
||||
"内存迁移使用`migratepages`将工作进程的现有页面移动到选定的NUMA节点。这使工作线程更接近其读取的内存，并减少远程NUMA内存读取延迟。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:157
|
||||
msgid ""
|
||||
"IRQ binding places NPU IRQ handling on the CPUs reserved for the "
|
||||
"corresponding NPU when `/proc/irq` is writable and IRQ files can be "
|
||||
"resolved. Ascend 950 skips this step."
|
||||
msgstr ""
|
||||
"当`/proc/irq`可写且IRQ文件可解析时，IRQ绑定将NPU IRQ处理放置在为相应NPU保留的CPU上。Ascend 950跳过此步骤。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:161
|
||||
msgid ""
|
||||
"These are conditional parts of CPU binding, not separate feature switches. "
|
||||
"If a host prerequisite is missing, that step is skipped while CPU thread "
|
||||
"binding still proceeds. Missing `migratepages` can still leave pages on "
|
||||
"remote NUMA nodes, so **latency or throughput may regress compared with a "
|
||||
"full CPU binding setup.**"
|
||||
msgstr ""
|
||||
"这些是CPU绑定的条件性部分，而非独立的功能开关。如果缺少某个主机前提条件，该步骤将被跳过，但CPU线程绑定仍会继续。缺少`migratepages`仍可能使页面留在远程NUMA节点上，因此**与完整的CPU绑定设置相比，延迟或吞吐量可能会下降。**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:167
|
||||
msgid "Examples"
|
||||
msgstr "示例"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:169
|
||||
msgid "A3 inference server with 640 CPUs and 16 NPUs"
|
||||
msgstr "具有640个CPU和16个NPU的A3推理服务器"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:171
|
||||
msgid "Inputs:"
|
||||
msgstr "输入："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:173
|
||||
msgid "`allowed_cpus = [0..639]`"
|
||||
msgstr "`allowed_cpus = [0..639]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:174
|
||||
msgid "`total_logic_npus = 16`"
|
||||
msgstr "`total_logic_npus = 16`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:175
|
||||
msgid "`running_npu_list = [0..15]`"
|
||||
msgstr "`running_npu_list = [0..15]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:177
|
||||
msgid "Computation:"
|
||||
msgstr "计算过程："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:179
|
||||
msgid "`base = 640 // 16 = 40`"
|
||||
msgstr "`base = 640 // 16 = 40`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:180
|
||||
msgid "`extra = 0`"
|
||||
msgstr "`extra = 0`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:181
|
||||
msgid ""
|
||||
"Worker `i` driving logical NPU `i` receives CPU slice `[i * 40 .. i * 40 + "
|
||||
"39]`."
|
||||
msgstr "驱动逻辑NPU `i`的工作线程`i`获得CPU切片`[i * 40 .. i * 40 + 39]`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:184
|
||||
msgid "Global slice view:"
|
||||
msgstr "全局切片视图："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:192
|
||||
msgid "Role split inside each worker slice:"
|
||||
msgstr "每个工作线程切片内部的角色拆分："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:200
|
||||
msgid "Concrete examples:"
|
||||
msgstr "具体示例："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Worker"
|
||||
msgstr "工作线程"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Logical NPU"
|
||||
msgstr "逻辑NPU"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "CPU pool"
|
||||
msgstr "CPU池"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "IRQ CPUs"
|
||||
msgstr "IRQ CPU"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Main CPUs"
|
||||
msgstr "主CPU"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "ACL CPU"
|
||||
msgstr "ACL CPU"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Release CPU"
|
||||
msgstr "释放CPU"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "0"
|
||||
msgstr "0"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "0-39"
|
||||
msgstr "0-39"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "0-1"
|
||||
msgstr "0-1"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "2-37"
|
||||
msgstr "2-37"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "38"
|
||||
msgstr "38"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "39"
|
||||
msgstr "39"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "1"
|
||||
msgstr "1"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "40-79"
|
||||
msgstr "40-79"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "40-41"
|
||||
msgstr "40-41"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "42-77"
|
||||
msgstr "42-77"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "78"
|
||||
msgstr "78"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "79"
|
||||
msgstr "79"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "..."
|
||||
msgstr "..."
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "15"
|
||||
msgstr "15"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "600-639"
|
||||
msgstr "600-639"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "600-601"
|
||||
msgstr "600-601"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "602-637"
|
||||
msgstr "602-637"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "638"
|
||||
msgstr "638"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "639"
|
||||
msgstr "639"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:209
|
||||
msgid ""
|
||||
"This layout remains deterministic even when different worker processes share"
|
||||
" the same cpuset, because slicing is based on the global logical NPU ID."
|
||||
msgstr "即使不同的工作进程共享相同的cpuset，此布局仍然保持确定性，因为切片基于全局逻辑NPU ID。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:212
|
||||
msgid "A2 topo_affinity with hidden same-affinity NPUs"
|
||||
msgstr "具有隐藏同亲和性NPU的A2 topo_affinity"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:214
|
||||
msgid "Inputs from an A2 topology:"
|
||||
msgstr "来自A2拓扑的输入："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:216
|
||||
msgid "NPU0 affinity: 144-167"
|
||||
msgstr "NPU0亲和性：144-167"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:217
|
||||
msgid "NPU2 affinity: 144-167"
|
||||
msgstr "NPU2亲和性：144-167"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:218
|
||||
msgid "Process A sees only NPU0"
|
||||
msgstr "进程A仅看到NPU0"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:219
|
||||
msgid "Process B sees only NPU2"
|
||||
msgstr "进程B仅看到NPU2"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:220
|
||||
msgid "Both processes have `allowed_cpus = [144..191]`"
|
||||
msgstr "两个进程均有 `allowed_cpus = [144..191]`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:222
|
||||
msgid ""
|
||||
"The allocator includes the hidden same-affinity NPU as a candidate in each "
|
||||
"process, splits the shared extended pool, and then keeps only the visible "
|
||||
"NPU in the final pool."
|
||||
msgstr "分配器将隐藏的同亲和性NPU作为候选纳入每个进程，拆分共享的扩展池，然后在最终池中仅保留可见的NPU。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:226
|
||||
msgid "Final pools:"
|
||||
msgstr "最终池："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Process"
|
||||
msgstr "进程"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Visible NPU"
|
||||
msgstr "可见NPU"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "Final CPU pool"
|
||||
msgstr "最终CPU池"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "A"
|
||||
msgstr "A"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "144-167"
|
||||
msgstr "144-167"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "B"
|
||||
msgstr "B"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "2"
|
||||
msgstr "2"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md
|
||||
msgid "168-191"
|
||||
msgstr "168-191"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:233
|
||||
msgid ""
|
||||
"This avoids overlapping CPU pools even when the two workers are launched as "
|
||||
"independent single-card services."
|
||||
msgstr "即使两个工作进程作为独立的单卡服务启动，这也能避免CPU池重叠。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:235
|
||||
msgid "Logs"
|
||||
msgstr "日志"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:237
|
||||
msgid "The allocator logs the selected mode and allocation plan:"
|
||||
msgstr "分配器记录所选模式和分配方案："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:245
|
||||
msgid ""
|
||||
"Ascend 950 uses a different role split, so its plan log does not include ACL"
|
||||
" or release fields. UVB polling thread binding is reported separately when "
|
||||
"matching threads are found:"
|
||||
msgstr "Ascend 950使用不同的角色拆分，因此其方案日志不包含ACL或释放字段。当找到匹配线程时，UVB轮询线程绑定会单独报告："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:256
|
||||
msgid "Limitations"
|
||||
msgstr "限制"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:258
|
||||
msgid "CPU binding runs only on ARM. It is skipped on x86_64."
|
||||
msgstr "CPU绑定仅在ARM上运行，在x86_64上跳过。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:259
|
||||
msgid ""
|
||||
"Each final NPU pool must have enough CPUs for its role split: at least 5 "
|
||||
"CPUs for devices with IRQ binding. Ascend 950 requires one complete CPU "
|
||||
"cluster per worker."
|
||||
msgstr ""
|
||||
"每个最终NPU池必须为其角色拆分提供足够的CPU：对于具有IRQ绑定的设备至少需要5个CPU。Ascend "
|
||||
"950要求每个工作进程拥有一个完整的CPU集群。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:261
|
||||
msgid ""
|
||||
"`global_slice` is deterministic and provides CPU/NUMA isolation when the "
|
||||
"cpuset is NUMA-aligned, but it cannot guarantee NUMA-local pools when CPU "
|
||||
"numbering or cpuset layout crosses NUMA boundaries."
|
||||
msgstr ""
|
||||
"`global_slice`是确定性的，当cpuset与NUMA对齐时提供CPU/NUMA隔离，但当CPU编号或cpuset布局跨越NUMA边界时，无法保证NUMA本地池。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:264
|
||||
msgid "`topo_affinity` depends on usable output from `npu-smi info -t topo`."
|
||||
msgstr "`topo_affinity`依赖于`npu-smi info -t topo`的可用输出。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:265
|
||||
msgid ""
|
||||
"IRQ binding requires writable `/proc/irq` and resolvable PCI/IRQ "
|
||||
"information. Ascend 950 skips IRQ binding even when `/proc/irq` is writable."
|
||||
msgstr ""
|
||||
"IRQ绑定需要可写的`/proc/irq`和可解析的PCI/IRQ信息。即使`/proc/irq`可写，Ascend 950也会跳过IRQ绑定。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:267
|
||||
msgid ""
|
||||
"Ascend 950 UVB polling thread binding requires visibility into the host PID "
|
||||
"namespace. Docker containers must be created with `--pid=host`; otherwise "
|
||||
"`uvb_poll_window_thread` may not be found."
|
||||
msgstr ""
|
||||
"Ascend 950 "
|
||||
"UVB轮询线程绑定需要可见的主机PID命名空间。Docker容器必须使用`--pid=host`创建；否则可能找不到`uvb_poll_window_thread`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:270
|
||||
msgid ""
|
||||
"Memory migration requires `migratepages`; otherwise only memory migration is"
|
||||
" skipped. CPU affinity still applies, but performance may degrade because "
|
||||
"existing pages are not moved to the target NUMA node and may be read through"
|
||||
" higher-latency remote NUMA access."
|
||||
msgstr ""
|
||||
"内存迁移需要`migratepages`；否则仅跳过内存迁移。CPU亲和性仍然生效，但性能可能下降，因为现有页面未移动到目标NUMA节点，可能通过更高延迟的远程NUMA访问读取。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:274
|
||||
msgid ""
|
||||
"If an exception escapes the binding flow, `NPUWorker` logs a warning and "
|
||||
"skips CPU binding for that rank."
|
||||
msgstr "如果异常逃逸出绑定流程，`NPUWorker`会记录警告并跳过该rank的CPU绑定。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:276
|
||||
msgid "References"
|
||||
msgstr "参考"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:278
|
||||
msgid "Implementation: `vllm_ascend/cpu_binding.py`"
|
||||
msgstr "实现：`vllm_ascend/cpu_binding.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:279
|
||||
msgid "Worker integration: `vllm_ascend/worker/worker.py`"
|
||||
msgstr "工作进程集成：`vllm_ascend/worker/worker.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:280
|
||||
msgid ""
|
||||
"Config: `vllm_ascend/ascend_config.py` and "
|
||||
"`docs/source/user_guide/configuration/additional_config.md`"
|
||||
msgstr ""
|
||||
"配置：`vllm_ascend/ascend_config.py` 和 "
|
||||
"`docs/source/user_guide/configuration/additional_config.md`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/cpu_binding.md:281
|
||||
msgid "Tests: `tests/ut/device_allocator/test_cpu_binding.py`"
|
||||
msgstr "测试：`tests/ut/device_allocator/test_cpu_binding.py`"
|
||||
@@ -0,0 +1,363 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:1
|
||||
msgid "Disaggregated-prefill"
|
||||
msgstr "分离式预填充"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:3
|
||||
msgid "Why disaggregated-prefill?"
|
||||
msgstr "为什么需要分离式预填充？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:5
|
||||
msgid ""
|
||||
"This feature addresses the need to optimize the **Time Per Output Token "
|
||||
"(TPOT)** and **Time To First Token (TTFT)** in large-scale inference tasks. "
|
||||
"The motivation is two-fold:"
|
||||
msgstr "该功能旨在优化大规模推理任务中的**每个输出令牌时间（TPOT）**和**首令牌时间（TTFT）**。其动机有两个方面："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:7
|
||||
msgid ""
|
||||
"**Adjusting Parallel Strategy and Instance Count for P and D Nodes** Using"
|
||||
" the disaggregated-prefill strategy, this feature allows the system to "
|
||||
"flexibly adjust the parallelization strategy (e.g., data parallelism (dp), "
|
||||
"tensor parallelism (tp), and expert parallelism (ep)) and the instance count"
|
||||
" for both P (Prefiller) and D (Decoder) nodes. This leads to better system "
|
||||
"performance tuning, particularly for **TTFT** and **TPOT**."
|
||||
msgstr ""
|
||||
"**调整P节点和D节点的并行策略与实例数量** "
|
||||
"通过使用分离式预填充策略，该功能允许系统灵活调整P（预填充器）和D（解码器）节点的并行化策略（如数据并行（dp）、张量并行（tp）和专家并行（ep））及实例数量。这有助于更好地调优系统性能，特别是针对**TTFT**和**TPOT**。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:10
|
||||
msgid ""
|
||||
"**Optimizing TPOT** Without the disaggregated-prefill strategy, prefill "
|
||||
"tasks are inserted during decoding, which results in inefficiencies and "
|
||||
"delays. Disaggregated-prefill solves this by allowing for better control "
|
||||
"over the system's **TPOT**. By managing chunked prefill tasks effectively, "
|
||||
"the system avoids the challenge of determining the optimal chunk size and "
|
||||
"provides more reliable control over the time taken for generating output "
|
||||
"tokens."
|
||||
msgstr ""
|
||||
"**优化TPOT** "
|
||||
"在没有分离式预填充策略的情况下，预填充任务会在解码过程中插入，导致效率低下和延迟。分离式预填充通过允许更好地控制系统**TPOT**来解决此问题。通过有效管理分块预填充任务，系统避免了确定最佳块大小的挑战，并提供了对生成输出令牌时间更可靠的控制。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:15
|
||||
msgid "Usage"
|
||||
msgstr "使用方法"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:17
|
||||
msgid ""
|
||||
"vLLM Ascend currently supports two types of connectors for handling KV cache"
|
||||
" management:"
|
||||
msgstr "vLLM Ascend 当前支持两种类型的连接器来处理KV缓存管理："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:19
|
||||
msgid "**MooncakeConnector**: D nodes pull KV cache from P nodes."
|
||||
msgstr "**MooncakeConnector**：D节点从P节点拉取KV缓存。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:20
|
||||
msgid ""
|
||||
"**MooncakeLayerwiseConnector**: P nodes push KV cache to D nodes in a "
|
||||
"layered manner."
|
||||
msgstr "**MooncakeLayerwiseConnector**：P节点以逐层方式将KV缓存推送到D节点。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:22
|
||||
msgid ""
|
||||
"For step-by-step deployment and configuration, refer to the following guide:"
|
||||
" "
|
||||
"[https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_disaggregation_mooncake_multi_node.html](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_disaggregation_mooncake_multi_node.html)"
|
||||
msgstr ""
|
||||
"有关逐步部署和配置，请参考以下指南： "
|
||||
"[https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_disaggregation_mooncake_multi_node.html](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_disaggregation_mooncake_multi_node.html)"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:27
|
||||
msgid "How It Works"
|
||||
msgstr "工作原理"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:29
|
||||
msgid "1. Design Approach"
|
||||
msgstr "1.设计方法"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:31
|
||||
msgid ""
|
||||
"Under the disaggregated-prefill, a global proxy receives external requests, "
|
||||
"forwarding prefill to P nodes and decode to D nodes; the KV cache (key-value"
|
||||
" cache) is exchanged between P and D nodes via peer-to-peer (P2P) "
|
||||
"communication."
|
||||
msgstr ""
|
||||
"在分离式预填充架构下，全局代理接收外部请求，将预填充转发至P节点，解码转发至D节点；KV缓存（键值缓存）通过点对点（P2P）通信在P节点和D节点之间交换。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:33
|
||||
msgid "2. Implementation Design"
|
||||
msgstr "2.实现设计"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:35
|
||||
msgid ""
|
||||
"Our design diagram is shown below, illustrating the pull and push schemes "
|
||||
"respectively.  "
|
||||
msgstr ""
|
||||
"我们的设计图如下所示，分别展示了拉取和推送方案。 "
|
||||
" "
|
||||
""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:35
|
||||
msgid "alt text"
|
||||
msgstr "替代文本"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:39
|
||||
msgid "Mooncake Connector"
|
||||
msgstr "Mooncake连接器"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:41
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:49
|
||||
msgid "The request is sent to the Proxy's `_handle_completions` endpoint."
|
||||
msgstr "请求被发送到代理的`_handle_completions`端点。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:42
|
||||
msgid ""
|
||||
"The Proxy calls `select_prefiller` to choose a P node and forwards the "
|
||||
"request, configuring `kv_transfer_params` with `do_remote_decode=True`, "
|
||||
"`max_completion_tokens=1`, and `min_tokens=1`."
|
||||
msgstr ""
|
||||
"代理调用`select_prefiller`选择一个P节点并转发请求，配置`kv_transfer_params`参数为`do_remote_decode=True`、`max_completion_tokens=1`和`min_tokens=1`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:43
|
||||
msgid ""
|
||||
"After the P node's scheduler finishes prefill, `update_from_output` invokes "
|
||||
"the schedule connector's `request_finished` to defer KV cache release, "
|
||||
"constructs `kv_transfer_params` with `do_remote_prefill=True`, and returns "
|
||||
"to the Proxy."
|
||||
msgstr ""
|
||||
"P节点调度器完成预填充后，`update_from_output`调用调度连接器的`request_finished`以延迟KV缓存释放，构造`kv_transfer_params`参数为`do_remote_prefill=True`，并返回给代理。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:44
|
||||
msgid ""
|
||||
"The Proxy calls `select_decoder` to choose a D node and forwards the "
|
||||
"request."
|
||||
msgstr "代理调用`select_decoder`选择一个D节点并转发请求。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:45
|
||||
msgid ""
|
||||
"On the D node, the scheduler marks the request as "
|
||||
"`RequestStatus.WAITING_FOR_REMOTE_KVS`, pre-allocates KV cache, calls "
|
||||
"`kv_connector_no_forward` to pull the remote KV cache, then notifies the P "
|
||||
"node to release KV cache and proceeds with decoding to return the result."
|
||||
msgstr ""
|
||||
"在D节点上，调度器将请求标记为`RequestStatus.WAITING_FOR_REMOTE_KVS`，预分配KV缓存，调用`kv_connector_no_forward`拉取远程KV缓存，然后通知P节点释放KV缓存并继续解码以返回结果。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:47
|
||||
msgid "Mooncake Layerwise Connector"
|
||||
msgstr "Mooncake逐层连接器"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:50
|
||||
msgid ""
|
||||
"The Proxy calls `select_decoder` to choose a D node and forwards the "
|
||||
"request, configuring `kv_transfer_params` with `do_remote_prefill=True` and "
|
||||
"setting the `metaserver` endpoint."
|
||||
msgstr ""
|
||||
"代理调用`select_decoder`选择一个D节点并转发请求，配置`kv_transfer_params`参数为`do_remote_prefill=True`并设置`metaserver`端点。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:51
|
||||
msgid ""
|
||||
"On the D node, the scheduler uses `kv_transfer_params` to mark the request "
|
||||
"as `RequestStatus.WAITING_FOR_REMOTE_KVS`, pre-allocates KV cache, then "
|
||||
"calls `kv_connector_no_forward` to send a request to the metaserver and "
|
||||
"waits for the KV cache transfer to complete."
|
||||
msgstr ""
|
||||
"在D节点上，调度器使用`kv_transfer_params`将请求标记为`RequestStatus.WAITING_FOR_REMOTE_KVS`，预分配KV缓存，然后调用`kv_connector_no_forward`向元服务器发送请求并等待KV缓存传输完成。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:52
|
||||
msgid ""
|
||||
"The Proxy's `metaserver` endpoint receives the request, calls "
|
||||
"`select_prefiller` to choose a P node, and forwards it with "
|
||||
"`kv_transfer_params` set to `do_remote_decode=True`, "
|
||||
"`max_completion_tokens=1`, and `min_tokens=1`."
|
||||
msgstr ""
|
||||
"代理的`metaserver`端点接收请求，调用`select_prefiller`选择一个P节点，并转发请求，设置`kv_transfer_params`参数为`do_remote_decode=True`、`max_completion_tokens=1`和`min_tokens=1`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:53
|
||||
msgid ""
|
||||
"During processing, the P node's scheduler pushes KV cache layer-wise; once "
|
||||
"all layers pushing is complete, it releases the request and notifies the D "
|
||||
"node to begin decoding."
|
||||
msgstr "处理过程中，P节点调度器逐层推送KV缓存；所有层推送完成后，释放请求并通知D节点开始解码。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:54
|
||||
msgid "The D node performs decoding and returns the result."
|
||||
msgstr "D节点执行解码并返回结果。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:56
|
||||
msgid "3. Interface Design"
|
||||
msgstr "3.接口设计"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:58
|
||||
msgid ""
|
||||
"Taking MooncakeConnector as an example, the system is organized into three "
|
||||
"primary classes:"
|
||||
msgstr "以MooncakeConnector为例，系统组织为三个主要类："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:60
|
||||
msgid "**MooncakeConnector**: Base class that provides core interfaces."
|
||||
msgstr "**MooncakeConnector**：提供核心接口的基类。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:61
|
||||
msgid ""
|
||||
"**MooncakeConnectorScheduler**: Interface for scheduling the connectors "
|
||||
"within the engine core, responsible for managing KV cache transfer "
|
||||
"requirements and completion."
|
||||
msgstr "**MooncakeConnectorScheduler**：用于在引擎核心内调度连接器的接口，负责管理KV缓存传输需求和完成。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:62
|
||||
msgid ""
|
||||
"**MooncakeConnectorWorker**: Interface for managing KV cache registration "
|
||||
"and transfer in worker processes."
|
||||
msgstr "**MooncakeConnectorWorker**：用于管理工作进程中KV缓存注册和传输的接口。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:64
|
||||
msgid "4. Specifications Design"
|
||||
msgstr "4.规格设计"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:66
|
||||
msgid ""
|
||||
"This feature is flexible and supports various configurations, including "
|
||||
"setups with MLA and GQA models. It is compatible with A2 and A3 hardware "
|
||||
"configurations and facilitates scenarios involving equal TP setups and "
|
||||
"certain unequal TP setups across multiple P and D nodes."
|
||||
msgstr ""
|
||||
"该功能灵活且支持多种配置，包括使用MLA和GQA模型的设置。它与A2和A3硬件配置兼容，并支持跨多个P和D节点的相等TP设置及某些不等TP设置场景。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md
|
||||
msgid "Feature"
|
||||
msgstr "功能"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md
|
||||
msgid "Status"
|
||||
msgstr "状态"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md
|
||||
msgid "A2"
|
||||
msgstr "A2"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md
|
||||
msgid "🟢 Functional"
|
||||
msgstr "🟢 功能正常"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md
|
||||
msgid "A3"
|
||||
msgstr "A3"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md
|
||||
msgid "equal TP configuration"
|
||||
msgstr "相等TP配置"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md
|
||||
msgid "unequal TP configuration"
|
||||
msgstr "不等TP配置"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md
|
||||
msgid "MLA"
|
||||
msgstr "MLA"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md
|
||||
msgid "GQA"
|
||||
msgstr "GQA"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:77
|
||||
msgid "🟢 Functional: Fully operational, with ongoing optimizations."
|
||||
msgstr "🟢 功能正常：完全可用，持续优化中。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:78
|
||||
msgid ""
|
||||
"🔵 Experimental: Experimental support, interfaces and functions may change."
|
||||
msgstr "🔵 实验性：实验性支持，接口和功能可能发生变化。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:79
|
||||
msgid "🚧 WIP: Under active development, will be supported soon."
|
||||
msgstr "🚧 开发中：正在积极开发中，即将支持。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:80
|
||||
msgid ""
|
||||
"🟡 Planned: Scheduled for future implementation (some may have open "
|
||||
"PRs/RFCs)."
|
||||
msgstr "🟡 已计划：已排入未来实现计划（部分可能有开放的 PR/RFC）。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:81
|
||||
msgid "🔴 NO plan/Deprecated: No plan or deprecated by vLLM."
|
||||
msgstr "🔴 无计划/已弃用：vLLM 暂无计划或已弃用。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:85
|
||||
msgid "DFX Analysis"
|
||||
msgstr "DFX 分析"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:87
|
||||
msgid "1. Config Parameter Validation"
|
||||
msgstr "1.配置参数验证"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:89
|
||||
msgid ""
|
||||
"Validate KV transfer config by checking whether the kv_connector type is "
|
||||
"supported. On transfer failures, emit clear error logs for diagnostics."
|
||||
msgstr "通过检查 kv_connector 类型是否受支持来验证 KV 传输配置。传输失败时，输出清晰的错误日志以辅助诊断。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:91
|
||||
msgid "2. Port Conflict Detection"
|
||||
msgstr "2.端口冲突检测"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:93
|
||||
msgid ""
|
||||
"Before startup, perform a port-usage check on configured ports (e.g., "
|
||||
"rpc_port, metrics_port, http_port/metaserver) by attempting to bind. If a "
|
||||
"port is already in use, fail fast and log an error."
|
||||
msgstr ""
|
||||
"启动前，通过尝试绑定对已配置端口（如 "
|
||||
"rpc_port、metrics_port、http_port/metaserver）执行端口占用检查。若端口已被占用，则快速失败并记录错误日志。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:95
|
||||
msgid "3. PD Ratio Validation"
|
||||
msgstr "3.PD 比例验证"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:97
|
||||
msgid ""
|
||||
"Under non-symmetric PD scenarios, validate the P-to-D tp ratio against "
|
||||
"expected and scheduling constraints to ensure correct and reliable "
|
||||
"operation."
|
||||
msgstr "在非对称 PD 场景下，根据预期和调度约束验证 P 与 D 的 tp 比例，以确保正确可靠的运行。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:101
|
||||
msgid "Limitations"
|
||||
msgstr "限制"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:103
|
||||
msgid ""
|
||||
"Heterogeneous P and D nodes are not supported, for example, running P nodes "
|
||||
"on A2 and D nodes on A3."
|
||||
msgstr "不支持异构 P 节点和 D 节点，例如在 A2 上运行 P 节点、在 A3 上运行 D 节点。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/disaggregated_prefill.md:105
|
||||
msgid ""
|
||||
"In non-symmetric TP configurations, only cases where the P nodes have a "
|
||||
"higher TP degree than the D nodes and the P TP count is an integer multiple "
|
||||
"of the D TP count are supported (i.e., P_tp > D_tp and P_tp % D_tp = 0)."
|
||||
msgstr ""
|
||||
"在非对称 TP 配置中，仅支持 P 节点的 TP 度数高于 D 节点且 P 的 TP 数量是 D 的 TP 数量的整数倍的情况（即 P_tp > "
|
||||
"D_tp 且 P_tp % D_tp = 0）。"
|
||||
@@ -0,0 +1,448 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:1
|
||||
msgid "Dynamic Chunked Pipeline Parallel (CPP)"
|
||||
msgstr "动态分块流水线并行（CPP）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:3
|
||||
msgid ""
|
||||
"TL;DR CPP uses profiling-based dynamic chunking to equalize per-chunk "
|
||||
"latency and eliminate pipeline bubbles in PP scenarios."
|
||||
msgstr "摘要：CPP 利用基于性能分析的动态分块，均衡每块延迟，消除 PP 场景中的流水线气泡。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:5
|
||||
msgid "Background"
|
||||
msgstr "背景"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:7
|
||||
msgid "Problem Statement"
|
||||
msgstr "问题描述"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:9
|
||||
msgid ""
|
||||
"In Pipeline Parallelism (PP) + Chunked Prefill scenarios, long sequences are"
|
||||
" split into fixed-size chunks that pass through the pipeline sequentially. "
|
||||
"Due to the O(n²) computational complexity of Self-Attention, **chunks of the"
|
||||
" same size take increasingly longer to process as the prefix sequence "
|
||||
"grows**:"
|
||||
msgstr ""
|
||||
"在流水线并行（PP）+ 分块预填充场景中，长序列被切分为固定大小的块，依次通过流水线。由于自注意力机制具有 O(n²) "
|
||||
"的计算复杂度，**随着前缀序列增长，相同大小的块处理时间会越来越长**："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:18
|
||||
msgid ""
|
||||
"This time variance propagates across pipeline stages, causing increased idle"
|
||||
" waiting (Pipeline Bubble) and significantly reducing GPU utilization."
|
||||
msgstr "这种时间差异会跨流水线阶段传播，导致空闲等待（流水线气泡）增加，并显著降低 GPU 利用率。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:20
|
||||
msgid "Solution Overview"
|
||||
msgstr "解决方案概述"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:22
|
||||
msgid ""
|
||||
"Dynamic Chunked Pipeline Parallel uses a **profile-first, then predict** "
|
||||
"strategy:"
|
||||
msgstr "动态分块流水线并行采用**先性能分析，后预测**的策略："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:38
|
||||
msgid ""
|
||||
"The core idea is borrowed from [SGLang's dynamic chunking "
|
||||
"mechanism](https://lmsys.org/blog/2026-01-15-chunked-pipeline/), with "
|
||||
"additional enhancements such as online calibration."
|
||||
msgstr ""
|
||||
"核心思想借鉴自 [SGLang 的动态分块机制](https://lmsys.org/blog/2026-01-15-chunked-"
|
||||
"pipeline/)，并增加了在线校准等增强功能。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:40
|
||||
msgid "Design"
|
||||
msgstr "设计"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:42
|
||||
msgid "Quadratic Latency Model"
|
||||
msgstr "二次延迟模型"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:44
|
||||
msgid ""
|
||||
"Transformer prefill latency grows quadratically with sequence length due to "
|
||||
"the O(n²) Self-Attention mechanism:"
|
||||
msgstr "由于 O(n²) 的自注意力机制，Transformer 预填充延迟随序列长度呈二次增长："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:46
|
||||
msgid "f(l) = a \\cdot l^2 + b \\cdot l + c"
|
||||
msgstr "f(l) = a \\cdot l^2 + b \\cdot l + c"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:48
|
||||
msgid "Where:"
|
||||
msgstr "其中："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:50
|
||||
msgid "$a \\cdot l^2$: Attention overhead (quadratic)"
|
||||
msgstr "$a \\cdot l^2$：注意力开销（二次）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:51
|
||||
msgid "$b \\cdot l$: Linear operations (FFN, projection)"
|
||||
msgstr "$b \\cdot l$：线性运算（FFN、投影）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:52
|
||||
msgid "$c$: Fixed overhead (kernel launch)"
|
||||
msgstr "$c$：固定开销（内核启动）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:54
|
||||
msgid "Startup Phase: Profiling"
|
||||
msgstr "启动阶段：性能分析"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:56
|
||||
msgid ""
|
||||
"During engine initialization, the system profiles actual model performance:"
|
||||
msgstr "在引擎初始化期间，系统对实际模型性能进行分析："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:58
|
||||
msgid ""
|
||||
"**Sampling**: Uniformly sample 64 different chunk sizes from "
|
||||
"`base_chunk_size` down to near 0"
|
||||
msgstr "**采样**：从 `base_chunk_size` 向下到接近 0，均匀采样 64 种不同的块大小"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:59
|
||||
msgid ""
|
||||
"**Execution**: Perform real model forward passes for each chunk size and "
|
||||
"precisely measure latency (milliseconds)"
|
||||
msgstr "**执行**：对每种块大小执行真实模型前向传播，并精确测量延迟（毫秒）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:60
|
||||
msgid "**Fitting**: Fit the quadratic model using least squares"
|
||||
msgstr "**拟合**：使用最小二乘法拟合二次模型"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:61
|
||||
msgid ""
|
||||
"**Target Setting**: Calculate target per-chunk latency based on "
|
||||
"`base_chunk_size`"
|
||||
msgstr "**目标设定**：基于 `base_chunk_size` 计算每块的目标延迟"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:63
|
||||
msgid ""
|
||||
"In PP mode, all workers execute forward passes to stay synchronized, but "
|
||||
"only the first PP rank's timing results are used for scheduling decisions."
|
||||
msgstr "在 PP 模式下，所有工作节点执行前向传播以保持同步，但仅使用第一个 PP 等级的计时结果进行调度决策。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:65
|
||||
msgid "Runtime Phase: Dynamic Prediction"
|
||||
msgstr "运行时阶段：动态预测"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:67
|
||||
#, python-brace-format
|
||||
msgid ""
|
||||
"Given current prefix length $L$ and target latency $T = "
|
||||
"f(\\text{base\\_chunk\\_size}) - f(0)$, the system solves for the next chunk"
|
||||
" size $x$:"
|
||||
msgstr ""
|
||||
"给定当前前缀长度 $L$ 和目标延迟 $T = f(\\text{base\\_chunk\\_size}) - f(0)$，系统求解下一个块大小 "
|
||||
"$x$："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:69
|
||||
msgid "f(L + x) - f(L) = T"
|
||||
msgstr "f(L + x) - f(L) = T"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:71
|
||||
msgid "Expanding to:"
|
||||
msgstr "展开为："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:73
|
||||
msgid "a \\cdot x^2 + (2aL + b) \\cdot x - T = 0"
|
||||
msgstr "a \\cdot x^2 + (2aL + b) \\cdot x - T = 0"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:75
|
||||
msgid "Solved using the quadratic formula:"
|
||||
msgstr "使用求根公式求解："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:77
|
||||
msgid "x = \\frac{-(2aL + b) + \\sqrt{(2aL + b)^2 + 4aT}}{2a}"
|
||||
msgstr "x = \\frac{-(2aL + b) + \\sqrt{(2aL + b)^2 + 4aT}}{2a}"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:79
|
||||
msgid "The result goes through post-processing:"
|
||||
msgstr "结果经过后处理："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:81
|
||||
msgid ""
|
||||
"**Smoothing**: Blend predicted chunk size with `base_chunk_size` using "
|
||||
"`smooth_factor`"
|
||||
msgstr "**平滑**：使用 `smooth_factor` 将预测的块大小与 `base_chunk_size` 混合"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:82
|
||||
msgid "**Alignment**: Round down to multiple of `page_size` (minimum 64)"
|
||||
msgstr "**对齐**：向下取整为 `page_size` 的倍数（最小 64）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:83
|
||||
msgid ""
|
||||
"**Constraints**: Not exceeding `max_model_len - history_len` and "
|
||||
"`max_num_scheduled_tokens`"
|
||||
msgstr "**约束**：不超过 `max_model_len - history_len` 和 `max_num_scheduled_tokens`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:85
|
||||
msgid "Online Calibration"
|
||||
msgstr "在线校准"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:87
|
||||
msgid ""
|
||||
"Since profiling only covers sequences up to `max_num_batched_tokens` "
|
||||
"(typically shorter than real workloads), the system continuously refines the"
|
||||
" model at runtime."
|
||||
msgstr "由于性能分析仅覆盖到 `max_num_batched_tokens` 的序列（通常比实际工作负载短），系统在运行时持续优化模型。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:89
|
||||
msgid "**Extended Model (two variables):**"
|
||||
msgstr "**扩展模型（双变量）：**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:91
|
||||
msgid "f(C, H) = a \\cdot C(C+H) + b \\cdot (C+H) + c"
|
||||
msgstr "f(C, H) = a \\cdot C(C+H) + b \\cdot (C+H) + c"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:93
|
||||
msgid "Where $C$ is chunk size and $H$ is prefix history length."
|
||||
msgstr "其中 $C$ 是块大小，$H$ 是前缀历史长度。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:95
|
||||
msgid ""
|
||||
"After each batch, feature vectors `[Σ(C+H)·C, Σ(C+H), N]` and actual "
|
||||
"execution time are recorded. Once enough data points accumulate (5-30), "
|
||||
"model parameters are updated using least squares."
|
||||
msgstr ""
|
||||
"每批处理后，记录特征向量 `[Σ(C+H)·C, Σ(C+H), N]` 和实际执行时间。一旦累积足够的数据点（5-30 "
|
||||
"个），使用最小二乘法更新模型参数。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:97
|
||||
msgid "Architecture"
|
||||
msgstr "架构"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:99
|
||||
msgid "Key Components"
|
||||
msgstr "关键组件"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Component"
|
||||
msgstr "组件"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Location"
|
||||
msgstr "位置"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Responsibility"
|
||||
msgstr "职责"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "**ChunkSizePredictor**"
|
||||
msgstr "**ChunkSizePredictor**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "`vllm_ascend/core/profiling_chunk_predictor.py`"
|
||||
msgstr "`vllm_ascend/core/profiling_chunk_predictor.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Quadratic model fitting and prediction"
|
||||
msgstr "二次模型拟合与预测"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "**ProfilingChunkManager**"
|
||||
msgstr "**ProfilingChunkManager**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Manage profiling workflow and predictor"
|
||||
msgstr "管理性能分析工作流和预测器"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "**Scheduler**"
|
||||
msgstr "**Scheduler**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "`vllm_ascend/core/scheduler_profiling_chunk.py`"
|
||||
msgstr "`vllm_ascend/core/scheduler_profiling_chunk.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Integrate CPP scheduling"
|
||||
msgstr "集成 CPP 调度"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "**EngineCore**"
|
||||
msgstr "**EngineCore**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "`vllm_ascend/patch/platform/patch_profiling_chunk.py`"
|
||||
msgstr "`vllm_ascend/patch/platform/patch_profiling_chunk.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Startup profiling, record execution time"
|
||||
msgstr "启动性能分析，记录执行时间"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "**NPUWorker**"
|
||||
msgstr "**NPUWorker**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "`vllm_ascend/worker/worker.py`"
|
||||
msgstr "`vllm_ascend/worker/worker.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Execute real forward pass profiling"
|
||||
msgstr "执行真实前向传播性能分析"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "**NPUModelRunner**"
|
||||
msgstr "**NPUModelRunner**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "`vllm_ascend/worker/model_runner_v1.py`"
|
||||
msgstr "`vllm_ascend/worker/model_runner_v1.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "`profile_cpp=True` mode"
|
||||
msgstr "`profile_cpp=True` 模式"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:110
|
||||
msgid "Workflow"
|
||||
msgstr "工作流"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:136
|
||||
msgid "Comparison with SGLang"
|
||||
msgstr "与 SGLang 对比"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Feature"
|
||||
msgstr "特性"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "SGLang Dynamic Chunking"
|
||||
msgstr "SGLang 动态分块"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Dynamic Chunked Pipeline Parallel"
|
||||
msgstr "动态分块流水线并行"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Profiling method"
|
||||
msgstr "性能分析方法"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Preset quadratic function"
|
||||
msgstr "预设二次函数"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Real forward pass profiling at startup"
|
||||
msgstr "启动时真实前向传播性能分析"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Model fitting"
|
||||
msgstr "模型拟合"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "$f(l) = a \\cdot l^2 + b \\cdot l + c$"
|
||||
msgstr "$f(l) = a \\cdot l^2 + b \\cdot l + c$"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Same + online calibration $f(C,H)$"
|
||||
msgstr "相同 + 在线校准 $f(C,H)$"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Online updates"
|
||||
msgstr "在线更新"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "None"
|
||||
msgstr "无"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "History-based fitting"
|
||||
msgstr "基于历史的拟合"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Accuracy"
|
||||
msgstr "准确性"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "May deviate on different hardware"
|
||||
msgstr "在不同硬件上可能有偏差"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Adapts to actual hardware performance"
|
||||
msgstr "自适应实际硬件性能"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "Startup cost"
|
||||
msgstr "启动开销"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md
|
||||
msgid "~64 forward passes (tens of seconds)"
|
||||
msgstr "约64次前向传播（数十秒）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:146
|
||||
msgid "Constraints"
|
||||
msgstr "约束条件"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:148
|
||||
msgid ""
|
||||
"**Pipeline Parallelism Required**: Must set `--pipeline-parallel-size > 1`"
|
||||
msgstr "**需要流水线并行**：必须设置 `--pipeline-parallel-size > 1`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:149
|
||||
msgid "**Chunked Prefill Required**: Must enable `--enable-chunked-prefill`"
|
||||
msgstr "**需要分块预填充**：必须启用 `--enable-chunked-prefill`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:150
|
||||
msgid ""
|
||||
"**Incompatible with Balance Scheduling**: Cannot enable "
|
||||
"`VLLM_ASCEND_BALANCE_SCHEDULING`"
|
||||
msgstr "**与均衡调度不兼容**：不能启用 `VLLM_ASCEND_BALANCE_SCHEDULING`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:151
|
||||
msgid ""
|
||||
"**Startup Overhead**: Profiling phase adds tens of seconds to initialization"
|
||||
msgstr "**启动开销**：性能分析阶段会增加数十秒的初始化时间"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:152
|
||||
msgid ""
|
||||
"**Memory**: No additional runtime memory overhead; profiling reuses existing"
|
||||
" dummy_run mechanism"
|
||||
msgstr "**内存**：无额外运行时内存开销；性能分析复用已有的 dummy_run 机制"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:154
|
||||
msgid "References"
|
||||
msgstr "参考文档"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:156
|
||||
msgid ""
|
||||
"[SGLang Dynamic Chunking Blog](https://lmsys.org/blog/2026-01-15-chunked-"
|
||||
"pipeline/)"
|
||||
msgstr "[SGLang 动态分块博客](https://lmsys.org/blog/2026-01-15-chunked-pipeline/)"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:157
|
||||
msgid ""
|
||||
"[User "
|
||||
"Guide](../../user_guide/feature_guide/dynamic_chunk_pipeline_parallel.md)"
|
||||
msgstr ""
|
||||
"[用户指南](../../user_guide/feature_guide/dynamic_chunk_pipeline_parallel.md)"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/dynamic_chunked_pipeline_parallel.md:158
|
||||
msgid ""
|
||||
"[Tutorial](../../tutorials/features/dynamic_chunked_pipeline_parallel.md)"
|
||||
msgstr "[教程](../../tutorials/features/dynamic_chunked_pipeline_parallel.md)"
|
||||
@@ -0,0 +1,465 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:1
|
||||
msgid "Expert Parallelism Load Balancer (EPLB)"
|
||||
msgstr "专家并行负载均衡器（EPLB）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:3
|
||||
msgid "Why We Need EPLB?"
|
||||
msgstr "为什么需要EPLB？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:5
|
||||
msgid ""
|
||||
"When using Expert Parallelism (EP), different experts are assigned to "
|
||||
"different NPUs. Given that the load of various experts may vary depending on"
|
||||
" the current workload, it is crucial to maintain balanced loads across "
|
||||
"different NPUs. We adopt a redundant experts strategy by duplicating "
|
||||
"heavily-loaded experts. Then, we heuristically pack these duplicated experts"
|
||||
" onto NPUs to ensure load balancing across them. Moreover, thanks to the "
|
||||
"group-limited expert routing used in MoE models, we also attempt to place "
|
||||
"experts of the same group on the same node to reduce inter-node data "
|
||||
"traffic, whenever possible."
|
||||
msgstr ""
|
||||
"在使用专家并行（EP）时，不同的专家被分配到不同的NPU上。由于不同专家的负载可能随当前工作负载变化，保持各NPU之间的负载均衡至关重要。我们采用冗余专家策略，通过复制高负载专家，然后启发式地将这些复制的专家打包到NPU上，以确保各NPU之间的负载均衡。此外，借助MoE模型中使用的组限制专家路由机制，我们尽可能将同一组的专家放置在同一节点上，以减少节点间的数据传输。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:7
|
||||
msgid ""
|
||||
"To facilitate reproduction and deployment, vLLM Ascend supports the deployed"
|
||||
" EP load balancing algorithm in `vllm_ascend/eplb/core/policy`. The "
|
||||
"algorithm computes a balanced expert replication and placement plan based on"
|
||||
" the estimated expert loads. Note that the exact method for predicting "
|
||||
"expert loads is outside the scope of this repository. A common method is to "
|
||||
"use a moving average of historical statistics."
|
||||
msgstr ""
|
||||
"为便于复现和部署，vLLM "
|
||||
"Ascend在`vllm_ascend/eplb/core/policy`中支持已部署的EP负载均衡算法。该算法根据估计的专家负载计算均衡的专家复制和放置方案。请注意，预测专家负载的具体方法不在本仓库范围内，常用方法是使用历史统计数据的移动平均值。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:9
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:9
|
||||
msgid "eplb"
|
||||
msgstr "eplb"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:11
|
||||
msgid "How to Use EPLB?"
|
||||
msgstr "如何使用EPLB？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:13
|
||||
msgid ""
|
||||
"Please refer to the EPLB section of the user guide for detailed information:"
|
||||
" [How to Use "
|
||||
"EPLB](../../user_guide/feature_guide/expert_parallelism_load_balancer.md)"
|
||||
msgstr ""
|
||||
"详细说明请参考用户指南中的EPLB章节：[如何使用EPLB](../../user_guide/feature_guide/expert_parallelism_load_balancer.md)"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:15
|
||||
msgid "How It Works?"
|
||||
msgstr "工作原理？"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:17
|
||||
msgid "**EPLB Module Architecture**"
|
||||
msgstr "**EPLB模块架构**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:40
|
||||
msgid ""
|
||||
"**1. Adaptor Module** *Handles registration and adaptation for different "
|
||||
"MoE model types*"
|
||||
msgstr "**1.适配器模块** *处理不同MoE模型类型的注册和适配*"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:43
|
||||
msgid ""
|
||||
"`vllm_adaptor.py` Implementation supporting Qwen3-MoE and DeepSeek models,"
|
||||
" standardizing parameter handling for policy algorithms"
|
||||
msgstr "`vllm_adaptor.py` 支持Qwen3-MoE和DeepSeek模型的实现，标准化策略算法的参数处理"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:46
|
||||
msgid ""
|
||||
"**2. Core Module** *Implements core algorithms, updates, and asynchronous "
|
||||
"processing*"
|
||||
msgstr "**2.核心模块** *实现核心算法、更新和异步处理*"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:49
|
||||
msgid ""
|
||||
"**Policy Submodule** *Load balancing algorithms with factory pattern "
|
||||
"instantiation*"
|
||||
msgstr "**策略子模块** *采用工厂模式实例化的负载均衡算法*"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:51
|
||||
msgid ""
|
||||
"`policy_abstract.py` Abstract class for load balancing strategy interfaces"
|
||||
msgstr "`policy_abstract.py` 负载均衡策略接口的抽象类"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:53
|
||||
msgid ""
|
||||
"`policy_default_eplb.py` Default implementation of open-source EPLB paper "
|
||||
"algorithm"
|
||||
msgstr "`policy_default_eplb.py` 开源EPLB论文算法的默认实现"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:55
|
||||
msgid ""
|
||||
"`policy_swift_balancer.py` Enhanced version optimizing expert swaps for "
|
||||
"low-bandwidth devices (e.g., A2)"
|
||||
msgstr "`policy_swift_balancer.py` 针对低带宽设备（如A2）优化专家交换的增强版本"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:57
|
||||
msgid ""
|
||||
"`policy_flashlb.py` Threshold-based adjustment reducing operational costs "
|
||||
"through layer-wise fluctuation detection"
|
||||
msgstr "`policy_flashlb.py` 基于阈值的调整，通过逐层波动检测降低运营成本"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:59
|
||||
msgid "`policy_random.py` Random policy for basic testing"
|
||||
msgstr "`policy_random.py` 用于基础测试的随机策略"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:61
|
||||
msgid ""
|
||||
"`policy_factory.py` Strategy factory for automatic algorithm instantiation"
|
||||
msgstr "`policy_factory.py` 用于自动实例化算法的策略工厂"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:64
|
||||
msgid ""
|
||||
"`eplb_device_transfer_loader.py` Manages expert table/weight transmission "
|
||||
"and updates"
|
||||
msgstr "`eplb_device_transfer_loader.py` 管理专家表/权重的传输和更新"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:66
|
||||
msgid ""
|
||||
"`eplb_utils.py` Utilities for expert table initialization and mapping"
|
||||
msgstr "`eplb_utils.py` 专家表初始化和映射的工具函数"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:68
|
||||
msgid ""
|
||||
"`eplb_worker.py` Asynchronous algorithm orchestration and result "
|
||||
"processing"
|
||||
msgstr "`eplb_worker.py` 异步算法编排和结果处理"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:71
|
||||
msgid "**3. System Components**"
|
||||
msgstr "**3.系统组件**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:73
|
||||
msgid ""
|
||||
"`eplb_updator.py` Central coordinator for load balancing during inference "
|
||||
"workflows"
|
||||
msgstr "`eplb_updator.py` 推理工作流中负载均衡的中央协调器"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:75
|
||||
msgid "`utils.py` General utilities for EPLB interface registration"
|
||||
msgstr "`utils.py` EPLB接口注册的通用工具函数"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:78
|
||||
msgid "*Key Optimizations:*"
|
||||
msgstr "*关键优化：*"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:80
|
||||
msgid "Maintained original structure while improving technical clarity"
|
||||
msgstr "在保持原有结构的同时提升技术清晰度"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:81
|
||||
msgid "Standardized terminology"
|
||||
msgstr "标准化术语"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:82
|
||||
msgid "Enhanced algorithm differentiation through concise descriptors"
|
||||
msgstr "通过简洁描述符增强算法区分度"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:83
|
||||
msgid "Improved scoping through hierarchical presentation"
|
||||
msgstr "通过分层展示改进范围界定"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:84
|
||||
msgid "Preserved file/class relationships while optimizing readability"
|
||||
msgstr "在优化可读性的同时保留文件/类关系"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:86
|
||||
msgid "Default Algorithm"
|
||||
msgstr "默认算法"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:88
|
||||
msgid "Hierarchical Load Balancing"
|
||||
msgstr "分层负载均衡"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:90
|
||||
msgid ""
|
||||
"When the number of server nodes evenly divides the number of expert groups, "
|
||||
"we use the hierarchical load balancing policy to leverage group-limited "
|
||||
"expert routing. We first pack the expert groups onto nodes evenly, ensuring "
|
||||
"balanced loads across different nodes. Then, we replicate the experts within"
|
||||
" each node. Finally, we pack the replicated experts onto individual NPUs to "
|
||||
"ensure load balancing across them. The hierarchical load balancing policy "
|
||||
"can be used in the prefilling stage with a smaller expert-parallel size."
|
||||
msgstr ""
|
||||
"当服务器节点数能整除专家组数时，我们使用分层负载均衡策略来利用组限制专家路由。首先将专家组均匀打包到节点上，确保各节点负载均衡；然后在每个节点内复制专家；最后将复制的专家打包到各个NPU上，确保NPU间负载均衡。分层负载均衡策略可用于预填充阶段，此时专家并行规模较小。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:92
|
||||
msgid "Global Load Balancing"
|
||||
msgstr "全局负载均衡"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:94
|
||||
msgid ""
|
||||
"In other cases, we use the global load balancing policy, which replicates "
|
||||
"experts globally regardless of expert groups, and packs the replicated "
|
||||
"experts onto individual NPUs. This policy can be adopted in the decoding "
|
||||
"stage with a larger expert-parallel size."
|
||||
msgstr ""
|
||||
"在其他情况下，我们使用全局负载均衡策略，该策略不考虑专家组，全局复制专家并将复制的专家打包到各个NPU上。此策略可用于解码阶段，此时专家并行规模较大。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:96
|
||||
msgid "Add a New EPLB Policy"
|
||||
msgstr "添加新的EPLB策略"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:98
|
||||
msgid ""
|
||||
"If you want to add a new eplb policy to vllm_ascend, you must follow these "
|
||||
"steps:"
|
||||
msgstr "如果你想向vllm_ascend添加新的EPLB策略，必须遵循以下步骤："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:100
|
||||
msgid ""
|
||||
"Inherit the `EplbPolicy` abstract class of `policy_abstract.py` and "
|
||||
"override the `rebalance_experts` interface, ensuring consistent input "
|
||||
"parameters `current_expert_table`, `expert_workload` and return types "
|
||||
"`newplacement`. For example:"
|
||||
msgstr ""
|
||||
"继承`policy_abstract.py`中的`EplbPolicy`抽象类，并重写`rebalance_experts`接口，确保输入参数`current_expert_table`、`expert_workload`和返回类型`newplacement`保持一致。例如："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:122
|
||||
msgid ""
|
||||
"To add a new EPLB algorithm, include the policy type and its corresponding "
|
||||
"implementation class in the `PolicyFactory` of `policy_factory.py`."
|
||||
msgstr "要添加新的EPLB算法，请在`policy_factory.py`的`PolicyFactory`中包含策略类型及其对应的实现类。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:124
|
||||
msgid "Add a New MoE Model"
|
||||
msgstr "添加新的MoE模型"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:126
|
||||
msgid "**Implementation Guide for Model Integration**"
|
||||
msgstr "**模型集成实现指南**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:128
|
||||
msgid "**Adapter File Modification**"
|
||||
msgstr "**适配器文件修改**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:129
|
||||
msgid "Inherit or modify `vllm_ascend/eplb/adaptor/vllm_adaptor.py`"
|
||||
msgstr "继承或修改`vllm_ascend/eplb/adaptor/vllm_adaptor.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:130
|
||||
msgid "Add processing logic for key parameters:"
|
||||
msgstr "添加关键参数的处理逻辑："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:131
|
||||
msgid "`num_dense_layers`"
|
||||
msgstr "`num_dense_layers`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:132
|
||||
msgid "`global_expert_num`"
|
||||
msgstr "`global_expert_num`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:133
|
||||
msgid "`num_roe_layers`"
|
||||
msgstr "`num_roe_layers`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:134
|
||||
msgid "Ensure parameter synchronization in the `model_register` function."
|
||||
msgstr "确保`model_register`函数中的参数同步。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:136
|
||||
msgid "For example:"
|
||||
msgstr "例如："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:138
|
||||
msgid ""
|
||||
"Modify `__init__` of `vllm_adaptor.py` to add a new moe model eplb params:"
|
||||
msgstr "修改`vllm_adaptor.py`的`__init__`方法，添加新的MoE模型EPLB参数："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:146
|
||||
msgid ""
|
||||
"Modify `model_register` of `vllm_adaptor.py` to register eplb params for new"
|
||||
" moe model:"
|
||||
msgstr "修改`vllm_adaptor.py`的`model_register`方法，为新的MoE模型注册EPLB参数："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:153
|
||||
msgid "**MoE Feature Integration**"
|
||||
msgstr "**MoE 特性集成**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:154
|
||||
msgid "Extend `vllm_ascend/eplb/utils.py` with MoE-specific methods"
|
||||
msgstr "使用 MoE 专用方法扩展 `vllm_ascend/eplb/utils.py`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:155
|
||||
msgid ""
|
||||
"Implement required functionality for expert routing or weight management"
|
||||
msgstr "实现专家路由或权重管理所需的功能"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:157
|
||||
msgid "**Registration Logic Update**"
|
||||
msgstr "**注册逻辑更新**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:158
|
||||
msgid "Add patch logic within the `model_register` function"
|
||||
msgstr "在 `model_register` 函数中添加补丁逻辑"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:159
|
||||
msgid "Maintain backward compatibility with existing model types"
|
||||
msgstr "保持与现有模型类型的向后兼容性"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:161
|
||||
msgid "**Validation & Testing**"
|
||||
msgstr "**验证与测试**"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:162
|
||||
msgid "Verify parameter consistency across layers"
|
||||
msgstr "验证各层之间的参数一致性"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:163
|
||||
msgid "Test cross-device communication for expert tables"
|
||||
msgstr "测试专家表的跨设备通信"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:164
|
||||
msgid "Benchmark against baseline implementations (e.g., Qwen3-MoE)"
|
||||
msgstr "与基线实现（例如 Qwen3-MoE）进行基准测试"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:166
|
||||
msgid "*Key Implementation Notes:*"
|
||||
msgstr "*关键实现说明：*"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:168
|
||||
msgid "Preserve existing interface contracts in abstract classes"
|
||||
msgstr "保留抽象类中的现有接口契约"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:169
|
||||
msgid "Use decorators for non-intrusive patch integration"
|
||||
msgstr "使用装饰器实现非侵入式补丁集成"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:170
|
||||
msgid "Leverage `eplb_utils.py` for shared expert mapping operations"
|
||||
msgstr "利用 `eplb_utils.py` 进行共享专家映射操作"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:172
|
||||
msgid "DFX"
|
||||
msgstr "DFX"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:174
|
||||
msgid "Parameter Validation"
|
||||
msgstr "参数校验"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:176
|
||||
msgid "Integer Parameters"
|
||||
msgstr "整数参数"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:178
|
||||
msgid ""
|
||||
"All integer input parameters must explicitly specify their maximum and "
|
||||
"minimum values and be subject to valid value validation. For example, "
|
||||
"`expert_heat_collection_interval` must be greater than 0:"
|
||||
msgstr ""
|
||||
"所有整数输入参数必须显式指定其最大值和最小值，并进行有效值校验。例如，`expert_heat_collection_interval` 必须大于 0："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:193
|
||||
msgid "File Path"
|
||||
msgstr "文件路径"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:195
|
||||
msgid ""
|
||||
"The file path for EPLB must be checked for legality, such as whether the "
|
||||
"file path is valid and whether it has appropriate read and write "
|
||||
"permissions. For example:"
|
||||
msgstr "EPLB 的文件路径必须进行合法性检查，例如文件路径是否有效以及是否具有适当的读写权限。例如："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:221
|
||||
msgid "Function Specifications"
|
||||
msgstr "函数规范"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:223
|
||||
msgid "Initialization Function"
|
||||
msgstr "初始化函数"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:225
|
||||
msgid ""
|
||||
"All EPLB parameters must be initialized by default during initialization, "
|
||||
"with specified parameter types and default values for proper handling."
|
||||
msgstr "所有 EPLB 参数必须在初始化时进行默认初始化，并指定参数类型和默认值以便正确处理。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:227
|
||||
msgid "General Functions"
|
||||
msgstr "通用函数"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:229
|
||||
msgid ""
|
||||
"All method arguments must specify parameter types and default values, and "
|
||||
"functions must include default return value handling for default arguments. "
|
||||
"It is recommended to use `try-except` blocks to handle the function body, "
|
||||
"specifying the type of exception captured and the failure handling (e.g., "
|
||||
"logging exceptions or returning a failure status)."
|
||||
msgstr ""
|
||||
"所有方法参数必须指定参数类型和默认值，并且函数必须包含默认参数的默认返回值处理。建议使用 `try-except` "
|
||||
"块处理函数体，指定捕获的异常类型和失败处理方式（例如记录异常或返回失败状态）。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:231
|
||||
msgid "Consistency"
|
||||
msgstr "一致性"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:233
|
||||
msgid "Expert Map"
|
||||
msgstr "专家映射"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:235
|
||||
msgid ""
|
||||
"The expert map must be globally unique during initialization and update. In "
|
||||
"a multi-node scenario during initialization, distributed communication "
|
||||
"should be used to verify the consistency of expert maps across each rank. If"
|
||||
" they are inconsistent, the user should be notified of which ranks have "
|
||||
"inconsistent maps. During the update process, if only a few layers or the "
|
||||
"expert table of a certain rank has been changed, the updated expert table "
|
||||
"must be synchronized with the EPLB's context to ensure global consistency."
|
||||
msgstr ""
|
||||
"专家映射在初始化和更新期间必须全局唯一。在初始化时的多节点场景中，应使用分布式通信来验证每个 rank 上专家映射的一致性。如果不一致，应通知用户哪些 "
|
||||
"rank 的映射不一致。在更新过程中，如果仅更改了少数层或某个 rank 的专家表，则必须将更新后的专家表与 EPLB 的上下文同步，以确保全局一致性。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:238
|
||||
msgid "Expert Weight"
|
||||
msgstr "专家权重"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:240
|
||||
msgid ""
|
||||
"When updating expert weights, ensure that the memory allocated for the "
|
||||
"expert weights has been released, or that the expert (referring to the old "
|
||||
"version) is no longer in use."
|
||||
msgstr "更新专家权重时，确保已释放为专家权重分配的内存，或者该专家（指旧版本）不再使用。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:242
|
||||
msgid "Limitations"
|
||||
msgstr "限制"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/eplb_swift_balancer.md:244
|
||||
msgid ""
|
||||
"Before using EPLB, start the script and add `export DYNAMIC_EPLB=\"true\"`. "
|
||||
"Before performing load data collection (or performance data collection), "
|
||||
"start the script and add `export EXPERT_MAP_RECORD=\"true\"`."
|
||||
msgstr ""
|
||||
"使用 EPLB 前，启动脚本并添加 `export DYNAMIC_EPLB=\"true\"`。在执行负载数据收集（或性能数据收集）前，启动脚本并添加"
|
||||
" `export EXPERT_MAP_RECORD=\"true\"`。"
|
||||
@@ -0,0 +1,33 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/index.md:1
|
||||
#: ../../source/developer_guide/Design_Documents/index.md:5
|
||||
msgid "Design Documents"
|
||||
msgstr "设计文档"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/index.md:3
|
||||
msgid ""
|
||||
"This section provides an overview of the features implemented in vLLM "
|
||||
"Ascend. Developers can refer to this guide to understand how vLLM Ascend "
|
||||
"works."
|
||||
msgstr "本节概述了 vLLM Ascend 中实现的功能。开发者可参考本指南了解 vLLM Ascend 的工作原理。"
|
||||
@@ -0,0 +1,236 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-08-04 01:39+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:1
|
||||
msgid "Npugraph_ex"
|
||||
msgstr "Npugraph_ex"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:3
|
||||
msgid "How Does It Work?"
|
||||
msgstr "工作原理"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:5
|
||||
msgid ""
|
||||
"This is an optimization based on FX graphs, which can be considered an "
|
||||
"acceleration solution for the aclgraph mode."
|
||||
msgstr "这是一种基于FX图的优化，可视为aclgraph模式的加速方案。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:7
|
||||
msgid "You can get its code [code](https://gitcode.com/Ascend/torchair)"
|
||||
msgstr "您可获取其代码[代码](https://gitcode.com/Ascend/torchair)"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:10
|
||||
#, python-brace-format
|
||||
msgid ""
|
||||
"Atlas 300I DUO and Atlas 200I Pro do not support `enable_npugraph_ex`. Set "
|
||||
"--additional-config '{\"ascend_compilation_config\": "
|
||||
"{\"enable_npugraph_ex\":false}}'."
|
||||
msgstr ""
|
||||
"Atlas 300I DUO 和 Atlas 200I Pro 不支持 `enable_npugraph_ex`。请设置 --additional-"
|
||||
"config '{\"ascend_compilation_config\": {\"enable_npugraph_ex\":false}}'。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:13
|
||||
msgid "Default FX Graph Optimization"
|
||||
msgstr "默认FX图优化"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:15
|
||||
msgid "FX Graph pass"
|
||||
msgstr "FX图优化Pass"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:17
|
||||
msgid ""
|
||||
"For the intermediate nodes of the model, replace the non-in-place operators "
|
||||
"contained in the nodes with in-place operators to reduce memory movement "
|
||||
"during computation and improve performance."
|
||||
msgstr "对于模型的中间节点，将节点中包含的非就地运算符替换为就地运算符，以减少计算过程中的内存搬运，提升性能。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:18
|
||||
msgid ""
|
||||
"For the original input parameters of the model, if they include in-place "
|
||||
"operators, Dynamo's Functionalize process will replace the in-place "
|
||||
"operators with a form of non-in-place operators + copy operators. "
|
||||
"npugraph_ex will reverse this process, restoring the in-place operators and "
|
||||
"reducing memory movement."
|
||||
msgstr ""
|
||||
"对于模型的原始输入参数，若包含就地运算符，Dynamo的Functionalize过程会将其替换为非就地运算符+拷贝运算符的形式。npugraph_ex将逆转此过程，恢复就地运算符，减少内存搬运。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:20
|
||||
msgid "FX fusion pass"
|
||||
msgstr "FX融合Pass"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:22
|
||||
msgid ""
|
||||
"npugraph_ex now provides some operator fusion passes, and more will be added"
|
||||
" in the future."
|
||||
msgstr "npugraph_ex目前已提供部分算子融合Pass，未来将持续增加。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:24
|
||||
msgid ""
|
||||
"Operator combinations that meet the replacement rules can be replaced with "
|
||||
"the corresponding fused operators."
|
||||
msgstr "满足替换规则的算子组合可替换为对应的融合算子。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:26
|
||||
msgid ""
|
||||
"You can get the default [fusion pass "
|
||||
"list](https://www.hiascend.com/document/detail/zh/Pytorch/2600/modthirdparty/torchairuseguide/docs/zh/npugraph_ex/basic/pattern_fusion_pass.md#功能简介)"
|
||||
msgstr ""
|
||||
"您可获取默认[融合Pass列表](https://www.hiascend.com/document/detail/zh/Pytorch/2600/modthirdparty/torchairuseguide/docs/zh/npugraph_ex/basic/pattern_fusion_pass.md#功能简介)"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:28
|
||||
msgid "Custom fusion pass"
|
||||
msgstr "自定义融合Pass"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:30
|
||||
msgid ""
|
||||
"Users can register a custom graph fusion pass in npugraph_ex to modify "
|
||||
"PyTorch FX graphs. The registration relies on the register_replacement API."
|
||||
msgstr ""
|
||||
"用户可在npugraph_ex中注册自定义图融合Pass，以修改PyTorch FX图。注册依赖于register_replacement API。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:32
|
||||
msgid "Below is the declaration of this API and a demo of its usage."
|
||||
msgstr "以下是该API的声明及使用示例。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "Parameter Name"
|
||||
msgstr "参数名"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "Input/Output"
|
||||
msgstr "输入/输出"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "Explanation"
|
||||
msgstr "说明"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "Is necessary"
|
||||
msgstr "是否必填"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "search_fn"
|
||||
msgstr "search_fn"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "Input"
|
||||
msgstr "输入"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid ""
|
||||
"This function is the operator combination or calculation logic that you want"
|
||||
" to recognize in the FX graph, such as the operator combination that needs "
|
||||
"to be fused"
|
||||
msgstr "该函数为您希望在FX图中识别的算子组合或计算逻辑，例如需要融合的算子组合"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "Yes"
|
||||
msgstr "是"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "replace_fn"
|
||||
msgstr "replace_fn"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid ""
|
||||
"When the combination corresponding to search_fn is found in the target "
|
||||
"graph, this function's computation logic will replace the original subgraph "
|
||||
"to achieve operator fusion or optimization."
|
||||
msgstr "当在目标图中找到search_fn对应的组合时，此函数的计算逻辑将替换原子图，实现算子融合或优化。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "example_inputs"
|
||||
msgstr "example_inputs"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid ""
|
||||
"Example input tensors used to track search_fn and replace_fn. The shape and "
|
||||
"dtype of the input should match the actual scenario."
|
||||
msgstr "用于追踪search_fn和replace_fn的示例输入张量。输入的shape和dtype应与实际场景匹配。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "trace_fn"
|
||||
msgstr "trace_fn"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid ""
|
||||
"By default, only the forward computation graph is tracked, which is suitable"
|
||||
" for optimization during the inference phase; if training scenarios need to "
|
||||
"be supported, a function that supports backward tracking can be provided."
|
||||
msgstr "默认仅追踪前向计算图，适用于推理阶段的优化；如需支持训练场景，可提供支持反向追踪的函数。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "No"
|
||||
msgstr "否"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "extra_check"
|
||||
msgstr "extra_check"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid ""
|
||||
"Find the extra verification function after operator fusion. The function's "
|
||||
"input parameter must be a Match object from torch._inductor.pattern_matcher,"
|
||||
" and it is used for further custom checks on the matching result, such as "
|
||||
"checking whether the fused operators are on the same stream, checking the "
|
||||
"device type, checking the input shapes, and so on."
|
||||
msgstr ""
|
||||
"算子融合后的额外校验函数。该函数的输入参数必须为torch._inductor.pattern_matcher中的Match对象，用于对匹配结果进行进一步自定义检查，例如检查融合算子是否在同一流上、检查设备类型、检查输入形状等。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid "search_fn_pattern"
|
||||
msgstr "search_fn_pattern"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:9
|
||||
msgid ""
|
||||
"A custom pattern object is generally unnecessary to provide. Its definition "
|
||||
"follows the rules of the native PyTorch MultiOutputPattern object. After "
|
||||
"passing this parameter, search_fn will no longer be used to match operator "
|
||||
"combinations; instead, this parameter will be used directly as the matching "
|
||||
"rule."
|
||||
msgstr ""
|
||||
"自定义模式对象，通常无需提供。其定义遵循原生PyTorch "
|
||||
"MultiOutputPattern对象的规则。传入此参数后，search_fn将不再用于匹配算子组合，而是直接使用此参数作为匹配规则。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:47
|
||||
msgid "Usage Example"
|
||||
msgstr "使用示例"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:101
|
||||
msgid ""
|
||||
"The default fusion pass in npugraph_ex is also implemented based on this "
|
||||
"API. You can see more examples of using this API in the vllm-ascend and "
|
||||
"npugraph_ex code repositories."
|
||||
msgstr ""
|
||||
"npugraph_ex中的默认融合Pass也基于此API实现。您可在vllm-ascend和npugraph_ex代码仓库中查看更多使用此API的示例。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:103
|
||||
msgid "DFX"
|
||||
msgstr "DFX"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/npugraph_ex.md:105
|
||||
msgid ""
|
||||
"By reusing the TORCH_COMPILE_DEBUG environment variable from the PyTorch "
|
||||
"community, when TORCH_COMPILE_DEBUG=1 is set, it will output the FX graphs "
|
||||
"throughout the entire process."
|
||||
msgstr ""
|
||||
"通过复用PyTorch社区的TORCH_COMPILE_DEBUG环境变量，当设置TORCH_COMPILE_DEBUG=1时，将输出全流程的FX图。"
|
||||
@@ -0,0 +1,221 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:1
|
||||
msgid "Patch in vLLM Ascend"
|
||||
msgstr "vLLM Ascend 中的补丁"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:3
|
||||
msgid ""
|
||||
"vLLM Ascend is a platform plugin for vLLM. Due to the different release "
|
||||
"cycle of vLLM and vLLM Ascend and their hardware limitations, we need to "
|
||||
"patch some code in vLLM to make it compatible with vLLM Ascend."
|
||||
msgstr ""
|
||||
"vLLM Ascend 是 vLLM 的平台插件。由于 vLLM 和 vLLM Ascend 的发布周期不同以及硬件限制，我们需要对 vLLM "
|
||||
"中的部分代码打补丁，使其与 vLLM Ascend 兼容。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:5
|
||||
msgid ""
|
||||
"In vLLM Ascend code, we provide a patch module `vllm_ascend/patch` to adapt "
|
||||
"to changes in vLLM."
|
||||
msgstr "在 vLLM Ascend 代码中，我们提供了一个补丁模块 `vllm_ascend/patch` 来适配 vLLM 的变更。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:7
|
||||
msgid "Principle"
|
||||
msgstr "原则"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:9
|
||||
msgid ""
|
||||
"We should keep in mind that Patch is not the best way to make vLLM Ascend "
|
||||
"compatible. It's just a temporary solution. The best way is to contribute "
|
||||
"the change to vLLM to make it compatible with vLLM Ascend initially. In vLLM"
|
||||
" Ascend, we have the basic principle for Patch strategy:"
|
||||
msgstr ""
|
||||
"我们应该牢记，补丁并非使 vLLM Ascend 兼容的最佳方式，它只是一个临时解决方案。最佳方式是将变更贡献给 vLLM，使其从一开始就与 vLLM "
|
||||
"Ascend 兼容。在 vLLM Ascend 中，我们对于补丁策略有以下基本原则："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:11
|
||||
msgid "Less is more. Please do not patch unless it's the only way currently."
|
||||
msgstr "少即是多。除非是当前唯一的方法，否则请不要打补丁。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:12
|
||||
msgid ""
|
||||
"Once a patch is added, it's required to describe the future plan for "
|
||||
"removing the patch."
|
||||
msgstr "一旦添加了补丁，就需要描述未来移除该补丁的计划。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:13
|
||||
msgid "Anytime, cleaning the patch code is welcome."
|
||||
msgstr "随时欢迎清理补丁代码。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:15
|
||||
msgid "How it works"
|
||||
msgstr "工作原理"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:17
|
||||
msgid "In `vllm_ascend/patch`, you can see the code structure as follows:"
|
||||
msgstr "在 `vllm_ascend/patch` 中，你可以看到如下代码结构："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:28
|
||||
msgid ""
|
||||
"**platform**: The patch code in this directory is for patching the code in "
|
||||
"vLLM Main process. It's called by "
|
||||
"`vllm_ascend/platform::NPUPlatform::pre_register_and_update` very early when"
|
||||
" vLLM is initialized."
|
||||
msgstr ""
|
||||
"**platform**：此目录中的补丁代码用于修补 vLLM 主进程中的代码。它在 vLLM 初始化时由 "
|
||||
"`vllm_ascend/platform::NPUPlatform::pre_register_and_update` 很早就调用。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:29
|
||||
msgid ""
|
||||
"For online mode, vLLM process calls the platform patch in "
|
||||
"`vllm/vllm/engine/arg_utils.py::AsyncEngineArgs.add_cli_args` when parsing "
|
||||
"the CLI args."
|
||||
msgstr ""
|
||||
"对于在线模式，vLLM 进程在解析 CLI 参数时调用 "
|
||||
"`vllm/vllm/engine/arg_utils.py::AsyncEngineArgs.add_cli_args` 中的平台补丁。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:30
|
||||
msgid ""
|
||||
"For offline mode, vLLM process calls the platform patch in "
|
||||
"`vllm/vllm/engine/arg_utils.py::EngineArgs.create_engine_config` when "
|
||||
"parsing the input parameters."
|
||||
msgstr ""
|
||||
"对于离线模式，vLLM 进程在解析输入参数时，会在 "
|
||||
"`vllm/vllm/engine/arg_utils.py::EngineArgs.create_engine_config` 中调用平台补丁。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:31
|
||||
msgid ""
|
||||
"**worker**: The patch code in this directory is for patching the code in "
|
||||
"vLLM worker process. It's called by "
|
||||
"`vllm_ascend/worker/worker::NPUWorker::__init__` when the vLLM Worker "
|
||||
"process is initialized."
|
||||
msgstr ""
|
||||
"**worker**：此目录中的补丁代码用于修补 vLLM 工作进程中的代码。它在 vLLM 工作进程初始化时由 "
|
||||
"`vllm_ascend/worker/worker::NPUWorker::__init__` 调用。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:32
|
||||
msgid ""
|
||||
"For both online and offline mode, vLLM EngineCore process calls the worker "
|
||||
"patch in `vllm/vllm/worker/worker_base.py::WorkerWrapperBase.init_worker` "
|
||||
"when initializing the worker process."
|
||||
msgstr ""
|
||||
"对于在线和离线模式，vLLM EngineCore 进程在初始化工作进程时调用 "
|
||||
"`vllm/vllm/worker/worker_base.py::WorkerWrapperBase.init_worker` 中的工作补丁。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:34
|
||||
msgid "How to write a patch"
|
||||
msgstr "如何编写补丁"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:36
|
||||
msgid ""
|
||||
"Before writing a patch, following the principle above, we should patch the "
|
||||
"least code. If it's necessary, we can patch the code in either **platform** "
|
||||
"or **worker** folder. Here is an example to patch `distributed` module in "
|
||||
"vLLM."
|
||||
msgstr ""
|
||||
"在编写补丁之前，遵循上述原则，我们应该修补最少的代码。如果有必要，我们可以在 **platform** 或 **worker** "
|
||||
"文件夹中修补代码。以下是一个修补 vLLM 中 `distributed` 模块的示例。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:38
|
||||
msgid ""
|
||||
"Decide which version of vLLM we should patch. For example, after analysis, "
|
||||
"here we want to patch both `0.10.0` and `main` of vLLM."
|
||||
msgstr "决定我们应该修补哪个版本的 vLLM。例如，经过分析，这里我们想要修补 vLLM 的 `0.10.0` 和 `main` 版本。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:39
|
||||
msgid ""
|
||||
"Decide which process we should patch. For example, here `distributed` "
|
||||
"belongs to the vLLM main process, so we should patch `platform`."
|
||||
msgstr "决定我们应该修补哪个进程。例如，这里的 `distributed` 属于 vLLM 主进程，所以我们应该修补 `platform`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:40
|
||||
#, python-brace-format
|
||||
msgid ""
|
||||
"Create the patch file in the right folder. The file should be named as "
|
||||
"`patch_{module_name}.py`. The example here is "
|
||||
"`vllm_ascend/patch/platform/patch_distributed.py`."
|
||||
msgstr ""
|
||||
"在正确的文件夹中创建补丁文件。文件应命名为 `patch_{module_name}.py`。此处的示例是 "
|
||||
"`vllm_ascend/patch/platform/patch_distributed.py`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:41
|
||||
msgid "Write your patch code in the new file. Here is an example:"
|
||||
msgstr "在新文件中编写你的补丁代码。示例如下："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:53
|
||||
msgid ""
|
||||
"Import the patch file in `__init__.py`. In this example, add `import "
|
||||
"vllm_ascend.patch.platform.patch_distributed` into "
|
||||
"`vllm_ascend/patch/platform/__init__.py`."
|
||||
msgstr ""
|
||||
"在 `__init__.py` 中导入补丁文件。在此示例中，将 `import "
|
||||
"vllm_ascend.patch.platform.patch_distributed` 添加到 "
|
||||
"`vllm_ascend/patch/platform/__init__.py` 中。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:54
|
||||
msgid ""
|
||||
"Add the description of the patch in `vllm_ascend/patch/__init__.py`. The "
|
||||
"description format is as follows:"
|
||||
msgstr "在 `vllm_ascend/patch/__init__.py` 中添加补丁的描述。描述格式如下："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:70
|
||||
msgid ""
|
||||
"Add the Unit Test and E2E Test. Any newly added code in vLLM Ascend should "
|
||||
"contain the Unit Test and E2E Test as well. You can find more details in "
|
||||
"[test guide](../contribution/testing.md)"
|
||||
msgstr ""
|
||||
"添加单元测试和端到端测试。vLLM Ascend "
|
||||
"中任何新添加的代码都应包含单元测试和端到端测试。你可以在[测试指南](../contribution/testing.md)中找到更多详细信息。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:72
|
||||
msgid "Limitations"
|
||||
msgstr "限制"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:74
|
||||
msgid ""
|
||||
"In V1 Engine, vLLM starts three kinds of processes: Main process, EngineCore"
|
||||
" process and Worker process. Now vLLM Ascend can only patch the code in Main"
|
||||
" process and Worker process by default. If you want to patch the code "
|
||||
"running in EngineCore process, you should patch EngineCore process entirely "
|
||||
"during setup. Find the entire code in `vllm.v1.engine.core`. Please override"
|
||||
" `EngineCoreProc` and `DPEngineCoreProc` entirely."
|
||||
msgstr ""
|
||||
"在 V1 引擎中，vLLM 启动三种进程：主进程、EngineCore 进程和工作进程。目前 vLLM Ascend "
|
||||
"默认只能修补主进程和工作进程中的代码。如果你想修补在 EngineCore 进程中运行的代码，你需要在设置期间完整地修补 EngineCore 进程。在"
|
||||
" `vllm.v1.engine.core` 中找到完整代码。请完整地重写 `EngineCoreProc` 和 `DPEngineCoreProc`。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/patch.md:75
|
||||
msgid ""
|
||||
"If you are running edited vLLM code, the version of vLLM may be changed "
|
||||
"automatically. For example, if you run the edited vLLM based on v0.9.n, the "
|
||||
"version of vLLM may be changed to v0.9.nxxx. In this case, the patch for "
|
||||
"v0.9.n in vLLM Ascend would not work as expected, because vLLM Ascend can't "
|
||||
"distinguish the version of the vLLM you're using. In this case, you can set "
|
||||
"the environment variable `VLLM_VERSION` to specify the version of the vLLM "
|
||||
"you're using, and then the patch for that version (e.g., v0.9.n) should "
|
||||
"work."
|
||||
msgstr ""
|
||||
"如果你运行的是经过编辑的 vLLM 代码，vLLM 的版本可能会自动更改。例如，如果你运行基于 v0.9.n 编辑的 vLLM，vLLM "
|
||||
"的版本可能会变为 v0.9.nxxx。在这种情况下，vLLM Ascend 中针对 v0.9.n 的补丁将无法按预期工作，因为 vLLM Ascend "
|
||||
"无法区分你正在使用的 vLLM 版本。在这种情况下，你可以设置环境变量 `VLLM_VERSION` 来指定你正在使用的 vLLM "
|
||||
"版本，然后针对该版本（例如 v0.9.n）的补丁应该可以正常工作。"
|
||||
@@ -0,0 +1,382 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:1
|
||||
msgid "Quantization Adaptation Guide"
|
||||
msgstr "量化适配指南"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:3
|
||||
msgid ""
|
||||
"This document provides guidance for adapting quantization algorithms and "
|
||||
"models related to **ModelSlim**."
|
||||
msgstr "本文档提供与 **ModelSlim** 相关的量化算法和模型适配的指导。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:5
|
||||
msgid "Quantization Feature Introduction"
|
||||
msgstr "量化特性介绍"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:7
|
||||
msgid "Quantization Inference Process"
|
||||
msgstr "量化推理流程"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:9
|
||||
msgid ""
|
||||
"The current process for registering and obtaining quantization methods in "
|
||||
"vLLM Ascend is as follows:"
|
||||
msgstr "vLLM Ascend 中注册和获取量化方法的当前流程如下："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:11
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:11
|
||||
msgid "get_quant_method"
|
||||
msgstr "get_quant_method"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:13
|
||||
msgid ""
|
||||
"vLLM Ascend registers a custom Ascend quantization method. By configuring "
|
||||
"the `--quantization ascend` parameter (or `quantization=\"ascend\"` for "
|
||||
"offline), the quantization feature is enabled. When constructing the "
|
||||
"`quant_config`, the registered `AscendModelSlimConfig` is initialized and "
|
||||
"`get_quant_method` is called to obtain the quantization method corresponding"
|
||||
" to each weight part, stored in the `quant_method` attribute."
|
||||
msgstr ""
|
||||
"vLLM Ascend 注册了自定义的 Ascend 量化方法。通过配置 `--quantization ascend` 参数（离线推理使用 "
|
||||
"`quantization=\"ascend\"`），即可启用量化功能。在构建 `quant_config` 时，会初始化已注册的 "
|
||||
"`AscendModelSlimConfig`，并调用 `get_quant_method` 获取每个权重部分对应的量化方法，存储在 "
|
||||
"`quant_method` 属性中。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:15
|
||||
msgid ""
|
||||
"Currently supported quantization methods include `AscendLinearMethod`, "
|
||||
"`AscendFusedMoEMethod`, `AscendEmbeddingMethod`, and their corresponding "
|
||||
"non-quantized methods:"
|
||||
msgstr ""
|
||||
"当前支持的量化方法包括 "
|
||||
"`AscendLinearMethod`、`AscendFusedMoEMethod`、`AscendEmbeddingMethod` "
|
||||
"及其对应的非量化方法："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:17
|
||||
msgid ""
|
||||
""
|
||||
msgstr ""
|
||||
""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:17
|
||||
msgid "quant_methods_overview"
|
||||
msgstr "quant_methods_overview"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:19
|
||||
msgid ""
|
||||
"The quantization method base class defined by vLLM and the overall call "
|
||||
"flow of quantization methods are as follows:"
|
||||
msgstr "vLLM 定义的量化方法基类及量化方法的整体调用流程如下："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:21
|
||||
msgid ""
|
||||
""
|
||||
msgstr ""
|
||||
""
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:21
|
||||
msgid "quant_method_call_flow"
|
||||
msgstr "quant_method_call_flow"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:23
|
||||
msgid ""
|
||||
"The `embedding` method is generally not implemented for quantization, "
|
||||
"focusing only on the other three methods."
|
||||
msgstr "`embedding` 方法通常不实现量化，仅关注其他三种方法。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:25
|
||||
msgid ""
|
||||
"The `create_weights` method is used for weight initialization; the "
|
||||
"`process_weights_after_loading` method is used for weight post-processing, "
|
||||
"such as transposition, format conversion, data type conversion, etc.; the "
|
||||
"`apply` method is used to perform activation quantization and quantized "
|
||||
"matrix multiplication calculations during the forward process."
|
||||
msgstr ""
|
||||
"`create_weights` 方法用于权重初始化；`process_weights_after_loading` "
|
||||
"方法用于权重后处理，例如转置、格式转换、数据类型转换等；`apply` 方法用于在前向过程中执行激活量化和量化矩阵乘法计算。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:27
|
||||
msgid ""
|
||||
"We need to implement the `create_weights`, `process_weights_after_loading`, "
|
||||
"and `apply` methods for different **layers** (**attention**, **mlp**, **MoE "
|
||||
"(Mixture of Experts)**)."
|
||||
msgstr ""
|
||||
"我们需要为不同的**层**（**attention**、**mlp**、**MoE（混合专家）**）实现 "
|
||||
"`create_weights`、`process_weights_after_loading` 和 `apply` 方法。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:29
|
||||
msgid ""
|
||||
"**Supplement**: When loading the model, the quantized model's description "
|
||||
"file **quant_model_description.json** needs to be read. This file describes "
|
||||
"the quantization configuration and parameters for each part of the model "
|
||||
"weights, for example:"
|
||||
msgstr ""
|
||||
"**补充**：加载模型时，需要读取量化模型的描述文件 "
|
||||
"**quant_model_description.json**。该文件描述了模型各权重部分的量化配置和参数，例如："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:49
|
||||
msgid ""
|
||||
"Based on the above content, we present a brief description of the adaptation"
|
||||
" process for quantization algorithms and quantized models."
|
||||
msgstr "基于以上内容，我们简要描述量化算法和量化模型的适配过程。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:51
|
||||
msgid "Quantization Algorithm Adaptation"
|
||||
msgstr "量化算法适配"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:53
|
||||
msgid ""
|
||||
"**Step 1: Algorithm Design**. Define the algorithm ID (e.g., "
|
||||
"`W4A8_DYNAMIC`), determine supported layers (linear, moe, attention), and "
|
||||
"design the quantization scheme (static/dynamic, "
|
||||
"pertensor/perchannel/pergroup)."
|
||||
msgstr ""
|
||||
"**步骤1：算法设计**。定义算法ID（例如 "
|
||||
"`W4A8_DYNAMIC`），确定支持的层（linear、moe、attention），并设计量化方案（静态/动态、pertensor/perchannel/pergroup）。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:54
|
||||
msgid ""
|
||||
"**Step 2: Registration**. Use the `@register_scheme` decorator in "
|
||||
"`vllm_ascend/quantization/methods/registry.py` to register your quantization"
|
||||
" scheme class."
|
||||
msgstr ""
|
||||
"**步骤2：注册**。在 `vllm_ascend/quantization/methods/registry.py` 中使用 "
|
||||
"`@register_scheme` 装饰器注册你的量化方案类。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:68
|
||||
msgid ""
|
||||
"**Step 3: Implementation**. Create an algorithm implementation file, such as"
|
||||
" `vllm_ascend/quantization/methods/w4a8.py`, and implement the method class "
|
||||
"and logic."
|
||||
msgstr ""
|
||||
"**步骤3：实现**。创建算法实现文件，例如 `vllm_ascend/quantization/methods/w4a8.py`，并实现方法类和逻辑。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:69
|
||||
msgid ""
|
||||
"**Step 4: Testing**. Use your algorithm to generate quantization "
|
||||
"configurations and verify correctness and performance on target models and "
|
||||
"hardware."
|
||||
msgstr "**步骤4：测试**。使用你的算法生成量化配置，并在目标模型和硬件上验证正确性和性能。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:71
|
||||
msgid "Quantized Model Adaptation"
|
||||
msgstr "量化模型适配"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:73
|
||||
msgid ""
|
||||
"Adapting a new quantized model requires ensuring the following three points:"
|
||||
msgstr "适配新的量化模型需要确保以下三点："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:75
|
||||
msgid "The original model has been successfully adapted in `vLLM Ascend`."
|
||||
msgstr "原始模型已在 `vLLM Ascend` 中成功适配。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:76
|
||||
msgid ""
|
||||
"**Fused Module Mapping**: Add the model's `model_type` to "
|
||||
"`packed_modules_model_mapping` in "
|
||||
"`vllm_ascend/quantization/modelslim_config.py` (e.g., `qkv_proj`, "
|
||||
"`gate_up_proj`, `experts`) to ensure sharding consistency and correct "
|
||||
"loading."
|
||||
msgstr ""
|
||||
"**融合模块映射**：将模型的 `model_type` 添加到 "
|
||||
"`vllm_ascend/quantization/modelslim_config.py` 的 "
|
||||
"`packed_modules_model_mapping` 中（例如 "
|
||||
"`qkv_proj`、`gate_up_proj`、`experts`），以确保分片一致性和正确加载。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:96
|
||||
msgid ""
|
||||
"All quantization algorithms used by the quantized model have been integrated"
|
||||
" into the `quantization` module."
|
||||
msgstr "量化模型使用的所有量化算法都已集成到 `quantization` 模块中。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:98
|
||||
msgid "Currently Supported Quantization Algorithms"
|
||||
msgstr "当前支持的量化算法"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:100
|
||||
msgid ""
|
||||
"vLLM Ascend supports multiple quantization algorithms. The following table "
|
||||
"provides an overview of each quantization algorithm based on the "
|
||||
"implementation in the `vllm_ascend.quantization` module:"
|
||||
msgstr "vLLM Ascend 支持多种量化算法。下表基于 `vllm_ascend.quantization` 模块的实现，概述了每种量化算法："
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Algorithm"
|
||||
msgstr "算法"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Weight"
|
||||
msgstr "权重"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Activation"
|
||||
msgstr "激活值"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Weight Granularity"
|
||||
msgstr "权重粒度"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Activation Granularity"
|
||||
msgstr "激活值粒度"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Type"
|
||||
msgstr "类型"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Description"
|
||||
msgstr "描述"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "`W4A16`"
|
||||
msgstr "`W4A16`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "INT4"
|
||||
msgstr "INT4"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "FP16/BF16"
|
||||
msgstr "FP16/BF16"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Per-Group"
|
||||
msgstr "Per-Group"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Per-Tensor"
|
||||
msgstr "Per-Tensor"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Static"
|
||||
msgstr "静态"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid ""
|
||||
"4-bit weight quantization with 16-bit activation precision, specifically "
|
||||
"designed for MoE model expert layers, supporting int32 format weight packing"
|
||||
msgstr "4位权重量化与16位激活精度，专为MoE模型专家层设计，支持int32格式权重打包"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "`W8A16`"
|
||||
msgstr "`W8A16`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "INT8"
|
||||
msgstr "INT8"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Per-Channel"
|
||||
msgstr "Per-Channel"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid ""
|
||||
"8-bit weight quantization with 16-bit activation precision, balancing "
|
||||
"accuracy and performance, suitable for linear layers"
|
||||
msgstr "8位权重量化与16位激活精度，平衡精度与性能，适用于线性层"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "`W8A8`"
|
||||
msgstr "`W8A8`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid ""
|
||||
"Static activation quantization, suitable for scenarios requiring high "
|
||||
"precision"
|
||||
msgstr "静态激活量化，适用于需要高精度的场景"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "`W8A8_DYNAMIC`"
|
||||
msgstr "`W8A8_DYNAMIC`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Per-Token"
|
||||
msgstr "Per-Token"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Dynamic"
|
||||
msgstr "动态"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid ""
|
||||
"Dynamic activation quantization with per-token scaling factor calculation"
|
||||
msgstr "动态激活量化，按token计算缩放因子"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "`W4A8_DYNAMIC`"
|
||||
msgstr "`W4A8_DYNAMIC`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid ""
|
||||
"Supports both direct per-channel quantization to 4-bit and two-step "
|
||||
"quantization (per-channel to 8-bit then per-group to 4-bit)"
|
||||
msgstr "支持直接按通道量化为4位，以及两步量化（先按通道量化为8位，再按组量化为4位）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "`W4A4_FLATQUANT_DYNAMIC`"
|
||||
msgstr "`W4A4_FLATQUANT_DYNAMIC`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid ""
|
||||
"Uses FlatQuant for activation distribution smoothing before 4-bit dynamic "
|
||||
"quantization, with additional matrix multiplications for precision "
|
||||
"preservation"
|
||||
msgstr "使用FlatQuant在4位动态量化前对激活分布进行平滑处理，并通过额外矩阵乘法保持精度"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "`W8A8_MIX`"
|
||||
msgstr "`W8A8_MIX`"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Per-Tensor/Token"
|
||||
msgstr "按张量/按Token"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid "Mixed"
|
||||
msgstr "混合"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md
|
||||
msgid ""
|
||||
"We support two deployment modes: PD Colocation (dynamic quantization for "
|
||||
"both P and D) and PD Disaggregation (dynamic-quant P and static-quant D)"
|
||||
msgstr "我们支持两种部署模式：PD共置（P和D均使用动态量化）和PD分离（P使用动态量化，D使用静态量化）"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:112
|
||||
msgid ""
|
||||
"**Static vs Dynamic:** Static quantization uses pre-computed scaling factors"
|
||||
" with better performance, while dynamic quantization computes scaling "
|
||||
"factors on-the-fly for each token/activation tensor with higher precision."
|
||||
msgstr ""
|
||||
"**静态量化 vs 动态量化：** 静态量化使用预计算的缩放因子，性能更优；动态量化则针对每个token/激活张量实时计算缩放因子，精度更高。"
|
||||
|
||||
#: ../../source/developer_guide/Design_Documents/quantization.md:114
|
||||
msgid ""
|
||||
"**Granularity:** Refers to the scope of scaling factor computation (e.g., "
|
||||
"per-tensor, per-channel, per-group)."
|
||||
msgstr "**粒度：** 指缩放因子计算的范围（例如按张量、按通道、按组）。"
|
||||
@@ -0,0 +1,540 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:1
|
||||
msgid "Doc writing guide"
|
||||
msgstr "文档编写指南"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:3
|
||||
msgid "Guide to Writing Model Tutorial Doc"
|
||||
msgstr "模型教程文档编写指南"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:5
|
||||
msgid ""
|
||||
"`docs/source/_templates/Model-Deployment-Tutorial-Template.md` is a template"
|
||||
" for writing model deployment tutorials. You can copy and modify it to "
|
||||
"create new docs."
|
||||
msgstr ""
|
||||
"`docs/source/_templates/Model-Deployment-Tutorial-Template.md` "
|
||||
"是编写模型部署教程的模板。您可以复制并修改它以创建新文档。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:7
|
||||
msgid "Testable doc code block generation (``model-code``)"
|
||||
msgstr "可测试文档代码块生成（``model-code``）"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:9
|
||||
msgid ""
|
||||
"For **documentation authors**: how to insert testable command blocks into "
|
||||
"docs"
|
||||
msgstr "面向**文档作者**：如何在文档中插入可测试的命令块"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:10
|
||||
msgid "For **developers**: how to add a new converter"
|
||||
msgstr "面向**开发者**：如何添加新的转换器"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:12
|
||||
msgid "Built-in supported `converter_tag` values:"
|
||||
msgstr "内置支持的 `converter_tag` 值："
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "converter_tag"
|
||||
msgstr "converter_tag"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "Renders"
|
||||
msgstr "渲染内容"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "YAML source"
|
||||
msgstr "YAML 源"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "`single_node`"
|
||||
msgstr "`single_node`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "A single node's env exports + `vllm serve` script"
|
||||
msgstr "单个节点的环境变量导出 + `vllm serve` 脚本"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "`test_cases[case_index]`"
|
||||
msgstr "`test_cases[case_index]`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "`multi_node`"
|
||||
msgstr "`multi_node`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "One host's env exports + `vllm serve` script"
|
||||
msgstr "单个主机的环境变量导出 + `vllm serve` 脚本"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "`deployment[host_index]`"
|
||||
msgstr "`deployment[host_index]`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "`external_dp_template`"
|
||||
msgstr "`external_dp_template`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "One external-DP node's env exports + `vllm serve` command"
|
||||
msgstr "单个外部 DP 节点的环境变量导出 + `vllm serve` 命令"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "`templates[host_index]`"
|
||||
msgstr "`templates[host_index]`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "`external_dp_launch`"
|
||||
msgstr "`external_dp_launch`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "One `launch_online_dp.py` line per node"
|
||||
msgstr "每个节点一行 `launch_online_dp.py`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "`config[]`"
|
||||
msgstr "`config[]`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "`external_dp_proxy`"
|
||||
msgstr "`external_dp_proxy`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "The load-balance proxy launch command"
|
||||
msgstr "负载均衡代理启动命令"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md
|
||||
msgid "`config[]` + `routing`"
|
||||
msgstr "`config[]` + `routing`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:22
|
||||
msgid "For authors: add a block"
|
||||
msgstr "面向作者：添加代码块"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:25
|
||||
msgid ""
|
||||
"By default, the generator scans only `.md` files under "
|
||||
"`docs/source/tutorials/models/` and produces artifacts. If you put ``model-"
|
||||
"code`` blocks in other directories, Sphinx builds will not automatically "
|
||||
"generate the corresponding scripts."
|
||||
msgstr ""
|
||||
"默认情况下，生成器仅扫描 `docs/source/tutorials/models/` 下的 `.md` 文件并生成产物。如果您在其他目录中放置 "
|
||||
"``model-code`` 代码块，Sphinx 构建将不会自动生成相应的脚本。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:29
|
||||
msgid "All ``model-code`` blocks need:"
|
||||
msgstr "所有 ``model-code`` 代码块需要："
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "Option"
|
||||
msgstr "选项"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "Required"
|
||||
msgstr "必需"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "Description"
|
||||
msgstr "描述"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "`block_name`"
|
||||
msgstr "`block_name`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "Yes"
|
||||
msgstr "是"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "Block name; must be unique within the current document"
|
||||
msgstr "代码块名称；在当前文档中必须唯一"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "`converter_tag`"
|
||||
msgstr "`converter_tag`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "Selects one of the built-in converters"
|
||||
msgstr "选择一个内置转换器"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "`test_case_path`"
|
||||
msgstr "`test_case_path`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid ""
|
||||
"Repository-relative YAML path that stays within the repo; the file must "
|
||||
"exist"
|
||||
msgstr "仓库相对路径的 YAML 路径，且必须在仓库内；文件必须存在"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:37
|
||||
msgid ""
|
||||
"Use the body of the block to add shell wrapper lines such as `set -eux`. "
|
||||
"Always place the `{{ generated }}` placeholder where the converter output "
|
||||
"should be inserted."
|
||||
msgstr ""
|
||||
"使用代码块主体添加 shell 包装行，例如 `set -eux`。始终将 `{{ generated }}` 占位符放置在转换器输出应插入的位置。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:41
|
||||
msgid "converter_tag: `single_node`"
|
||||
msgstr "converter_tag: `single_node`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:43
|
||||
msgid ""
|
||||
"`single_node` reads one item from `test_cases`. The optional `case_index` "
|
||||
"metadata selects the item; when omitted, it defaults to `0`."
|
||||
msgstr ""
|
||||
"`single_node` 从 `test_cases` 中读取一个条目。可选的 `case_index` 元数据用于选择条目；省略时默认为 `0`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:46
|
||||
msgid ""
|
||||
"Only the fields read by this converter are expanded below. Other test "
|
||||
"metadata can be left in the YAML and is ignored by this converter."
|
||||
msgstr "仅此转换器读取的字段会在下方展开。其他测试元数据可以保留在 YAML 中，此转换器会忽略它们。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:67
|
||||
msgid ""
|
||||
"`envs` is rendered as `export` lines. `SERVER_PORT: DEFAULT_PORT` is "
|
||||
"resolved to the default single-node port `8000`. `model` becomes `vllm serve"
|
||||
" <model>`, and `server_cmd` plus optional `server_cmd_extra` become command "
|
||||
"arguments. Both command fields can be either a shell string or a flat token "
|
||||
"list."
|
||||
msgstr ""
|
||||
"`envs` 渲染为 `export` 行。`SERVER_PORT: DEFAULT_PORT` 解析为默认单节点端口 `8000`。`model` "
|
||||
"变为 `vllm serve <model>`，`server_cmd` 加上可选的 `server_cmd_extra` "
|
||||
"成为命令参数。两个命令字段可以是 shell 字符串或扁平令牌列表。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:72
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:132
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:211
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:280
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:338
|
||||
msgid "Write the doc block like this:"
|
||||
msgstr "按如下方式编写文档代码块："
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:86
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:293
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:351
|
||||
msgid "Generated shell script:"
|
||||
msgstr "生成的 shell 脚本："
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:100
|
||||
msgid "converter_tag: `multi_node`"
|
||||
msgstr "converter_tag: `multi_node`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:102
|
||||
msgid ""
|
||||
"`multi_node` reads one item from `deployment`. The required `host_index` "
|
||||
"metadata selects which host to render."
|
||||
msgstr "`multi_node` 从 `deployment` 中读取一个条目。必需的 `host_index` 元数据用于选择要渲染的主机。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:129
|
||||
msgid ""
|
||||
"`server_cmd` must be a complete command starting with `vllm serve <model>`. "
|
||||
"It can be written as a shell string or a flat token list."
|
||||
msgstr ""
|
||||
"`server_cmd` 必须是完整的命令，以 `vllm serve <model>` 开头。它可以写成 shell 字符串或扁平令牌列表。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:146
|
||||
msgid "Generated shell script for `host_index: 1`:"
|
||||
msgstr "为 `host_index: 1` 生成的 shell 脚本："
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:162
|
||||
msgid "converter_tag: `external_dp_template`"
|
||||
msgstr "converter_tag: `external_dp_template`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:164
|
||||
msgid ""
|
||||
"`external_dp_template` reads one item from `templates`. The required "
|
||||
"`host_index` metadata selects which template to render. The top-level "
|
||||
"`model` field is also required because the converter builds `vllm serve "
|
||||
"<model>`."
|
||||
msgstr ""
|
||||
"`external_dp_template` 从 `templates` 中读取一个条目。必需的 `host_index` "
|
||||
"元数据用于选择要渲染的模板。顶级 `model` 字段也是必需的，因为转换器会构建 `vllm serve <model>`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:195
|
||||
msgid ""
|
||||
"Known braced template variables are rewritten to the positional shell "
|
||||
"arguments that `run_dp_template.sh` receives from `launch_online_dp.py`:"
|
||||
msgstr ""
|
||||
"已知的花括号模板变量会被重写为 `run_dp_template.sh` 从 `launch_online_dp.py` 接收的位置 shell 参数："
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "Template variable"
|
||||
msgstr "模板变量"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "Rendered positional"
|
||||
msgstr "渲染后的位置参数"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
#, python-brace-format
|
||||
msgid "`${VISIBLE_DEVICES}`"
|
||||
msgstr "`${VISIBLE_DEVICES}`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "`$1`"
|
||||
msgstr "`$1`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
#, python-brace-format
|
||||
msgid "`${PORT}`"
|
||||
msgstr "`${PORT}`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "`$2`"
|
||||
msgstr "`$2`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
#, python-brace-format
|
||||
msgid "`${DP_SIZE}`"
|
||||
msgstr "`${DP_SIZE}`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "`$3`"
|
||||
msgstr "`$3`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
#, python-brace-format
|
||||
msgid "`${DP_RANK}`"
|
||||
msgstr "`${DP_RANK}`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "`$4`"
|
||||
msgstr "`$4`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
#, python-brace-format
|
||||
msgid "`${DP_ADDRESS}`"
|
||||
msgstr "`${DP_ADDRESS}`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "`$5`"
|
||||
msgstr "`$5`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
#, python-brace-format
|
||||
msgid "`${DP_RPC_PORT}`"
|
||||
msgstr "`${DP_RPC_PORT}`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "`$6`"
|
||||
msgstr "`$6`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
#, python-brace-format
|
||||
msgid "`${TP_SIZE}`"
|
||||
msgstr "`${TP_SIZE}`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:24
|
||||
msgid "`$7`"
|
||||
msgstr "`$7`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:208
|
||||
msgid ""
|
||||
"Unknown braced variables and unbraced shell references such as "
|
||||
"`$SERVER_PORT` are left unchanged."
|
||||
msgstr "未知的花括号变量和无花括号的 shell 引用（例如 `$SERVER_PORT`）保持不变。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:230
|
||||
msgid "Generated shell script for `host_index: 0`:"
|
||||
msgstr "为 `host_index: 0` 生成的 shell 脚本："
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:253
|
||||
msgid "converter_tag: `external_dp_launch`"
|
||||
msgstr "converter_tag: `external_dp_launch`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:255
|
||||
msgid ""
|
||||
"`external_dp_launch` reads the full `config` list and renders one "
|
||||
"`launch_online_dp.py` command per node. It does not take an index option."
|
||||
msgstr ""
|
||||
"`external_dp_launch` 读取完整的 `config` 列表，并为每个节点渲染一条 `launch_online_dp.py` "
|
||||
"命令。它不接受索引选项。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:302
|
||||
msgid "converter_tag: `external_dp_proxy`"
|
||||
msgstr "converter_tag: `external_dp_proxy`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:304
|
||||
msgid ""
|
||||
"`external_dp_proxy` reads `config` and `routing`. It renders the "
|
||||
"`load_balance_proxy_server_example.py` command for `routing.type: "
|
||||
"disaggregated_prefill`. It does not take an index option."
|
||||
msgstr ""
|
||||
"`external_dp_proxy` 读取 `config` 和 `routing`。它为 `routing.type: "
|
||||
"disaggregated_prefill` 渲染 `load_balance_proxy_server_example.py` "
|
||||
"命令。它不接受索引选项。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:334
|
||||
#, python-brace-format
|
||||
msgid ""
|
||||
"`routing.groups.prefiller` and `routing.groups.decoder` contain indices into"
|
||||
" `config`. Each referenced node expands to `dp_size_local` host and port "
|
||||
"entries. The proxy itself is rendered on `${NODE_0_IP}:1999`."
|
||||
msgstr ""
|
||||
"`routing.groups.prefiller` 和 `routing.groups.decoder` 包含指向 `config` "
|
||||
"的索引。每个被引用的节点展开为 `dp_size_local` 主机和端口条目。代理本身渲染在 `${NODE_0_IP}:1999` 上。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:376
|
||||
msgid "Local debugging and generation"
|
||||
msgstr "本地调试与生成"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:378
|
||||
msgid "Generate only (without building the full site)"
|
||||
msgstr "仅生成（不构建完整站点）"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:393
|
||||
msgid ""
|
||||
"By default, artifacts are written to: "
|
||||
"`docs/_build/doc_codegen/<doc_stem>/<block_name>.sh`."
|
||||
msgstr "默认情况下，产物写入：`docs/_build/doc_codegen/<doc_stem>/<block_name>.sh`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:396
|
||||
msgid ""
|
||||
"After the script is generated, please make sure to check whether the "
|
||||
"generated content is runnable, especially key parts such as environment "
|
||||
"variables and command-line parameters."
|
||||
msgstr "脚本生成后，请务必检查生成的内容是否可运行，尤其是环境变量和命令行参数等关键部分。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:399
|
||||
msgid "Build the site & preview locally"
|
||||
msgstr "构建站点并在本地预览"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:421
|
||||
msgid "For developers: add a new converter"
|
||||
msgstr "面向开发者：添加新转换器"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:423
|
||||
msgid ""
|
||||
"A converter turns one loaded YAML file plus one parsed `ModelCodeBlock` into"
|
||||
" a `GeneratedScript`. The current pipeline is:"
|
||||
msgstr "转换器将一个已加载的YAML文件加上一个解析后的`ModelCodeBlock`转换为`GeneratedScript`。当前流程为："
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:426
|
||||
msgid ""
|
||||
"`BlockScanner` parses ``model-code`` fences and accepts only options listed "
|
||||
"in `MODEL_CODE_OPTION_NAMES`."
|
||||
msgstr "`BlockScanner`解析``model-code``围栏，仅接受`MODEL_CODE_OPTION_NAMES`中列出的选项。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:428
|
||||
msgid "`YamlLoader` loads `test_case_path`."
|
||||
msgstr "`YamlLoader`加载`test_case_path`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:429
|
||||
msgid ""
|
||||
"`get_converter()` looks up `block.converter_tag` from "
|
||||
"`build_default_converters()`."
|
||||
msgstr ""
|
||||
"`get_converter()`从`build_default_converters()`中查找`block.converter_tag`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:431
|
||||
msgid ""
|
||||
"The selected converter returns `GeneratedScript(content=..., "
|
||||
"language=\"shell\")`."
|
||||
msgstr "选定的转换器返回`GeneratedScript(content=..., language=\"shell\")`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:432
|
||||
msgid ""
|
||||
"`GeneratorService` replaces `{{ generated }}` in the block body, validates "
|
||||
"that the final script is non-empty, and writes "
|
||||
"`docs/_build/doc_codegen/<doc_stem>/<block_name>.sh`."
|
||||
msgstr ""
|
||||
"`GeneratorService`替换块体中的`{{ generated "
|
||||
"}}`，验证最终脚本非空，并写入`docs/_build/doc_codegen/<doc_stem>/<block_name>.sh`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:436
|
||||
msgid "To add a converter:"
|
||||
msgstr "要添加转换器："
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:438
|
||||
msgid ""
|
||||
"In `tools/docs_codegen/converters.py`, add a `BaseConverter` subclass with a"
|
||||
" unique `name`. That name is the value authors put in `:converter_tag:`."
|
||||
msgstr ""
|
||||
"在`tools/docs_codegen/converters.py`中，添加一个具有唯一`name`的`BaseConverter`子类。该名称是作者在`:converter_tag:`中填入的值。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:440
|
||||
msgid ""
|
||||
"Implement `convert(self, loaded_yaml, *, block) -> GeneratedScript`. Use "
|
||||
"`make_docs_codegen_error(..., block=block)` for user-facing validation "
|
||||
"errors so the CLI and Sphinx output include document context."
|
||||
msgstr ""
|
||||
"实现`convert(self, loaded_yaml, *, block) -> "
|
||||
"GeneratedScript`。对于面向用户的验证错误，使用`make_docs_codegen_error(..., "
|
||||
"block=block)`，以便CLI和Sphinx输出包含文档上下文。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:443
|
||||
msgid ""
|
||||
"Reuse helpers from `tools/docs_codegen/utils.py`, such as `require_mapping`,"
|
||||
" `require_mapping_list`, `require_scalar_mapping`, "
|
||||
"`require_indexed_mapping`, `require_node_field`, `parse_command_tokens`, "
|
||||
"`substitute_template_positionals`, and `render_cli_command`."
|
||||
msgstr ""
|
||||
"复用`tools/docs_codegen/utils.py`中的辅助函数，例如`require_mapping`、`require_mapping_list`、`require_scalar_mapping`、`require_indexed_mapping`、`require_node_field`、`parse_command_tokens`、`substitute_template_positionals`和`render_cli_command`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:447
|
||||
msgid ""
|
||||
"Register the converter in `build_default_converters()`. If it is not "
|
||||
"registered, `get_converter()` will reject the new `converter_tag`."
|
||||
msgstr ""
|
||||
"在`build_default_converters()`中注册该转换器。如果未注册，`get_converter()`将拒绝新的`converter_tag`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:449
|
||||
msgid ""
|
||||
"If the converter needs new directive metadata, add the option name to "
|
||||
"`MODEL_CODE_OPTION_NAMES` in `tools/docs_codegen/scanner.py` and to "
|
||||
"`ModelCodeDirective.option_spec` in "
|
||||
"`tools/docs_codegen/sphinx_extension.py`. Read the option with "
|
||||
"`block.get_option(\"<option_name>\")`."
|
||||
msgstr ""
|
||||
"如果转换器需要新的指令元数据，请将选项名称添加到`tools/docs_codegen/scanner.py`中的`MODEL_CODE_OPTION_NAMES`以及`tools/docs_codegen/sphinx_extension.py`中的`ModelCodeDirective.option_spec`。使用`block.get_option(\"<option_name>\")`读取该选项。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:454
|
||||
msgid ""
|
||||
"Add or update tests in `tests/ut/tools/test_docs_codegen.py`. Cover the "
|
||||
"successful render path, required option validation, YAML shape validation, "
|
||||
"and any CLI/Sphinx scanner behavior affected by new metadata."
|
||||
msgstr ""
|
||||
"在`tests/ut/tools/test_docs_codegen.py`中添加或更新测试。覆盖成功渲染路径、必需选项验证、YAML结构验证，以及受新元数据影响的任何CLI/Sphinx扫描器行为。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:457
|
||||
msgid ""
|
||||
"Add a real ``model-code`` example in a model tutorial, preferably under "
|
||||
"`docs/source/tutorials/models/`, and point it to an existing YAML file under"
|
||||
" `tests/`."
|
||||
msgstr ""
|
||||
"在模型教程中添加一个真实的``model-"
|
||||
"code``示例，最好放在`docs/source/tutorials/models/`下，并将其指向`tests/`下的现有YAML文件。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:460
|
||||
msgid "Validate with the CLI:"
|
||||
msgstr "使用CLI验证："
|
||||
|
||||
#: ../../source/developer_guide/contribution/doc_writing.md:467
|
||||
msgid ""
|
||||
"If a converter should render something other than shell, set "
|
||||
"`GeneratedScript.language` accordingly so Sphinx can highlight the generated"
|
||||
" literal block correctly."
|
||||
msgstr "如果转换器应渲染非shell内容，请相应设置`GeneratedScript.language`，以便Sphinx正确高亮生成的文字块。"
|
||||
@@ -0,0 +1,402 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:1
|
||||
msgid "E2E CI Test"
|
||||
msgstr "E2E CI 测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:3
|
||||
msgid ""
|
||||
"This document explains how to trigger specific E2E tests against your PR "
|
||||
"code via a comment command, without running the full E2E test suite."
|
||||
msgstr "本文档说明如何通过评论命令针对您的 PR 代码触发特定的 E2E 测试，而无需运行完整的 E2E 测试套件。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:6
|
||||
msgid "Background"
|
||||
msgstr "背景"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:8
|
||||
msgid ""
|
||||
"The `E2E-Full` workflow ([`pr_test.yaml`](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/.github/workflows/pr_test.yaml)) normally runs"
|
||||
" the complete E2E test suite when a PR has `ready` label. This is expensive "
|
||||
"in CI resources and time."
|
||||
msgstr ""
|
||||
"`E2E-Full` 工作流（[`pr_test.yaml`](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/pr_test.yaml)）通常在 PR 带有 `ready` 标签时运行完整的 "
|
||||
"E2E 测试套件。这会消耗大量的 CI 资源和时间。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:12
|
||||
msgid ""
|
||||
"Authorized users can trigger only the specific test files they care about by"
|
||||
" posting a `/e2e` comment on the PR, then adding the `ready` label."
|
||||
msgstr "授权用户可以通过在 PR 上发布 `/e2e` 评论，然后添加 `ready` 标签，仅触发他们关心的特定测试文件。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:15
|
||||
msgid "How to Trigger"
|
||||
msgstr "如何触发"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:17
|
||||
msgid "1. Post a comment"
|
||||
msgstr "1.发布评论"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:19
|
||||
msgid "First, post a comment on the PR specifying which test paths to run:"
|
||||
msgstr "首先，在 PR 上发布一条评论，指定要运行的测试路径："
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:25
|
||||
msgid "Each path must be a valid pytest path relative to the repository root."
|
||||
msgstr "每个路径必须是相对于仓库根目录的有效 pytest 路径。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:26
|
||||
msgid "Multiple paths can be listed in a single comment, separated by spaces."
|
||||
msgstr "可以在单条评论中列出多个路径，用空格分隔。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:27
|
||||
msgid "A specific test case can be targeted using `::` notation."
|
||||
msgstr "可以使用 `::` 符号定位特定的测试用例。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md
|
||||
msgid "Comment format"
|
||||
msgstr "评论格式"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md
|
||||
msgid "Effect"
|
||||
msgstr "效果"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md
|
||||
msgid "`/e2e tests/e2e/pull_request/one_card/test_foo.py`"
|
||||
msgstr "`/e2e tests/e2e/pull_request/one_card/test_foo.py`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md
|
||||
msgid "Run one test file on one_card"
|
||||
msgstr "在 one_card 上运行一个测试文件"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md
|
||||
msgid "`/e2e tests/e2e/pull_request/two_card/test_bar.py`"
|
||||
msgstr "`/e2e tests/e2e/pull_request/two_card/test_bar.py`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md
|
||||
msgid "Run one test file on two_card"
|
||||
msgstr "在 two_card 上运行一个测试文件"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md
|
||||
msgid "`/e2e path1 path2 path3`"
|
||||
msgstr "`/e2e path1 path2 path3`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md
|
||||
msgid "Run multiple files, routed by path pattern"
|
||||
msgstr "运行多个文件，按路径模式路由"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md
|
||||
msgid "`/e2e tests/e2e/pull_request/one_card/test_foo.py::test_case`"
|
||||
msgstr "`/e2e tests/e2e/pull_request/one_card/test_foo.py::test_case`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md
|
||||
msgid "Run a specific test case"
|
||||
msgstr "运行特定的测试用例"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:36
|
||||
msgid "2. Add the label"
|
||||
msgstr "2.添加标签"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:38
|
||||
msgid ""
|
||||
"After posting the comment, add the **`ready`** label to your PR. Adding the "
|
||||
"label is what actually **triggers** the workflow — at that point the "
|
||||
"workflow reads the existing comments to find the `/e2e` command."
|
||||
msgstr ""
|
||||
"发布评论后，为您的 PR 添加 **`ready`** 标签。添加标签才是实际**触发**工作流的操作——此时工作流会读取现有评论以查找 `/e2e` "
|
||||
"命令。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:43
|
||||
msgid ""
|
||||
"Only repository **Contributors** (Triage role) and **Maintainers** (Write "
|
||||
"role) can add labels. If you do not have this permission, ask a maintainer "
|
||||
"to add the label for you. You can find the list of maintainers and "
|
||||
"contributors by checking the [CODEOWNERS](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/.github/CODEOWNERS) file."
|
||||
msgstr ""
|
||||
"只有仓库的**贡献者**（Triage 角色）和**维护者**（Write 角色）可以添加标签。如果您没有此权限，请让维护者为您添加标签。您可以通过查看"
|
||||
" [CODEOWNERS](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/CODEOWNERS) 文件找到维护者和贡献者列表。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:51
|
||||
msgid ""
|
||||
"The comment must be posted **before** the label is added. If you add the "
|
||||
"label first, the workflow will find no `/e2e` comment and will not trigger "
|
||||
"any per-test runs."
|
||||
msgstr "评论必须在添加标签**之前**发布。如果您先添加标签，工作流将找不到 `/e2e` 评论，也不会触发任何按测试运行。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:56
|
||||
msgid ""
|
||||
"Additionally, only the **PR author** or collaborators with **write or "
|
||||
"admin** repository access can trigger tests via comment. The workflow "
|
||||
"validates the commenter's permission before proceeding."
|
||||
msgstr "此外，只有 **PR 作者**或具有仓库**写入或管理员**权限的协作者才能通过评论触发测试。工作流会在继续之前验证评论者的权限。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:61
|
||||
msgid "3. Wait for results"
|
||||
msgstr "3.等待结果"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:63
|
||||
msgid ""
|
||||
"GitHub Actions will trigger the `E2E-Full` workflow. Only the hardware jobs "
|
||||
"matching the provided test paths will run, which saves CI resources."
|
||||
msgstr "GitHub Actions 将触发 `E2E-Full` 工作流。只有与提供的测试路径匹配的硬件任务会运行，从而节省 CI 资源。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:66
|
||||
msgid "Path Routing Rules"
|
||||
msgstr "路径路由规则"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:68
|
||||
msgid ""
|
||||
"The workflow automatically routes each test path to the correct hardware "
|
||||
"runner based on path patterns:"
|
||||
msgstr "工作流会根据路径模式自动将每个测试路径路由到正确的硬件运行器："
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Path pattern"
|
||||
msgstr "路径模式"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Hardware"
|
||||
msgstr "硬件"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Runner"
|
||||
msgstr "运行器"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "`two_card` in path"
|
||||
msgstr "路径中包含 `two_card`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "two_card A3 NPU"
|
||||
msgstr "two_card A3 NPU"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "`linux-aarch64-a3-2`"
|
||||
msgstr "`linux-aarch64-a3-2`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "`four_card` in path"
|
||||
msgstr "路径中包含 `four_card`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "four_card A3 NPU"
|
||||
msgstr "four_card A3 NPU"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "`linux-aarch64-a3-4`"
|
||||
msgstr "`linux-aarch64-a3-4`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "`_310p` in filename under one/two_card"
|
||||
msgstr "one/two_card 下的文件名中包含 `_310p`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Ascend 310P x1"
|
||||
msgstr "Ascend 310P x1"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "`linux-aarch64-310p-*`"
|
||||
msgstr "`linux-aarch64-310p-*`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "`_310p` in filename under four_card"
|
||||
msgstr "four_card 下的文件名中包含 `_310p`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Ascend 310P x4"
|
||||
msgstr "Ascend 310P x4"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "All other paths"
|
||||
msgstr "所有其他路径"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "one_card A2 NPU"
|
||||
msgstr "one_card A2 NPU"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "`linux-aarch64-a2b3-1`"
|
||||
msgstr "`linux-aarch64-a2b3-1`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:79
|
||||
msgid ""
|
||||
"When paths from multiple categories are listed in a single comment, each "
|
||||
"category's tests run on its respective hardware in parallel."
|
||||
msgstr "当单条评论中列出多个类别的路径时，每个类别的测试会在其各自的硬件上并行运行。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:82
|
||||
msgid "Test Path Reference"
|
||||
msgstr "测试路径参考"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:84
|
||||
msgid ""
|
||||
"The `tests/e2e/pull_request/` directory is organized by hardware category:"
|
||||
msgstr "`tests/e2e/pull_request/` 目录按硬件类别组织："
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:93
|
||||
msgid ""
|
||||
"310P tests use `_310p` subdirectories or `_310p.py` filename suffix under "
|
||||
"the corresponding card directory:"
|
||||
msgstr "310P 测试在相应的卡目录下使用 `_310p` 子目录或 `_310p.py` 文件名后缀："
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:101
|
||||
msgid "Comparison with Full E2E Suite"
|
||||
msgstr "与完整 E2E 套件对比"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Aspect"
|
||||
msgstr "方面"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Full E2E suite"
|
||||
msgstr "完整 E2E 套件"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Per-test comment trigger"
|
||||
msgstr "按测试评论触发"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Trigger"
|
||||
msgstr "触发方式"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "`ready` labels"
|
||||
msgstr "`ready` 标签"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "`/e2e` comment + `ready` label"
|
||||
msgstr "`/e2e` 评论 + `ready` 标签"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Scope"
|
||||
msgstr "范围"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "All E2E tests"
|
||||
msgstr "所有 E2E 测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Only specified test paths"
|
||||
msgstr "仅指定的测试路径"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Who can trigger"
|
||||
msgstr "谁可以触发"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Anyone who can add labels"
|
||||
msgstr "任何可以添加标签的人"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "PR author or write/admin collaborator"
|
||||
msgstr "PR 作者或写入/管理员协作者"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Use case"
|
||||
msgstr "使用场景"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Pre-merge validation"
|
||||
msgstr "合并前验证"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:55
|
||||
msgid "Iterative debugging of specific tests"
|
||||
msgstr "特定测试的迭代调试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:110
|
||||
msgid "Examples"
|
||||
msgstr "示例"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:112
|
||||
msgid "Run a single one_card test:"
|
||||
msgstr "运行单个 one_card 测试："
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:118
|
||||
msgid "Run a two_card test:"
|
||||
msgstr "运行 two_card 测试："
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:124
|
||||
msgid "Run tests across multiple hardware categories in one comment:"
|
||||
msgstr "在一条评论中运行跨多个硬件类别的测试："
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:130
|
||||
msgid ""
|
||||
"Re-trigger after fixing an issue: just push a new commit. The `synchronize` "
|
||||
"event re-runs the workflow and picks up the existing `/e2e` comment "
|
||||
"automatically — no need to post a new comment."
|
||||
msgstr ""
|
||||
"修复问题后重新触发：只需推送新的提交。`synchronize` 事件会重新运行工作流并自动获取现有的 `/e2e` 评论——无需发布新评论。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:134
|
||||
msgid "Troubleshooting"
|
||||
msgstr "故障排除"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:136
|
||||
msgid "**The workflow did not start after I added the label.**"
|
||||
msgstr "**添加标签后工作流未启动。**"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:138
|
||||
msgid ""
|
||||
"Make sure the `/e2e` comment was posted **before** the label was added. If "
|
||||
"the label was added first, remove it and re-add it after posting the "
|
||||
"comment."
|
||||
msgstr "请确保 `/e2e` 评论是在添加标签 **之前** 发布的。如果标签已先添加，请先移除标签，在发布评论后重新添加。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:140
|
||||
msgid ""
|
||||
"Check that the comment starts exactly with `/e2e` followed by at least one "
|
||||
"path, with no leading spaces or extra characters before the slash."
|
||||
msgstr "检查评论是否以 `/e2e` 开头，后跟至少一个路径，且斜杠前没有前导空格或额外字符。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:142
|
||||
msgid ""
|
||||
"To re-trigger after fixing an issue, simply push a new commit — the workflow"
|
||||
" will reuse the existing `/e2e` comment automatically."
|
||||
msgstr "修复问题后如需重新触发，只需推送新提交——工作流将自动复用已有的 `/e2e` 评论。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:145
|
||||
msgid "**Tests ran on the wrong hardware.**"
|
||||
msgstr "**测试在错误的硬件上运行。**"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:147
|
||||
msgid ""
|
||||
"Check that the path includes the expected directory segment (`one_card`, "
|
||||
"`two_card`, `four_card`, or `_310p`). Paths that do not match any of these "
|
||||
"patterns are routed to the one_card runner by default."
|
||||
msgstr ""
|
||||
"检查路径是否包含预期的目录段（`one_card`、`two_card`、`four_card` 或 `_310p`）。不匹配这些模式的路径默认路由到 "
|
||||
"one_card 运行器。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:151
|
||||
msgid "**The `parse-comment` job skipped with a permission error.**"
|
||||
msgstr "**`parse-comment` 任务因权限错误被跳过。**"
|
||||
|
||||
#: ../../source/developer_guide/contribution/e2e_ci_test.md:153
|
||||
msgid ""
|
||||
"Only the PR author or write/admin collaborators can use the comment trigger."
|
||||
" Ask a maintainer to post the `/e2e` comment instead."
|
||||
msgstr "只有 PR 作者或具有写入/管理员权限的协作者才能使用评论触发器。请让维护者代为发布 `/e2e` 评论。"
|
||||
@@ -2,14 +2,14 @@
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2025.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-07-18 09:01+0800\n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
@@ -18,170 +18,171 @@ msgstr ""
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.17.0\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:107
|
||||
#: ../../source/developer_guide/contribution/index.md:107
|
||||
msgid "Index"
|
||||
msgstr "索引"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:1
|
||||
#: ../../source/developer_guide/contribution/index.md:1
|
||||
msgid "Contributing"
|
||||
msgstr "贡献"
|
||||
msgstr "贡献指南"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:3
|
||||
msgid "Building and testing"
|
||||
#: ../../source/developer_guide/contribution/index.md:3
|
||||
msgid "Building and Testing"
|
||||
msgstr "构建与测试"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:4
|
||||
#: ../../source/developer_guide/contribution/index.md:5
|
||||
msgid ""
|
||||
"It's recommended to set up a local development environment to build and test"
|
||||
" before you submit a PR."
|
||||
msgstr "建议先搭建本地开发环境来进行构建和测试，再提交 PR。"
|
||||
"It's recommended to set up a local development environment to build vllm-"
|
||||
"ascend and run tests before you submit a PR."
|
||||
msgstr "建议在提交 PR 前搭建本地开发环境，用于构建 vllm-ascend 并运行测试。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:7
|
||||
msgid "Setup development environment"
|
||||
#: ../../source/developer_guide/contribution/index.md:8
|
||||
msgid "Set up a development environment"
|
||||
msgstr "搭建开发环境"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:9
|
||||
#: ../../source/developer_guide/contribution/index.md:10
|
||||
msgid ""
|
||||
"Theoretically, the vllm-ascend build is only supported on Linux because "
|
||||
"`vllm-ascend` dependency `torch_npu` only supports Linux."
|
||||
msgstr ""
|
||||
"理论上，vllm-ascend 构建仅支持 Linux，因为 `vllm-ascend` 的依赖项 `torch_npu` 只支持 Linux。"
|
||||
msgstr "理论上，vllm-ascend 的构建仅支持 Linux，因为其依赖 `torch_npu` 仅支持 Linux。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:12
|
||||
#: ../../source/developer_guide/contribution/index.md:13
|
||||
msgid ""
|
||||
"But you can still set up dev env on Linux/Windows/macOS for linting and "
|
||||
"basic test as following commands:"
|
||||
msgstr "但你仍然可以在 Linux/Windows/macOS 上按照以下命令设置开发环境，用于代码规约检查和基本测试："
|
||||
"But you can still set up a development environment on Linux/Windows/macOS "
|
||||
"for linting and running basic tests."
|
||||
msgstr "但你仍可在 Linux/Windows/macOS 上搭建开发环境，用于代码检查与运行基础测试。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:15
|
||||
#: ../../source/developer_guide/contribution/index.md:16
|
||||
msgid "Run lint locally"
|
||||
msgstr "在本地运行 lint"
|
||||
msgstr "本地运行代码检查"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:33
|
||||
#: ../../source/developer_guide/contribution/index.md:35
|
||||
msgid "Run CI locally"
|
||||
msgstr "本地运行CI"
|
||||
msgstr "本地运行 CI"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:35
|
||||
msgid "After complete \"Run lint\" setup, you can run CI locally:"
|
||||
msgstr "在完成“运行 lint”设置后，你可以在本地运行 CI："
|
||||
#: ../../source/developer_guide/contribution/index.md:37
|
||||
msgid ""
|
||||
"After completing \"Run lint\" setup, you can run CI (Continuous integration)"
|
||||
" locally:"
|
||||
msgstr "完成“运行代码检查”设置后，你可以在本地运行 CI（持续集成）："
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:61
|
||||
#: ../../source/developer_guide/contribution/index.md:62
|
||||
msgid "Submit the commit"
|
||||
msgstr "提交该提交"
|
||||
msgstr "提交代码"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:68
|
||||
#: ../../source/developer_guide/contribution/index.md:69
|
||||
msgid ""
|
||||
"🎉 Congratulations! You have completed the development environment setup."
|
||||
msgstr "🎉 恭喜！你已经完成了开发环境的搭建。"
|
||||
msgstr "🎉 恭喜！你已完成开发环境搭建。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:70
|
||||
msgid "Test locally"
|
||||
#: ../../source/developer_guide/contribution/index.md:71
|
||||
msgid "Testing locally"
|
||||
msgstr "本地测试"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:72
|
||||
#: ../../source/developer_guide/contribution/index.md:73
|
||||
msgid ""
|
||||
"You can refer to [Testing](./testing.md) doc to help you setup testing "
|
||||
"environment and running tests locally."
|
||||
msgstr "你可以参考 [测试](./testing.md) 文档，帮助你搭建测试环境并在本地运行测试。"
|
||||
"You can refer to [Testing](./testing.md) to set up a testing environment "
|
||||
"and running tests locally."
|
||||
msgstr "可参考 [测试](./testing.md) 搭建测试环境并在本地运行测试。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:74
|
||||
#: ../../source/developer_guide/contribution/index.md:75
|
||||
msgid "DCO and Signed-off-by"
|
||||
msgstr "DCO 和签名确认"
|
||||
msgstr "DCO 与 Signed-off-by"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:76
|
||||
#: ../../source/developer_guide/contribution/index.md:77
|
||||
msgid ""
|
||||
"When contributing changes to this project, you must agree to the DCO. "
|
||||
"Commits must include a `Signed-off-by:` header which certifies agreement "
|
||||
"with the terms of the DCO."
|
||||
msgstr "当为本项目贡献更改时，您必须同意 DCO。提交必须包含 `Signed-off-by:` 头部，以证明您同意 DCO 的条款。"
|
||||
"with the terms of the DCO (Developer Certificate of Origin)."
|
||||
msgstr ""
|
||||
"向本项目贡献代码时，你必须同意 DCO。提交信息必须包含 `Signed-off-by:` 头，以证明同意 DCO（开发者原创证书）的条款。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:78
|
||||
#: ../../source/developer_guide/contribution/index.md:79
|
||||
msgid "Using `-s` with `git commit` will automatically add this header."
|
||||
msgstr "在使用 `git commit` 时加上 `-s` 参数会自动添加这个头部信息。"
|
||||
msgstr "使用 `git commit -s` 将自动添加此头信息。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:80
|
||||
#: ../../source/developer_guide/contribution/index.md:81
|
||||
msgid "PR Title and Classification"
|
||||
msgstr "PR 标题与分类"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:82
|
||||
#: ../../source/developer_guide/contribution/index.md:83
|
||||
msgid ""
|
||||
"Only specific types of PRs will be reviewed. The PR title is prefixed "
|
||||
"appropriately to indicate the type of change. Please use one of the "
|
||||
"following:"
|
||||
msgstr "只有特定类型的 PR 会被审核。PR 标题应使用合适的前缀以指明更改类型。请使用以下之一："
|
||||
msgstr "仅特定类型的 PR 会被审查。PR 标题需添加适当前缀以指示变更类型。请使用以下前缀之一："
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:84
|
||||
#: ../../source/developer_guide/contribution/index.md:85
|
||||
msgid "`[Attention]` for new features or optimization in attention."
|
||||
msgstr "`[Attention]` 用于注意力机制中新特性或优化。"
|
||||
msgstr "`[Attention]` 用于注意力机制的新功能或优化。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:85
|
||||
#: ../../source/developer_guide/contribution/index.md:86
|
||||
msgid "`[Communicator]` for new features or optimization in communicators."
|
||||
msgstr "`[Communicator]` 适用于通信器中的新特性或优化。"
|
||||
msgstr "`[Communicator]` 用于通信器的新功能或优化。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:86
|
||||
#: ../../source/developer_guide/contribution/index.md:87
|
||||
msgid "`[ModelRunner]` for new features or optimization in model runner."
|
||||
msgstr "`[ModelRunner]` 用于模型运行器中的新功能或优化。"
|
||||
msgstr "`[ModelRunner]` 用于模型运行器的新功能或优化。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:87
|
||||
#: ../../source/developer_guide/contribution/index.md:88
|
||||
msgid "`[Platform]` for new features or optimization in platform."
|
||||
msgstr "`[Platform]` 用于平台中新功能或优化。"
|
||||
msgstr "`[Platform]` 用于平台的新功能或优化。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:88
|
||||
#: ../../source/developer_guide/contribution/index.md:89
|
||||
msgid "`[Worker]` for new features or optimization in worker."
|
||||
msgstr "`[Worker]` 用于 worker 的新功能或优化。"
|
||||
msgstr "`[Worker]` 用于工作节点的新功能或优化。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:89
|
||||
#: ../../source/developer_guide/contribution/index.md:90
|
||||
msgid ""
|
||||
"`[Core]` for new features or optimization in the core vllm-ascend logic "
|
||||
"(such as platform, attention, communicators, model runner)"
|
||||
msgstr "`[Core]` 用于核心 vllm-ascend 逻辑中的新特性或优化（例如平台、注意力机制、通信器、模型运行器）。"
|
||||
msgstr "`[Core]` 用于 vllm-ascend 核心逻辑（如平台、注意力机制、通信器、模型运行器）的新功能或优化。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:90
|
||||
msgid "`[Kernel]` changes affecting compute kernels and ops."
|
||||
msgstr "`[Kernel]` 影响计算内核和操作的更改。"
|
||||
#: ../../source/developer_guide/contribution/index.md:91
|
||||
msgid "`[Kernel]` for changes affecting compute kernels and ops."
|
||||
msgstr "`[Kernel]` 用于影响计算内核与算子的变更。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:91
|
||||
msgid "`[Bugfix]` for bug fixes."
|
||||
msgstr "`[Bugfix]` 用于表示错误修复。"
|
||||
#: ../../source/developer_guide/contribution/index.md:92
|
||||
msgid "`[BugFix]` for bug fixes."
|
||||
msgstr "`[BugFix]` 用于错误修复。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:92
|
||||
#: ../../source/developer_guide/contribution/index.md:93
|
||||
msgid "`[Doc]` for documentation fixes and improvements."
|
||||
msgstr "`[Doc]` 用于文档修复和改进。"
|
||||
msgstr "`[Doc]` 用于文档修复与改进。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:93
|
||||
#: ../../source/developer_guide/contribution/index.md:94
|
||||
msgid "`[Test]` for tests (such as unit tests)."
|
||||
msgstr "`[Test]` 用于测试（如单元测试）。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:94
|
||||
#: ../../source/developer_guide/contribution/index.md:95
|
||||
msgid "`[CI]` for build or continuous integration improvements."
|
||||
msgstr "`[CI]` 用于构建或持续集成的改进。"
|
||||
msgstr "`[CI]` 用于构建或持续集成改进。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:95
|
||||
#: ../../source/developer_guide/contribution/index.md:96
|
||||
msgid ""
|
||||
"`[Misc]` for PRs that do not fit the above categories. Please use this "
|
||||
"sparingly."
|
||||
msgstr "对于不属于上述类别的 PR，请使用 `[Misc]`。请谨慎使用此标签。"
|
||||
msgstr "`[Misc]` 用于不属于上述类别的 PR。请谨慎使用。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:98
|
||||
#: ../../source/developer_guide/contribution/index.md:99
|
||||
msgid ""
|
||||
"If the PR spans more than one category, please include all relevant "
|
||||
"prefixes."
|
||||
msgstr "如果拉取请求（PR）涵盖多个类别，请包含所有相关的前缀。"
|
||||
msgstr "如果 PR 涉及多个类别，请包含所有相关前缀。"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:101
|
||||
#: ../../source/developer_guide/contribution/index.md:102
|
||||
msgid "Others"
|
||||
msgstr "其他"
|
||||
|
||||
#: ../../developer_guide/contribution/index.md:103
|
||||
#: ../../source/developer_guide/contribution/index.md:104
|
||||
msgid ""
|
||||
"You may find more information about contributing to vLLM Ascend backend "
|
||||
"plugin on "
|
||||
"[<u>docs.vllm.ai</u>](https://docs.vllm.ai/en/latest/contributing/overview.html)."
|
||||
" If you find any problem when contributing, you can feel free to submit a PR"
|
||||
" to improve the doc to help other developers."
|
||||
"[<u>docs.vllm.ai</u>](https://docs.vllm.ai/en/latest/contributing). If you "
|
||||
"encounter any problems while contributing, feel free to submit a PR to "
|
||||
"improve the documentation to help other developers."
|
||||
msgstr ""
|
||||
"你可以在 "
|
||||
"[<u>docs.vllm.ai</u>](https://docs.vllm.ai/en/latest/contributing/overview.html)"
|
||||
" 上找到有关为 vLLM Ascend 后端插件做贡献的更多信息。如果你在贡献过程中遇到任何问题，欢迎随时提交 PR 来改进文档，以帮助其他开发者。"
|
||||
"你可以在 [<u>docs.vllm.ai</u>](https://docs.vllm.ai/en/latest/contributing) "
|
||||
"上找到更多关于贡献 vLLM Ascend 后端插件的信息。如果在贡献过程中遇到任何问题，欢迎提交 PR 改进文档以帮助其他开发者。"
|
||||
|
||||
@@ -0,0 +1,384 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:1
|
||||
msgid "Multi Node Test"
|
||||
msgstr "多节点测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:3
|
||||
msgid ""
|
||||
"Multi-Node CI is designed to test distributed scenarios of very large "
|
||||
"models, for example, disaggregated_prefill multi DP across multi nodes and "
|
||||
"so on."
|
||||
msgstr "多节点CI旨在测试超大规模模型的分布式场景，例如跨多节点的disaggregated_prefill多DP等。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:5
|
||||
msgid "How it works"
|
||||
msgstr "工作原理"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:7
|
||||
msgid ""
|
||||
"The following picture shows the basic deployment view of the multi-node CI "
|
||||
"mechanism. It shows how the GitHub action interacts with "
|
||||
"[lws](https://lws.sigs.k8s.io/docs/overview/) (a kind of kubernetes crd "
|
||||
"resource)."
|
||||
msgstr ""
|
||||
"下图展示了多节点CI机制的基本部署视图，说明了GitHub "
|
||||
"Action如何与[lws](https://lws.sigs.k8s.io/docs/overview/)（一种Kubernetes "
|
||||
"CRD资源）交互。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:9
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:9
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:13
|
||||
msgid "alt text"
|
||||
msgstr "替代文本"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:11
|
||||
msgid ""
|
||||
"From the workflow perspective, we can see how the final test script is "
|
||||
"executed. The key point is that the shared files "
|
||||
"`tests/e2e/nightly/multi_node/scripts/lws.yaml.jinja2` and "
|
||||
"`tests/e2e/nightly/multi_node/scripts/run.sh` define the cluster template "
|
||||
"and pod entry script. Each node executes different logic according to the "
|
||||
"[LWS_WORKER_INDEX](https://lws.sigs.k8s.io/docs/reference/labels-"
|
||||
"annotations-and-environment-variables/) environment variable, so that "
|
||||
"multiple nodes can form a distributed cluster to perform tasks. `run.sh` "
|
||||
"selects the pytest entrypoint from the config path: internal DP configs use "
|
||||
"`internal_dp/scripts/test_multi_node.py`, while external DP configs use "
|
||||
"`external_dp/scripts/test_external_dp.py`."
|
||||
msgstr ""
|
||||
"从工作流的角度来看，我们可以看到最终测试脚本是如何执行的。关键在于共享文件`tests/e2e/nightly/multi_node/scripts/lws.yaml.jinja2`和`tests/e2e/nightly/multi_node/scripts/run.sh`定义了集群模板和pod入口脚本。每个节点根据[LWS_WORKER_INDEX](https://lws.sigs.k8s.io/docs/reference/labels-"
|
||||
"annotations-and-environment-"
|
||||
"variables/)环境变量执行不同的逻辑，从而使多个节点能够组成分布式集群来执行任务。`run.sh`从配置路径中选择pytest入口点：内部DP配置使用`internal_dp/scripts/test_multi_node.py`，而外部DP配置使用`external_dp/scripts/test_external_dp.py`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:13
|
||||
msgid ""
|
||||
msgstr ""
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:15
|
||||
msgid "How to contribute"
|
||||
msgstr "如何贡献"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:17
|
||||
msgid "Upload custom weights"
|
||||
msgstr "上传自定义权重"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:19
|
||||
msgid ""
|
||||
"If you need customized weights, for example, you quantized a w8a8 weight for"
|
||||
" DeepSeek-V3 and you want your weight to run on CI, uploading weights to "
|
||||
"ModelScope's [vllm-ascend](https://www.modelscope.cn/organization/vllm-"
|
||||
"ascend) organization is welcome. If you do not have permission to upload, "
|
||||
"please contact @Potabk"
|
||||
msgstr ""
|
||||
"如果您需要自定义权重，例如您为DeepSeek-V3量化了w8a8权重并希望在CI上运行，欢迎将权重上传至ModelScope的[vllm-"
|
||||
"ascend](https://www.modelscope.cn/organization/vllm-"
|
||||
"ascend)组织。如果您没有上传权限，请联系@Potabk"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:21
|
||||
msgid "Add config yaml"
|
||||
msgstr "添加配置yaml"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:23
|
||||
msgid ""
|
||||
"For the normal internal DP multi-node flow, add the config yaml to "
|
||||
"`tests/e2e/nightly/multi_node/internal_dp/config/`, like `DeepSeek-V3.yaml`."
|
||||
" External DP cases use the separate "
|
||||
"`tests/e2e/nightly/multi_node/external_dp/config/` directory and should pass"
|
||||
" that directory through `config_base_path` in workflow or `CONFIG_BASE_PATH`"
|
||||
" locally."
|
||||
msgstr ""
|
||||
"对于常规的内部DP多节点流程，将配置yaml添加到`tests/e2e/nightly/multi_node/internal_dp/config/`，例如`DeepSeek-V3.yaml`。外部DP用例使用独立的`tests/e2e/nightly/multi_node/external_dp/config/`目录，并应通过工作流中的`config_base_path`或本地的`CONFIG_BASE_PATH`传递该目录。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:25
|
||||
msgid ""
|
||||
"Suppose you have **2 nodes** running a 1P1D setup (1 Prefillers + 1 "
|
||||
"Decoder):"
|
||||
msgstr "假设您有**2个节点**运行1P1D配置（1个Prefiller + 1个Decoder）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:27
|
||||
msgid "you may add a config file looks like:"
|
||||
msgstr "您可以添加一个如下所示的配置文件："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:77
|
||||
msgid "Add the case to nightly workflow"
|
||||
msgstr "将用例添加到夜间工作流"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:79
|
||||
msgid ""
|
||||
"Currently, the multi-node test workflow is defined in "
|
||||
"`.github/workflows/schedule_nightly_test_a3.yaml`."
|
||||
msgstr "目前，多节点测试工作流定义在`.github/workflows/schedule_nightly_test_a3.yaml`中。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:114
|
||||
msgid ""
|
||||
"The matrix above defines all the parameters required to add a multi-machine "
|
||||
"use case. The parameters worth noting are `size`, `config_file_path`, and "
|
||||
"`config_base_path`. `size` defines the number of nodes required for your use"
|
||||
" case. `config_file_path` is the yaml file name, and `config_base_path` "
|
||||
"tells the loader which config directory to use. For internal DP cases, use "
|
||||
"an empty `config_base_path` so the loader uses its default internal DP "
|
||||
"config directory. For external DP cases, set it to "
|
||||
"`tests/e2e/nightly/multi_node/external_dp/config/`."
|
||||
msgstr ""
|
||||
"上述矩阵定义了添加多机用例所需的所有参数。值得注意的参数有`size`、`config_file_path`和`config_base_path`。`size`定义用例所需的节点数量。`config_file_path`是yaml文件名，`config_base_path`告诉加载器使用哪个配置目录。对于内部DP用例，使用空的`config_base_path`，以便加载器使用其默认的内部DP配置目录。对于外部DP用例，将其设置为`tests/e2e/nightly/multi_node/external_dp/config/`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:123
|
||||
msgid "Run Multi-Node tests locally"
|
||||
msgstr "本地运行多节点测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:125
|
||||
msgid "1. Use kubernetes"
|
||||
msgstr "1.使用Kubernetes"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:127
|
||||
msgid ""
|
||||
"This section assumes that you already have a "
|
||||
"[Kubernetes](https://kubernetes.io/docs/setup/) NPU cluster environment "
|
||||
"locally. Then you can easily start our test with one click."
|
||||
msgstr ""
|
||||
"本节假设您本地已经有一个[Kubernetes](https://kubernetes.io/docs/setup/) "
|
||||
"NPU集群环境。然后您可以一键轻松启动测试。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:129
|
||||
msgid "Step 1. Install LWS CRD resources"
|
||||
msgstr "步骤1.安装LWS CRD资源"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:131
|
||||
msgid ""
|
||||
"See <https://lws.sigs.k8s.io/docs/installation/> Which can be used as a "
|
||||
"reference"
|
||||
msgstr "参见<https://lws.sigs.k8s.io/docs/installation/> 可作为参考"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:133
|
||||
msgid "Step 2. Deploy the following yaml file `lws.yaml` as needed"
|
||||
msgstr "步骤2.根据需要部署以下yaml文件`lws.yaml`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:273
|
||||
msgid "Verify the status of the pods:"
|
||||
msgstr "验证Pod的状态："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:279
|
||||
msgid "Should get an output similar to this:"
|
||||
msgstr "应得到类似如下的输出："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:287
|
||||
msgid "Verify that the distributed inference works:"
|
||||
msgstr "验证分布式推理是否正常工作："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:293
|
||||
msgid "Should get something similar to this:"
|
||||
msgstr "应得到类似如下的结果："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:328
|
||||
msgid "2. Test without Kubernetes"
|
||||
msgstr "2.不使用Kubernetes进行测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:330
|
||||
msgid ""
|
||||
"The same `tests/e2e/nightly/multi_node/scripts/run.sh` entrypoint can be "
|
||||
"used on prepared bare-metal or container hosts. Without LWS, set the values "
|
||||
"that Kubernetes normally injects yourself:"
|
||||
msgstr ""
|
||||
"相同的`tests/e2e/nightly/multi_node/scripts/run.sh`入口点可以在准备好的裸机或容器主机上使用。在没有LWS的情况下，自行设置Kubernetes通常注入的值："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:334
|
||||
msgid ""
|
||||
"`cluster_hosts` in the config yaml, using IPs reachable from every node."
|
||||
msgstr "配置yaml中的`cluster_hosts`，使用每个节点可达的IP。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:335
|
||||
msgid "`LWS_WORKER_INDEX` on each node, starting from `0`."
|
||||
msgstr "每个节点上的`LWS_WORKER_INDEX`，从`0`开始。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:336
|
||||
msgid ""
|
||||
"`CONFIG_YAML_PATH` as the config file name and `CONFIG_BASE_PATH` as the "
|
||||
"config directory."
|
||||
msgstr "`CONFIG_YAML_PATH`作为配置文件名，`CONFIG_BASE_PATH`作为配置目录。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:339
|
||||
msgid ""
|
||||
"Use the host NIC IPs that can reach each other, for example addresses shown "
|
||||
"by `ip addr` or `ifconfig` on the active network interface. Do not use per-"
|
||||
"host Docker bridge addresses such as `172.17.0.1`, because each host has its"
|
||||
" own local bridge."
|
||||
msgstr ""
|
||||
"使用可以相互通信的主机网卡IP，例如活动网络接口上`ip "
|
||||
"addr`或`ifconfig`显示的地址。不要使用每个主机的Docker桥接地址，例如`172.17.0.1`，因为每个主机都有自己的本地桥接。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:344
|
||||
msgid ""
|
||||
"Local `cluster_hosts` edits should be removed before submitting a PR unless "
|
||||
"the hosts are part of a committed test environment."
|
||||
msgstr "在提交PR之前，应移除本地的`cluster_hosts`编辑，除非这些主机是已提交测试环境的一部分。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:347
|
||||
msgid "2.1 Internal DP local run"
|
||||
msgstr "2.1 内部DP本地运行"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:349
|
||||
msgid "2.1.1 Add cluster hosts"
|
||||
msgstr "2.1.1 添加集群主机"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:351
|
||||
msgid "Edit the internal DP config you want to run, for example:"
|
||||
msgstr "编辑您要运行的内部DP配置，例如："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:357
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:439
|
||||
msgid ""
|
||||
"Add `cluster_hosts` as a top-level field, for example near `num_nodes` and "
|
||||
"`npu_per_node`:"
|
||||
msgstr "添加`cluster_hosts`作为顶级字段，例如在`num_nodes`和`npu_per_node`附近："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:366
|
||||
msgid "2.1.2 Prepare the environment"
|
||||
msgstr "2.1.2 准备环境"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:368
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:452
|
||||
msgid "Install vllm-ascend development dependencies on every cluster host:"
|
||||
msgstr "在每个集群主机上安装vllm-ascend开发依赖："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:375
|
||||
msgid ""
|
||||
"Install AISBench on the first host, which is the node with "
|
||||
"`LWS_WORKER_INDEX=0`:"
|
||||
msgstr "在第一个主机上安装AISBench，即`LWS_WORKER_INDEX=0`的节点："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:388
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:471
|
||||
msgid ""
|
||||
"If your local image already contains the model, benchmark data, Ascend "
|
||||
"runtime, and AISBench, you only need the run-time exports in the next step."
|
||||
msgstr "如果您的本地镜像已包含模型、基准数据、Ascend运行环境和AISBench，则只需下一步中的运行时导出。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:391
|
||||
msgid "2.1.3 Start each node"
|
||||
msgstr "2.1.3 启动每个节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:393
|
||||
msgid ""
|
||||
"Run the script on each node separately. Start worker nodes first, then start"
|
||||
" node 0."
|
||||
msgstr "在每个节点上分别运行脚本。先启动工作节点，然后启动节点0。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:396
|
||||
msgid "On node 1:"
|
||||
msgstr "在节点1上："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:409
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:497
|
||||
msgid "On node 0:"
|
||||
msgstr "在节点0上："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:422
|
||||
msgid ""
|
||||
"Internal DP logs are mainly printed to the terminal running `run.sh`. When "
|
||||
"`LOG_PREFIX` is set, the shared script also backs up Ascend logs to:"
|
||||
msgstr "内部DP日志主要打印到运行`run.sh`的终端。当设置了`LOG_PREFIX`时，共享脚本还会将Ascend日志备份到："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:429
|
||||
msgid "2.2 External DP local run"
|
||||
msgstr "2.2 外部DP本地运行"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:431
|
||||
msgid "2.2.1 Add cluster hosts"
|
||||
msgstr "2.2.1 添加集群主机"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:433
|
||||
msgid "Edit the external DP config you want to run. For example:"
|
||||
msgstr "编辑要运行的外部DP配置。例如："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:450
|
||||
msgid "2.2.2 Prepare the environment"
|
||||
msgstr "2.2.2 准备环境"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:459
|
||||
msgid "Install AISBench on node 0:"
|
||||
msgstr "在节点0上安装AISBench："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:474
|
||||
msgid "2.2.3 Start each node"
|
||||
msgstr "2.2.3 启动每个节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:476
|
||||
msgid ""
|
||||
"External DP uses the same shared `run.sh`. Set `CONFIG_BASE_PATH` to the "
|
||||
"external DP config directory so the script chooses "
|
||||
"`external_dp/scripts/test_external_dp.py`."
|
||||
msgstr ""
|
||||
"外部DP使用相同的共享`run.sh`。将`CONFIG_BASE_PATH`设置为外部DP配置目录，以便脚本选择`external_dp/scripts/test_external_dp.py`。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:480
|
||||
msgid ""
|
||||
"Then start non-master nodes first, and start node 0 last. The following "
|
||||
"example uses `GLM5_1-W8A8-EP-external.yaml`, which is a 4-node disaggregated"
|
||||
" prefill case."
|
||||
msgstr ""
|
||||
"然后先启动非主节点，最后启动节点0。以下示例使用`GLM5_1-W8A8-EP-external.yaml`，这是一个4节点解耦预填充场景。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:484
|
||||
msgid "On node 1, node 2, and node 3, set the matching `LWS_WORKER_INDEX`:"
|
||||
msgstr "在节点1、节点2和节点3上，设置匹配的`LWS_WORKER_INDEX`："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:510
|
||||
msgid ""
|
||||
"For `GLM5_1-W8A8-EP-external.yaml`, node 0 and node 1 start prefiller ranks,"
|
||||
" node 2 and node 3 start decoder ranks, and node 0 also starts the proxy and"
|
||||
" benchmark."
|
||||
msgstr ""
|
||||
"对于`GLM5_1-W8A8-EP-"
|
||||
"external.yaml`，节点0和节点1启动预填充器rank，节点2和节点3启动解码器rank，节点0还启动代理和基准测试。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:514
|
||||
msgid "2.2.4 Read logs while the test is running"
|
||||
msgstr "2.2.4 测试运行时读取日志"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:516
|
||||
msgid ""
|
||||
"The terminal running `run.sh` prints pytest orchestration logs. For external"
|
||||
" DP, AISBench output is also printed on node 0, while rank and proxy "
|
||||
"stdout/stderr are written to `EXTERNAL_DP_LOG_DIR`. The default layout is:"
|
||||
msgstr ""
|
||||
"运行`run.sh`的终端会打印pytest编排日志。对于外部DP，AISBench输出也会在节点0上打印，而rank和代理的stdout/stderr会写入`EXTERNAL_DP_LOG_DIR`。默认布局为："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:531
|
||||
msgid ""
|
||||
"The first line of each rank log records the exact command and environment "
|
||||
"used to start that rank. `proxy.log` exists only on the configured proxy "
|
||||
"node, usually node 0."
|
||||
msgstr "每个rank日志的第一行记录了启动该rank时使用的确切命令和环境。`proxy.log`仅存在于配置的代理节点上，通常是节点0。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:535
|
||||
msgid "Use a separate log directory when running multiple local experiments:"
|
||||
msgstr "运行多个本地实验时使用单独的日志目录："
|
||||
|
||||
#: ../../source/developer_guide/contribution/multi_node_test.md:541
|
||||
msgid ""
|
||||
"To watch logs in real time, run these commands in another terminal on the "
|
||||
"corresponding node:"
|
||||
msgstr "要实时查看日志，请在对应节点的另一个终端中运行以下命令："
|
||||
@@ -0,0 +1,929 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:1
|
||||
msgid "Nightly CI Test"
|
||||
msgstr "Nightly CI 测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:3
|
||||
msgid ""
|
||||
"This document explains how to trigger nightly hardware CI tests against your"
|
||||
" own PR code on Ascend NPU hardware (A2/A3), without waiting for the "
|
||||
"scheduled nightly run."
|
||||
msgstr "本文档说明如何在 Ascend NPU 硬件（A2/A3）上针对您自己的 PR 代码触发夜间硬件 CI 测试，而无需等待预定的夜间运行。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:6
|
||||
msgid "Background"
|
||||
msgstr "背景"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:8
|
||||
msgid ""
|
||||
"By default, nightly CI tests run on a fixed schedule using pre-built nightly"
|
||||
" images. Contributors can self-service trigger these tests directly against "
|
||||
"their PR changes by combining a GitHub label with a comment command."
|
||||
msgstr ""
|
||||
"默认情况下，夜间 CI 测试按固定计划使用预构建的夜间镜像运行。贡献者可以通过结合 GitHub 标签和评论命令，自助直接针对其 PR "
|
||||
"更改触发这些测试。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:12
|
||||
msgid "How to Trigger"
|
||||
msgstr "如何触发"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:14
|
||||
msgid "1. Post a comment"
|
||||
msgstr "1.发表评论"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:16
|
||||
msgid ""
|
||||
"Post one of the following comments in the PR to specify which tests to run. "
|
||||
"The comment itself triggers the workflow — no label is required."
|
||||
msgstr "在 PR 中发表以下评论之一以指定要运行的测试。评论本身即可触发工作流——无需标签。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md
|
||||
msgid "Comment"
|
||||
msgstr "评论"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md
|
||||
msgid "Effect"
|
||||
msgstr "效果"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md
|
||||
msgid "`/nightly`"
|
||||
msgstr "`/nightly`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md
|
||||
msgid "Run **all** nightly tests"
|
||||
msgstr "运行**所有**夜间测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md
|
||||
msgid "`/nightly all`"
|
||||
msgstr "`/nightly all`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md
|
||||
msgid "Run **all** nightly tests (same as above)"
|
||||
msgstr "运行**所有**夜间测试（同上）"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md
|
||||
msgid "`/nightly test1 test2 ...`"
|
||||
msgstr "`/nightly test1 test2 ...`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md
|
||||
msgid "Run only the **named** tests"
|
||||
msgstr "仅运行**指定名称**的测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:26
|
||||
msgid ""
|
||||
"Only repository **Contributors** (Triage role) and **Maintainers** (Write "
|
||||
"role) can trigger the `/nightly` command. If you do not have this "
|
||||
"permission, ask a maintainer to post the comment for you. You can find the "
|
||||
"list of maintainers and contributors in the project's "
|
||||
"[Governance](../../community/governance.md) page or by checking the "
|
||||
"[CODEOWNERS](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/CODEOWNERS) file."
|
||||
msgstr ""
|
||||
"只有仓库的**贡献者**（Triage 角色）和**维护者**（Write 角色）才能触发 `/nightly` "
|
||||
"命令。如果您没有此权限，请让维护者为您发表评论。您可以在项目的[治理](../../community/governance.md)页面或查看 "
|
||||
"[CODEOWNERS](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/CODEOWNERS) 文件中找到维护者和贡献者列表。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:34
|
||||
msgid "2. Wait for results"
|
||||
msgstr "2.等待结果"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:36
|
||||
msgid ""
|
||||
"GitHub Actions will trigger the `Nightly-A2` or `Nightly-A3` workflow. Only "
|
||||
"tests matching the filter will be dispatched, which saves hardware "
|
||||
"resources."
|
||||
msgstr ""
|
||||
"GitHub Actions 将触发 `Nightly-A2` 或 `Nightly-A3` 工作流。仅会调度与过滤器匹配的测试，从而节省硬件资源。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:39
|
||||
msgid "Differences Between PR and Scheduled Runs"
|
||||
msgstr "PR 运行与计划运行之间的差异"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Scheduled / Manual Dispatch"
|
||||
msgstr "计划/手动调度"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "PR-triggered"
|
||||
msgstr "PR 触发"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Trigger"
|
||||
msgstr "触发方式"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Cron (daily) or `workflow_dispatch`"
|
||||
msgstr "Cron（每日）或 `workflow_dispatch`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`/nightly` comment"
|
||||
msgstr "`/nightly` 评论"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Code tested"
|
||||
msgstr "测试的代码"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Pre-built nightly image"
|
||||
msgstr "预构建的夜间镜像"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Your PR's HEAD commit (source installed fresh)"
|
||||
msgstr "您的 PR 的 HEAD 提交（全新安装的源码）"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Test scope"
|
||||
msgstr "测试范围"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "All tests"
|
||||
msgstr "所有测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Configurable via `/nightly <names>`"
|
||||
msgstr "可通过 `/nightly <names>` 配置"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "vLLM + vllm-ascend"
|
||||
msgstr "vLLM + vllm-ascend"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "From image"
|
||||
msgstr "来自镜像"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Checked out and installed from source"
|
||||
msgstr "从源码检出并安装"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Test matrix"
|
||||
msgstr "测试矩阵"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "From main branch's matrix YAML"
|
||||
msgstr "来自主分支的矩阵 YAML"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "From PR branch's matrix YAML"
|
||||
msgstr "来自 PR 分支的矩阵 YAML"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:49
|
||||
msgid ""
|
||||
"When a PR run is detected (`is_pr_test: true`), the workflow additionally:"
|
||||
msgstr "当检测到 PR 运行时（`is_pr_test: true`），工作流还会执行以下操作："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:51
|
||||
msgid "Uninstalls any existing vllm packages in the container."
|
||||
msgstr "卸载容器中所有现有的 vllm 包。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:52
|
||||
msgid ""
|
||||
"Checks out the specific vllm version and your PR's vllm-ascend commit from "
|
||||
"source."
|
||||
msgstr "从源码检出特定的 vllm 版本和您的 PR 的 vllm-ascend 提交。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:53
|
||||
msgid "Installs all dependencies from source."
|
||||
msgstr "从源码安装所有依赖项。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:54
|
||||
msgid "Installs the `aisbench` benchmark suite."
|
||||
msgstr "安装 `aisbench` 基准测试套件。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:56
|
||||
msgid "Test Matrix Data Source"
|
||||
msgstr "测试矩阵数据源"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:58
|
||||
msgid ""
|
||||
"The set of nightly test cases (their names, runners, test paths, model "
|
||||
"configs) is declared in a single data file:"
|
||||
msgstr "夜间测试用例集（其名称、运行器、测试路径、模型配置）在单个数据文件中声明："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:65
|
||||
msgid ""
|
||||
"The file is organized as `a2:` and `a3:` top-level keys (one per SoC). Under"
|
||||
" each SoC, tests are grouped by execution shape (single-node, multi-node, "
|
||||
"double-node, multi-card, accuracy) and each group holds a `test_config` (or "
|
||||
"`nightly` / `pr_only` for accuracy) list whose entries carry a `name` plus "
|
||||
"the fields consumed by the downstream reusable workflows (`os`, `tests`, "
|
||||
"`config_file_path`, `size`, etc.)."
|
||||
msgstr ""
|
||||
"该文件组织为 `a2:` 和 `a3:` 顶级键（每个 SoC 一个）。在每个 SoC "
|
||||
"下，测试按执行形态（单节点、多节点、双节点、多卡、精度）分组，每个组包含一个 `test_config`（或用于精度的 `nightly` / "
|
||||
"`pr_only`）列表，其条目带有 `name` "
|
||||
"以及下游可复用工作流使用的字段（`os`、`tests`、`config_file_path`、`size` 等）。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:71
|
||||
msgid ""
|
||||
"Both the `Nightly-A2` and `Nightly-A3` workflows dynamically read this file "
|
||||
"at run time — there is no hardcoded test matrix in the workflow YAMLs. The "
|
||||
"`/nightly <name>` slash command resolves names by walking the same file from"
|
||||
" the PR branch, so newly added entries can be exercised on a PR before they "
|
||||
"land on main."
|
||||
msgstr ""
|
||||
"`Nightly-A2` 和 `Nightly-A3` 工作流都在运行时动态读取此文件——工作流 YAML 中没有硬编码的测试矩阵。`/nightly "
|
||||
"<name>` 斜杠命令通过从 PR 分支遍历同一文件来解析名称，因此新添加的条目可以在合并到主分支之前在 PR 上进行测试。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:77
|
||||
msgid "Adding a New Nightly Test Case"
|
||||
msgstr "添加新的夜间测试用例"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:79
|
||||
msgid "To add a new test case (no need to touch the workflow YAMLs):"
|
||||
msgstr "要添加新的测试用例（无需修改工作流 YAML）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:81
|
||||
msgid ""
|
||||
"Append an entry under the appropriate section in "
|
||||
"`.github/workflows/configs/nightly_config.yaml`. Each entry needs at least:"
|
||||
msgstr ""
|
||||
"在 `.github/workflows/configs/nightly_config.yaml` 的相应部分下追加一个条目。每个条目至少需要："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:83
|
||||
msgid "`name`: unique identifier used in `/nightly <name>` filters"
|
||||
msgstr "`name`：用于 `/nightly <name>` 过滤器的唯一标识符"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:84
|
||||
msgid ""
|
||||
"`os` (for single-node / multi-card pytest+yaml tests) or `runner` is "
|
||||
"inferred"
|
||||
msgstr "`os`（用于单节点/多卡 pytest+yaml 测试）或 `runner`（可推断）"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:85
|
||||
msgid ""
|
||||
"one of `tests:` (pytest directory) or `config_file_path:` (YAML-driven model"
|
||||
" config)"
|
||||
msgstr "`tests:`（pytest 目录）或 `config_file_path:`（YAML 驱动的模型配置）之一"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:86
|
||||
msgid "`size` (multi-node / double-node only)"
|
||||
msgstr "`size`（仅限多节点/双节点）"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:87
|
||||
msgid ""
|
||||
"Add the actual test files (pytest modules under `tests/e2e/nightly/...` or "
|
||||
"YAML model configs in `tests/e2e/nightly/.../configs/`)."
|
||||
msgstr ""
|
||||
"添加实际的测试文件（`tests/e2e/nightly/...` 下的 pytest 模块或 "
|
||||
"`tests/e2e/nightly/.../configs/` 中的 YAML 模型配置）。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:89
|
||||
msgid ""
|
||||
"Open a PR. Once CI is green, you can validate the new entry against real NPU"
|
||||
" hardware **without** merging the PR — see *Examples* below."
|
||||
msgstr "打开一个 PR。一旦 CI 通过，您可以在**不**合并 PR 的情况下，针对真实 NPU 硬件验证新条目——请参阅下面的*示例*。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:92
|
||||
msgid "Available Test Names"
|
||||
msgstr "可用的测试名称"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:94
|
||||
msgid ""
|
||||
"The test names you can pass to `/nightly` correspond to the `name` fields "
|
||||
"under the matching section in "
|
||||
"`.github/workflows/configs/nightly_config.yaml`. The tables below mirror the"
|
||||
" current contents of that file."
|
||||
msgstr ""
|
||||
"您可以传递给 `/nightly` 的测试名称对应于 `.github/workflows/configs/nightly_config.yaml` "
|
||||
"中匹配部分下的 `name` 字段。下表反映了该文件的当前内容。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:98
|
||||
msgid "A2 workflow (`.github/workflows/schedule_nightly_test_a2.yaml`)"
|
||||
msgstr "A2 工作流（`.github/workflows/schedule_nightly_test_a2.yaml`）"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:100
|
||||
msgid "**Single-node tests** (`a2.single_node.test_config`):"
|
||||
msgstr "**单节点测试**（`a2.single_node.test_config`）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Test name"
|
||||
msgstr "测试名称"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Description"
|
||||
msgstr "描述"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`test_custom_op_multi_card`"
|
||||
msgstr "`test_custom_op_multi_card`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Custom operator tests (multi card)"
|
||||
msgstr "自定义算子测试（多卡）"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`qwen3-vl-32b-instruct-w8a8`"
|
||||
msgstr "`qwen3-vl-32b-instruct-w8a8`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-VL-32B-Instruct W8A8"
|
||||
msgstr "Qwen3-VL-32B-Instruct W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`qwen3-32b-int8`"
|
||||
msgstr "`qwen3-32b-int8`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-32B INT8 quantization"
|
||||
msgstr "Qwen3-32B INT8 量化"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`Qwen3.5-27B-w8a8-A2`"
|
||||
msgstr "`Qwen3.5-27B-w8a8-A2`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3.5-27B W8A8"
|
||||
msgstr "Qwen3.5-27B W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`Qwen3.5-397B-A17B-w4a8-mtp`"
|
||||
msgstr "`Qwen3.5-397B-A17B-w4a8-mtp`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3.5-397B-A17B W4A8 + MTP"
|
||||
msgstr "Qwen3.5-397B-A17B W4A8 + MTP"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:110
|
||||
msgid "**Multi-node tests** (`a2.multi_node.test_config`):"
|
||||
msgstr "**多节点测试**（`a2.multi_node.test_config`）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-qwen3-235b-dp`"
|
||||
msgstr "`multi-node-qwen3-235b-dp`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-235B-A22B, 2-node DP"
|
||||
msgstr "Qwen3-235B-A22B，2 节点 DP"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-GLM-5.1-w8a8-A2`"
|
||||
msgstr "`multi-node-GLM-5.1-w8a8-A2`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "GLM-5.1 W8A8, 2 nodes"
|
||||
msgstr "GLM-5.1 W8A8，2节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-Kimi-K2.5-W4A8-A2`"
|
||||
msgstr "`multi-node-Kimi-K2.5-W4A8-A2`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Kimi-K2.5 W4A8, 2 nodes"
|
||||
msgstr "Kimi-K2.5 W4A8，2节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:118
|
||||
msgid "**Accuracy tests** (`a2.accuracy.nightly` and `a2.accuracy.pr_only`):"
|
||||
msgstr "**精度测试**（`a2.accuracy.nightly` 和 `a2.accuracy.pr_only`）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Scope"
|
||||
msgstr "范围"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`accuracy-group-1`"
|
||||
msgstr "`accuracy-group-1`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-VL-8B, Qwen3-8B, Qwen2-Audio-7B, etc."
|
||||
msgstr "Qwen3-VL-8B、Qwen3-8B、Qwen2-Audio-7B 等"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "nightly"
|
||||
msgstr "nightly"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`accuracy-group-2`"
|
||||
msgstr "`accuracy-group-2`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "ERNIE-4.5, Molmo-7B, Llama-3.2-3B, etc."
|
||||
msgstr "ERNIE-4.5、Molmo-7B、Llama-3.2-3B 等"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`accuracy-group-3`"
|
||||
msgstr "`accuracy-group-3`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-30B-A3B, Qwen3-VL-30B-A3B, etc."
|
||||
msgstr "Qwen3-30B-A3B、Qwen3-VL-30B-A3B 等"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`accuracy-group-4`"
|
||||
msgstr "`accuracy-group-4`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-Next-80B-A3B, Qwen3-Omni-30B-A3B, etc."
|
||||
msgstr "Qwen3-Next-80B-A3B、Qwen3-Omni-30B-A3B 等"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`pr-accuracy-group-1`"
|
||||
msgstr "`pr-accuracy-group-1`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "gemma-3-4b-it, internlm3-8b-instruct, etc."
|
||||
msgstr "gemma-3-4b-it、internlm3-8b-instruct 等"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "pr_only"
|
||||
msgstr "pr_only"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`pr-accuracy-group-2`"
|
||||
msgstr "`pr-accuracy-group-2`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen2.5-Math-RM-72B, Hunyuan-A13B-Instruct"
|
||||
msgstr "Qwen2.5-Math-RM-72B、Hunyuan-A13B-Instruct"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:129
|
||||
msgid ""
|
||||
"The `pr-accuracy-group-*` entries only run on `/nightly` (PR-triggered) "
|
||||
"runs; `/nightly all` on the schedule skips them."
|
||||
msgstr ""
|
||||
"`pr-accuracy-group-*` 条目仅在 `/nightly`（PR 触发）运行中执行；定时触发的 `/nightly all` "
|
||||
"会跳过它们。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:132
|
||||
msgid "A3 workflow (`.github/workflows/schedule_nightly_test_a3.yaml`)"
|
||||
msgstr "A3 工作流（`.github/workflows/schedule_nightly_test_a3.yaml`）"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:134
|
||||
msgid "**Multi-node tests** (`a3.multi_node.test_config`, 4-node):"
|
||||
msgstr "**多节点测试**（`a3.multi_node.test_config`，4节点）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-deepseek-v3.2-W8A8-EP`"
|
||||
msgstr "`multi-node-deepseek-v3.2-W8A8-EP`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "DeepSeek-V3.2-W8A8 with EP, 4-node"
|
||||
msgstr "DeepSeek-V3.2-W8A8 带 EP，4节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:140
|
||||
msgid ""
|
||||
"**Double-node tests** (`a3.double_node.test_config`, 2-node, run after "
|
||||
"multi-node):"
|
||||
msgstr "**双节点测试**（`a3.double_node.test_config`，2节点，在多节点测试后运行）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-deepseek-r1-w8a8-longseq`"
|
||||
msgstr "`multi-node-deepseek-r1-w8a8-longseq`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "DeepSeek-R1-W8A8 long sequence, 2-node"
|
||||
msgstr "DeepSeek-R1-W8A8 长序列，2节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-qwen3-dp`"
|
||||
msgstr "`multi-node-qwen3-dp`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-qwenw8a8-2node-eplb`"
|
||||
msgstr "`multi-node-qwenw8a8-2node-eplb`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-235B-W8A8 with EPLB, 2-node"
|
||||
msgstr "Qwen3-235B-W8A8 带 EPLB，2节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-dpsk3.2-2node`"
|
||||
msgstr "`multi-node-dpsk3.2-2node`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "DeepSeek-V3.2-W8A8, 2-node"
|
||||
msgstr "DeepSeek-V3.2-W8A8，2节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-qwenw8a8-2node-longseq`"
|
||||
msgstr "`multi-node-qwenw8a8-2node-longseq`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-235B-W8A8 long sequence, 2-node"
|
||||
msgstr "Qwen3-235B-W8A8 长序列，2节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-qwen-disagg-pd`"
|
||||
msgstr "`multi-node-qwen-disagg-pd`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-235B disaggregated PD, 2-node"
|
||||
msgstr "Qwen3-235B 分离式 PD，2节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-qwen-vl-disagg-pd`"
|
||||
msgstr "`multi-node-qwen-vl-disagg-pd`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-VL-235B disaggregated PD, 2-node"
|
||||
msgstr "Qwen3-VL-235B 分离式 PD，2节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-deepseek-v3.1`"
|
||||
msgstr "`multi-node-deepseek-v3.1`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "DeepSeek-V3.1-BF16, 2-node"
|
||||
msgstr "DeepSeek-V3.1-BF16，2节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`multi-node-glm-5.2`"
|
||||
msgstr "`multi-node-glm-5.2`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "GLM-5.1-W8A8, 2-node"
|
||||
msgstr "GLM-5.1-W8A8，2节点"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:155
|
||||
msgid "**Single-node tests** (`a3.single_node.test_config`):"
|
||||
msgstr "**单节点测试**（`a3.single_node.test_config`）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`mtpx-deepseek-r1-0528-w8a8`"
|
||||
msgstr "`mtpx-deepseek-r1-0528-w8a8`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "MTP-X + DeepSeek-R1-0528-W8A8"
|
||||
msgstr "MTP-X + DeepSeek-R1-0528-W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`deepseek-r1-0528-w8a8`"
|
||||
msgstr "`deepseek-r1-0528-w8a8`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "DeepSeek-R1-0528-W8A8"
|
||||
msgstr "DeepSeek-R1-0528-W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`kimi-k2-thinking`"
|
||||
msgstr "`kimi-k2-thinking`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Kimi-K2-Thinking"
|
||||
msgstr "Kimi-K2-Thinking"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`qwen3-vl-235b-a22b-instruct-w8a8`"
|
||||
msgstr "`qwen3-vl-235b-a22b-instruct-w8a8`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-VL-235B-A22B-Instruct-W8A8"
|
||||
msgstr "Qwen3-VL-235B-A22B-Instruct-W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`deepseek-r1-0528-w8a8-prefix-cache`"
|
||||
msgstr "`deepseek-r1-0528-w8a8-prefix-cache`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "DeepSeek-R1-0528-W8A8 prefix cache"
|
||||
msgstr "DeepSeek-R1-0528-W8A8 前缀缓存"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`deepseek-v3-2-w8a8`"
|
||||
msgstr "`deepseek-v3-2-w8a8`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "DeepSeek-V3.2-W8A8"
|
||||
msgstr "DeepSeek-V3.2-W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`glm-4.7-w8a8`"
|
||||
msgstr "`glm-4.7-w8a8`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "GLM-4.7 W8A8"
|
||||
msgstr "GLM-4.7 W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`kimi-k2.5`"
|
||||
msgstr "`kimi-k2.5`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Kimi-K2.5"
|
||||
msgstr "Kimi-K2.5"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`qwen3-235b-a22b-w8a8`"
|
||||
msgstr "`qwen3-235b-a22b-w8a8`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-235B-A22B-W8A8"
|
||||
msgstr "Qwen3-235B-A22B-W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`Qwen3.5-397B-A17B-w8a8-mtp`"
|
||||
msgstr "`Qwen3.5-397B-A17B-w8a8-mtp`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3.5-397B-A17B W8A8 + MTP"
|
||||
msgstr "Qwen3.5-397B-A17B W8A8 + MTP"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`MiniMax-M2.5-w8a8-QuaRot-A3`"
|
||||
msgstr "`MiniMax-M2.5-w8a8-QuaRot-A3`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "MiniMax-M2.5 W8A8 + QuaRot"
|
||||
msgstr "MiniMax-M2.5 W8A8 + QuaRot"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`Qwen3.5-27B-w8a8-A3`"
|
||||
msgstr "`Qwen3.5-27B-w8a8-A3`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`Qwen3.5-122B-A10B-W8A8-A3`"
|
||||
msgstr "`Qwen3.5-122B-A10B-W8A8-A3`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3.5-122B-A10B W8A8"
|
||||
msgstr "Qwen3.5-122B-A10B W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`DeepSeek-V4-Flash-W8A8-A3`"
|
||||
msgstr "`DeepSeek-V4-Flash-W8A8-A3`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "DeepSeek-V4-Flash W8A8"
|
||||
msgstr "DeepSeek-V4-Flash W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:174
|
||||
msgid "**Multi-card tests** (`a3.multi_card.test_config`):"
|
||||
msgstr "**多卡测试**（`a3.multi_card.test_config`）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`qwen3-30b-acc`"
|
||||
msgstr "`qwen3-30b-acc`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-30B accuracy test"
|
||||
msgstr "Qwen3-30B 精度测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`qwen3-30b-a3b-w8a8`"
|
||||
msgstr "`qwen3-30b-a3b-w8a8`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-30B-A3B-W8A8"
|
||||
msgstr "Qwen3-30B-A3B-W8A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-32B-Int8"
|
||||
msgstr "Qwen3-32B-Int8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`qwen3-32b-int8-prefix-cache`"
|
||||
msgstr "`qwen3-32b-int8-prefix-cache`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-32B-Int8 prefix cache"
|
||||
msgstr "Qwen3-32B-Int8 前缀缓存"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`Qwen3-30B-A3B-W4A8-llm-compressor`"
|
||||
msgstr "`Qwen3-30B-A3B-W4A8-llm-compressor`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-30B-A3B W4A8 via llm-compressor"
|
||||
msgstr "通过 llm-compressor 实现的 Qwen3-30B-A3B W4A8"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`Qwen3-30B-QuaRot`"
|
||||
msgstr "`Qwen3-30B-QuaRot`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-30B QuaRot + eagle3"
|
||||
msgstr "Qwen3-30B QuaRot + eagle3"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "`Qwen3-32B-QuaRot`"
|
||||
msgstr "`Qwen3-32B-QuaRot`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:25
|
||||
msgid "Qwen3-32B QuaRot + eagle3"
|
||||
msgstr "Qwen3-32B QuaRot + eagle3"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:187
|
||||
msgid ""
|
||||
"The A3 resource pool has a maximum concurrency of **5×16 NPUs**. Multi-node "
|
||||
"tests run with `max-parallel: 2` to avoid resource exhaustion. Running "
|
||||
"`/nightly all` on A3 will queue a large number of jobs — prefer targeting "
|
||||
"specific test names when possible."
|
||||
msgstr ""
|
||||
"A3 资源池的最大并发数为 **5×16 NPU**。多节点测试使用 `max-parallel: 2` 运行，以避免资源耗尽。在 A3 上运行 "
|
||||
"`/nightly all` 会排队大量任务——请尽可能指定具体的测试名称。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:193
|
||||
msgid "Examples"
|
||||
msgstr "示例"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:195
|
||||
msgid "Run all available nightly tests against your PR:"
|
||||
msgstr "针对你的 PR 运行所有可用的 nightly 测试："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:201
|
||||
msgid "Run only the custom operator multi-card test:"
|
||||
msgstr "仅运行自定义算子多卡测试："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:207
|
||||
msgid "Run two specific tests at once (one per SoC):"
|
||||
msgstr "同时运行两个特定测试（每个 SoC 一个）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:213
|
||||
msgid "Run a single accuracy group (with all of its models):"
|
||||
msgstr "运行单个精度组（包含其所有模型）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:219
|
||||
msgid "Run a single accuracy model (only that model from a group):"
|
||||
msgstr "运行单个精度模型（仅运行组中的该模型）："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:225
|
||||
msgid ""
|
||||
"Re-trigger after fixing an issue: just push a new commit. The `synchronize` "
|
||||
"event re-runs the workflow and picks up the existing `/nightly` comment "
|
||||
"automatically — no need to post a new comment."
|
||||
msgstr ""
|
||||
"修复问题后重新触发：只需推送一个新提交。`synchronize` 事件会重新运行工作流，并自动拾取已有的 `/nightly` "
|
||||
"评论——无需发布新评论。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:229
|
||||
msgid "Adding a New Test Case — Worked Example"
|
||||
msgstr "添加新测试用例——操作示例"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:231
|
||||
msgid "To add `my-new-test` to the A2 single-node section:"
|
||||
msgstr "要将 `my-new-test` 添加到 A2 单节点部分："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:233
|
||||
msgid ""
|
||||
"Edit `.github/workflows/configs/nightly_config.yaml`, append under "
|
||||
"`a2.single_node.test_config`:"
|
||||
msgstr ""
|
||||
"编辑 `.github/workflows/configs/nightly_config.yaml`，在 "
|
||||
"`a2.single_node.test_config` 下追加："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:242
|
||||
msgid "Commit the new pytest file (`test_my_new.py`) in the same PR."
|
||||
msgstr "在同一 PR 中提交新的 pytest 文件（`test_my_new.py`）。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:244
|
||||
msgid "Trigger from the PR:"
|
||||
msgstr "从 PR 触发："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:250
|
||||
msgid "The workflow will:"
|
||||
msgstr "工作流将："
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:252
|
||||
msgid ""
|
||||
"`pr_nightly_command.yml` reads your PR's `nightly_config.yaml` and resolves "
|
||||
"`my-new-test` → dispatch A2 only."
|
||||
msgstr ""
|
||||
"`pr_nightly_command.yml` 读取你 PR 中的 `nightly_config.yaml`，并将 `my-new-test` "
|
||||
"解析为仅调度 A2。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:254
|
||||
msgid ""
|
||||
"`Nightly-A2` is dispatched at `main`, but `generate-a2-matrix` checks out "
|
||||
"your PR commit and reads the new entry from the matrix."
|
||||
msgstr ""
|
||||
"`Nightly-A2` 在 `main` 上被调度，但 `generate-a2-matrix` 会检出你的 PR 提交并从矩阵中读取新条目。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:256
|
||||
msgid ""
|
||||
"`single-node-tests` runs one matrix job for `my-new-test`, with "
|
||||
"`should_run=true`. The reusable workflow checks out your PR code (via "
|
||||
"`vllm_ascend_ref`) and runs your pytest."
|
||||
msgstr ""
|
||||
"`single-node-tests` 为 `my-new-test` 运行一个矩阵任务，其中 "
|
||||
"`should_run=true`。可复用工作流会检出你的 PR 代码（通过 `vllm_ascend_ref`）并运行你的 pytest。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:260
|
||||
msgid "Troubleshooting"
|
||||
msgstr "故障排除"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:262
|
||||
msgid "**The workflow didn't start after I posted the comment.**"
|
||||
msgstr "**发布评论后工作流未启动。**"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:264
|
||||
msgid ""
|
||||
"Check that the comment starts exactly with `/nightly` with no leading spaces"
|
||||
" or extra characters before the slash."
|
||||
msgstr "请检查评论是否以 `/nightly` 开头，且斜杠前无前导空格或额外字符。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:266
|
||||
msgid ""
|
||||
"Confirm you have at least Triage permission on the repository; unauthorized "
|
||||
"users' comments are ignored."
|
||||
msgstr "确认您对该仓库至少拥有 Triage 权限；未经授权用户的评论将被忽略。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:268
|
||||
msgid ""
|
||||
"To re-trigger after fixing an issue, simply push a new commit — the workflow"
|
||||
" will reuse the existing `/nightly` comment automatically."
|
||||
msgstr "修复问题后重新触发：只需推送新提交——工作流将自动复用已有的 `/nightly` 评论。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:271
|
||||
msgid "**Only some tests ran, not the ones I expected.**"
|
||||
msgstr "**仅运行了部分测试，并非预期测试。**"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:273
|
||||
msgid ""
|
||||
"Test names are case-sensitive and must match the `name` field in "
|
||||
"`.github/workflows/configs/nightly_config.yaml` exactly (see the tables "
|
||||
"above)."
|
||||
msgstr ""
|
||||
"测试名称区分大小写，必须与 `.github/workflows/configs/nightly_config.yaml` 中的 `name` "
|
||||
"字段完全一致（参见上表）。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:275
|
||||
msgid ""
|
||||
"For a PR-triggered run, the matrix is loaded from your PR's "
|
||||
"`nightly_config.yaml`, not main. If a name isn't in your PR's file, it won't"
|
||||
" be recognized and the dispatch will be skipped."
|
||||
msgstr ""
|
||||
"对于 PR 触发的运行，矩阵从您 PR 的 `nightly_config.yaml` 加载，而非主分支。若名称不在您的 PR "
|
||||
"文件中，将无法识别并跳过调度。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:278
|
||||
msgid ""
|
||||
"Check the `parse-trigger` job output in GitHub Actions for the resolved "
|
||||
"`test_filter` value."
|
||||
msgstr "请检查 GitHub Actions 中 `parse-trigger` 任务的输出，以获取解析后的 `test_filter` 值。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:281
|
||||
msgid "**The workflow ran with the scheduled image, not my PR code.**"
|
||||
msgstr "**工作流使用了定时镜像运行，而非我的 PR 代码。**"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:283
|
||||
msgid ""
|
||||
"Confirm the workflow was triggered by `repository_dispatch` (slash command),"
|
||||
" not bare `workflow_dispatch`. The `pr_nightly_command.yml` workflow is what"
|
||||
" actually dispatches `schedule_nightly_test_a2.yaml` / `_a3.yaml` with "
|
||||
"`vllm_ascend_ref` pointing at your PR SHA."
|
||||
msgstr ""
|
||||
"确认工作流由 `repository_dispatch`（斜杠命令）触发，而非裸 `workflow_dispatch`。实际由 "
|
||||
"`pr_nightly_command.yml` 工作流调度 `schedule_nightly_test_a2.yaml` / "
|
||||
"`_a3.yaml`，其中 `vllm_ascend_ref` 指向您的 PR SHA。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:288
|
||||
msgid "**A new test I added isn't being recognized.**"
|
||||
msgstr "**我添加的新测试未被识别。**"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:290
|
||||
msgid ""
|
||||
"Confirm the entry is well-formed YAML under "
|
||||
"`.github/workflows/configs/nightly_config.yaml`. The `name` field is "
|
||||
"required and must be unique within the SoC's section."
|
||||
msgstr ""
|
||||
"确认条目在 `.github/workflows/configs/nightly_config.yaml` 中为格式正确的 YAML。`name` "
|
||||
"字段为必填项，且在 SoC 部分内必须唯一。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:293
|
||||
msgid ""
|
||||
"The matrix is loaded from your PR branch, so make sure the file is committed"
|
||||
" to the same branch the `/nightly` comment was posted on."
|
||||
msgstr "矩阵从您的 PR 分支加载，请确保文件已提交至发布 `/nightly` 评论的同一分支。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:296
|
||||
msgid ""
|
||||
"**How to obtain more detailed logs to pinpoint problems for multi-node "
|
||||
"tests**"
|
||||
msgstr "**如何获取更详细的日志以定位多节点测试问题**"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:298
|
||||
msgid ""
|
||||
"For most issues, the stdout pop-up logs from GitHub actions are sufficient "
|
||||
"(this log always represents the logs from the first node)."
|
||||
msgstr "对于大多数问题，GitHub Actions 的标准输出弹出日志已足够（该日志始终代表第一个节点的日志）。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/nightly_ci_test.md:299
|
||||
msgid ""
|
||||
"If the logs from a first node are no longer sufficient to provide effective "
|
||||
"logging information, see the summary of your jobs to download log archive "
|
||||
"for the corresponding test, which includes the framework-side logs and plog "
|
||||
"information for each node, structured as follows:"
|
||||
msgstr ""
|
||||
"若第一个节点的日志不足以提供有效日志信息，请查看任务摘要以下载对应测试的日志归档，其中包含每个节点的框架侧日志和 plog 信息，结构如下："
|
||||
@@ -2,14 +2,14 @@
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2025.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-07-18 09:01+0800\n"
|
||||
"POT-Creation-Date: 2026-08-05 06:27+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
@@ -18,88 +18,88 @@ msgstr ""
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.17.0\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:1
|
||||
#: ../../source/developer_guide/contribution/testing.md:1
|
||||
msgid "Testing"
|
||||
msgstr "测试"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:3
|
||||
#: ../../source/developer_guide/contribution/testing.md:3
|
||||
msgid ""
|
||||
"This secition explains how to write e2e tests and unit tests to verify the "
|
||||
"implementation of your feature."
|
||||
msgstr "本节介绍如何编写端到端测试和单元测试，以验证你的功能实现。"
|
||||
"This document explains how to write unit tests, E2E tests, and nightly tests"
|
||||
" to verify your feature implementation."
|
||||
msgstr "本文档说明如何编写单元测试、端到端测试和夜间测试，以验证您的功能实现。"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:5
|
||||
msgid "Setup test environment"
|
||||
#: ../../source/developer_guide/contribution/testing.md:5
|
||||
msgid "Set up a test environment"
|
||||
msgstr "设置测试环境"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:7
|
||||
#: ../../source/developer_guide/contribution/testing.md:7
|
||||
msgid ""
|
||||
"The fastest way to setup test environment is to use the main branch "
|
||||
"The fastest way to set up a test environment is to use the main branch's "
|
||||
"container image:"
|
||||
msgstr "搭建测试环境最快的方法是使用 main 分支的容器镜像："
|
||||
msgstr "设置测试环境最快的方法是使用主分支的容器镜像："
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md
|
||||
#: ../../source/developer_guide/contribution/testing.md
|
||||
msgid "Local (CPU)"
|
||||
msgstr "本地（CPU）"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:18
|
||||
msgid "You can run the unit tests on CPU with the following steps:"
|
||||
msgstr "你可以按照以下步骤在 CPU 上运行单元测试："
|
||||
#: ../../source/developer_guide/contribution/testing.md:18
|
||||
msgid "You can run the unit tests on CPUs with the following steps:"
|
||||
msgstr "您可以通过以下步骤在CPU上运行单元测试："
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md
|
||||
msgid "Single card"
|
||||
msgstr "单张卡片"
|
||||
#: ../../source/developer_guide/contribution/testing.md
|
||||
msgid "Single-card"
|
||||
msgstr "单卡"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:85
|
||||
#: ../../developer_guide/contribution/testing.md:123
|
||||
#: ../../source/developer_guide/contribution/testing.md:97
|
||||
#: ../../source/developer_guide/contribution/testing.md:139
|
||||
msgid ""
|
||||
"After starting the container, you should install the required packages:"
|
||||
msgstr "启动容器后，你应该安装所需的软件包："
|
||||
msgstr "启动容器后，您应安装所需的软件包："
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md
|
||||
msgid "Multi cards"
|
||||
#: ../../source/developer_guide/contribution/testing.md
|
||||
msgid "Multi-cards"
|
||||
msgstr "多卡"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:137
|
||||
#: ../../source/developer_guide/contribution/testing.md:153
|
||||
msgid "Running tests"
|
||||
msgstr "运行测试"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:139
|
||||
msgid "Unit test"
|
||||
#: ../../source/developer_guide/contribution/testing.md:155
|
||||
msgid "Unit tests"
|
||||
msgstr "单元测试"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:141
|
||||
#: ../../source/developer_guide/contribution/testing.md:157
|
||||
msgid "There are several principles to follow when writing unit tests:"
|
||||
msgstr "编写单元测试时需要遵循几个原则："
|
||||
msgstr "编写单元测试时应遵循以下几个原则："
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:143
|
||||
#: ../../source/developer_guide/contribution/testing.md:159
|
||||
msgid ""
|
||||
"The test file path should be consistent with source file and start with "
|
||||
"`test_` prefix, such as: `vllm_ascend/worker/worker_v1.py` --> "
|
||||
"`tests/ut/worker/test_worker_v1.py`"
|
||||
"The test file path should be consistent with the source file and start with "
|
||||
"the `test_` prefix, such as: `vllm_ascend/worker/worker.py` --> "
|
||||
"`tests/ut/worker/test_worker.py`"
|
||||
msgstr ""
|
||||
"测试文件的路径应与源文件保持一致，并以 `test_` 前缀开头，例如：`vllm_ascend/worker/worker_v1.py` --> "
|
||||
"`tests/ut/worker/test_worker_v1.py`"
|
||||
"测试文件路径应与源文件一致，并以 `test_` 前缀开头，例如：`vllm_ascend/worker/worker.py` --> "
|
||||
"`tests/ut/worker/test_worker.py`"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:144
|
||||
#: ../../source/developer_guide/contribution/testing.md:160
|
||||
msgid ""
|
||||
"The vLLM Ascend test are using unittest framework, see "
|
||||
"[here](https://docs.python.org/3/library/unittest.html#module-unittest) to "
|
||||
"understand how to write unit tests."
|
||||
"The vLLM Ascend test uses unittest framework. See [the Python unittest "
|
||||
"documentation](https://docs.python.org/3/library/unittest.html#module-"
|
||||
"unittest) to understand how to write unit tests."
|
||||
msgstr ""
|
||||
"vLLM Ascend 测试使用 unittest "
|
||||
"框架，参见[这里](https://docs.python.org/3/library/unittest.html#module-"
|
||||
"unittest)了解如何编写单元测试。"
|
||||
"vLLM Ascend 测试使用 unittest 框架。请参阅 [Python unittest "
|
||||
"文档](https://docs.python.org/3/library/unittest.html#module-unittest) "
|
||||
"了解如何编写单元测试。"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:145
|
||||
#: ../../source/developer_guide/contribution/testing.md:161
|
||||
msgid ""
|
||||
"All unit tests can be run on CPU, so you must mock the device-related "
|
||||
"function to host."
|
||||
msgstr "所有单元测试都可以在 CPU 上运行，因此你必须将与设备相关的函数模拟为 host。"
|
||||
"All unit tests can be run on CPUs, so you must mock the device-related "
|
||||
"functions on the host."
|
||||
msgstr "所有单元测试都可以在CPU上运行，因此您必须在主机上模拟与设备相关的函数。"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:146
|
||||
#: ../../source/developer_guide/contribution/testing.md:162
|
||||
msgid ""
|
||||
"Example: [tests/ut/test_ascend_config.py](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/tests/ut/test_ascend_config.py)."
|
||||
@@ -107,104 +107,105 @@ msgstr ""
|
||||
"示例：[tests/ut/test_ascend_config.py](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/tests/ut/test_ascend_config.py)。"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:147
|
||||
#: ../../source/developer_guide/contribution/testing.md:163
|
||||
msgid "You can run the unit tests using `pytest`:"
|
||||
msgstr "你可以使用 `pytest` 运行单元测试："
|
||||
msgstr "您可以使用 `pytest` 运行单元测试："
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md
|
||||
msgid "Multi cards test"
|
||||
msgstr "多卡测试"
|
||||
#: ../../source/developer_guide/contribution/testing.md
|
||||
msgid "Multi-card"
|
||||
msgstr "多卡"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:192
|
||||
#: ../../source/developer_guide/contribution/testing.md:210
|
||||
msgid "E2E test"
|
||||
msgstr "端到端测试"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:194
|
||||
#: ../../source/developer_guide/contribution/testing.md:212
|
||||
msgid ""
|
||||
"Although vllm-ascend CI provide [e2e test](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/.github/workflows/vllm_ascend_test.yaml) on "
|
||||
"Ascend CI, you can run it locally."
|
||||
"Although vllm-ascend CI provides E2E tests on Ascend CI (for example, "
|
||||
"[schedule_nightly_test_a2.yaml](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/schedule_nightly_test_a2.yaml), "
|
||||
"[schedule_nightly_test_a3.yaml](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/schedule_nightly_test_a3.yaml), "
|
||||
"[pr_test.yaml](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/pr_test.yaml)), you can run them locally."
|
||||
msgstr ""
|
||||
"虽然 vllm-ascend CI 在 Ascend CI 上提供了 [端到端测试](https://github.com/vllm-"
|
||||
"尽管 vllm-ascend CI 在 Ascend CI "
|
||||
"上提供了端到端测试（例如，[schedule_nightly_test_a2.yaml](https://github.com/vllm-"
|
||||
"project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/vllm_ascend_test.yaml)，你也可以在本地运行它。"
|
||||
"ascend/blob/main/.github/workflows/schedule_nightly_test_a2.yaml), "
|
||||
"[schedule_nightly_test_a3.yaml](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/schedule_nightly_test_a3.yaml), "
|
||||
"[pr_test.yaml](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/pr_test.yaml)），您也可以在本地运行它们。"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:204
|
||||
msgid "You can't run e2e test on CPU."
|
||||
msgstr "你无法在 CPU 上运行 e2e 测试。"
|
||||
#: ../../source/developer_guide/contribution/testing.md:215
|
||||
msgid "PR-triggered E2E test"
|
||||
msgstr "PR 触发的端到端测试"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:240
|
||||
#: ../../source/developer_guide/contribution/testing.md:217
|
||||
#: ../../source/developer_guide/contribution/testing.md:268
|
||||
msgid "You can run tests with `pytest` as well. Typical examples:"
|
||||
msgstr "您也可以使用 `pytest` 运行测试。典型示例："
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:225
|
||||
#: ../../source/developer_guide/contribution/testing.md:277
|
||||
msgid "You can't run the E2E test on CPUs."
|
||||
msgstr "您无法在CPU上运行端到端测试。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:264
|
||||
msgid "This will reproduce the E2E test behavior."
|
||||
msgstr "这将重现端到端测试的行为。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:266
|
||||
msgid "Nightly-triggered E2E test"
|
||||
msgstr "夜间触发的端到端测试"
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:307
|
||||
msgid ""
|
||||
"This will reproduce e2e test: "
|
||||
"[vllm_ascend_test.yaml](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/vllm_ascend_test.yaml)."
|
||||
msgstr ""
|
||||
"这将复现端到端测试：[vllm_ascend_test.yaml](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/vllm_ascend_test.yaml)。"
|
||||
"For running nightly single-node model test cases locally, refer to the "
|
||||
"following example."
|
||||
msgstr "要在本地运行夜间单节点模型测试用例，请参考以下示例。"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:242
|
||||
msgid "E2E test example:"
|
||||
msgstr "E2E 测试示例："
|
||||
#: ../../source/developer_guide/contribution/testing.md:314
|
||||
msgid ""
|
||||
"For running nightly multi-node model test cases locally, refer to the "
|
||||
"`Running Locally` section in [Multi Node Test](./multi_node_test.md)."
|
||||
msgstr "要在本地运行夜间多节点模型测试用例，请参考 [多节点测试](./multi_node_test.md) 中的 `本地运行` 部分。"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:244
|
||||
#: ../../source/developer_guide/contribution/testing.md:316
|
||||
msgid "E2E test examples"
|
||||
msgstr "端到端测试示例"
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:318
|
||||
msgid ""
|
||||
"Offline test example: "
|
||||
"[`tests/e2e/singlecard/test_offline_inference.py`](https://github.com/vllm-"
|
||||
"project/vllm-"
|
||||
"ascend/blob/main/tests/e2e/singlecard/test_offline_inference.py)"
|
||||
"[`tests/e2e/pull_request/one_card/test_camem.py`](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/tests/e2e/pull_request/one_card/test_camem.py)"
|
||||
msgstr ""
|
||||
"离线测试示例：[`tests/e2e/singlecard/test_offline_inference.py`](https://github.com/vllm-"
|
||||
"project/vllm-"
|
||||
"ascend/blob/main/tests/e2e/singlecard/test_offline_inference.py)"
|
||||
"离线测试示例：[`tests/e2e/pull_request/one_card/test_camem.py`](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/tests/e2e/pull_request/one_card/test_camem.py)"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:245
|
||||
#: ../../source/developer_guide/contribution/testing.md:320
|
||||
msgid ""
|
||||
"Online test examples: "
|
||||
"[`tests/e2e/singlecard/test_prompt_embedding.py`](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/tests/e2e/singlecard/test_prompt_embedding.py)"
|
||||
msgstr ""
|
||||
"在线测试示例：[`tests/e2e/singlecard/test_prompt_embedding.py`](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/tests/e2e/singlecard/test_prompt_embedding.py)"
|
||||
"The CI resource is limited, and you might need to reduce the number of "
|
||||
"layers of a model. Below is an example of how to generate a reduced layer "
|
||||
"model:"
|
||||
msgstr "CI 资源有限，您可能需要减少模型的层数。以下是如何生成精简层模型的示例："
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:246
|
||||
#: ../../source/developer_guide/contribution/testing.md:322
|
||||
msgid ""
|
||||
"Correctness test example: "
|
||||
"[`tests/e2e/singlecard/test_aclgraph.py`](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/tests/e2e/singlecard/test_aclgraph.py)"
|
||||
msgstr ""
|
||||
"正确性测试示例：[`tests/e2e/singlecard/test_aclgraph.py`](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/tests/e2e/singlecard/test_aclgraph.py)"
|
||||
"Fork the original model repo in modelscope. All the files in the repo except"
|
||||
" for weights are required."
|
||||
msgstr "在 modelscope 中复刻原始模型仓库。需要仓库中除权重文件外的所有文件。"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:247
|
||||
msgid ""
|
||||
"Reduced Layer model test example: [test_torchair_graph_mode.py - "
|
||||
"DeepSeek-V3-Pruning](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/20767a043cccb3764214930d4695e53941de87ec/tests/e2e/multicard/test_torchair_graph_mode.py#L48)"
|
||||
msgstr ""
|
||||
"简化层模型测试示例：[test_torchair_graph_mode.py - "
|
||||
"DeepSeek-V3-Pruning](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/20767a043cccb3764214930d4695e53941de87ec/tests/e2e/multicard/test_torchair_graph_mode.py#L48)"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:249
|
||||
msgid ""
|
||||
"The CI resource is limited, you might need to reduce layer number of the "
|
||||
"model, below is an example of how to generate a reduced layer model:"
|
||||
msgstr "CI 资源有限，您可能需要减少模型的层数，下面是一个生成减少层数模型的示例："
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:250
|
||||
msgid ""
|
||||
"Fork the original model repo in modelscope, we need all the files in the "
|
||||
"repo except for weights."
|
||||
msgstr "在 modelscope 中 fork 原始模型仓库，我们需要仓库中的所有文件，除了权重文件。"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:251
|
||||
#: ../../source/developer_guide/contribution/testing.md:323
|
||||
#, python-brace-format
|
||||
msgid ""
|
||||
"Set `num_hidden_layers` to the expected number of layers, e.g., "
|
||||
"`{\"num_hidden_layers\": 2,}`"
|
||||
msgstr "将 `num_hidden_layers` 设置为期望的层数，例如 `{\"num_hidden_layers\": 2,}`"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:252
|
||||
#: ../../source/developer_guide/contribution/testing.md:324
|
||||
msgid ""
|
||||
"Copy the following python script as `generate_random_weight.py`. Set the "
|
||||
"relevant parameters `MODEL_LOCAL_PATH`, `DIST_DTYPE` and `DIST_MODEL_PATH` "
|
||||
@@ -213,25 +214,79 @@ msgstr ""
|
||||
"将以下 Python 脚本复制为 `generate_random_weight.py`。根据需要设置相关参数 "
|
||||
"`MODEL_LOCAL_PATH`、`DIST_DTYPE` 和 `DIST_MODEL_PATH`："
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:270
|
||||
#: ../../source/developer_guide/contribution/testing.md:342
|
||||
msgid "Run doctest"
|
||||
msgstr "运行 doctest"
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:272
|
||||
#: ../../source/developer_guide/contribution/testing.md:344
|
||||
msgid ""
|
||||
"vllm-ascend provides a `vllm-ascend/tests/e2e/run_doctests.sh` command to "
|
||||
"run all doctests in the doc files. The doctest is a good way to make sure "
|
||||
"the docs are up to date and the examples are executable, you can run it "
|
||||
"locally as follows:"
|
||||
"docs stay current and examples remain executable, which can be run locally "
|
||||
"as follows:"
|
||||
msgstr ""
|
||||
"vllm-ascend 提供了一个 `vllm-ascend/tests/e2e/run_doctests.sh` 命令，用于运行文档文件中的所有 "
|
||||
"doctest。doctest 是确保文档保持最新且示例可执行的好方法，你可以按照以下方式在本地运行它："
|
||||
"vllm-ascend 提供了一个 `vllm-ascend/tests/e2e/run_doctests.sh` 命令来运行文档文件中的所有 "
|
||||
"doctest。Doctest 是确保文档保持最新且示例保持可执行的好方法，可以在本地按如下方式运行："
|
||||
|
||||
#: ../../developer_guide/contribution/testing.md:280
|
||||
#: ../../source/developer_guide/contribution/testing.md:352
|
||||
msgid ""
|
||||
"This will reproduce the same environment as the CI: "
|
||||
"[vllm_ascend_doctest.yaml](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/vllm_ascend_doctest.yaml)."
|
||||
"This will reproduce the same environment as the CI. See "
|
||||
"[labeled_doctest.yaml](https://github.com/vllm-project/vllm-"
|
||||
"ascend/blob/main/.github/workflows/labeled_doctest.yaml)."
|
||||
msgstr ""
|
||||
"这将复现与 CI 相同的环境：[vllm_ascend_doctest.yaml](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/.github/workflows/vllm_ascend_doctest.yaml)。"
|
||||
"这将重现与 CI 相同的环境。请参阅 [labeled_doctest.yaml](https://github.com/vllm-"
|
||||
"project/vllm-ascend/blob/main/.github/workflows/labeled_doctest.yaml)。"
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:354
|
||||
msgid "Run docs link check"
|
||||
msgstr "运行文档链接检查"
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:356
|
||||
msgid "You can validate external links in the Sphinx docs locally with:"
|
||||
msgstr "您可以在本地使用以下命令验证 Sphinx 文档中的外部链接："
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:362
|
||||
msgid ""
|
||||
"To check links in a specific Markdown file, pass the file to `sphinx-build`."
|
||||
" For example, to check only `docs/source/user_guide/release_notes.md`:"
|
||||
msgstr ""
|
||||
"要检查特定 Markdown 文件中的链接，请将该文件传递给 `sphinx-build`。例如，仅检查 "
|
||||
"`docs/source/user_guide/release_notes.md`："
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:371
|
||||
msgid "The detailed report will be written to:"
|
||||
msgstr "详细报告将写入："
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:373
|
||||
msgid "`docs/_build/linkcheck/output.txt`"
|
||||
msgstr "`docs/_build/linkcheck/output.txt`"
|
||||
|
||||
#: ../../source/developer_guide/contribution/testing.md:374
|
||||
msgid "`docs/_build/linkcheck/output.json`"
|
||||
msgstr "`docs/_build/linkcheck/output.json`"
|
||||
|
||||
#~ msgid "Single card"
|
||||
#~ msgstr "单卡"
|
||||
|
||||
#~ msgid "Multi cards"
|
||||
#~ msgstr "多卡"
|
||||
|
||||
#~ msgid ""
|
||||
#~ "Online test example: "
|
||||
#~ "[`tests/e2e/pull_request/two_card/aclgraph/test_single_request_aclgraph.py`](https://github.com/vllm-"
|
||||
#~ "project/vllm-"
|
||||
#~ "ascend/blob/main/tests/e2e/pull_request/two_card/aclgraph/test_single_request_aclgraph.py)"
|
||||
#~ msgstr ""
|
||||
#~ "在线测试示例：[`tests/e2e/pull_request/two_card/aclgraph/test_single_request_aclgraph.py`](https://github.com/vllm-"
|
||||
#~ "project/vllm-"
|
||||
#~ "ascend/blob/main/tests/e2e/pull_request/two_card/aclgraph/test_single_request_aclgraph.py)"
|
||||
|
||||
#~ msgid ""
|
||||
#~ "Correctness test example: "
|
||||
#~ "[`tests/e2e/pull_request/one_card/aclgraph/test_aclgraph_accuracy.py`](https://github.com/vllm-"
|
||||
#~ "project/vllm-"
|
||||
#~ "ascend/blob/main/tests/e2e/pull_request/one_card/aclgraph/test_aclgraph_accuracy.py)"
|
||||
#~ msgstr ""
|
||||
#~ "正确性测试示例：[`tests/e2e/pull_request/one_card/aclgraph/test_aclgraph_accuracy.py`](https://github.com/vllm-"
|
||||
#~ "project/vllm-"
|
||||
#~ "ascend/blob/main/tests/e2e/pull_request/one_card/aclgraph/test_aclgraph_accuracy.py)"
|
||||
|
||||
@@ -2,14 +2,14 @@
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2025.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-07-18 09:01+0800\n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
@@ -18,9 +18,9 @@ msgstr ""
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.17.0\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../developer_guide/evaluation/index.md:1
|
||||
#: ../../developer_guide/evaluation/index.md:3
|
||||
#: ../../source/developer_guide/evaluation/index.md:1
|
||||
#: ../../source/developer_guide/evaluation/index.md:3
|
||||
msgid "Accuracy"
|
||||
msgstr "准确性"
|
||||
msgstr "准确率"
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:1
|
||||
msgid "Using AISBench"
|
||||
msgstr "使用AISBench"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:3
|
||||
msgid ""
|
||||
"This document guides you to conduct accuracy testing using "
|
||||
"[AISBench](https://github.com/AISBench/benchmark/tree/master). AISBench "
|
||||
"provides accuracy and performance evaluation for many datasets."
|
||||
msgstr ""
|
||||
"本文档指导您使用[AISBench](https://github.com/AISBench/benchmark/tree/master)进行精度测试。AISBench为多个数据集提供精度和性能评估。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:5
|
||||
msgid "Online Server"
|
||||
msgstr "在线服务器"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:7
|
||||
msgid "1. Start the vLLM server"
|
||||
msgstr "1.启动vLLM服务器"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:9
|
||||
msgid "You can run docker container to start the vLLM server on a single NPU:"
|
||||
msgstr "您可以通过运行Docker容器在单个NPU上启动vLLM服务器："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:37
|
||||
msgid "Run the vLLM server in the docker."
|
||||
msgstr "在Docker中运行vLLM服务器。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:45
|
||||
msgid ""
|
||||
"`--max-model-len` should be greater than `35000`, this will be suitable for "
|
||||
"most datasets. Otherwise the accuracy evaluation may be affected."
|
||||
msgstr "`--max-model-len`应大于`35000`，此设置适用于大多数数据集，否则可能影响精度评估。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:48
|
||||
msgid "The vLLM server is started successfully, if you see logs as below:"
|
||||
msgstr "如果看到如下日志，则表示vLLM服务器启动成功："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:56
|
||||
msgid "2. Run different datasets using AISBench"
|
||||
msgstr "2.使用AISBench运行不同数据集"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:58
|
||||
msgid "Install AISBench"
|
||||
msgstr "安装AISBench"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:60
|
||||
msgid ""
|
||||
"Refer to [AISBench](https://github.com/AISBench/benchmark/tree/master) for "
|
||||
"details. Install AISBench from source."
|
||||
msgstr ""
|
||||
"详情请参考[AISBench](https://github.com/AISBench/benchmark/tree/master)。从源码安装AISBench。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:69
|
||||
msgid "Install extra AISBench dependencies."
|
||||
msgstr "安装AISBench额外依赖。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:76
|
||||
msgid "Run `ais_bench -h` to check the installation."
|
||||
msgstr "运行`ais_bench -h`检查安装。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:78
|
||||
msgid "Download Dataset"
|
||||
msgstr "下载数据集"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:80
|
||||
msgid ""
|
||||
"You can choose one or multiple datasets to execute accuracy evaluation."
|
||||
msgstr "您可以选择一个或多个数据集执行精度评估。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:82
|
||||
msgid "`C-Eval` dataset."
|
||||
msgstr "`C-Eval`数据集。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:84
|
||||
msgid ""
|
||||
"Take `C-Eval` dataset as an example. You can refer to "
|
||||
"[Datasets](https://github.com/AISBench/benchmark/tree/master/ais_bench/benchmark/configs/datasets)"
|
||||
" for more datasets. Each dataset has a `README.md` with detailed download "
|
||||
"and installation instructions."
|
||||
msgstr ""
|
||||
"以`C-Eval`数据集为例。更多数据集请参考[Datasets](https://github.com/AISBench/benchmark/tree/master/ais_bench/benchmark/configs/datasets)。每个数据集都有`README.md`文件，包含详细的下载和安装说明。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:86
|
||||
msgid "Download dataset and install it to specific path."
|
||||
msgstr "下载数据集并安装到指定路径。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:98
|
||||
msgid "`MMLU` dataset."
|
||||
msgstr "`MMLU`数据集。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:107
|
||||
msgid "`GPQA` dataset."
|
||||
msgstr "`GPQA`数据集。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:116
|
||||
msgid "`MATH` dataset."
|
||||
msgstr "`MATH`数据集。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:125
|
||||
msgid "`LiveCodeBench` dataset."
|
||||
msgstr "`LiveCodeBench`数据集。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:133
|
||||
msgid "`AIME 2024` dataset."
|
||||
msgstr "`AIME 2024`数据集。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:144
|
||||
msgid "`GSM8K` dataset."
|
||||
msgstr "`GSM8K`数据集。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:153
|
||||
msgid "Configuration"
|
||||
msgstr "配置"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:155
|
||||
msgid ""
|
||||
"Update the file "
|
||||
"`benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`."
|
||||
" There are several arguments that you should update according to your "
|
||||
"environment."
|
||||
msgstr ""
|
||||
"更新文件`benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`。您需要根据环境更新以下参数。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:158
|
||||
msgid ""
|
||||
"`attr`: Identifier for the inference backend type, fixed as `service` "
|
||||
"(serving-based inference) or `local` (local model)."
|
||||
msgstr "`attr`：推理后端类型标识符，固定为`service`（基于服务的推理）或`local`（本地模型）。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:159
|
||||
msgid "`type`: Used to select different backend API types."
|
||||
msgstr "`type`：用于选择不同的后端API类型。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:160
|
||||
msgid ""
|
||||
"`abbr`: Unique identifier for a local task, used to distinguish between "
|
||||
"multiple tasks."
|
||||
msgstr "`abbr`：本地任务的唯一标识符，用于区分多个任务。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:161
|
||||
msgid "`path`: Update to your model weight path."
|
||||
msgstr "`path`：更新为您的模型权重路径。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:162
|
||||
msgid "`model`: Update to your model name in vLLM."
|
||||
msgstr "`model`：更新为vLLM中的模型名称。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:163
|
||||
msgid "`host_ip` and `host_port`: Update to your vLLM server ip and port."
|
||||
msgstr "`host_ip`和`host_port`：更新为您的vLLM服务器IP和端口。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:164
|
||||
msgid ""
|
||||
"`max_out_len`: Note `max_out_len` + LLM input length should be less than "
|
||||
"`max_model_len` (config in your vllm server), `32768` will be suitable for "
|
||||
"most datasets."
|
||||
msgstr ""
|
||||
"`max_out_len`：请注意，`max_out_len` 加上 LLM 输入长度应小于 `max_model_len`（在您的 vllm "
|
||||
"服务器中配置），`32768` 适用于大多数数据集。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:165
|
||||
msgid "`batch_size`: Update according to your dataset."
|
||||
msgstr "`batch_size`：根据数据集更新。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:166
|
||||
msgid "`temperature`: Update inference argument."
|
||||
msgstr "`temperature`：更新推理参数。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:199
|
||||
msgid "Execute Accuracy Evaluation"
|
||||
msgstr "执行精度评估"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:201
|
||||
msgid "Run the following code to execute different accuracy evaluation."
|
||||
msgstr "运行以下代码执行不同的精度评估。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:227
|
||||
msgid ""
|
||||
"After each dataset execution, you can get the result from saved files such "
|
||||
"as `outputs/default/20250628_151326`, there is an example as follows:"
|
||||
msgstr "每个数据集执行完成后，您可以从保存的文件（如`outputs/default/20250628_151326`）中获取结果，示例如下："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:252
|
||||
msgid "Execute Performance Evaluation"
|
||||
msgstr "执行性能评估"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:254
|
||||
msgid "Text-only benchmarks:"
|
||||
msgstr "纯文本基准："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:279
|
||||
msgid "Multi-modal benchmarks (text + images):"
|
||||
msgstr "多模态基准（文本+图像）："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:286
|
||||
msgid ""
|
||||
"After execution, you can get the result from saved files, there is an "
|
||||
"example as follows:"
|
||||
msgstr "执行完成后，您可以从保存的文件中获取结果，示例如下："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:306
|
||||
msgid "3. Troubleshooting"
|
||||
msgstr "3.故障排除"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:308
|
||||
msgid "Invalid Image Path Error"
|
||||
msgstr "无效图像路径错误"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:310
|
||||
msgid ""
|
||||
"If you download the TextVQA dataset following the AISBench documentation:"
|
||||
msgstr "如果您按照AISBench文档下载TextVQA数据集："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:322
|
||||
msgid "you may encounter the following error:"
|
||||
msgstr "您可能会遇到以下错误："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_ais_bench.md:328
|
||||
msgid ""
|
||||
"You need to manually replace the dataset image paths with absolute paths, "
|
||||
"changing `/path/to/benchmark/ais_bench/datasets/textvqa/train_images/` to "
|
||||
"the actual absolute directory where the images are stored:"
|
||||
msgstr ""
|
||||
"您需要手动将数据集图像路径替换为绝对路径，将`/path/to/benchmark/ais_bench/datasets/textvqa/train_images/`更改为存储图像的实际绝对目录："
|
||||
@@ -2,14 +2,14 @@
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2025.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-07-18 09:01+0800\n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
@@ -18,95 +18,98 @@ msgstr ""
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.17.0\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:1
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:1
|
||||
msgid "Using EvalScope"
|
||||
msgstr "使用 EvalScope"
|
||||
msgstr "使用EvalScope"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:3
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:3
|
||||
msgid ""
|
||||
"This document will guide you have model inference stress testing and "
|
||||
"This document will guide you through model inference stress testing and "
|
||||
"accuracy testing using [EvalScope](https://github.com/modelscope/evalscope)."
|
||||
msgstr ""
|
||||
"本文档将指导您如何使用 [EvalScope](https://github.com/modelscope/evalscope) "
|
||||
"进行模型推理压力测试和精度测试。"
|
||||
"本文档将指导您使用[EvalScope](https://github.com/modelscope/evalscope)进行模型推理压力测试和精度测试。"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:5
|
||||
msgid "1. Online serving"
|
||||
msgstr "1. 在线服务"
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:5
|
||||
msgid "1. Online server"
|
||||
msgstr "1.在线服务器"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:7
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:7
|
||||
msgid "You can run docker container to start the vLLM server on a single NPU:"
|
||||
msgstr "你可以运行 docker 容器，在单个 NPU 上启动 vLLM 服务器："
|
||||
msgstr "您可以运行docker容器在单个NPU上启动vLLM服务器："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:34
|
||||
msgid "If your service start successfully, you can see the info shown below:"
|
||||
msgstr "如果你的服务启动成功，你会看到如下所示的信息："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:42
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:35
|
||||
msgid ""
|
||||
"Once your server is started, you can query the model with input prompts in "
|
||||
"new terminal:"
|
||||
msgstr "一旦你的服务器启动后，你可以在新的终端中用输入提示词查询模型："
|
||||
"If the vLLM server is started successfully, you can see information shown "
|
||||
"below:"
|
||||
msgstr "如果vLLM服务器启动成功，您将看到如下信息："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:55
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:43
|
||||
msgid ""
|
||||
"Once your server is started, you can query the model with input prompts in a"
|
||||
" new terminal:"
|
||||
msgstr "服务器启动后，您可以在新终端中使用输入提示查询模型："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:56
|
||||
msgid "2. Install EvalScope using pip"
|
||||
msgstr "2. 使用 pip 安装 EvalScope"
|
||||
msgstr "2.使用pip安装EvalScope"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:57
|
||||
msgid "You can install EvalScope by using:"
|
||||
msgstr "你可以使用以下方式安装 EvalScope："
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:58
|
||||
msgid "You can install EvalScope as follows:"
|
||||
msgstr "您可以按如下方式安装EvalScope："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:65
|
||||
msgid "3. Run gsm8k accuracy test using EvalScope"
|
||||
msgstr "3. 使用 EvalScope 运行 gsm8k 准确率测试"
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:66
|
||||
msgid "3. Run GSM8K using EvalScope for accuracy testing"
|
||||
msgstr "3.使用EvalScope运行GSM8K进行精度测试"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:67
|
||||
msgid "You can `evalscope eval` run gsm8k accuracy test:"
|
||||
msgstr "你可以使用 `evalscope eval` 运行 gsm8k 准确率测试："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:78
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:114
|
||||
msgid "After 1-2 mins, the output is as shown below:"
|
||||
msgstr "1-2 分钟后，输出如下所示："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:88
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:68
|
||||
msgid ""
|
||||
"See more detail in: [EvalScope doc - Model API Service "
|
||||
"You can use `evalscope eval` to run GSM8K (a grade-school math benchmark "
|
||||
"dataset) for accuracy testing:"
|
||||
msgstr "您可以使用`evalscope eval`运行GSM8K（小学数学基准数据集）进行精度测试："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:80
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:117
|
||||
msgid "After 1 to 2 minutes, the output is shown below:"
|
||||
msgstr "1到2分钟后，输出如下所示："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:90
|
||||
msgid ""
|
||||
"See more details in [EvalScope doc - Model API Service "
|
||||
"Evaluation](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#model-"
|
||||
"api-service-evaluation)."
|
||||
msgstr ""
|
||||
"更多详情请见：[EvalScope 文档 - 模型 API "
|
||||
"服务评测](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#model-"
|
||||
"更多详情请参见[EvalScope文档 - "
|
||||
"模型API服务评估](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#model-"
|
||||
"api-service-evaluation)。"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:90
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:92
|
||||
msgid "4. Run model inference stress testing using EvalScope"
|
||||
msgstr "4. 使用 EvalScope 运行模型推理压力测试"
|
||||
msgstr "4.使用EvalScope运行模型推理压力测试"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:92
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:94
|
||||
msgid "Install EvalScope[perf] using pip"
|
||||
msgstr "使用 pip 安装 EvalScope[perf]"
|
||||
msgstr "使用pip安装EvalScope[perf]"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:98
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:100
|
||||
msgid "Basic usage"
|
||||
msgstr "基本用法"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:100
|
||||
msgid "You can use `evalscope perf` run perf test:"
|
||||
msgstr "你可以使用 `evalscope perf` 运行性能测试："
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:102
|
||||
msgid "You can use `evalscope perf` to run perf testing:"
|
||||
msgstr "您可以使用`evalscope perf`运行性能测试："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:112
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:115
|
||||
msgid "Output results"
|
||||
msgstr "输出结果"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_evalscope.md:173
|
||||
#: ../../source/developer_guide/evaluation/using_evalscope.md:176
|
||||
msgid ""
|
||||
"See more detail in: [EvalScope doc - Model Inference Stress "
|
||||
"See more detail in [EvalScope doc - Model Inference Stress "
|
||||
"Testing](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#basic-"
|
||||
"usage)."
|
||||
msgstr ""
|
||||
"更多详情见：[EvalScope 文档 - "
|
||||
"更多详情请参见[EvalScope文档 - "
|
||||
"模型推理压力测试](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#basic-"
|
||||
"usage)。"
|
||||
|
||||
@@ -2,14 +2,14 @@
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2025.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-07-18 09:01+0800\n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
@@ -18,48 +18,112 @@ msgstr ""
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.17.0\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_lm_eval.md:1
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:1
|
||||
msgid "Using lm-eval"
|
||||
msgstr "使用 lm-eval"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_lm_eval.md:2
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:3
|
||||
msgid ""
|
||||
"This document will guide you have a accuracy testing using [lm-"
|
||||
"eval](https://github.com/EleutherAI/lm-evaluation-harness)."
|
||||
"This document guides you to conduct accuracy testing using [lm-eval][1]."
|
||||
msgstr "本文档指导您使用 [lm-eval][1] 进行精度测试。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:5
|
||||
msgid "Online Server"
|
||||
msgstr "在线服务器"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:7
|
||||
msgid "1. Start the vLLM server"
|
||||
msgstr "1.启动 vLLM 服务器"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:9
|
||||
msgid ""
|
||||
"You can run a docker container to start the vLLM server on a single NPU:"
|
||||
msgstr "您可以运行一个docker容器在单个NPU上启动vLLM服务器："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:38
|
||||
msgid "The vLLM server is started successfully, if you see logs as below:"
|
||||
msgstr "如果看到如下日志，则表示 vLLM 服务器启动成功："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:46
|
||||
msgid ""
|
||||
"2. Run GSM8K using the vLLM server (curl) and then run lm-eval for accuracy "
|
||||
"testing"
|
||||
msgstr "2.使用 vLLM 服务器（curl）运行 GSM8K，然后运行 lm-eval 进行精度测试"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:48
|
||||
msgid "You can query the result with input prompts:"
|
||||
msgstr "您可以通过输入提示词查询结果："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:83
|
||||
msgid "The output format matches the following:"
|
||||
msgstr "输出格式与以下内容匹配："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:113
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:185
|
||||
msgid "Install lm-eval in the container:"
|
||||
msgstr "在容器中安装 lm-eval："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:122
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:194
|
||||
msgid ""
|
||||
"The Docker container is launched with `VLLM_USE_MODELSCOPE=True`, which may "
|
||||
"cause lm-eval to download datasets from ModelScope instead of HuggingFace. "
|
||||
"Setting `USE_MODELSCOPE_HUB=0` disables this behavior so that lm-eval can "
|
||||
"fetch datasets from HuggingFace correctly."
|
||||
msgstr ""
|
||||
"本文将指导你如何使用 [lm-eval](https://github.com/EleutherAI/lm-evaluation-harness) "
|
||||
"进行准确率测试。"
|
||||
"Docker 容器启动时设置了 `VLLM_USE_MODELSCOPE=True`，这可能导致 lm-eval 从 ModelScope 而非 "
|
||||
"HuggingFace 下载数据集。设置 `USE_MODELSCOPE_HUB=0` 可禁用此行为，使 lm-eval 能够正确地从 "
|
||||
"HuggingFace 获取数据集。"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_lm_eval.md:4
|
||||
msgid "1. Run docker container"
|
||||
msgstr "1. 运行 docker 容器"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_lm_eval.md:6
|
||||
msgid "You can run docker container on a single NPU:"
|
||||
msgstr "你可以在单个NPU上运行docker容器："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_lm_eval.md:33
|
||||
msgid "2. Run ceval accuracy test using lm-eval"
|
||||
msgstr "2. 使用 lm-eval 运行 ceval 准确性测试"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_lm_eval.md:34
|
||||
msgid "Install lm-eval in the container."
|
||||
msgstr "在容器中安装 lm-eval。"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_lm_eval.md:39
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:128
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:200
|
||||
msgid "Run the following command:"
|
||||
msgstr "运行以下命令："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_lm_eval.md:50
|
||||
msgid "After 1-2 mins, the output is as shown below:"
|
||||
msgstr "1-2 分钟后，输出如下所示："
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:139
|
||||
msgid "After 30 minutes, the output is as shown below:"
|
||||
msgstr "30 分钟后，输出如下所示："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_lm_eval.md:62
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:151
|
||||
msgid "Offline Server"
|
||||
msgstr "离线服务器"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:153
|
||||
msgid "1. Run docker container"
|
||||
msgstr "1.运行 Docker 容器"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:155
|
||||
msgid "You can run docker container on a single NPU:"
|
||||
msgstr "您可以在单个 NPU 上运行 Docker 容器："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:183
|
||||
msgid "2. Run GSM8K using lm-eval for accuracy testing"
|
||||
msgstr "2.使用 lm-eval 运行 GSM8K 进行精度测试"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:211
|
||||
msgid "After 1 to 2 minutes, the output is shown below:"
|
||||
msgstr "1 到 2 分钟后，输出如下所示："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:223
|
||||
msgid "Use Offline Datasets"
|
||||
msgstr "使用离线数据集"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:225
|
||||
msgid ""
|
||||
"You can see more usage on [Lm-eval Docs](https://github.com/EleutherAI/lm-"
|
||||
"evaluation-harness/blob/main/docs/README.md)."
|
||||
msgstr ""
|
||||
"你可以在 [Lm-eval 文档](https://github.com/EleutherAI/lm-evaluation-"
|
||||
"harness/blob/main/docs/README.md) 上查看更多用法。"
|
||||
"Take GSM8K (single dataset) and MMLU (multi-subject dataset) as examples, "
|
||||
"and you can see more from [using-local-datasets][2]."
|
||||
msgstr "以 GSM8K（单一数据集）和 MMLU（多学科数据集）为例，更多信息请参见 [using-local-datasets][2]。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:239
|
||||
msgid "Set [gsm8k.yaml][3] as follows:"
|
||||
msgstr "将 [gsm8k.yaml][3] 设置如下："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:302
|
||||
msgid "Set [_default_template_yaml][4] as follows:"
|
||||
msgstr "将 [_default_template_yaml][4] 设置如下："
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_lm_eval.md:325
|
||||
msgid "You can see more usage on [Lm-eval Docs][5]."
|
||||
msgstr "更多用法请参见 [Lm-eval Docs][5]。"
|
||||
|
||||
@@ -2,14 +2,14 @@
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2025.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2025-07-18 09:01+0800\n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
@@ -18,66 +18,68 @@ msgstr ""
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.17.0\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:1
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:1
|
||||
msgid "Using OpenCompass"
|
||||
msgstr "使用 OpenCompass"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:2
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:3
|
||||
msgid ""
|
||||
"This document will guide you have a accuracy testing using "
|
||||
"This document guides you to conduct accuracy testing using "
|
||||
"[OpenCompass](https://github.com/open-compass/opencompass)."
|
||||
msgstr ""
|
||||
"本文档将指导你如何使用 [OpenCompass](https://github.com/open-compass/opencompass) "
|
||||
"进行准确率测试。"
|
||||
"本文档指导您使用 [OpenCompass](https://github.com/open-compass/opencompass) 进行精度测试。"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:4
|
||||
msgid "1. Online Serving"
|
||||
msgstr "1. 在线服务"
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:5
|
||||
msgid "1. Online Server"
|
||||
msgstr "1.在线服务器"
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:6
|
||||
msgid "You can run docker container to start the vLLM server on a single NPU:"
|
||||
msgstr "你可以运行 docker 容器，在单个 NPU 上启动 vLLM 服务器："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:32
|
||||
msgid "If your service start successfully, you can see the info shown below:"
|
||||
msgstr "如果你的服务启动成功，你会看到如下所示的信息："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:39
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:7
|
||||
msgid ""
|
||||
"Once your server is started, you can query the model with input prompts in "
|
||||
"new terminal:"
|
||||
msgstr "一旦你的服务器启动后，你可以在新的终端中用输入提示词查询模型："
|
||||
"You can run a docker container to start the vLLM server on a single NPU:"
|
||||
msgstr "您可以通过运行 Docker 容器在单个 NPU 上启动 vLLM 服务器："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:51
|
||||
msgid "2. Run ceval accuracy test using OpenCompass"
|
||||
msgstr "2. 使用 OpenCompass 运行 ceval 准确率测试"
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:35
|
||||
msgid ""
|
||||
"The vLLM server is started successfully, if you see information as below:"
|
||||
msgstr "如果看到如下信息，则表示 vLLM 服务器启动成功："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:52
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:43
|
||||
msgid ""
|
||||
"Once your server is started, you can query the model with input prompts in a"
|
||||
" new terminal."
|
||||
msgstr "服务器启动后，您可以在新终端中使用输入提示查询模型。"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:56
|
||||
msgid ""
|
||||
"2. Run C-Eval (a Chinese language model evaluation benchmark) using "
|
||||
"OpenCompass for accuracy testing"
|
||||
msgstr "2.使用 OpenCompass 运行 C-Eval（中文语言模型评估基准）进行精度测试"
|
||||
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:58
|
||||
msgid ""
|
||||
"Install OpenCompass and configure the environment variables in the "
|
||||
"container."
|
||||
msgstr "在容器中安装 OpenCompass 并配置环境变量。"
|
||||
"container:"
|
||||
msgstr "在容器中安装 OpenCompass 并配置环境变量："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:64
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:70
|
||||
msgid ""
|
||||
"Add `opencompass/configs/eval_vllm_ascend_demo.py` with the following "
|
||||
"content:"
|
||||
msgstr "添加 `opencompass/configs/eval_vllm_ascend_demo.py`，内容如下："
|
||||
"Add the following content to `opencompass/configs/eval_vllm_ascend_demo.py`:"
|
||||
msgstr "将以下内容添加到 `opencompass/configs/eval_vllm_ascend_demo.py`："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:104
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:110
|
||||
msgid "Run the following command:"
|
||||
msgstr "运行以下命令："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:110
|
||||
msgid "After 1-2 mins, the output is as shown below:"
|
||||
msgstr "1-2 分钟后，输出如下所示："
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:116
|
||||
msgid "After 1 to 2 minutes, the output is shown below:"
|
||||
msgstr "1 到 2 分钟后，输出如下所示："
|
||||
|
||||
#: ../../developer_guide/evaluation/using_opencompass.md:120
|
||||
#: ../../source/developer_guide/evaluation/using_opencompass.md:126
|
||||
msgid ""
|
||||
"You can see more usage on [OpenCompass "
|
||||
"Docs](https://opencompass.readthedocs.io/en/latest/index.html)."
|
||||
msgstr ""
|
||||
"你可以在 [OpenCompass "
|
||||
"文档](https://opencompass.readthedocs.io/en/latest/index.html) 查看更多用法。"
|
||||
"更多用法请参见 [OpenCompass "
|
||||
"文档](https://opencompass.readthedocs.io/en/latest/index.html)。"
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-29 10:14+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/index.md:1
|
||||
#: ../../source/developer_guide/performance_and_debug/index.md:3
|
||||
msgid "Performance and Debug"
|
||||
msgstr "性能与调试"
|
||||
@@ -0,0 +1,683 @@
|
||||
# SOME DESCRIPTIVE TITLE.
|
||||
# Copyright (C) 2025, vllm-ascend team
|
||||
# This file is distributed under the same license as the vllm-ascend
|
||||
# package.
|
||||
# FIRST AUTHOR <EMAIL@ADDRESS>, 2026.
|
||||
#
|
||||
#, fuzzy
|
||||
msgid ""
|
||||
msgstr ""
|
||||
"Project-Id-Version: vllm-ascend\n"
|
||||
"Report-Msgid-Bugs-To: \n"
|
||||
"POT-Creation-Date: 2026-07-31 09:58+0000\n"
|
||||
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
|
||||
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
|
||||
"Language-Team: zh_CN <LL@li.org>\n"
|
||||
"Language: zh_CN\n"
|
||||
"MIME-Version: 1.0\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n"
|
||||
"Content-Transfer-Encoding: 8bit\n"
|
||||
"Plural-Forms: nplurals=1; plural=0;\n"
|
||||
"Generated-By: Babel 2.18.0\n"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:1
|
||||
msgid "MSProbe Debugging Guide"
|
||||
msgstr "MSProbe 调试指南"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:3
|
||||
msgid ""
|
||||
"During inference or training runs we often encounter accuracy anomalies such"
|
||||
" as outputs drifting away from the expectation, unstable numerical behavior "
|
||||
"(NaN/Inf), or predictions that no longer match the labels. To pinpoint the "
|
||||
"root cause we have to monitor and capture intermediate data produced while "
|
||||
"the model executes—feature maps, weights, activations, and layer outputs. By"
|
||||
" capturing key tensors at specific stages, logging I/O pairs for the core "
|
||||
"layers, and retaining contextual metadata (prompts, tensor dtypes, hardware "
|
||||
"configuration, etc.), we can systematically trace where the accuracy "
|
||||
"degradation or numerical error started. This guide describes the end-to-end "
|
||||
"workflow for diagnosing accuracy issues for AI models (with a focus on vllm-"
|
||||
"ascend services): preparation, data capture, and analysis & verification."
|
||||
msgstr ""
|
||||
"在推理或训练过程中，我们经常会遇到精度异常，例如输出偏离预期、数值行为不稳定（NaN/Inf）或预测结果不再匹配标签。为了定位根本原因，我们需要监控并捕获模型执行过程中产生的中间数据——特征图、权重、激活值和层输出。通过在特定阶段捕获关键张量、记录核心层的输入/输出对，并保留上下文元数据（提示词、张量数据类型、硬件配置等），我们可以系统地追踪精度下降或数值错误开始的位置。本指南描述了诊断AI模型（重点关注vllm-"
|
||||
"ascend服务）精度问题的端到端工作流程：准备、数据捕获以及分析与验证。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:5
|
||||
msgid ""
|
||||
"For more details, see [Ascend/msprobe](https://gitcode.com/Ascend/msprobe)."
|
||||
msgstr "更多详情，请参见 [Ascend/msprobe](https://gitcode.com/Ascend/msprobe)。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:7
|
||||
msgid "0. Background Concepts"
|
||||
msgstr "0.背景概念"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:9
|
||||
msgid "`msprobe` supports three accuracy levels:"
|
||||
msgstr "`msprobe` 支持三种精度级别："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:11
|
||||
msgid ""
|
||||
"**L0**: dumps tensors at the module level and generates `construct.json` so "
|
||||
"that visualization tools can rebuild the network structure. A model or "
|
||||
"submodule handle must be passed in."
|
||||
msgstr "**L0**：在模块级别转储张量并生成 `construct.json`，以便可视化工具重建网络结构。必须传入模型或子模块句柄。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:12
|
||||
msgid ""
|
||||
"**L1**: collects operator-level statistics only, which is suitable for "
|
||||
"lightweight troubleshooting."
|
||||
msgstr "**L1**：仅收集算子级别的统计信息，适用于轻量级故障排查。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:13
|
||||
msgid ""
|
||||
"**mix**: captures both structural information and operator statistics, which"
|
||||
" is useful when you need both graph reconstruction and numerical "
|
||||
"comparisons."
|
||||
msgstr "**mix**：同时捕获结构信息和算子统计信息，适用于需要同时进行图重建和数值比较的场景。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:15
|
||||
msgid "1. Prerequisites"
|
||||
msgstr "1.前提条件"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:17
|
||||
msgid "1.1 Install `msprobe`"
|
||||
msgstr "1.1 安装 `msprobe`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:19
|
||||
msgid "Install msprobe with pip:"
|
||||
msgstr "使用 pip 安装 msprobe："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:25
|
||||
msgid "1.2 Graph mode dump (optional)"
|
||||
msgstr "1.2 图模式转储（可选）"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:27
|
||||
msgid ""
|
||||
"If you need to dump cudagraph graphs, you need to install from source code:"
|
||||
msgstr "如果需要转储 cudagraph 图，则需要从源代码安装："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:29
|
||||
msgid "Install `aclgraph_dump` from source code:"
|
||||
msgstr "从源代码安装 `aclgraph_dump`："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:38
|
||||
msgid "2. Collecting Data with `msprobe`"
|
||||
msgstr "2.使用 `msprobe` 收集数据"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:40
|
||||
msgid ""
|
||||
"We generally follow a coarse-to-fine strategy when capturing data. First, "
|
||||
"identify the token where the issue shows up, and then decide which range "
|
||||
"needs to be sampled around that token. The typical workflow is described "
|
||||
"below."
|
||||
msgstr ""
|
||||
"我们在捕获数据时通常遵循从粗到细的策略。首先，确定问题出现的 token，然后决定需要在该 token 周围采样的范围。典型的工作流程如下所述。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:42
|
||||
msgid "2.1 Prepare the dump configuration content"
|
||||
msgstr "2.1 准备转储配置内容"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:44
|
||||
msgid ""
|
||||
"Prepare configuration content that can be parsed by `PrecisionDebugger`. You"
|
||||
" can use either of the following ways:"
|
||||
msgstr "准备可由 `PrecisionDebugger` 解析的配置内容。您可以使用以下任一方式："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:46
|
||||
msgid ""
|
||||
"Pass the config object directly through `--additional-config.dump_config`."
|
||||
msgstr "通过 `--additional-config.dump_config` 直接传递配置对象。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:47
|
||||
msgid ""
|
||||
"Pass a config file path through `--additional-config.dump_config_path`."
|
||||
msgstr "通过 `--additional-config.dump_config_path` 传递配置文件路径。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:49
|
||||
msgid "Common fields are:"
|
||||
msgstr "常见字段包括："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "Field"
|
||||
msgstr "字段"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "Description"
|
||||
msgstr "描述"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "Required"
|
||||
msgstr "是否必填"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "Eager Mode"
|
||||
msgstr "Eager 模式"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "Graph Mode"
|
||||
msgstr "Graph 模式"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "`task`"
|
||||
msgstr "`task`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid ""
|
||||
"Type of dump task. Common PyTorch values include `\"statistics\"` and "
|
||||
"`\"tensor\"`. A statistics task collects tensor statistics (mean, variance, "
|
||||
"max, min, etc.) while a tensor task captures arbitrary tensors."
|
||||
msgstr ""
|
||||
"转储任务的类型。常见的 PyTorch 值包括 `\"statistics\"` 和 "
|
||||
"`\"tensor\"`。统计任务收集张量统计信息（均值、方差、最大值、最小值等），而张量任务则捕获任意张量。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "Yes"
|
||||
msgstr "是"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "✅"
|
||||
msgstr "✅"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "`dump_path`"
|
||||
msgstr "`dump_path`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid ""
|
||||
"Directory where dump results are stored. When omitted, `msprobe` uses its "
|
||||
"default path."
|
||||
msgstr "存储转储结果的目录。省略时，`msprobe` 使用其默认路径。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "No"
|
||||
msgstr "否"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "`rank`"
|
||||
msgstr "`rank`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid ""
|
||||
"Ranks to sample. An empty list collects every rank. For single-card tasks, "
|
||||
"you must set this field to `[]`."
|
||||
msgstr "需要采样的 rank。空列表表示收集所有 rank。对于单卡任务，必须将此字段设置为 `[]`。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "`step`"
|
||||
msgstr "`step`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "Token iteration(s) to sample. An empty list means every iteration."
|
||||
msgstr "需要采样的 token 迭代次数。空列表表示每个迭代都采样。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "❌"
|
||||
msgstr "❌"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "`level`"
|
||||
msgstr "`level`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid ""
|
||||
"Dump level string (`\"L0\"`, `\"L1\"`, or `\"mix\"`). `L0` targets "
|
||||
"`nn.Module`, `L1` targets `torch.api`, and `mix` collects both."
|
||||
msgstr ""
|
||||
"转储级别字符串（`\"L0\"`、`\"L1\"` 或 `\"mix\"`）。`L0` 针对 `nn.Module`，`L1` 针对 "
|
||||
"`torch.api`，`mix` 同时收集两者。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "`async_dump`"
|
||||
msgstr "`async_dump`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid ""
|
||||
"Whether to enable asynchronous dump (supported for PyTorch "
|
||||
"`statistics`/`tensor` tasks). Defaults to `false`."
|
||||
msgstr "是否启用异步转储（支持 PyTorch `statistics`/`tensor` 任务）。默认为 `false`。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "`scope`"
|
||||
msgstr "`scope`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "Module range to sample. An empty list collects every module."
|
||||
msgstr "需要采样的模块范围。空列表表示收集所有模块。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "`dump_enable`"
|
||||
msgstr "`dump_enable`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid ""
|
||||
"Dynamic switch for enabling/disabling dump in `PrecisionDebugger` during one"
|
||||
" running training/inference job. This allows turning dump on or off on "
|
||||
"demand in the same job."
|
||||
msgstr ""
|
||||
"在单个运行的训练/推理任务中，用于启用/禁用 `PrecisionDebugger` 中转储的动态开关。这允许在同一任务中按需打开或关闭转储。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "`list`"
|
||||
msgstr "`list`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md
|
||||
msgid "Operator range to sample. An empty list collects every operator."
|
||||
msgstr "需要采样的算子范围。空列表表示收集所有算子。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:63
|
||||
msgid ""
|
||||
"To restrict the operators that are captured, configure the `list` block:"
|
||||
msgstr "要限制捕获的算子，请配置 `list` 块："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:65
|
||||
msgid ""
|
||||
"`scope` (list[str]): In PyTorch PyNative scenarios this field restricts the "
|
||||
"dump range. Provide two module or API names that follow the tool's naming "
|
||||
"convention to lock a range; only data between the two names will be dumped. "
|
||||
"Examples:"
|
||||
msgstr ""
|
||||
"`scope` (list[str])：在 PyTorch PyNative 场景中，此字段限制转储范围。提供两个遵循工具命名约定的模块或 API "
|
||||
"名称来锁定一个范围；仅转储这两个名称之间的数据。示例："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:73
|
||||
msgid ""
|
||||
"The `level` setting determines what can be provided—modules when `level=L0`,"
|
||||
" APIs when `level=L1`, and either modules or APIs when `level=mix`."
|
||||
msgstr ""
|
||||
"`level` 设置决定了可以提供什么——当 `level=L0` 时提供模块，当 `level=L1` 时提供 API，当 `level=mix` "
|
||||
"时提供模块或 API。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:75
|
||||
msgid "`list` (list[str]): Custom operator list. Options include:"
|
||||
msgstr "`list` (list[str])：自定义算子列表。选项包括："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:76
|
||||
msgid ""
|
||||
"Supply the full names of specific APIs in PyTorch pynative scenarios to only"
|
||||
" dump those APIs. Example: `\"list\": [\"Tensor.permute.1.forward\", "
|
||||
"\"Tensor.transpose.2.forward\", \"Torch.relu.3.forward\"]`."
|
||||
msgstr ""
|
||||
"在 PyTorch pynative 场景中提供特定 API 的全名，以仅转储这些 API。示例：`\"list\": "
|
||||
"[\"Tensor.permute.1.forward\", \"Tensor.transpose.2.forward\", "
|
||||
"\"Torch.relu.3.forward\"]`。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:77
|
||||
msgid ""
|
||||
"When `level=mix`, you can provide module names so that the dump expands to "
|
||||
"everything produced while the module is running. Example: `\"list\": "
|
||||
"[\"Module.module.language_model.encoder.layers.0.mlp.ParallelMlp.forward.0\"]`."
|
||||
msgstr ""
|
||||
"当 `level=mix` 时，您可以提供模块名称，以便转储扩展到模块运行时产生的所有内容。示例：`\"list\": "
|
||||
"[\"Module.module.language_model.encoder.layers.0.mlp.ParallelMlp.forward.0\"]`。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:78
|
||||
msgid ""
|
||||
"Provide a substring such as `\"list\": [\"relu\"]` to dump every API whose "
|
||||
"name contains the substring. When `level=mix`, modules whose names contain "
|
||||
"the substring are also expanded."
|
||||
msgstr ""
|
||||
"提供一个子字符串，例如 `\"list\": [\"relu\"]`，以转储名称包含该子字符串的所有 API。当 `level=mix` "
|
||||
"时，名称包含该子字符串的模块也会被展开。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:80
|
||||
msgid "Example configuration: eager mode:"
|
||||
msgstr "示例配置：eager 模式："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:102
|
||||
msgid "Graph mode:"
|
||||
msgstr "Graph 模式："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:115
|
||||
msgid "3. Enable `msprobe` in vllm-ascend"
|
||||
msgstr "3.在 vllm-ascend 中启用 `msprobe`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:117
|
||||
msgid ""
|
||||
"Start vLLM and pass the dump config content through `--additional-config`:"
|
||||
msgstr "启动 vLLM 并通过 `--additional-config` 传递转储配置内容："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:136
|
||||
msgid "Compatibility mode (legacy) is still supported:"
|
||||
msgstr "兼容模式（旧版）仍受支持："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:146
|
||||
msgid "4. Send requests and collect dumps"
|
||||
msgstr "4.发送请求并收集转储"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:148
|
||||
msgid "Send inference requests as usual, for example:"
|
||||
msgstr "照常发送推理请求，例如："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:161
|
||||
msgid ""
|
||||
"Each request drives the sequence `msprobe: start -> forward -> stop -> "
|
||||
"step`. The runner invokes `step()` on every code path, so you always get a "
|
||||
"complete dataset even if inference returns early."
|
||||
msgstr ""
|
||||
"每个请求驱动序列 `msprobe: start -> forward -> stop -> step`。运行器会在每条代码路径上调用 "
|
||||
"`step()`，因此即使推理提前返回，您也始终能获得完整的数据集。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:163
|
||||
msgid "Dump files are written into `dump_path`. They usually contain:"
|
||||
msgstr "转储文件写入 `dump_path`。通常包含："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:164
|
||||
msgid "Tensor files grouped by operator/module."
|
||||
msgstr "按算子/模块分组的张量文件。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:165
|
||||
msgid ""
|
||||
"`dump.json`, which records metadata such as dtype, shape, min/max, and "
|
||||
"`requires_grad`."
|
||||
msgstr "`dump.json`，记录 dtype、shape、min/max 和 `requires_grad` 等元数据。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:166
|
||||
msgid ""
|
||||
"`construct.json`, which is generated when `level` is `L0` or `mix` (required"
|
||||
" for visualization)."
|
||||
msgstr "`construct.json`，当 `level` 为 `L0` 或 `mix` 时生成（可视化所需）。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:168
|
||||
msgid "Example directory layout: eager mode:"
|
||||
msgstr "示例目录结构：eager 模式："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:200
|
||||
#, python-brace-format
|
||||
msgid ""
|
||||
"`rank`: Device ID. Each card writes its data to the corresponding `rank{ID}`"
|
||||
" directory. In non-distributed scenarios the directory is simply named "
|
||||
"`rank`."
|
||||
msgstr "`rank`：设备 ID。每张卡将其数据写入对应的 `rank{ID}` 目录。在非分布式场景中，该目录仅命名为 `rank`。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:201
|
||||
msgid "`dump_tensor_data`: Tensor payloads that were collected."
|
||||
msgstr "`dump_tensor_data`：已收集的张量载荷。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:202
|
||||
msgid ""
|
||||
"`dump.json`: Statistics for the forward data of each API or module, "
|
||||
"including names, dtype, shape, max, min, mean, L2 norm (square root of the "
|
||||
"L2 variance), and CRC-32 when `summary_mode=\"md5\"`. See [dump.json file "
|
||||
"description](#dumpjson-file-description) for details."
|
||||
msgstr ""
|
||||
"`dump.json`：每个 API 或模块前向数据的统计信息，包括名称、dtype、shape、max、min、mean、L2 范数（L2 "
|
||||
"方差的平方根），以及当 `summary_mode=\"md5\"` 时的 CRC-32。详见 [dump.json 文件描述](#dumpjson-"
|
||||
"file-description)。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:203
|
||||
msgid ""
|
||||
"`dump_error_info.log`: Present only when the dump tool encountered an error "
|
||||
"and records the failure log."
|
||||
msgstr "`dump_error_info.log`：仅在转储工具遇到错误时存在，记录失败日志。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:204
|
||||
msgid "`stack.json`: Call stacks for APIs/modules."
|
||||
msgstr "`stack.json`：API/模块的调用栈。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:205
|
||||
msgid ""
|
||||
"`construct.json`: Hierarchical structure description. Empty when `level=L1`."
|
||||
msgstr "`construct.json`：层次结构描述。当 `level=L1` 时为空。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:207
|
||||
msgid "graph mode:"
|
||||
msgstr "graph 模式："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:231
|
||||
msgid ""
|
||||
"`dump.json`: See [dump.json file description](#dumpjson-file-description) "
|
||||
"for details."
|
||||
msgstr "`dump.json`：有关详细信息，请参阅[dump.json文件描述](#dumpjson-file-description)。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:233
|
||||
msgid "5. Analyze the results"
|
||||
msgstr "5.分析结果"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:235
|
||||
msgid "5.1 Prerequisites"
|
||||
msgstr "5.1 前提条件"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:237
|
||||
msgid ""
|
||||
"You typically need two dump datasets: one from the \"problem side\" (the run"
|
||||
" that exposes the accuracy or numerical error) and another from the "
|
||||
"\"benchmark side\" (a good baseline). These datasets do not have to be "
|
||||
"identical—they can come from different branches, framework versions, or even"
|
||||
" alternative implementations (operator substitutions, different graph-"
|
||||
"optimization switches, etc.). As long as they use the same or similar "
|
||||
"inputs, hardware topology, and sampling points (step/token), `msprobe` can "
|
||||
"compare them and locate the divergent nodes. If you cannot find a perfectly "
|
||||
"clean benchmark, start by capturing the problem-side data, craft the "
|
||||
"smallest reproducible case by hand, and perform a self-comparison. Below we "
|
||||
"assume the problem dump is `problem_dump` and the benchmark dump is "
|
||||
"`bench_dump`."
|
||||
msgstr ""
|
||||
"您通常需要两个转储数据集：一个来自“问题侧”（暴露精度或数值误差的运行），另一个来自“基准侧”（良好的基线）。这些数据集不必完全相同——它们可以来自不同的分支、框架版本，甚至不同的实现（算子替换、不同的图优化开关等）。只要它们使用相同或相似的输入、硬件拓扑和采样点（step/token），`msprobe`"
|
||||
" 就可以比较它们并定位差异节点。如果找不到完全干净的基准，请先捕获问题侧数据，手动构建最小的可复现案例，并进行自比较。下面我们假设问题转储为 "
|
||||
"`problem_dump`，基准转储为 `bench_dump`。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:239
|
||||
msgid "5.2 Visualization"
|
||||
msgstr "5.2 可视化"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:241
|
||||
msgid ""
|
||||
"Use `msprobe graph_visualize` to build or compare graphs, then open the "
|
||||
"generated `*.vis.db` file(s) with TensorBoard (`tb_graph_ascend` plugin)."
|
||||
msgstr ""
|
||||
"使用 `msprobe graph_visualize` 构建或比较图，然后用 TensorBoard（`tb_graph_ascend` "
|
||||
"插件）打开生成的 `*.vis.db` 文件。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:243
|
||||
msgid "Ensure dump data is visualization-ready:"
|
||||
msgstr "确保转储数据可用于可视化："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:244
|
||||
msgid "Dump level must be `L0` or `mix` so `construct.json` is non-empty."
|
||||
msgstr "转储级别必须为 `L0` 或 `mix`，以确保 `construct.json` 非空。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:245
|
||||
msgid ""
|
||||
"Each rank directory should contain `dump.json`, `stack.json`, and "
|
||||
"`construct.json`."
|
||||
msgstr "每个 rank 目录应包含 `dump.json`、`stack.json` 和 `construct.json`。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:247
|
||||
msgid "Choose command mode:"
|
||||
msgstr "选择命令模式："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:248
|
||||
msgid "Single-graph build:"
|
||||
msgstr "单图构建："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:254
|
||||
msgid "Graph comparison:"
|
||||
msgstr "图比较："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:260
|
||||
msgid "Common optional flags:"
|
||||
msgstr "常用可选标志："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:261
|
||||
msgid "`-oc` / `--overflow_check`: enable overflow marking"
|
||||
msgstr "`-oc` / `--overflow_check`：启用溢出标记"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:262
|
||||
msgid "`-fm` / `--fuzzy_match`: enable fuzzy matching for node mapping"
|
||||
msgstr "`-fm` / `--fuzzy_match`：启用节点映射的模糊匹配"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:263
|
||||
msgid ""
|
||||
"`-lm` / `--layer_mapping [mapping.yaml]`: cross-framework/layer mapping "
|
||||
"compare"
|
||||
msgstr "`-lm` / `--layer_mapping [mapping.yaml]`：跨框架/层映射比较"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:264
|
||||
msgid "`-tensor_log`: print per-node compare log (tensor dump scenarios)"
|
||||
msgstr "`-tensor_log`：打印每个节点的比较日志（张量转储场景）"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:265
|
||||
msgid "`-progress_log`: print detailed progress log"
|
||||
msgstr "`-progress_log`：打印详细进度日志"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:267
|
||||
msgid "Path granularity is auto-detected by `graph_visualize`:"
|
||||
msgstr "路径粒度由 `graph_visualize` 自动检测："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:268
|
||||
msgid "Single-rank: `.../step0/rank0`"
|
||||
msgstr "单 rank：`.../step0/rank0`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:269
|
||||
msgid "Multi-rank (batch): `.../step0`"
|
||||
msgstr "多 rank（批量）：`.../step0`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:270
|
||||
msgid "Multi-step (batch): dump root path containing `step*`"
|
||||
msgstr "多 step（批量）：包含 `step*` 的转储根路径"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:272
|
||||
msgid "Output files:"
|
||||
msgstr "输出文件："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:273
|
||||
#, python-brace-format
|
||||
msgid "Single-graph build: `build_{timestamp}.vis.db`"
|
||||
msgstr "单图构建：`build_{timestamp}.vis.db`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:274
|
||||
#, python-brace-format
|
||||
msgid "Graph comparison: `compare_{timestamp}.vis.db`"
|
||||
msgstr "图比较：`compare_{timestamp}.vis.db`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:276
|
||||
msgid "Launch TensorBoard with the output directory:"
|
||||
msgstr "使用输出目录启动 TensorBoard："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:282
|
||||
msgid "In the visualization UI, inspect structure and numeric differences:"
|
||||
msgstr "在可视化界面中，检查结构和数值差异："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:283
|
||||
msgid "Switch rank/step to locate unstable nodes quickly."
|
||||
msgstr "切换 rank/step 以快速定位不稳定节点。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:284
|
||||
msgid "Use search/filter to focus on target ops/modules."
|
||||
msgstr "使用搜索/过滤聚焦于目标算子/模块。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:285
|
||||
msgid ""
|
||||
"For compare mode, prioritize highlighted high-difference nodes and trace "
|
||||
"surrounding I/O/parameters."
|
||||
msgstr "对于比较模式，优先处理高亮显示的高差异节点，并追踪周围的输入/输出/参数。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:287
|
||||
msgid "6. Troubleshooting"
|
||||
msgstr "6.故障排除"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:289
|
||||
msgid ""
|
||||
"`RuntimeError: Please enforce eager mode`: Restart vLLM and add the "
|
||||
"`--enforce-eager` flag."
|
||||
msgstr ""
|
||||
"`RuntimeError: Please enforce eager mode`：重启 vLLM 并添加 `--enforce-eager` 标志。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:290
|
||||
msgid ""
|
||||
"No dump files: Confirm that the JSON path is correct and every node has "
|
||||
"write permission. In distributed scenarios set `keep_all_ranks` so that "
|
||||
"every rank writes its own dump."
|
||||
msgstr ""
|
||||
"无转储文件：确认 JSON 路径正确且每个节点都有写入权限。在分布式场景中设置 `keep_all_ranks`，以便每个 rank 写入自己的转储。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:291
|
||||
msgid ""
|
||||
"Dumps are too large: Start with a `statistics` task to locate abnormal "
|
||||
"tensors, then narrow the scope with `scope`/`list`/`tensor_list`, `filters`,"
|
||||
" `token_range`, etc."
|
||||
msgstr ""
|
||||
"转储过大：从 `statistics` 任务开始定位异常张量，然后使用 "
|
||||
"`scope`/`list`/`tensor_list`、`filters`、`token_range` 等缩小范围。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:295
|
||||
msgid "Appendix"
|
||||
msgstr "附录"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:297
|
||||
msgid "dump.json file description"
|
||||
msgstr "dump.json 文件描述"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:299
|
||||
msgid "L0 level"
|
||||
msgstr "L0 级别"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:301
|
||||
msgid ""
|
||||
"An L0 `dump.json` contains forward I/O for modules together with parameters."
|
||||
" Using PyTorch's `Conv2d` as an example, the network code looks like:"
|
||||
msgstr "L0 级别的 `dump.json` 包含模块的前向输入/输出以及参数。以 PyTorch 的 `Conv2d` 为例，网络代码如下："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:303
|
||||
msgid ""
|
||||
"`output = self.conv2(input) # self.conv2 = torch.nn.Conv2d(64, 128, 5, "
|
||||
"padding=2, bias=True)`"
|
||||
msgstr ""
|
||||
"`output = self.conv2(input) # self.conv2 = torch.nn.Conv2d(64, 128, 5, "
|
||||
"padding=2, bias=True)`"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:305
|
||||
msgid "`dump.json` contains the following entries:"
|
||||
msgstr "`dump.json` 包含以下条目："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:307
|
||||
msgid ""
|
||||
"`Module.conv2.Conv2d.forward.0`: Forward data of the module. `input_args` "
|
||||
"represents positional inputs, `input_kwargs` represents keyword inputs, "
|
||||
"`output` stores forward outputs, and `parameters` stores weights/biases."
|
||||
msgstr ""
|
||||
"`Module.conv2.Conv2d.forward.0`：模块的前向数据。`input_args` 表示位置输入，`input_kwargs` "
|
||||
"表示关键字输入，`output` 存储前向输出，`parameters` 存储权重/偏置。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:309
|
||||
#, python-brace-format
|
||||
msgid ""
|
||||
"**Note**: When the `model` parameter passed to the dump API is "
|
||||
"`List[torch.nn.Module]` or `Tuple[torch.nn.Module]`, module-level names "
|
||||
"include the index inside the list (`{Module}.{index}.*`). Example: "
|
||||
"`Module.0.conv1.Conv2d.forward.0`."
|
||||
msgstr ""
|
||||
"**注意**：当传递给 dump API 的 `model` 参数为 `List[torch.nn.Module]` 或 "
|
||||
"`Tuple[torch.nn.Module]` "
|
||||
"时，模块级名称包含列表内的索引（`{Module}.{index}.*`）。例如：`Module.0.conv1.Conv2d.forward.0`。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:392
|
||||
msgid "L1 level"
|
||||
msgstr "L1 级别"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:394
|
||||
msgid ""
|
||||
"An L1 `dump.json` records forward I/O for APIs. Using PyTorch's `relu` "
|
||||
"function as an example (`output = torch.nn.functional.relu(input)`), the "
|
||||
"file contains:"
|
||||
msgstr ""
|
||||
"L1 级别的 `dump.json` 记录 API 的前向输入/输出。以 PyTorch 的 `relu` 函数为例（`output = "
|
||||
"torch.nn.functional.relu(input)`），该文件包含："
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:396
|
||||
msgid ""
|
||||
"`Functional.relu.0.forward`: Forward data of the API. `input_args` are "
|
||||
"positional inputs, `input_kwargs` are keyword inputs, and `output` stores "
|
||||
"the forward outputs."
|
||||
msgstr ""
|
||||
"`Functional.relu.0.forward`：API 的前向数据。`input_args` 为位置输入，`input_kwargs` "
|
||||
"为关键字输入，`output` 存储前向输出。"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:448
|
||||
msgid "mix level"
|
||||
msgstr "混合级别"
|
||||
|
||||
#: ../../source/developer_guide/performance_and_debug/msprobe_guide.md:450
|
||||
msgid ""
|
||||
"A `mix` dump.json contains both L0 and L1 level data; the file format is the"
|
||||
" same as the examples above."
|
||||
msgstr "`mix` 级别的 dump.json 同时包含 L0 和 L1 级别的数据；文件格式与上述示例相同。"
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user