icer/qwen36_01
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

record moe tiny batch matrix results

Browse Source
This commit is contained in:
the magician
2026-07-15 01:28:06 +08:00
parent af375830c7
commit d1f8048b6e
30 changed files with 7740 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
worklogs/decode_analysis_report_2026-07-14.md
View File

@@ -765,3 +765,63 @@ ENGINEX_MOE_TINY_MAX=4
### 影响 ### 影响
这次优先目标是恢复平台可构建、可启动、可进入 benchmark-agent 验证流程。为降低平台构建风险，直接采用成功案例中已验证的一组补丁文件；此前本地 MoE tiny-batch 实验代码可能被成功案例的 native MoE/GDN 优化实现覆盖，后续如果平台构建通过，再基于这套可提交基线继续做性能优化分支。 这次优先目标是恢复平台可构建、可启动、可进入 benchmark-agent 验证流程。为降低平台构建风险，直接采用成功案例中已验证的一组补丁文件；此前本地 MoE tiny-batch 实验代码可能被成功案例的 native MoE/GDN 优化实现覆盖，后续如果平台构建通过，再基于这套可提交基线继续做性能优化分支。
## 2026-07-15：MoE tiny-batch 矩阵实验
### 实验过程
远端实验目录：
- `/root/work/logs/moe_tiny_matrix_20260715_codex_05`
本地归档：
- `worklogs/remote_results/2026-07-15-moe-tiny-matrix/`
前两次启动没有形成有效结果：
- `codex_03`：未加 `--enforce-eager`，服务加载后卡在 CUDA Graph capture，`/health` 一直拒绝连接。
- `codex_04`：已加 `--enforce-eager`，但旧的 `codex_03` 服务仍占用显存，导致新服务加载 MoE 权重时 OOM。
有效实验为 `codex_05`，先清理旧进程，再用 `--enforce-eager` 重跑 5 组矩阵。
固定压测口径：
- `prompt_mode=short`
- `concurrency=2`
- `requests=4`
- `max_tokens=128`
- `max_num_seqs=2`
- `max_model_len=100000`
- `tp=4`
- `enable_prefix_caching`
- `enable_chunked_prefill`
### 结果
| case | success | TTFT P90 | Output TPS P10 | Output TPS P50 | Aggregate Output TPS | Avg GPU util | Max GPU util |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `bmm_max16` | 100.00% | 1.401 | 6.245 | 6.381 | 11.958 | 53.2 | 100.0 |
| `bmm_max8` | 100.00% | 1.424 | 6.900 | 6.957 | 12.907 | 54.7 | 100.0 |
| `tokenwise_max16` | 100.00% | 1.312 | 6.292 | 6.387 | 12.003 | 54.8 | 100.0 |
| `tokenwise_max4` | 100.00% | 1.671 | 6.377 | 6.416 | 11.868 | 51.4 | 100.0 |
| `tokenwise_max8` | 100.00% | 1.589 | 6.347 | 6.357 | 11.788 | 54.6 | 100.0 |
### 结论
本轮最佳结果是：
- `ENGINEX_MOE_TINY_IMPL=bmm`
- `ENGINEX_MOE_TINY_MAX=8`
- Aggregate Output TPS = `12.907`
- Output TPS P10 = `6.900`
相对上一轮 `tokenwise_max4` 的有效结果 `11.28`，本轮 `bmm_max8` 聚合吞吐约提升 `14.4%`。但要注意两轮测试的启动环境不同：本轮强制 `--enforce-eager`，且经过多次重启，不能把这个提升完全归因于 `bmm` 实现。
更重要的是，这个矩阵**没有真正验证 `T > TINY_MAX` 后按块拆分是否有收益**。原因是本轮压测固定 `max_num_seqs=2`，decode 阶段 `T` 基本是 1-2，因此 `TINY_MAX=4/8/16` 都会命中同一 tiny fast path。也就是说：
- `tokenwise_max4/8/16` 之间的差异主要是重启和运行噪声。
- `bmm_max8/16` 之间的差异也主要是噪声。
- 本轮更有价值的对比是 `tokenwise` 与 `bmm` 两种 tiny implementation。
从本轮结果看，在 `T≈2` 的短输出 decode 场景里，`bmm` 路径不差于 `tokenwise`，并且本次最佳是 `bmm_max8`。下一步如果要验证“`T` 大于阈值后分块计算”的思路，需要把 `max_num_seqs` 提到 8 或 16，或构造能让单步 decode 聚合更多 token 的压测，否则 `TINY_MAX` 参数不会真正生效。

1122
worklogs/remote_results/2026-07-15-moe-tiny-matrix/bmm_max16_ixsmi.json Normal file
View File

File diff suppressed because it is too large Load Diff

146
worklogs/remote_results/2026-07-15-moe-tiny-matrix/bmm_max16_server.log Normal file
View File

@@ -0,0 +1,146 @@
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 17:16:18 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-14 17:16:20.381623: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-07-14 17:16:20.432980: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
INFO 07-14 17:16:25 api_server.py:530] vLLM API server version 0.6.3
INFO 07-14 17:16:25 api_server.py:531] args: Namespace(host='0.0.0.0', port=1111, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=100000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
INFO 07-14 17:16:25 config.py:1670] Downcasting torch.float32 to torch.float16.
INFO 07-14 17:16:37 config.py:887] Defaulting to use mp for distributed inference
INFO 07-14 17:16:37 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 07-14 17:16:37 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 07-14 17:16:37 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=100000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=True, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 07-14 17:16:38 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 07-14 17:16:38 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 07-14 17:16:38 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 17:16:38 selector.py:115] Using XFormers backend.
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 17:16:40 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 17:16:40 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 17:16:40 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
(VllmWorkerProcess pid=23518) INFO 07-14 17:16:47 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=23518) INFO 07-14 17:16:47 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=23517) INFO 07-14 17:16:47 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=23517) INFO 07-14 17:16:47 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=23519) INFO 07-14 17:16:47 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=23519) INFO 07-14 17:16:47 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=23517) INFO 07-14 17:16:47 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=23518) INFO 07-14 17:16:47 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=23519) INFO 07-14 17:16:47 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
INFO 07-14 17:16:48 shm_broadcast.py:242] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7efbc6469390>, local_subscribe_port=39903, remote_subscribe_port=None)
INFO 07-14 17:16:48 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=23517) INFO 07-14 17:16:48 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=23518) INFO 07-14 17:16:48 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=23519) INFO 07-14 17:16:48 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=23517) INFO 07-14 17:16:48 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=23517) INFO 07-14 17:16:48 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=23519) INFO 07-14 17:16:48 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=23519) INFO 07-14 17:16:48 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=23518) INFO 07-14 17:16:48 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=23518) INFO 07-14 17:16:48 selector.py:115] Using XFormers backend.
INFO 07-14 17:16:48 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 17:16:48 selector.py:115] Using XFormers backend.
Loading safetensors checkpoint shards: 0% Completed | 0/26 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 4% Completed | 1/26 [00:01<00:40, 1.61s/it]
Loading safetensors checkpoint shards: 8% Completed | 2/26 [00:02<00:22, 1.07it/s]
Loading safetensors checkpoint shards: 12% Completed | 3/26 [00:03<00:29, 1.30s/it]
Loading safetensors checkpoint shards: 15% Completed | 4/26 [00:05<00:32, 1.50s/it]
Loading safetensors checkpoint shards: 19% Completed | 5/26 [00:06<00:27, 1.29s/it]
Loading safetensors checkpoint shards: 23% Completed | 6/26 [00:08<00:31, 1.57s/it]
Loading safetensors checkpoint shards: 27% Completed | 7/26 [00:09<00:22, 1.18s/it]
Loading safetensors checkpoint shards: 31% Completed | 8/26 [00:10<00:24, 1.38s/it]
Loading safetensors checkpoint shards: 35% Completed | 9/26 [00:11<00:21, 1.24s/it]
Loading safetensors checkpoint shards: 38% Completed | 10/26 [00:13<00:24, 1.52s/it]
Loading safetensors checkpoint shards: 42% Completed | 11/26 [00:14<00:17, 1.13s/it]
Loading safetensors checkpoint shards: 46% Completed | 12/26 [00:16<00:19, 1.40s/it]
Loading safetensors checkpoint shards: 50% Completed | 13/26 [00:18<00:21, 1.63s/it]
Loading safetensors checkpoint shards: 54% Completed | 14/26 [00:20<00:21, 1.82s/it]
Loading safetensors checkpoint shards: 58% Completed | 15/26 [00:21<00:16, 1.53s/it]
Loading safetensors checkpoint shards: 62% Completed | 16/26 [00:23<00:17, 1.80s/it]
Loading safetensors checkpoint shards: 65% Completed | 17/26 [00:24<00:12, 1.39s/it]
Loading safetensors checkpoint shards: 69% Completed | 18/26 [00:26<00:12, 1.53s/it]
Loading safetensors checkpoint shards: 73% Completed | 19/26 [00:28<00:11, 1.65s/it]
Loading safetensors checkpoint shards: 77% Completed | 20/26 [00:30<00:11, 1.94s/it]
Loading safetensors checkpoint shards: 81% Completed | 21/26 [00:31<00:08, 1.69s/it]
Loading safetensors checkpoint shards: 85% Completed | 22/26 [00:32<00:05, 1.41s/it]
Loading safetensors checkpoint shards: 88% Completed | 23/26 [00:34<00:05, 1.70s/it]
Loading safetensors checkpoint shards: 92% Completed | 24/26 [00:36<00:03, 1.75s/it]
Loading safetensors checkpoint shards: 96% Completed | 25/26 [00:38<00:01, 1.80s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:39<00:00, 1.43s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:39<00:00, 1.51s/it]
(VllmWorkerProcess pid=23518) INFO 07-14 17:17:28 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=23517) INFO 07-14 17:17:28 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=23519) INFO 07-14 17:17:28 model_runner.py:1123] Loading model weights took 16.2303 GB
INFO 07-14 17:17:28 model_runner.py:1123] Loading model weights took 16.2303 GB
INFO 07-14 17:17:36 distributed_gpu_executor.py:57] # GPU blocks: 18502, # CPU blocks: 6553
INFO 07-14 17:17:36 distributed_gpu_executor.py:61] Maximum concurrency for 100000 tokens per request: 2.96x
INFO 07-14 17:17:41 serving_chat.py:79] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
INFO 07-14 17:17:41 serving_chat.py:101] Reasoning parser 'qwen3' enabled.
WARNING 07-14 17:17:41 serving_embedding.py:199] embedding_mode is False. Embedding API will not work.
INFO 07-14 17:17:41 launcher.py:19] Available routes are:
INFO 07-14 17:17:41 launcher.py:27] Route: /openapi.json, Methods: HEAD, GET
INFO 07-14 17:17:41 launcher.py:27] Route: /docs, Methods: HEAD, GET
INFO 07-14 17:17:41 launcher.py:27] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 07-14 17:17:41 launcher.py:27] Route: /redoc, Methods: HEAD, GET
INFO 07-14 17:17:41 launcher.py:27] Route: /health, Methods: GET
INFO 07-14 17:17:41 launcher.py:27] Route: /tokenize, Methods: POST
INFO 07-14 17:17:41 launcher.py:27] Route: /detokenize, Methods: POST
INFO 07-14 17:17:41 launcher.py:27] Route: /v1/models, Methods: GET
INFO 07-14 17:17:41 launcher.py:27] Route: /version, Methods: GET
INFO 07-14 17:17:41 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 07-14 17:17:41 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 07-14 17:17:41 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO: Started server process [23172]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on socket ('0.0.0.0', 1111) (Press CTRL+C to quit)
INFO: 127.0.0.1:34692 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/usr/local/lib/python3.10/site-packages/pyairports/airports.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import resource_string
INFO 07-14 17:17:51 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 17:17:51 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO: 127.0.0.1:45748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:45764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-14 17:17:57 metrics.py:345] Avg prompt throughput: 20.4 tokens/s, Avg generation throughput: 5.9 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 17:17:57 metrics.py:361] Prefix cache hit rate: GPU: 66.67%, CPU: 0.00%
INFO 07-14 17:18:02 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.8 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

1
worklogs/remote_results/2026-07-15-moe-tiny-matrix/bmm_max16_server.pid Normal file
View File

@@ -0,0 +1 @@
23172

84
worklogs/remote_results/2026-07-15-moe-tiny-matrix/bmm_max16_short_c2_t128_r4.json Normal file
View File

@@ -0,0 +1,84 @@
{
"created_at": "2026-07-14T17:18:38",
"label": "bmm_max16_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 42.81683264859021,
"success_rate": 1.0,
"ttft_p50_sec": 1.3186510931700468,
"ttft_p90_sec": 1.4008520478382707,
"output_tps_p10_per_request": 6.244800032698569,
"output_tps_p50_per_request": 6.38071535566192,
"aggregate_output_tps": 11.957913940111546,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1839,
"monitor": {
"records": 32,
"parsed_samples": 128,
"avg_gpu_util_pct": 53.1953125,
"max_gpu_util_pct": 100,
"avg_mem_mib": 29039.0,
"max_mem_mib": 29123,
"avg_power_w": 56.03125,
"max_power_w": 58
},
"results": [
{
"ok": true,
"elapsed_sec": 21.897162649780512,
"ttft_sec": 1.4002014975994825,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.244828150361198,
"chars": 486,
"error": null
},
{
"ok": true,
"elapsed_sec": 21.898223849013448,
"ttft_sec": 1.4011308550834656,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.2447879822717285,
"chars": 452,
"error": null
},
{
"ok": true,
"elapsed_sec": 20.879237646237016,
"ttft_sec": 1.237100688740611,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.516602560962641,
"chars": 452,
"error": null
},
{
"ok": true,
"elapsed_sec": 20.879044523462653,
"ttft_sec": 1.2369663435965776,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.516622061468281,
"chars": 449,
"error": null
}
]
}

39
worklogs/remote_results/2026-07-15-moe-tiny-matrix/bmm_max16_warmup.json Normal file
View File

@@ -0,0 +1,39 @@
{
"created_at": "2026-07-14T17:17:55",
"label": "bmm_max16_warmup",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 1,
"max_tokens": 32,
"wall_sec": 8.268167091533542,
"success_rate": 1.0,
"ttft_p50_sec": 4.3172661755234,
"ttft_p90_sec": 4.3172661755234,
"output_tps_p10_per_request": 8.103515357530624,
"output_tps_p50_per_request": 8.103515357530624,
"aggregate_output_tps": 3.8702652771455766,
"prompt_tokens": 39,
"cached_tokens": 0,
"completion_tokens": 32,
"reasoning_tokens": 32,
"chars": 111,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 8.266169656068087,
"ttft_sec": 4.3172661755234,
"completion_tokens": 32,
"prompt_tokens": 39,
"cached_tokens": 0,
"reasoning_tokens": 32,
"output_tps": 8.103515357530624,
"chars": 111,
"error": null
}
]
}

1087
worklogs/remote_results/2026-07-15-moe-tiny-matrix/bmm_max8_ixsmi.json Normal file
View File

File diff suppressed because it is too large Load Diff

144
worklogs/remote_results/2026-07-15-moe-tiny-matrix/bmm_max8_server.log Normal file
View File

@@ -0,0 +1,144 @@
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 17:13:44 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-14 17:13:46.085639: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-07-14 17:13:46.136960: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
INFO 07-14 17:13:52 api_server.py:530] vLLM API server version 0.6.3
INFO 07-14 17:13:52 api_server.py:531] args: Namespace(host='0.0.0.0', port=1111, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=100000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
INFO 07-14 17:13:52 config.py:1670] Downcasting torch.float32 to torch.float16.
INFO 07-14 17:14:03 config.py:887] Defaulting to use mp for distributed inference
INFO 07-14 17:14:03 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 07-14 17:14:03 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 07-14 17:14:03 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=100000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=True, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 07-14 17:14:04 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 07-14 17:14:04 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 07-14 17:14:04 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 17:14:04 selector.py:115] Using XFormers backend.
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 17:14:06 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 17:14:06 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 17:14:06 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
(VllmWorkerProcess pid=22420) INFO 07-14 17:14:13 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=22420) INFO 07-14 17:14:13 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=22420) INFO 07-14 17:14:13 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=22419) INFO 07-14 17:14:13 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=22419) INFO 07-14 17:14:13 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=22419) INFO 07-14 17:14:13 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=22421) INFO 07-14 17:14:14 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=22421) INFO 07-14 17:14:14 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=22421) INFO 07-14 17:14:14 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
INFO 07-14 17:14:14 shm_broadcast.py:242] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f8887d0e0e0>, local_subscribe_port=56475, remote_subscribe_port=None)
INFO 07-14 17:14:14 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=22419) INFO 07-14 17:14:14 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=22420) INFO 07-14 17:14:14 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=22421) INFO 07-14 17:14:14 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
INFO 07-14 17:14:14 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 17:14:14 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=22421) INFO 07-14 17:14:14 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=22421) INFO 07-14 17:14:14 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=22419) INFO 07-14 17:14:14 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=22419) INFO 07-14 17:14:14 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=22420) INFO 07-14 17:14:14 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=22420) INFO 07-14 17:14:14 selector.py:115] Using XFormers backend.
Loading safetensors checkpoint shards: 0% Completed | 0/26 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 4% Completed | 1/26 [00:01<00:44, 1.77s/it]
Loading safetensors checkpoint shards: 8% Completed | 2/26 [00:02<00:22, 1.08it/s]
Loading safetensors checkpoint shards: 12% Completed | 3/26 [00:03<00:29, 1.29s/it]
Loading safetensors checkpoint shards: 15% Completed | 4/26 [00:05<00:31, 1.42s/it]
Loading safetensors checkpoint shards: 19% Completed | 5/26 [00:06<00:26, 1.24s/it]
Loading safetensors checkpoint shards: 23% Completed | 6/26 [00:08<00:28, 1.44s/it]
Loading safetensors checkpoint shards: 27% Completed | 7/26 [00:08<00:21, 1.11s/it]
Loading safetensors checkpoint shards: 31% Completed | 8/26 [00:10<00:22, 1.26s/it]
Loading safetensors checkpoint shards: 35% Completed | 9/26 [00:11<00:20, 1.19s/it]
Loading safetensors checkpoint shards: 38% Completed | 10/26 [00:13<00:22, 1.41s/it]
Loading safetensors checkpoint shards: 42% Completed | 11/26 [00:13<00:16, 1.07s/it]
Loading safetensors checkpoint shards: 46% Completed | 12/26 [00:15<00:17, 1.27s/it]
Loading safetensors checkpoint shards: 50% Completed | 13/26 [00:16<00:18, 1.42s/it]
Loading safetensors checkpoint shards: 54% Completed | 14/26 [00:18<00:19, 1.59s/it]
Loading safetensors checkpoint shards: 58% Completed | 15/26 [00:19<00:14, 1.34s/it]
Loading safetensors checkpoint shards: 62% Completed | 16/26 [00:21<00:16, 1.60s/it]
Loading safetensors checkpoint shards: 65% Completed | 17/26 [00:22<00:11, 1.25s/it]
Loading safetensors checkpoint shards: 69% Completed | 18/26 [00:24<00:11, 1.43s/it]
Loading safetensors checkpoint shards: 73% Completed | 19/26 [00:25<00:10, 1.46s/it]
Loading safetensors checkpoint shards: 77% Completed | 20/26 [00:27<00:10, 1.71s/it]
Loading safetensors checkpoint shards: 81% Completed | 21/26 [00:29<00:07, 1.54s/it]
Loading safetensors checkpoint shards: 85% Completed | 22/26 [00:29<00:05, 1.29s/it]
Loading safetensors checkpoint shards: 88% Completed | 23/26 [00:32<00:04, 1.64s/it]
Loading safetensors checkpoint shards: 92% Completed | 24/26 [00:34<00:03, 1.71s/it]
Loading safetensors checkpoint shards: 96% Completed | 25/26 [00:35<00:01, 1.71s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:36<00:00, 1.39s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:36<00:00, 1.40s/it]
(VllmWorkerProcess pid=22420) INFO 07-14 17:14:51 model_runner.py:1123] Loading model weights took 16.2303 GB
INFO 07-14 17:14:51 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=22419) INFO 07-14 17:14:52 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=22421) INFO 07-14 17:14:54 model_runner.py:1123] Loading model weights took 16.2303 GB
INFO 07-14 17:15:03 distributed_gpu_executor.py:57] # GPU blocks: 18502, # CPU blocks: 6553
INFO 07-14 17:15:03 distributed_gpu_executor.py:61] Maximum concurrency for 100000 tokens per request: 2.96x
INFO 07-14 17:15:07 serving_chat.py:79] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
INFO 07-14 17:15:07 serving_chat.py:101] Reasoning parser 'qwen3' enabled.
WARNING 07-14 17:15:07 serving_embedding.py:199] embedding_mode is False. Embedding API will not work.
INFO 07-14 17:15:07 launcher.py:19] Available routes are:
INFO 07-14 17:15:07 launcher.py:27] Route: /openapi.json, Methods: HEAD, GET
INFO 07-14 17:15:07 launcher.py:27] Route: /docs, Methods: HEAD, GET
INFO 07-14 17:15:07 launcher.py:27] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 07-14 17:15:07 launcher.py:27] Route: /redoc, Methods: HEAD, GET
INFO 07-14 17:15:07 launcher.py:27] Route: /health, Methods: GET
INFO 07-14 17:15:07 launcher.py:27] Route: /tokenize, Methods: POST
INFO 07-14 17:15:07 launcher.py:27] Route: /detokenize, Methods: POST
INFO 07-14 17:15:07 launcher.py:27] Route: /v1/models, Methods: GET
INFO 07-14 17:15:07 launcher.py:27] Route: /version, Methods: GET
INFO 07-14 17:15:07 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 07-14 17:15:07 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 07-14 17:15:07 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO: Started server process [22074]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on socket ('0.0.0.0', 1111) (Press CTRL+C to quit)
INFO: 127.0.0.1:55232 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:55234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/usr/local/lib/python3.10/site-packages/pyairports/airports.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import resource_string
INFO 07-14 17:15:16 metrics.py:345] Avg prompt throughput: 4.1 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 17:15:16 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO: 127.0.0.1:45424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:45430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-14 17:15:22 metrics.py:345] Avg prompt throughput: 14.8 tokens/s, Avg generation throughput: 6.3 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

1
worklogs/remote_results/2026-07-15-moe-tiny-matrix/bmm_max8_server.pid Normal file
View File

@@ -0,0 +1 @@
22074

84
worklogs/remote_results/2026-07-15-moe-tiny-matrix/bmm_max8_short_c2_t128_r4.json Normal file
View File

@@ -0,0 +1,84 @@
{
"created_at": "2026-07-14T17:16:00",
"label": "bmm_max8_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 39.6690470892936,
"success_rate": 1.0,
"ttft_p50_sec": 1.4093408677726984,
"ttft_p90_sec": 1.4237070180475713,
"output_tps_p10_per_request": 6.899902704972039,
"output_tps_p50_per_request": 6.956794946027678,
"aggregate_output_tps": 12.906788480386393,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1801,
"monitor": {
"records": 31,
"parsed_samples": 124,
"avg_gpu_util_pct": 54.66129032258065,
"max_gpu_util_pct": 100,
"avg_mem_mib": 28876.25,
"max_mem_mib": 28955,
"avg_power_w": 56.58064516129032,
"max_power_w": 58
},
"results": [
{
"ok": true,
"elapsed_sec": 19.97476913779974,
"ttft_sec": 1.4238783307373524,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.899938193333011,
"chars": 449,
"error": null
},
{
"ok": true,
"elapsed_sec": 19.9743344001472,
"ttft_sec": 1.4233072884380817,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.89988749567448,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 19.64549656584859,
"ttft_sec": 1.395374447107315,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 7.013651698722346,
"chars": 449,
"error": null
},
{
"ok": true,
"elapsed_sec": 19.64533146843314,
"ttft_sec": 1.3952654153108597,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 7.013673245204575,
"chars": 449,
"error": null
}
]
}

39
worklogs/remote_results/2026-07-15-moe-tiny-matrix/bmm_max8_warmup.json Normal file
View File

@@ -0,0 +1,39 @@
{
"created_at": "2026-07-14T17:15:20",
"label": "bmm_max8_warmup",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 1,
"max_tokens": 32,
"wall_sec": 7.85333033464849,
"success_rate": 1.0,
"ttft_p50_sec": 4.084131080657244,
"ttft_p90_sec": 4.084131080657244,
"output_tps_p10_per_request": 8.494017912057346,
"output_tps_p50_per_request": 8.494017912057346,
"aggregate_output_tps": 4.074704442116441,
"prompt_tokens": 39,
"cached_tokens": 0,
"completion_tokens": 32,
"reasoning_tokens": 32,
"chars": 111,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 7.851488335058093,
"ttft_sec": 4.084131080657244,
"completion_tokens": 32,
"prompt_tokens": 39,
"cached_tokens": 0,
"reasoning_tokens": 32,
"output_tps": 8.494017912057346,
"chars": 111,
"error": null
}
]
}

455
worklogs/remote_results/2026-07-15-moe-tiny-matrix/driver.log Normal file
View File

@@ -0,0 +1,455 @@
[matrix] out_dir=/root/work/logs/moe_tiny_matrix_20260715_codex_05
[matrix] bench_script=/root/work/decode_microbench.py
[matrix] model_path=/root/public-storage/models/Qwen/Qwen3.6-35B-A3B
[matrix] starting tokenwise_max4
[matrix] waiting health: 18s
[matrix] waiting health: 28s
[matrix] waiting health: 39s
[matrix] waiting health: 49s
[matrix] waiting health: 59s
[matrix] waiting health: 69s
[matrix] waiting health: 79s
[matrix] waiting health: 89s
[matrix] waiting health: 99s
[matrix] health ok
[matrix] warmup tokenwise_max4
done 1/1 ok=True tps=7.896776108883328
{
"created_at": "2026-07-14T17:07:27",
"label": "tokenwise_max4_warmup",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 1,
"max_tokens": 32,
"wall_sec": 8.475353751331568,
"success_rate": 1.0,
"ttft_p50_sec": 4.421089859679341,
"ttft_p90_sec": 4.421089859679341,
"output_tps_p10_per_request": 7.896776108883328,
"output_tps_p50_per_request": 7.896776108883328,
"aggregate_output_tps": 3.7756536115052968,
"prompt_tokens": 39,
"cached_tokens": 0,
"completion_tokens": 32,
"reasoning_tokens": 32,
"chars": 111,
"monitor": null
}
RESULT_FILE /root/work/logs/moe_tiny_matrix_20260715_codex_05/tokenwise_max4_warmup.json
[matrix] benchmark tokenwise_max4
done 1/4 ok=True tps=6.377104297707733
done 2/4 ok=True tps=6.377036995390967
done 3/4 ok=True tps=6.454291150916464
done 4/4 ok=True tps=6.454309006035291
{
"created_at": "2026-07-14T17:08:10",
"label": "tokenwise_max4_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 43.14171364530921,
"success_rate": 1.0,
"ttft_p50_sec": 1.6165212774649262,
"ttft_p90_sec": 1.671257301606238,
"output_tps_p10_per_request": 6.377057186085997,
"output_tps_p50_per_request": 6.415697724312098,
"aggregate_output_tps": 11.867864225547972,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1811,
"monitor": {
"records": 33,
"parsed_samples": 132,
"avg_gpu_util_pct": 51.39393939393939,
"max_gpu_util_pct": 100,
"avg_mem_mib": 28876.25,
"max_mem_mib": 28955,
"avg_power_w": 55.734848484848484,
"max_power_w": 58
}
}
RESULT_FILE /root/work/logs/moe_tiny_matrix_20260715_codex_05/tokenwise_max4_short_c2_t128_r4.json
[matrix] starting tokenwise_max8
[matrix] waiting health: 177s
[matrix] waiting health: 187s
[matrix] waiting health: 197s
[matrix] waiting health: 207s
[matrix] waiting health: 217s
[matrix] waiting health: 227s
[matrix] waiting health: 237s
[matrix] waiting health: 247s
[matrix] waiting health: 257s
[matrix] health ok
[matrix] warmup tokenwise_max8
done 1/1 ok=True tps=8.113631034297331
{
"created_at": "2026-07-14T17:10:05",
"label": "tokenwise_max8_warmup",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 1,
"max_tokens": 32,
"wall_sec": 8.34307736903429,
"success_rate": 1.0,
"ttft_p50_sec": 4.3972604889422655,
"ttft_p90_sec": 4.3972604889422655,
"output_tps_p10_per_request": 8.113631034297331,
"output_tps_p50_per_request": 8.113631034297331,
"aggregate_output_tps": 3.835515192364085,
"prompt_tokens": 39,
"cached_tokens": 0,
"completion_tokens": 32,
"reasoning_tokens": 32,
"chars": 111,
"monitor": null
}
RESULT_FILE /root/work/logs/moe_tiny_matrix_20260715_codex_05/tokenwise_max8_warmup.json
[matrix] benchmark tokenwise_max8
done 1/4 ok=True tps=6.347329155987261
done 2/4 ok=True tps=6.3472829583524115
done 3/4 ok=True tps=6.366059179012696
done 4/4 ok=True tps=6.366082294004555
{
"created_at": "2026-07-14T17:10:48",
"label": "tokenwise_max8_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 43.433731293305755,
"success_rate": 1.0,
"ttft_p50_sec": 1.5782105717808008,
"ttft_p90_sec": 1.5890035681426526,
"output_tps_p10_per_request": 6.3472968176428655,
"output_tps_p50_per_request": 6.356694167499978,
"aggregate_output_tps": 11.788073111713343,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1809,
"monitor": {
"records": 33,
"parsed_samples": 132,
"avg_gpu_util_pct": 54.56818181818182,
"max_gpu_util_pct": 100,
"avg_mem_mib": 28876.25,
"max_mem_mib": 28955,
"avg_power_w": 55.60606060606061,
"max_power_w": 57
}
}
RESULT_FILE /root/work/logs/moe_tiny_matrix_20260715_codex_05/tokenwise_max8_short_c2_t128_r4.json
[matrix] starting tokenwise_max16
[matrix] waiting health: 335s
[matrix] waiting health: 345s
[matrix] waiting health: 356s
[matrix] waiting health: 366s
[matrix] waiting health: 376s
[matrix] waiting health: 386s
[matrix] waiting health: 396s
[matrix] waiting health: 406s
[matrix] waiting health: 416s
[matrix] health ok
[matrix] warmup tokenwise_max16
done 1/1 ok=True tps=8.604076337909753
{
"created_at": "2026-07-14T17:12:43",
"label": "tokenwise_max16_warmup",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 1,
"max_tokens": 32,
"wall_sec": 7.908331776037812,
"success_rate": 1.0,
"ttft_p50_sec": 4.187222994863987,
"ttft_p90_sec": 4.187222994863987,
"output_tps_p10_per_request": 8.604076337909753,
"output_tps_p50_per_request": 8.604076337909753,
"aggregate_output_tps": 4.046365391113176,
"prompt_tokens": 39,
"cached_tokens": 0,
"completion_tokens": 32,
"reasoning_tokens": 32,
"chars": 111,
"monitor": null
}
RESULT_FILE /root/work/logs/moe_tiny_matrix_20260715_codex_05/tokenwise_max16_warmup.json
[matrix] benchmark tokenwise_max16
done 1/4 ok=True tps=6.292255993870769
done 2/4 ok=True tps=6.292149220757748
done 3/4 ok=True tps=6.4821665164576485
done 4/4 ok=True tps=6.482185873169073
{
"created_at": "2026-07-14T17:13:26",
"label": "tokenwise_max16_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 42.65718542598188,
"success_rate": 1.0,
"ttft_p50_sec": 1.2818848732858896,
"ttft_p90_sec": 1.3119126919656992,
"output_tps_p10_per_request": 6.292181252691654,
"output_tps_p50_per_request": 6.387211255164209,
"aggregate_output_tps": 12.002667191636796,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1908,
"monitor": {
"records": 32,
"parsed_samples": 128,
"avg_gpu_util_pct": 54.7890625,
"max_gpu_util_pct": 100,
"avg_mem_mib": 28876.25,
"max_mem_mib": 28955,
"avg_power_w": 55.8671875,
"max_power_w": 58
}
}
RESULT_FILE /root/work/logs/moe_tiny_matrix_20260715_codex_05/tokenwise_max16_short_c2_t128_r4.json
[matrix] starting bmm_max8
[matrix] waiting health: 493s
[matrix] waiting health: 503s
[matrix] waiting health: 513s
[matrix] waiting health: 523s
[matrix] waiting health: 533s
[matrix] waiting health: 543s
[matrix] waiting health: 553s
[matrix] waiting health: 563s
[matrix] waiting health: 573s
[matrix] health ok
[matrix] warmup bmm_max8
done 1/1 ok=True tps=8.494017912057346
{
"created_at": "2026-07-14T17:15:20",
"label": "bmm_max8_warmup",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 1,
"max_tokens": 32,
"wall_sec": 7.85333033464849,
"success_rate": 1.0,
"ttft_p50_sec": 4.084131080657244,
"ttft_p90_sec": 4.084131080657244,
"output_tps_p10_per_request": 8.494017912057346,
"output_tps_p50_per_request": 8.494017912057346,
"aggregate_output_tps": 4.074704442116441,
"prompt_tokens": 39,
"cached_tokens": 0,
"completion_tokens": 32,
"reasoning_tokens": 32,
"chars": 111,
"monitor": null
}
RESULT_FILE /root/work/logs/moe_tiny_matrix_20260715_codex_05/bmm_max8_warmup.json
[matrix] benchmark bmm_max8
done 1/4 ok=True tps=6.899938193333011
done 2/4 ok=True tps=6.89988749567448
done 3/4 ok=True tps=7.013651698722346
done 4/4 ok=True tps=7.013673245204575
{
"created_at": "2026-07-14T17:16:00",
"label": "bmm_max8_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 39.6690470892936,
"success_rate": 1.0,
"ttft_p50_sec": 1.4093408677726984,
"ttft_p90_sec": 1.4237070180475713,
"output_tps_p10_per_request": 6.899902704972039,
"output_tps_p50_per_request": 6.956794946027678,
"aggregate_output_tps": 12.906788480386393,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1801,
"monitor": {
"records": 31,
"parsed_samples": 124,
"avg_gpu_util_pct": 54.66129032258065,
"max_gpu_util_pct": 100,
"avg_mem_mib": 28876.25,
"max_mem_mib": 28955,
"avg_power_w": 56.58064516129032,
"max_power_w": 58
}
}
RESULT_FILE /root/work/logs/moe_tiny_matrix_20260715_codex_05/bmm_max8_short_c2_t128_r4.json
[matrix] starting bmm_max16
[matrix] waiting health: 647s
[matrix] waiting health: 657s
[matrix] waiting health: 667s
[matrix] waiting health: 677s
[matrix] waiting health: 687s
[matrix] waiting health: 697s
[matrix] waiting health: 708s
[matrix] waiting health: 718s
[matrix] waiting health: 728s
[matrix] health ok
[matrix] warmup bmm_max16
done 1/1 ok=True tps=8.103515357530624
{
"created_at": "2026-07-14T17:17:55",
"label": "bmm_max16_warmup",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 1,
"max_tokens": 32,
"wall_sec": 8.268167091533542,
"success_rate": 1.0,
"ttft_p50_sec": 4.3172661755234,
"ttft_p90_sec": 4.3172661755234,
"output_tps_p10_per_request": 8.103515357530624,
"output_tps_p50_per_request": 8.103515357530624,
"aggregate_output_tps": 3.8702652771455766,
"prompt_tokens": 39,
"cached_tokens": 0,
"completion_tokens": 32,
"reasoning_tokens": 32,
"chars": 111,
"monitor": null
}
RESULT_FILE /root/work/logs/moe_tiny_matrix_20260715_codex_05/bmm_max16_warmup.json
[matrix] benchmark bmm_max16
done 1/4 ok=True tps=6.244828150361198
done 2/4 ok=True tps=6.2447879822717285
done 3/4 ok=True tps=6.516602560962641
done 4/4 ok=True tps=6.516622061468281
{
"created_at": "2026-07-14T17:18:38",
"label": "bmm_max16_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 42.81683264859021,
"success_rate": 1.0,
"ttft_p50_sec": 1.3186510931700468,
"ttft_p90_sec": 1.4008520478382707,
"output_tps_p10_per_request": 6.244800032698569,
"output_tps_p50_per_request": 6.38071535566192,
"aggregate_output_tps": 11.957913940111546,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1839,
"monitor": {
"records": 32,
"parsed_samples": 128,
"avg_gpu_util_pct": 53.1953125,
"max_gpu_util_pct": 100,
"avg_mem_mib": 29039.0,
"max_mem_mib": 29123,
"avg_power_w": 56.03125,
"max_power_w": 58
}
}
RESULT_FILE /root/work/logs/moe_tiny_matrix_20260715_codex_05/bmm_max16_short_c2_t128_r4.json
{
"out_dir": "/root/work/logs/moe_tiny_matrix_20260715_codex_05",
"rows": [
{
"label": "bmm_max16_short_c2_t128_r4",
"success_rate": 1.0,
"ttft_p90_sec": 1.4008520478382707,
"output_tps_p10_per_request": 6.244800032698569,
"output_tps_p50_per_request": 6.38071535566192,
"aggregate_output_tps": 11.957913940111546,
"avg_gpu_util_pct": 53.1953125,
"max_gpu_util_pct": 100
},
{
"label": "bmm_max8_short_c2_t128_r4",
"success_rate": 1.0,
"ttft_p90_sec": 1.4237070180475713,
"output_tps_p10_per_request": 6.899902704972039,
"output_tps_p50_per_request": 6.956794946027678,
"aggregate_output_tps": 12.906788480386393,
"avg_gpu_util_pct": 54.66129032258065,
"max_gpu_util_pct": 100
},
{
"label": "tokenwise_max16_short_c2_t128_r4",
"success_rate": 1.0,
"ttft_p90_sec": 1.3119126919656992,
"output_tps_p10_per_request": 6.292181252691654,
"output_tps_p50_per_request": 6.387211255164209,
"aggregate_output_tps": 12.002667191636796,
"avg_gpu_util_pct": 54.7890625,
"max_gpu_util_pct": 100
},
{
"label": "tokenwise_max4_short_c2_t128_r4",
"success_rate": 1.0,
"ttft_p90_sec": 1.671257301606238,
"output_tps_p10_per_request": 6.377057186085997,
"output_tps_p50_per_request": 6.415697724312098,
"aggregate_output_tps": 11.867864225547972,
"avg_gpu_util_pct": 51.39393939393939,
"max_gpu_util_pct": 100
},
{
"label": "tokenwise_max8_short_c2_t128_r4",
"success_rate": 1.0,
"ttft_p90_sec": 1.5890035681426526,
"output_tps_p10_per_request": 6.3472968176428655,
"output_tps_p50_per_request": 6.356694167499978,
"aggregate_output_tps": 11.788073111713343,
"avg_gpu_util_pct": 54.56818181818182,
"max_gpu_util_pct": 100
}
]
}
[matrix] done
[matrix] summary=/root/work/logs/moe_tiny_matrix_20260715_codex_05/summary.md

55
worklogs/remote_results/2026-07-15-moe-tiny-matrix/summary.json Normal file
View File

@@ -0,0 +1,55 @@
{
"out_dir": "/root/work/logs/moe_tiny_matrix_20260715_codex_05",
"rows": [
{
"label": "bmm_max16_short_c2_t128_r4",
"success_rate": 1.0,
"ttft_p90_sec": 1.4008520478382707,
"output_tps_p10_per_request": 6.244800032698569,
"output_tps_p50_per_request": 6.38071535566192,
"aggregate_output_tps": 11.957913940111546,
"avg_gpu_util_pct": 53.1953125,
"max_gpu_util_pct": 100
},
{
"label": "bmm_max8_short_c2_t128_r4",
"success_rate": 1.0,
"ttft_p90_sec": 1.4237070180475713,
"output_tps_p10_per_request": 6.899902704972039,
"output_tps_p50_per_request": 6.956794946027678,
"aggregate_output_tps": 12.906788480386393,
"avg_gpu_util_pct": 54.66129032258065,
"max_gpu_util_pct": 100
},
{
"label": "tokenwise_max16_short_c2_t128_r4",
"success_rate": 1.0,
"ttft_p90_sec": 1.3119126919656992,
"output_tps_p10_per_request": 6.292181252691654,
"output_tps_p50_per_request": 6.387211255164209,
"aggregate_output_tps": 12.002667191636796,
"avg_gpu_util_pct": 54.7890625,
"max_gpu_util_pct": 100
},
{
"label": "tokenwise_max4_short_c2_t128_r4",
"success_rate": 1.0,
"ttft_p90_sec": 1.671257301606238,
"output_tps_p10_per_request": 6.377057186085997,
"output_tps_p50_per_request": 6.415697724312098,
"aggregate_output_tps": 11.867864225547972,
"avg_gpu_util_pct": 51.39393939393939,
"max_gpu_util_pct": 100
},
{
"label": "tokenwise_max8_short_c2_t128_r4",
"success_rate": 1.0,
"ttft_p90_sec": 1.5890035681426526,
"output_tps_p10_per_request": 6.3472968176428655,
"output_tps_p50_per_request": 6.356694167499978,
"aggregate_output_tps": 11.788073111713343,
"avg_gpu_util_pct": 54.56818181818182,
"max_gpu_util_pct": 100
}
]
}

9
worklogs/remote_results/2026-07-15-moe-tiny-matrix/summary.md Normal file
View File

@@ -0,0 +1,9 @@
# MoE tiny-batch matrix
| case | success | TTFT P90 | Output TPS P10 | Output TPS P50 | Aggregate Output TPS | Avg GPU util | Max GPU util |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| bmm_max16_short_c2_t128_r4 | 100.00% | 1.401 | 6.245 | 6.381 | 11.958 | 53.2 | 100.0 |
| bmm_max8_short_c2_t128_r4 | 100.00% | 1.424 | 6.900 | 6.957 | 12.907 | 54.7 | 100.0 |
| tokenwise_max16_short_c2_t128_r4 | 100.00% | 1.312 | 6.292 | 6.387 | 12.003 | 54.8 | 100.0 |
| tokenwise_max4_short_c2_t128_r4 | 100.00% | 1.671 | 6.377 | 6.416 | 11.868 | 51.4 | 100.0 |
| tokenwise_max8_short_c2_t128_r4 | 100.00% | 1.589 | 6.347 | 6.357 | 11.788 | 54.6 | 100.0 |

1122
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max16_ixsmi.json Normal file
View File

File diff suppressed because it is too large Load Diff

146
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max16_server.log Normal file
View File

@@ -0,0 +1,146 @@
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 17:11:07 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-14 17:11:08.715124: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-07-14 17:11:08.765999: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
INFO 07-14 17:11:14 api_server.py:530] vLLM API server version 0.6.3
INFO 07-14 17:11:14 api_server.py:531] args: Namespace(host='0.0.0.0', port=1111, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=100000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
INFO 07-14 17:11:14 config.py:1670] Downcasting torch.float32 to torch.float16.
INFO 07-14 17:11:25 config.py:887] Defaulting to use mp for distributed inference
INFO 07-14 17:11:25 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 07-14 17:11:25 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 07-14 17:11:25 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=100000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=True, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 07-14 17:11:26 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 07-14 17:11:26 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 07-14 17:11:26 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 17:11:26 selector.py:115] Using XFormers backend.
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 17:11:28 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 17:11:28 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 17:11:28 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
(VllmWorkerProcess pid=21301) INFO 07-14 17:11:35 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=21301) INFO 07-14 17:11:35 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=21302) INFO 07-14 17:11:35 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=21302) INFO 07-14 17:11:35 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=21301) INFO 07-14 17:11:35 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=21302) INFO 07-14 17:11:35 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=21303) INFO 07-14 17:11:36 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=21303) INFO 07-14 17:11:36 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=21303) INFO 07-14 17:11:36 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
INFO 07-14 17:11:36 shm_broadcast.py:242] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f7dcd651450>, local_subscribe_port=41787, remote_subscribe_port=None)
INFO 07-14 17:11:36 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=21302) INFO 07-14 17:11:36 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=21301) INFO 07-14 17:11:36 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=21303) INFO 07-14 17:11:36 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
INFO 07-14 17:11:36 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 17:11:36 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=21302) INFO 07-14 17:11:36 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=21302) INFO 07-14 17:11:36 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=21301) INFO 07-14 17:11:36 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=21301) INFO 07-14 17:11:36 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=21303) INFO 07-14 17:11:36 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=21303) INFO 07-14 17:11:36 selector.py:115] Using XFormers backend.
Loading safetensors checkpoint shards: 0% Completed | 0/26 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 4% Completed | 1/26 [00:01<00:42, 1.68s/it]
Loading safetensors checkpoint shards: 8% Completed | 2/26 [00:02<00:21, 1.10it/s]
Loading safetensors checkpoint shards: 12% Completed | 3/26 [00:03<00:29, 1.26s/it]
Loading safetensors checkpoint shards: 15% Completed | 4/26 [00:05<00:32, 1.49s/it]
Loading safetensors checkpoint shards: 19% Completed | 5/26 [00:06<00:26, 1.29s/it]
Loading safetensors checkpoint shards: 23% Completed | 6/26 [00:08<00:30, 1.52s/it]
Loading safetensors checkpoint shards: 27% Completed | 7/26 [00:08<00:22, 1.16s/it]
Loading safetensors checkpoint shards: 31% Completed | 8/26 [00:10<00:24, 1.35s/it]
Loading safetensors checkpoint shards: 35% Completed | 9/26 [00:11<00:20, 1.23s/it]
Loading safetensors checkpoint shards: 38% Completed | 10/26 [00:13<00:23, 1.48s/it]
Loading safetensors checkpoint shards: 42% Completed | 11/26 [00:13<00:16, 1.10s/it]
Loading safetensors checkpoint shards: 46% Completed | 12/26 [00:15<00:18, 1.35s/it]
Loading safetensors checkpoint shards: 50% Completed | 13/26 [00:17<00:19, 1.52s/it]
Loading safetensors checkpoint shards: 54% Completed | 14/26 [00:19<00:20, 1.71s/it]
Loading safetensors checkpoint shards: 58% Completed | 15/26 [00:20<00:15, 1.43s/it]
Loading safetensors checkpoint shards: 62% Completed | 16/26 [00:23<00:17, 1.74s/it]
Loading safetensors checkpoint shards: 65% Completed | 17/26 [00:23<00:12, 1.35s/it]
Loading safetensors checkpoint shards: 69% Completed | 18/26 [00:25<00:11, 1.49s/it]
Loading safetensors checkpoint shards: 73% Completed | 19/26 [00:27<00:10, 1.56s/it]
Loading safetensors checkpoint shards: 77% Completed | 20/26 [00:29<00:11, 1.87s/it]
Loading safetensors checkpoint shards: 81% Completed | 21/26 [00:30<00:08, 1.63s/it]
Loading safetensors checkpoint shards: 85% Completed | 22/26 [00:31<00:05, 1.37s/it]
Loading safetensors checkpoint shards: 88% Completed | 23/26 [00:33<00:05, 1.68s/it]
Loading safetensors checkpoint shards: 92% Completed | 24/26 [00:35<00:03, 1.78s/it]
Loading safetensors checkpoint shards: 96% Completed | 25/26 [00:37<00:01, 1.80s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:38<00:00, 1.41s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:38<00:00, 1.47s/it]
(VllmWorkerProcess pid=21303) INFO 07-14 17:12:15 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=21302) INFO 07-14 17:12:15 model_runner.py:1123] Loading model weights took 16.2303 GB
INFO 07-14 17:12:15 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=21301) INFO 07-14 17:12:15 model_runner.py:1123] Loading model weights took 16.2303 GB
INFO 07-14 17:12:23 distributed_gpu_executor.py:57] # GPU blocks: 18502, # CPU blocks: 6553
INFO 07-14 17:12:23 distributed_gpu_executor.py:61] Maximum concurrency for 100000 tokens per request: 2.96x
INFO 07-14 17:12:28 serving_chat.py:79] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
INFO 07-14 17:12:28 serving_chat.py:101] Reasoning parser 'qwen3' enabled.
WARNING 07-14 17:12:28 serving_embedding.py:199] embedding_mode is False. Embedding API will not work.
INFO 07-14 17:12:28 launcher.py:19] Available routes are:
INFO 07-14 17:12:28 launcher.py:27] Route: /openapi.json, Methods: GET, HEAD
INFO 07-14 17:12:28 launcher.py:27] Route: /docs, Methods: GET, HEAD
INFO 07-14 17:12:28 launcher.py:27] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 07-14 17:12:28 launcher.py:27] Route: /redoc, Methods: GET, HEAD
INFO 07-14 17:12:28 launcher.py:27] Route: /health, Methods: GET
INFO 07-14 17:12:28 launcher.py:27] Route: /tokenize, Methods: POST
INFO 07-14 17:12:28 launcher.py:27] Route: /detokenize, Methods: POST
INFO 07-14 17:12:28 launcher.py:27] Route: /v1/models, Methods: GET
INFO 07-14 17:12:28 launcher.py:27] Route: /version, Methods: GET
INFO 07-14 17:12:28 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 07-14 17:12:28 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 07-14 17:12:28 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO: Started server process [20956]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on socket ('0.0.0.0', 1111) (Press CTRL+C to quit)
INFO: 127.0.0.1:38504 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:38520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/usr/local/lib/python3.10/site-packages/pyairports/airports.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import resource_string
INFO 07-14 17:12:38 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 17:12:38 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 07-14 17:12:43 metrics.py:345] Avg prompt throughput: 7.7 tokens/s, Avg generation throughput: 5.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 17:12:43 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO: 127.0.0.1:38534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:38546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-14 17:12:48 metrics.py:345] Avg prompt throughput: 15.2 tokens/s, Avg generation throughput: 9.2 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

1
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max16_server.pid Normal file
View File

@@ -0,0 +1 @@
20956

84
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max16_short_c2_t128_r4.json Normal file
View File

@@ -0,0 +1,84 @@
{
"created_at": "2026-07-14T17:13:26",
"label": "tokenwise_max16_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 42.65718542598188,
"success_rate": 1.0,
"ttft_p50_sec": 1.2818848732858896,
"ttft_p90_sec": 1.3119126919656992,
"output_tps_p10_per_request": 6.292181252691654,
"output_tps_p50_per_request": 6.387211255164209,
"aggregate_output_tps": 12.002667191636796,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1908,
"monitor": {
"records": 32,
"parsed_samples": 128,
"avg_gpu_util_pct": 54.7890625,
"max_gpu_util_pct": 100,
"avg_mem_mib": 28876.25,
"max_mem_mib": 28955,
"avg_power_w": 55.8671875,
"max_power_w": 58
},
"results": [
{
"ok": true,
"elapsed_sec": 21.593617534264922,
"ttft_sec": 1.2511521093547344,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.292255993870769,
"chars": 486,
"error": null
},
{
"ok": true,
"elapsed_sec": 21.594850908964872,
"ttft_sec": 1.2520402874797583,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.292149220757748,
"chars": 486,
"error": null
},
{
"ok": true,
"elapsed_sec": 21.058475621044636,
"ttft_sec": 1.3119912203401327,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.4821665164576485,
"chars": 449,
"error": null
},
{
"ok": true,
"elapsed_sec": 21.058154894039035,
"ttft_sec": 1.311729459092021,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.482185873169073,
"chars": 487,
"error": null
}
]
}

39
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max16_warmup.json Normal file
View File

@@ -0,0 +1,39 @@
{
"created_at": "2026-07-14T17:12:43",
"label": "tokenwise_max16_warmup",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 1,
"max_tokens": 32,
"wall_sec": 7.908331776037812,
"success_rate": 1.0,
"ttft_p50_sec": 4.187222994863987,
"ttft_p90_sec": 4.187222994863987,
"output_tps_p10_per_request": 8.604076337909753,
"output_tps_p50_per_request": 8.604076337909753,
"aggregate_output_tps": 4.046365391113176,
"prompt_tokens": 39,
"cached_tokens": 0,
"completion_tokens": 32,
"reasoning_tokens": 32,
"chars": 111,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 7.906390368938446,
"ttft_sec": 4.187222994863987,
"completion_tokens": 32,
"prompt_tokens": 39,
"cached_tokens": 0,
"reasoning_tokens": 32,
"output_tps": 8.604076337909753,
"chars": 111,
"error": null
}
]
}

1157
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max4_ixsmi.json Normal file
View File

File diff suppressed because it is too large Load Diff

144
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max4_server.log Normal file
View File

@@ -0,0 +1,144 @@
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 17:05:49 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-14 17:05:51.416940: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-07-14 17:05:51.468510: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
INFO 07-14 17:05:56 api_server.py:530] vLLM API server version 0.6.3
INFO 07-14 17:05:56 api_server.py:531] args: Namespace(host='0.0.0.0', port=1111, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=100000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
INFO 07-14 17:05:56 config.py:1670] Downcasting torch.float32 to torch.float16.
INFO 07-14 17:06:07 config.py:887] Defaulting to use mp for distributed inference
INFO 07-14 17:06:07 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 07-14 17:06:07 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 07-14 17:06:07 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=100000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=True, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 07-14 17:06:08 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 07-14 17:06:08 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 07-14 17:06:08 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 17:06:08 selector.py:115] Using XFormers backend.
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 17:06:10 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 17:06:10 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 17:06:10 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
(VllmWorkerProcess pid=19084) INFO 07-14 17:06:18 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=19084) INFO 07-14 17:06:18 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=19084) INFO 07-14 17:06:18 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=19083) INFO 07-14 17:06:18 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=19083) INFO 07-14 17:06:18 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=19083) INFO 07-14 17:06:18 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=19085) INFO 07-14 17:06:18 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=19085) INFO 07-14 17:06:18 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=19085) INFO 07-14 17:06:18 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
INFO 07-14 17:06:19 shm_broadcast.py:242] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f253b4a9570>, local_subscribe_port=47013, remote_subscribe_port=None)
(VllmWorkerProcess pid=19083) INFO 07-14 17:06:19 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
INFO 07-14 17:06:19 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=19084) INFO 07-14 17:06:19 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=19085) INFO 07-14 17:06:19 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
INFO 07-14 17:06:19 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 17:06:19 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=19083) INFO 07-14 17:06:19 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=19085) INFO 07-14 17:06:19 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=19083) INFO 07-14 17:06:19 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=19085) INFO 07-14 17:06:19 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=19084) INFO 07-14 17:06:19 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=19084) INFO 07-14 17:06:19 selector.py:115] Using XFormers backend.
Loading safetensors checkpoint shards: 0% Completed | 0/26 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 4% Completed | 1/26 [00:01<00:48, 1.95s/it]
Loading safetensors checkpoint shards: 8% Completed | 2/26 [00:02<00:26, 1.10s/it]
Loading safetensors checkpoint shards: 12% Completed | 3/26 [00:04<00:31, 1.36s/it]
Loading safetensors checkpoint shards: 15% Completed | 4/26 [00:05<00:33, 1.50s/it]
Loading safetensors checkpoint shards: 19% Completed | 5/26 [00:06<00:28, 1.35s/it]
Loading safetensors checkpoint shards: 23% Completed | 6/26 [00:08<00:31, 1.58s/it]
Loading safetensors checkpoint shards: 27% Completed | 7/26 [00:09<00:22, 1.20s/it]
Loading safetensors checkpoint shards: 31% Completed | 8/26 [00:11<00:24, 1.35s/it]
Loading safetensors checkpoint shards: 35% Completed | 9/26 [00:12<00:20, 1.23s/it]
Loading safetensors checkpoint shards: 38% Completed | 10/26 [00:13<00:22, 1.43s/it]
Loading safetensors checkpoint shards: 42% Completed | 11/26 [00:14<00:16, 1.08s/it]
Loading safetensors checkpoint shards: 46% Completed | 12/26 [00:16<00:18, 1.32s/it]
Loading safetensors checkpoint shards: 50% Completed | 13/26 [00:18<00:19, 1.52s/it]
Loading safetensors checkpoint shards: 54% Completed | 14/26 [00:20<00:20, 1.74s/it]
Loading safetensors checkpoint shards: 58% Completed | 15/26 [00:21<00:16, 1.46s/it]
Loading safetensors checkpoint shards: 62% Completed | 16/26 [00:23<00:17, 1.77s/it]
Loading safetensors checkpoint shards: 65% Completed | 17/26 [00:24<00:12, 1.37s/it]
Loading safetensors checkpoint shards: 69% Completed | 18/26 [00:25<00:11, 1.46s/it]
Loading safetensors checkpoint shards: 73% Completed | 19/26 [00:27<00:10, 1.50s/it]
Loading safetensors checkpoint shards: 77% Completed | 20/26 [00:29<00:10, 1.75s/it]
Loading safetensors checkpoint shards: 81% Completed | 21/26 [00:30<00:07, 1.51s/it]
Loading safetensors checkpoint shards: 85% Completed | 22/26 [00:31<00:05, 1.27s/it]
Loading safetensors checkpoint shards: 88% Completed | 23/26 [00:33<00:04, 1.56s/it]
Loading safetensors checkpoint shards: 92% Completed | 24/26 [00:35<00:03, 1.66s/it]
Loading safetensors checkpoint shards: 96% Completed | 25/26 [00:37<00:01, 1.82s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:38<00:00, 1.44s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:38<00:00, 1.47s/it]
INFO 07-14 17:06:58 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=19083) INFO 07-14 17:07:00 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=19085) INFO 07-14 17:07:01 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=19084) INFO 07-14 17:07:01 model_runner.py:1123] Loading model weights took 16.2303 GB
INFO 07-14 17:07:09 distributed_gpu_executor.py:57] # GPU blocks: 18502, # CPU blocks: 6553
INFO 07-14 17:07:09 distributed_gpu_executor.py:61] Maximum concurrency for 100000 tokens per request: 2.96x
INFO 07-14 17:07:13 serving_chat.py:79] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
INFO 07-14 17:07:13 serving_chat.py:101] Reasoning parser 'qwen3' enabled.
WARNING 07-14 17:07:13 serving_embedding.py:199] embedding_mode is False. Embedding API will not work.
INFO 07-14 17:07:13 launcher.py:19] Available routes are:
INFO 07-14 17:07:13 launcher.py:27] Route: /openapi.json, Methods: GET, HEAD
INFO 07-14 17:07:13 launcher.py:27] Route: /docs, Methods: GET, HEAD
INFO 07-14 17:07:13 launcher.py:27] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 07-14 17:07:13 launcher.py:27] Route: /redoc, Methods: GET, HEAD
INFO 07-14 17:07:13 launcher.py:27] Route: /health, Methods: GET
INFO 07-14 17:07:13 launcher.py:27] Route: /tokenize, Methods: POST
INFO 07-14 17:07:13 launcher.py:27] Route: /detokenize, Methods: POST
INFO 07-14 17:07:13 launcher.py:27] Route: /v1/models, Methods: GET
INFO 07-14 17:07:13 launcher.py:27] Route: /version, Methods: GET
INFO 07-14 17:07:13 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 07-14 17:07:13 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 07-14 17:07:13 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO: Started server process [18738]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on socket ('0.0.0.0', 1111) (Press CTRL+C to quit)
INFO: 127.0.0.1:37776 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:37780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/usr/local/lib/python3.10/site-packages/pyairports/airports.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import resource_string
INFO 07-14 17:07:22 metrics.py:345] Avg prompt throughput: 4.3 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 17:07:22 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO: 127.0.0.1:33406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:33408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-14 17:07:28 metrics.py:345] Avg prompt throughput: 13.4 tokens/s, Avg generation throughput: 5.7 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

1
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max4_server.pid Normal file
View File

@@ -0,0 +1 @@
18738

84
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max4_short_c2_t128_r4.json Normal file
View File

@@ -0,0 +1,84 @@
{
"created_at": "2026-07-14T17:08:10",
"label": "tokenwise_max4_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 43.14171364530921,
"success_rate": 1.0,
"ttft_p50_sec": 1.6165212774649262,
"ttft_p90_sec": 1.671257301606238,
"output_tps_p10_per_request": 6.377057186085997,
"output_tps_p50_per_request": 6.415697724312098,
"aggregate_output_tps": 11.867864225547972,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1811,
"monitor": {
"records": 33,
"parsed_samples": 132,
"avg_gpu_util_pct": 51.39393939393939,
"max_gpu_util_pct": 100,
"avg_mem_mib": 28876.25,
"max_mem_mib": 28955,
"avg_power_w": 55.734848484848484,
"max_power_w": 58
},
"results": [
{
"ok": true,
"elapsed_sec": 21.74245741777122,
"ttft_sec": 1.6706514656543732,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.377104297707733,
"chars": 449,
"error": null
},
{
"ok": true,
"elapsed_sec": 21.743534732609987,
"ttft_sec": 1.6715169455856085,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.377036995390967,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 21.394158359616995,
"ttft_sec": 1.5623910892754793,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.454291150916464,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 21.393961790949106,
"ttft_sec": 1.5622493829578161,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.454309006035291,
"chars": 454,
"error": null
}
]
}

39
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max4_warmup.json Normal file
View File

@@ -0,0 +1,39 @@
{
"created_at": "2026-07-14T17:07:27",
"label": "tokenwise_max4_warmup",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 1,
"max_tokens": 32,
"wall_sec": 8.475353751331568,
"success_rate": 1.0,
"ttft_p50_sec": 4.421089859679341,
"ttft_p90_sec": 4.421089859679341,
"output_tps_p10_per_request": 7.896776108883328,
"output_tps_p50_per_request": 7.896776108883328,
"aggregate_output_tps": 3.7756536115052968,
"prompt_tokens": 39,
"cached_tokens": 0,
"completion_tokens": 32,
"reasoning_tokens": 32,
"chars": 111,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 8.473376458510756,
"ttft_sec": 4.421089859679341,
"completion_tokens": 32,
"prompt_tokens": 39,
"cached_tokens": 0,
"reasoning_tokens": 32,
"output_tps": 7.896776108883328,
"chars": 111,
"error": null
}
]
}

1157
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max8_ixsmi.json Normal file
View File

File diff suppressed because it is too large Load Diff

144
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max8_server.log Normal file
View File

@@ -0,0 +1,144 @@
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 17:08:28 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
2026-07-14 17:08:30.205600: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-07-14 17:08:30.256904: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
INFO 07-14 17:08:36 api_server.py:530] vLLM API server version 0.6.3
INFO 07-14 17:08:36 api_server.py:531] args: Namespace(host='0.0.0.0', port=1111, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='qwen3_coder', tool_parser_plugin='', reasoning_parser='qwen3', model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=100000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=2, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['llm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False)
INFO 07-14 17:08:36 config.py:1670] Downcasting torch.float32 to torch.float16.
INFO 07-14 17:08:48 config.py:887] Defaulting to use mp for distributed inference
INFO 07-14 17:08:48 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 07-14 17:08:48 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 07-14 17:08:48 llm_engine.py:237] Initializing an LLM engine (v0.6.3) with config: model='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='/root/public-storage/models/Qwen/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=100000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=llm, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=True, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 07-14 17:08:49 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 07-14 17:08:49 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 07-14 17:08:49 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 17:08:49 selector.py:115] Using XFormers backend.
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/usr/local/corex/lib/python3/dist-packages/torch/cuda/__init__.py:51: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 07-14 17:08:51 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 17:08:51 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
INFO 07-14 17:08:51 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
(VllmWorkerProcess pid=20203) INFO 07-14 17:08:58 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=20203) INFO 07-14 17:08:58 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=20203) INFO 07-14 17:08:58 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=20202) INFO 07-14 17:08:58 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=20202) INFO 07-14 17:08:58 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=20201) INFO 07-14 17:08:58 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=20201) INFO 07-14 17:08:58 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=20202) INFO 07-14 17:08:58 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=20201) INFO 07-14 17:08:58 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
INFO 07-14 17:08:59 shm_broadcast.py:242] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7ff383d62050>, local_subscribe_port=34695, remote_subscribe_port=None)
INFO 07-14 17:08:59 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=20202) INFO 07-14 17:08:59 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=20201) INFO 07-14 17:08:59 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=20203) INFO 07-14 17:08:59 model_runner.py:1112] Starting to load model /root/public-storage/models/Qwen/Qwen3.6-35B-A3B...
(VllmWorkerProcess pid=20202) INFO 07-14 17:08:59 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=20202) INFO 07-14 17:08:59 selector.py:115] Using XFormers backend.
INFO 07-14 17:08:59 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
INFO 07-14 17:08:59 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=20203) INFO 07-14 17:08:59 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=20203) INFO 07-14 17:08:59 selector.py:115] Using XFormers backend.
(VllmWorkerProcess pid=20201) INFO 07-14 17:08:59 selector.py:266] Cannot use FlashAttention-2 backend because the vllm.vllm_flash_attn package is not found. Make sure that vllm_flash_attn was built and installed (on by default).
(VllmWorkerProcess pid=20201) INFO 07-14 17:08:59 selector.py:115] Using XFormers backend.
Loading safetensors checkpoint shards: 0% Completed | 0/26 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 4% Completed | 1/26 [00:02<00:50, 2.03s/it]
Loading safetensors checkpoint shards: 8% Completed | 2/26 [00:02<00:27, 1.14s/it]
Loading safetensors checkpoint shards: 12% Completed | 3/26 [00:04<00:34, 1.50s/it]
Loading safetensors checkpoint shards: 15% Completed | 4/26 [00:06<00:38, 1.75s/it]
Loading safetensors checkpoint shards: 19% Completed | 5/26 [00:07<00:31, 1.49s/it]
Loading safetensors checkpoint shards: 23% Completed | 6/26 [00:09<00:34, 1.73s/it]
Loading safetensors checkpoint shards: 27% Completed | 7/26 [00:10<00:25, 1.32s/it]
Loading safetensors checkpoint shards: 31% Completed | 8/26 [00:12<00:26, 1.48s/it]
Loading safetensors checkpoint shards: 35% Completed | 9/26 [00:13<00:22, 1.31s/it]
Loading safetensors checkpoint shards: 38% Completed | 10/26 [00:15<00:24, 1.55s/it]
Loading safetensors checkpoint shards: 42% Completed | 11/26 [00:15<00:17, 1.18s/it]
Loading safetensors checkpoint shards: 46% Completed | 12/26 [00:17<00:19, 1.41s/it]
Loading safetensors checkpoint shards: 50% Completed | 13/26 [00:19<00:20, 1.60s/it]
Loading safetensors checkpoint shards: 54% Completed | 14/26 [00:21<00:21, 1.82s/it]
Loading safetensors checkpoint shards: 58% Completed | 15/26 [00:22<00:17, 1.55s/it]
Loading safetensors checkpoint shards: 62% Completed | 16/26 [00:25<00:18, 1.83s/it]
Loading safetensors checkpoint shards: 65% Completed | 17/26 [00:25<00:13, 1.45s/it]
Loading safetensors checkpoint shards: 69% Completed | 18/26 [00:27<00:12, 1.60s/it]
Loading safetensors checkpoint shards: 73% Completed | 19/26 [00:29<00:11, 1.64s/it]
Loading safetensors checkpoint shards: 77% Completed | 20/26 [00:32<00:11, 1.93s/it]
Loading safetensors checkpoint shards: 81% Completed | 21/26 [00:33<00:08, 1.67s/it]
Loading safetensors checkpoint shards: 85% Completed | 22/26 [00:33<00:05, 1.41s/it]
Loading safetensors checkpoint shards: 88% Completed | 23/26 [00:36<00:05, 1.76s/it]
(VllmWorkerProcess pid=20203) INFO 07-14 17:09:37 model_runner.py:1123] Loading model weights took 16.2303 GB
Loading safetensors checkpoint shards: 92% Completed | 24/26 [00:38<00:03, 1.86s/it]
Loading safetensors checkpoint shards: 96% Completed | 25/26 [00:40<00:01, 1.89s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:41<00:00, 1.47s/it]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:41<00:00, 1.58s/it]
INFO 07-14 17:09:40 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=20201) INFO 07-14 17:09:41 model_runner.py:1123] Loading model weights took 16.2303 GB
(VllmWorkerProcess pid=20202) INFO 07-14 17:09:41 model_runner.py:1123] Loading model weights took 16.2303 GB
INFO 07-14 17:09:50 distributed_gpu_executor.py:57] # GPU blocks: 18502, # CPU blocks: 6553
INFO 07-14 17:09:50 distributed_gpu_executor.py:61] Maximum concurrency for 100000 tokens per request: 2.96x
INFO 07-14 17:09:54 serving_chat.py:79] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
INFO 07-14 17:09:54 serving_chat.py:101] Reasoning parser 'qwen3' enabled.
WARNING 07-14 17:09:54 serving_embedding.py:199] embedding_mode is False. Embedding API will not work.
INFO 07-14 17:09:54 launcher.py:19] Available routes are:
INFO 07-14 17:09:54 launcher.py:27] Route: /openapi.json, Methods: GET, HEAD
INFO 07-14 17:09:54 launcher.py:27] Route: /docs, Methods: GET, HEAD
INFO 07-14 17:09:54 launcher.py:27] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 07-14 17:09:54 launcher.py:27] Route: /redoc, Methods: GET, HEAD
INFO 07-14 17:09:54 launcher.py:27] Route: /health, Methods: GET
INFO 07-14 17:09:54 launcher.py:27] Route: /tokenize, Methods: POST
INFO 07-14 17:09:54 launcher.py:27] Route: /detokenize, Methods: POST
INFO 07-14 17:09:54 launcher.py:27] Route: /v1/models, Methods: GET
INFO 07-14 17:09:54 launcher.py:27] Route: /version, Methods: GET
INFO 07-14 17:09:54 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 07-14 17:09:54 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 07-14 17:09:54 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO: Started server process [19856]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on socket ('0.0.0.0', 1111) (Press CTRL+C to quit)
INFO: 127.0.0.1:43078 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:43092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/usr/local/lib/python3.10/site-packages/pyairports/airports.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import resource_string
INFO 07-14 17:10:01 metrics.py:345] Avg prompt throughput: 5.7 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-14 17:10:01 metrics.py:361] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO: 127.0.0.1:43094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:43110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-14 17:10:06 metrics.py:345] Avg prompt throughput: 13.9 tokens/s, Avg generation throughput: 5.9 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

1
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max8_server.pid Normal file
View File

@@ -0,0 +1 @@
19856

84
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max8_short_c2_t128_r4.json Normal file
View File

@@ -0,0 +1,84 @@
{
"created_at": "2026-07-14T17:10:48",
"label": "tokenwise_max8_short_c2_t128_r4",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 2,
"requests": 4,
"max_tokens": 128,
"wall_sec": 43.433731293305755,
"success_rate": 1.0,
"ttft_p50_sec": 1.5782105717808008,
"ttft_p90_sec": 1.5890035681426526,
"output_tps_p10_per_request": 6.3472968176428655,
"output_tps_p50_per_request": 6.356694167499978,
"aggregate_output_tps": 11.788073111713343,
"prompt_tokens": 156,
"cached_tokens": 128,
"completion_tokens": 512,
"reasoning_tokens": 512,
"chars": 1809,
"monitor": {
"records": 33,
"parsed_samples": 132,
"avg_gpu_util_pct": 54.56818181818182,
"max_gpu_util_pct": 100,
"avg_mem_mib": 28876.25,
"max_mem_mib": 28955,
"avg_power_w": 55.60606060606061,
"max_power_w": 57
},
"results": [
{
"ok": true,
"elapsed_sec": 21.755138302221894,
"ttft_sec": 1.5891760755330324,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.347329155987261,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 21.75471005216241,
"ttft_sec": 1.5886010508984327,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.3472829583524115,
"chars": 454,
"error": null
},
{
"ok": true,
"elapsed_sec": 21.674450647085905,
"ttft_sec": 1.567820092663169,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.366059179012696,
"chars": 452,
"error": null
},
{
"ok": true,
"elapsed_sec": 21.67425451427698,
"ttft_sec": 1.5676969662308693,
"completion_tokens": 128,
"prompt_tokens": 39,
"cached_tokens": 32,
"reasoning_tokens": 128,
"output_tps": 6.366082294004555,
"chars": 449,
"error": null
}
]
}

39
worklogs/remote_results/2026-07-15-moe-tiny-matrix/tokenwise_max8_warmup.json Normal file
View File

@@ -0,0 +1,39 @@
{
"created_at": "2026-07-14T17:10:05",
"label": "tokenwise_max8_warmup",
"url": "http://127.0.0.1:1111",
"model": "llm",
"prompt_mode": "short",
"with_tools": false,
"tool_count": 0,
"concurrency": 1,
"requests": 1,
"max_tokens": 32,
"wall_sec": 8.34307736903429,
"success_rate": 1.0,
"ttft_p50_sec": 4.3972604889422655,
"ttft_p90_sec": 4.3972604889422655,
"output_tps_p10_per_request": 8.113631034297331,
"output_tps_p50_per_request": 8.113631034297331,
"aggregate_output_tps": 3.835515192364085,
"prompt_tokens": 39,
"cached_tokens": 0,
"completion_tokens": 32,
"reasoning_tokens": 32,
"chars": 111,
"monitor": null,
"results": [
{
"ok": true,
"elapsed_sec": 8.341240670531988,
"ttft_sec": 4.3972604889422655,
"completion_tokens": 32,
"prompt_tokens": 39,
"cached_tokens": 0,
"reasoning_tokens": 32,
"output_tps": 8.113631034297331,
"chars": 111,
"error": null
}
]
}

172
worklogs/run_moe_tiny_matrix_remote.sh Normal file
View File

@@ -0,0 +1,172 @@
#!/usr/bin/env bash
set -euo pipefail
OUT_DIR="${1:-/root/work/logs/moe_tiny_matrix_$(date +%Y%m%d_%H%M%S)}"
BENCH_SCRIPT="${BENCH_SCRIPT:-/root/work/decode_microbench.py}"
MODEL_PATH="${MODEL_PATH:-/root/public-storage/models/Qwen/Qwen3.6-35B-A3B}"
BASE_URL="${BASE_URL:-http://127.0.0.1:1111}"
mkdir -p "$OUT_DIR"
echo "[matrix] out_dir=$OUT_DIR"
echo "[matrix] bench_script=$BENCH_SCRIPT"
echo "[matrix] model_path=$MODEL_PATH"
stop_server() {
pkill -f "vllm.entrypoints.openai.api_server" 2>/dev/null || true
pkill -f "VllmWorkerProcess" 2>/dev/null || true
sleep 8
}
wait_health() {
local deadline=$((SECONDS + 1200))
while (( SECONDS < deadline )); do
if python3 - <<'PY'
import urllib.request
try:
urllib.request.urlopen("http://127.0.0.1:1111/health", timeout=5)
except Exception:
raise SystemExit(1)
PY
then
echo "[matrix] health ok"
return 0
fi
sleep 10
echo "[matrix] waiting health: ${SECONDS}s"
done
echo "[matrix] ERROR: service did not become healthy" >&2
return 1
}
start_server() {
local impl="$1"
local max_t="$2"
local label="$3"
local server_log="$OUT_DIR/${label}_server.log"
echo "[matrix] starting $label"
(
cd /root
ENGINEX_MOE_TINY_IMPL="$impl" \
ENGINEX_MOE_TINY_MAX="$max_t" \
VLLM_ENGINE_ITERATION_TIMEOUT_S=3600 \
PYTHONPATH="/usr/local/corex/lib/python3/dist-packages:/usr/local/corex/lib64/python3/dist-packages:${PYTHONPATH:-}" \
LD_LIBRARY_PATH="/usr/local/openmpi/lib:/usr/local/corex-3.2.3/lib64:/usr/local/corex/lib64:/usr/local/corex/lib:/usr/local/iluvatar/lib64:${LD_LIBRARY_PATH:-}" \
nohup python3 -m vllm.entrypoints.openai.api_server \
--model "$MODEL_PATH" \
--host 0.0.0.0 \
--port 1111 \
--served-model-name llm \
--max-model-len 100000 \
--enforce-eager \
--trust-remote-code \
-tp 4 \
--gpu-memory-utilization 0.90 \
--max-num-seqs 2 \
--disable-log-requests \
--disable-frontend-multiprocessing \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--reasoning-parser qwen3 \
> "$server_log" 2>&1 &
echo $! > "$OUT_DIR/${label}_server.pid"
)
wait_health
}
run_case() {
local impl="$1"
local max_t="$2"
local label="${impl}_max${max_t}"
local result="$OUT_DIR/${label}_short_c2_t128_r4.json"
local monitor="$OUT_DIR/${label}_ixsmi.json"
local warmup="$OUT_DIR/${label}_warmup.json"
stop_server
start_server "$impl" "$max_t" "$label"
echo "[matrix] warmup $label"
python3 "$BENCH_SCRIPT" \
--url "$BASE_URL" \
--model llm \
--label "${label}_warmup" \
--prompt-mode short \
--concurrency 1 \
--requests 1 \
--max-tokens 32 \
--timeout 900 \
--out "$warmup"
echo "[matrix] benchmark $label"
python3 "$BENCH_SCRIPT" \
--url "$BASE_URL" \
--model llm \
--label "${label}_short_c2_t128_r4" \
--prompt-mode short \
--concurrency 2 \
--requests 4 \
--max-tokens 128 \
--timeout 900 \
--monitor-out "$monitor" \
--monitor-interval 1.0 \
--out "$result"
stop_server
}
run_case tokenwise 4
run_case tokenwise 8
run_case tokenwise 16
run_case bmm 8
run_case bmm 16
python3 - "$OUT_DIR" <<'PY'
import json
import sys
from pathlib import Path
out_dir = Path(sys.argv[1])
rows = []
for path in sorted(out_dir.glob("*_short_c2_t128_r4.json")):
data = json.loads(path.read_text(encoding="utf-8"))
rows.append({
"label": data.get("label"),
"success_rate": data.get("success_rate"),
"ttft_p90_sec": data.get("ttft_p90_sec"),
"output_tps_p10_per_request": data.get("output_tps_p10_per_request"),
"output_tps_p50_per_request": data.get("output_tps_p50_per_request"),
"aggregate_output_tps": data.get("aggregate_output_tps"),
"avg_gpu_util_pct": (data.get("monitor") or {}).get("avg_gpu_util_pct"),
"max_gpu_util_pct": (data.get("monitor") or {}).get("max_gpu_util_pct"),
})
summary = {
"out_dir": str(out_dir),
"rows": rows,
}
(out_dir / "summary.json").write_text(
json.dumps(summary, ensure_ascii=False, indent=2),
encoding="utf-8",
)
lines = [
"# MoE tiny-batch matrix",
"",
"| case | success | TTFT P90 | Output TPS P10 | Output TPS P50 | Aggregate Output TPS | Avg GPU util | Max GPU util |",
"| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |",
]
for r in rows:
lines.append(
f"| {r['label']} | {r['success_rate']:.2%} | "
f"{r['ttft_p90_sec']:.3f} | {r['output_tps_p10_per_request']:.3f} | "
f"{r['output_tps_p50_per_request']:.3f} | {r['aggregate_output_tps']:.3f} | "
f"{r['avg_gpu_util_pct']:.1f} | {r['max_gpu_util_pct']:.1f} |"
)
(out_dir / "summary.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY
echo "[matrix] done"
echo "[matrix] summary=$OUT_DIR/summary.md"