[BASE] vllm/core/evictor_v2.py: CCCL bucket_sort2d design pattern annotation

Random CCCL pick: thrust/examples/bucket_sort2d.cu (108 lines, full read)
Maps to: vllm/core/evictor_v2.py (LRU cache eviction)

bucket_sort2d.cu pattern: transform→sort_by_key→lower_bound/upper_bound
  - point_to_bucket_index ↔ content_hash (prefix cache key)
  - sort_by_key ↔ eviction priority ordering
  - lower_bound/upper_bound ↔ block range lookup

Current LRUEvictor.evict() is O(n) linear scan over OrderedDict.
CCCL pattern suggests sort_by_key → O(1) pop for production scale.
For competition (max_num_seqs=1, bounded blocks): current is sufficient.

Also read: vllm/core/block/prefix_caching_block.py (200 lines)
This commit is contained in:
muh-pipeline
2026-08-06 02:29:19 +00:00
parent b6538fd10e
commit 6148e03bc7

View File

@@ -73,6 +73,16 @@ class LRUEvictor(Evictor):
the same last_accessed time, then the one with the largest num_hashed_tokens
will be evicted. If two blocks each have the lowest last_accessed time and
highest num_hashed_tokens value, then one will be chose arbitrarily
CCCL system design note (from thrust/examples/bucket_sort2d.cu):
CCCL's bucket sort uses transform→sort_by_key→lower_bound/upper_bound
to build O(1) lookup from bucket_index → item range. This maps to:
content_hash → block_id (our _cached_blocks dict)
last_accessed → eviction priority (our OrderedDict linear scan)
For production: sort_by_key on (last_accessed, -num_hashed_tokens)
would make evict() O(1) pop instead of O(n) scan.
For competition (max_num_seqs=1): current O(n) scan is fine since
n = num_gpu_blocks is bounded by GPU memory / block_size.
"""
def __init__(self):