Source: CCCL dispatch_merge_sort.cuh alias_temporaries() pattern
- 4 allocations (partitions + keys + values + vsmem) packed into 1 cudaMalloc
- Principle: never allocate throwaway intermediates in the hot path
- dispatch_merge_sort uses ping-pong buffer to avoid copying between passes
Changes to vllm/model_executor/layers/sampler.py _apply_penalties():
Old: repetition_penalties[:, None].repeat(1, vocab_size)
→ Creates full (num_seqs, 152064) float32 tensor = 608KB
→ Then masks most values to 1.0 (wasted allocation)
→ Then torch.where over entire vocab (wasted compute on masked positions)
New: Broadcasting with unsqueeze(1) + conditional torch.where
→ rep_pen shape: (num_seqs, 1) broadcasts to (num_seqs, vocab_size)
→ Zero intermediate allocation
→ token_mask selects only prompt/output tokens (typically <1% of vocab)
→ Nested torch.where applies divide/multiply only where needed
Memory saving per decode step: 608KB (vocab=152064, num_seqs=1, float32)
This is in the penalties hot path that runs every decode step when
repetition_penalty != 1.0.
Also in this commit (from previous edit):
- Fixed _sampler_cache -> _sampler_temp_storage module-level declaration
- CCCL alias_temporaries pattern for bin_counts pre-allocation
322f5553e1