2d1588d261
Random CCCL pick: cub/cub/device/dispatch/dispatch_topk.cuh (480 lines, full read) CCCL's DeviceTopK uses DoubleBuffer<key_in_t> to ping-pong between two pre-allocated buffers across radix passes, achieving zero allocation in the hot loop. Our sampler.py's _apply_top_k_top_p was allocating 2 new tensors (logits_sort + logits_idx, each vocab_size×4B = 600KB) on every single decode step via torch.sort(). Change: cache sort output tensors keyed on (batch, vocab, device) and reuse them via torch.sort(..., out=(cached_sort, cached_idx)). This eliminates 1.2MB of GPU allocation per decode step. For competition max_num_seqs=1, vocab=152064: Before: 2 × 152064 × 4B = 1.2MB allocated per step After: 0 bytes allocated per step (reuse cached buffers) At 395 tokens/sec target: saves 474MB/sec of allocator pressure. BI-V100 has no async CUDA allocator, so this is synchronous overhead. CCCL architecture insight used: dispatch_topk.cuh line ~430: DoubleBuffer<key_in_t> key_bufs(alloc[3], alloc[2]) for pass: key_bufs.Current() → read, key_bufs.Alternate() → write, swap Base file modified: qwen3_6_scripts/sampler.py (deployed via patch_ops.sh)