36 lines
2.5 KiB
Plaintext
36 lines
2.5 KiB
Plaintext
|
/usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
|
|
registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
|
||
|
|
dispatch key: ADInplaceOrView
|
||
|
|
previous kernel: no debug info
|
||
|
|
new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
|
|
self.m.impl(
|
||
|
|
Loading model from: checkpoints/korean_3b_fp8_run1/checkpoint-0057000
|
||
|
|
Model parameters: 3015.4M
|
||
|
|
|
||
|
|
Perplexity config: seq_len=2048, stride=512, batch_size=32
|
||
|
|
Loaded 15,159,838 tokens from data/korean_c4_val.bin
|
||
|
|
Evaluating perplexity: 0%| | 0/926 [00:00<?, ?batch/s]
Evaluating perplexity: 0%| | 0/926 [00:00<?, ?batch/s]
|
||
|
|
Traceback (most recent call last):
|
||
|
|
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/perplexity.py", line 263, in <module>
|
||
|
|
main()
|
||
|
|
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/perplexity.py", line 247, in main
|
||
|
|
ppl = compute_perplexity(
|
||
|
|
^^^^^^^^^^^^^^^^^^^
|
||
|
|
File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
|
||
|
|
return func(*args, **kwargs)
|
||
|
|
^^^^^^^^^^^^^^^^^^^^^
|
||
|
|
File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/perplexity.py", line 157, in compute_perplexity
|
||
|
|
ce = F.cross_entropy(
|
||
|
|
^^^^^^^^^^^^^^^^
|
||
|
|
File "/usr/local/lib/python3.12/dist-packages/torch/nn/functional.py", line 3499, in cross_entropy
|
||
|
|
return torch._C._nn.cross_entropy_loss(
|
||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
|
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
|
||
|
|
Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
|
||
|
|
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
|
||
|
|
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
|
||
|
|
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
|
||
|
|
|