CUDA full GPU acceleration, KV cache in VRAM (#1827)

Browse Source

* Fixed CUDA RoPE

* ggml_cuda_mul_mat_vec_p021

* ggml_cuda_scale

* ggml_cuda_diag_mask_inf

* ggml_is_permuted

* ggml_cuda_cpy

* flatten rows for ggml_cuda_op

* Added a --low-vram option

* Fixed Windows performance

* Fixed LLAMA_CUDA_DMMV_Y > 1 for WizardLM

...

This commit is contained in:

Johannes Gäßler

2023-06-14 19:47:19 +02:00

committed by GitHub

parent 9254920265

commit 254a7a7a5f

11 changed files with 853 additions and 149 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

797

ggml-cuda.cu

File diff suppressed because it is too large Load Diff