ggml/llama.cpp releases: b10255
Sentiment: neutral
TL;DR
The ggml/llama.cpp project updated its SYCL oneDNN SDPA to support Q4_0-Q8_0 and FP32 KV caches, extending the handling of non-FP16 key-value pairs. This update is significant as it enhances the flexibility and performance of the model in various computational scenarios.
Detailed Summary
The ggml/llama.cpp project has updated its SYCL oneDNN SDPA implementation to support non-FP16 key-value (KV) caches, specifically Q4_0-Q8_0 and FP32 types. This update was made through pull request #25874, which extends the existing path for handling KV caches by dequantizing or converting them to dense FP16 on-device before processing. The broader impact of this change enhances the flexibility and efficiency of the library in various computational scenarios involving different data precisions.
Key Points
- • Extends oneDNN SDPA for Q4_0-Q8_0 and FP32 KV caches
- • Dequantizes or converts K/V to dense FP16 on-device
- • Handles non-FP16 KV caches in SYCL environment