ggml/llama.cpp releases: b10643
Sentiment: neutral
TL;DR
The ggml/llama.cpp project released an update that enhances support for multi-NPU devices, making the backend fully asynchronous by default. This update is significant as it improves performance and flexibility in deploying models across multiple hardware accelerators.
Detailed Summary
The ggml/llama.cpp project released an update (commit b10643) that enhances support for multi-NPU devices, specifically IQ9 and IQ10, by implementing fully asynchronous backend functionality. This update removes optional host buffer support in hexagon and relaxes supported unary checks, improving performance and compatibility. The broader impact includes better utilization of multi-NPU hardware, potentially leading to faster processing times for large language models.
Key Points
- • hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend (#26501)
- • hexagon: use non-host bufs by default and make the backend fully async
- • hex-hb: remove optional hostbuf support and fix async copy