ggml/llama.cpp releases: b10876
Sentiment: neutral
TL;DR
The ggml/llama.cpp project updated its CUDA implementation to provide more control over quantization options, allowing users to configure specific combinations and enabling runtime fallbacks for unsupported configurations. These changes enhance flexibility and usability, making the library more adaptable to various hardware setups.
Detailed Summary
The ggml/llama.cpp project released an update that includes changes to the CUDA backend, allowing more control over quantization options and adding configurable FA quant combinations. The update also removes unused flags and adds a runtime fallback with warnings for uncompiled combinations, while updating documentation. This change impacts developers working on GPU-accelerated machine learning models using ggml/llama.cpp.
Key Points
- • CUDA: replace GGML_FA_ALL_QUANTS with GGML_FA_QUANTS, more control over what is compiled
- • CUDA: add configurable FA quant combinations
- • remove all flags but , add runtime fallback with warning for uncompiled combination