ggml/llama.cpp releases: b10829

TL;DR

The ggml/llama.cpp project updated its models to use `rsqrt` normalization for GDN instead of `max`, and implemented an L2 norm for gated delta net q/k, enhancing model performance according to changes from flash-linear-attention. These updates are significant as they improve the efficiency and accuracy of the models used in natural language processing tasks.

Detailed Summary

The ggml/llama.cpp project released a new version (b10829) that includes updates to the GDN normalization from `max` to `rsqrt`, and uses flash-linear-attention's l2norm for gated delta net q/k. This update impacts model performance by improving normalization techniques, potentially enhancing the efficiency and accuracy of natural language processing tasks.

Key Points

• • models use flash-linear-attention's l2norm for gated delta net q/k
• • GDN q/k normalization uses rsqrt(sum(x*x) + eps)
• • GDN call includes eps inside the root