Zyphra ZYFA: FP8 FlashAttention for AMD MI355X, Up to 1.76x Faster

TL;DR

Zyphra released open-source FP8 and BF16 prefill attention kernels for AMD's MI355X, achieving up to 1.76x faster performance with significantly lower error rates compared to AMD’s AITER, highlighting potential advancements in GPU efficiency for AI models.

Detailed Summary

Zyphra released ZYFA, open-source FP8 and BF16 prefill attention kernels for AMD Instinct MI355X, which run 1.16-1.76 times faster than AMD’s AITER on real model activations while maintaining low error rates. The broader impact includes potential improvements in the efficiency of AI model training, particularly benefiting users of AMD GPUs.

Key Points

• • Zyphra released ZYFA, open-source FP8 and BF16 prefill attention kernels for AMD Instinct MI355X.
• • FP8 kernels run 1.16-1.76x faster than AMD’s AITER on real model activations.
• • Error is reduced by 62-78% at head dimension 128 compared to AMD’s solution.