Try   HackMD

FLAT-Attention v.s. FlashAttention

FLAT-Attention was public in Arxiv in July 2021 and published in ASPLOS March 2023. We are aware of concurrent work FlashAttention. In short, we are taking different route to tackle the same problem. The proposed solutions are different but the key idea is the same (tiling and scheudling). We summmarize the key difference in the following. To see the detail difference, please refer to our colab demo.

Qualtively Comparisons


Comparisons of FLAT-Attention and FlashAttention

Tiling Strategy Comparisons


The tiling strategy difference between FLAT-Attention and FlashAttention. FlashAttention uses block tiling and weight stationary. FLAT-Attention uses row tiling (row-granularity) and output stationary.

Scheduling Strategy (Dataflow) Comparisons

FlashAttention

FLAT-Attention

Last changed by 
 
Felix
0
1
1016

Read more

FLAT-Attention

Flat-Attention v.s. FlashAttention FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks, ASPLOS'23 Summary of FLAT-Attention FLAT (Fused Logit and Attend Tiling) The quadratic complexity of Logit and Attend operator in Attention layer causing two major challenges: Low performance from memory boundedness Large on-chip buffer requirement for staging intermediate activations

Jun 14, 2023
Sheng-Chun Kao

Felix Kao Up-to-date website Github, Linkedin Skills Proficient: Python, Pytorch, JAX, GCP, Cloud TPU, Verilog Experienced: Tensorflow, C/C++, Matlab Research Interest and Experience [ML] ML-based automation, RLs, GA-based optimization, Transformer, Efficient attention for long sequence, Pruning, Quantization, Neural architecture search

Jun 10, 2022
Read more from Felix
Published on HackMD