# Transformer

###### tags: `Accelerator`
## Repo
## 未來計畫
## 會議記錄彙整
::: spoiler 2024 summer
[2024/7/11](/0oUtwSu6TJaGPsG2bL0a1g)
[2024/7/29](/SkQPhwNt0)
[2024/8/15](/@XggIbVZOTWWoUb59rhalCg/Bke8Lyoc0)
[2024/8/29](/@XggIbVZOTWWoUb59rhalCg/B1W0HuaoC)

:::
::: spoiler 2024 fall
[2024/9/23](/@XggIbVZOTWWoUb59rhalCg/rkyKcLCpA)
[2024/10/7](/@XggIbVZOTWWoUb59rhalCg/SyK_Zpl1kg)
[2024/10/28](/@XggIbVZOTWWoUb59rhalCg/SyVbktnl1e)
[2024/11/11](/@Mickeyyayyaya/HJX0bxJz1e)
[2024/11/25](/@XggIbVZOTWWoUb59rhalCg/H1-Xl7eXkx)
[2024/12/9](/@KevinChou0518/Bk6DSR74Jl)
[2024/12/23](/@Mickeyyayyaya/H1cdZLIHyg)
[2025/01/06](/@XggIbVZOTWWoUb59rhalCg/rkijzTd81g)
[2025/01/20](/@XggIbVZOTWWoUb59rhalCg/rJLqsEjv1g)
:::
:::spoiler 2025 spring
[2025/02/05](/@XggIbVZOTWWoUb59rhalCg/B1DwPAetke)
[2025/02/17](/@Mickeyyayyaya/Hk2OxVx51x)
[2025/02/24](/@XggIbVZOTWWoUb59rhalCg/S1IqIst5kx)
[2025/03/10](/@Mickeyyayyaya/B1HEhe2s1e)
[2025/03/24](/@XggIbVZOTWWoUb59rhalCg/rkLJCK02ye)
[2025/05/05](/@KevinChou0518/ryxABCSlgl)
[2025/05/19](/@KevinChou0518/ByfTPr_-xl)
[2025/06/02](/@KevinChou0518/SyeNbkofxl)
:::
:::spoiler 2025 fall
[2025/09/24](/@XggIbVZOTWWoUb59rhalCg/rJAEIPbhle)
:::

## 研究數據
### [Data Sheet](https://docs.google.com/spreadsheets/d/1LgZrz_RaV6M-10-fFQlOEgkBjQIWKQh-_MXb8A_MDGU/edit?gid=73368035#gid=73368035)
[Experiment#1: Model Parameter explantion](/ByRlLSiOR)
[Experiment#2: Data Transfer(no sram)](/By1TaLouC)
[Experiment#3: Impact of SRAM Size](/ryVYrYEtA)
#### NEXT TODO
- [ ] Add systolic array
- [ ] Add another model
- [ ] Add flash attention
- [ ] Add e2e
## 筆記
[Transformer note: Calculate of param](/S1ssbSSU6)
[Transformer note: Data Transfer Analysis](/B1J7hIj_C)
https://hackmd.io/@Mickeyyayyaya/H1IMJIRTA
## 資源
* Transformer model and paper
[Transformers-Tutorials](https://github.com/NielsRogge/Transformers-Tutorials)
[labml.ai Deep Learning Paper Implementations](https://github.com/labmlai/annotated_deep_learning_paper_implementations)
[Scalable MatMul-free Language Modeling](https://github.com/ridgerchu/matmulfreellm)
https://huggingface.co/blog/hf-bitsandbytes-integration
## 相關論文
[Scalable MatMul-free Language Modeling](https://arxiv.org/pdf/2406.02528)