---
# System prepended metadata

title: '[CWZ+23]Multimodal Federated Learning: A Survey'

---

*Multimodal Federated Learning: A Survey*
https://www.mdpi.com/1424-8220/23/15/6986 

Multimodal Federated Learning(MFL)，傳統的FL訓練data sample只有一種modality(型態。例如: image vs text)。在MLF中所要處理的問題是有多modality。(<font color="#f00">vs multi-task FL 則是system中一次有多個model owner同時訓練</font>)

CONTRIBUTION:
1. 對現有modality data FL依據進行分類
2. 介紹多個不同情境的dataset(看paper)
3. 此領域非常新很少人研究

PROBLEM:
* modality heterogeneity challenge, 主要包含:
    1. significant differences in model structures
    2. significant differences in local tasks
    3. significant differences in parameter spaces among clients(可以想像image與text的CNN會長得不一樣)

ALGORITHM OUTLINE:
* 大致上是由unimodel的FedAvg改良而來。其中可能改良的地方是在local計算loss時，可以多加入weigh於不同modality上(例如在更著重於image便是的task中，可以調高image modality的比重減少text的比重)
    1. 紅色-不同modality的weight
    2. 綠色-不同modality所對應的不同model，output維predict label，y-label
* **loss function**
    ![image](https://hackmd.io/_uploads/S1IeQGjVp.png)
* **aggregate**
    ![image](https://hackmd.io/_uploads/SkDGXGi4p.png)

CATEGORY of MFL
* 使用每個clinet之間lable space、feature space以及modality space分布方式來區分成四種
* the more congruent modality combinations the clients hold, the less heterogeneous the modality distribution of the system.
    1. Congruent MFL: all the clients hold the SAME modality set
        * horizontal settings (traditional unimodal FL) ⇨ SAME AS TRADITIONAL FL
    2. Incongruent MFL: the clients hold the DIFFERENT modality set
        * Vertical (traditional unimodal FL) ⇨ SAME AS TRADITIONAL FL
        * Transfer (traditional unimodal FL) ⇨ SAME AS TRADITIONAL FL
        * hybrid MFL  ⇨ NEW!!!
    ![image](https://hackmd.io/_uploads/rkZEVzsN6.png)
    1. horizontal MFL
        ![image](https://hackmd.io/_uploads/HJQU4zsNa.png)
    2. Vertical MFL
        ![image](https://hackmd.io/_uploads/Sktd4MsVp.png)
    3. Transfer MFL
        ![image](https://hackmd.io/_uploads/S195NGiN6.png)
    4. hybrid MFL
        ![image](https://hackmd.io/_uploads/Hk_hEGjVT.png)


