*Multimodal Federated Learning: A Survey*
https://www.mdpi.com/1424-8220/23/15/6986

Multimodal Federated Learning(MFL)，傳統的FL訓練data sample只有一種modality(型態。例如: image vs text)。在MLF中所要處理的問題是有多modality。(<font color="#f00">vs multi-task FL 則是system中一次有多個model owner同時訓練</font>)

CONTRIBUTION:
1. 對現有modality data FL依據進行分類
2. 介紹多個不同情境的dataset(看paper)
3. 此領域非常新很少人研究

PROBLEM:
* modality heterogeneity challenge, 主要包含:
1. significant differences in model structures
2. significant differences in local tasks
3. significant differences in parameter spaces among clients(可以想像image與text的CNN會長得不一樣)

ALGORITHM OUTLINE:
* 大致上是由unimodel的FedAvg改良而來。其中可能改良的地方是在local計算loss時，可以多加入weigh於不同modality上(例如在更著重於image便是的task中，可以調高image modality的比重減少text的比重)
1. 紅色-不同modality的weight
2. 綠色-不同modality所對應的不同model，output維predict label，y-label
* **loss function**
* **aggregate**

CATEGORY of MFL
* 使用每個clinet之間lable space、feature space以及modality space分布方式來區分成四種
* the more congruent modality combinations the clients hold, the less heterogeneous the modality distribution of the system.
1. Congruent MFL: all the clients hold the SAME modality set
* horizontal settings (traditional unimodal FL) ⇨ SAME AS TRADITIONAL FL
2. Incongruent MFL: the clients hold the DIFFERENT modality set
* Vertical (traditional unimodal FL) ⇨ SAME AS TRADITIONAL FL
* Transfer (traditional unimodal FL) ⇨ SAME AS TRADITIONAL FL
* hybrid MFL ⇨ NEW!!!
1. horizontal MFL
2. Vertical MFL
3. Transfer MFL
4. hybrid MFL