MLE - Outfit Compatibility Learning
===

# Summary

This project is to reimplement an outfit recommendation system from a paper [Dressing as a Whole: Outfit Compatibility Learning Based on Node-wise Graph Neural Networks](https://paperswithcode.com/paper/dressing-as-a-whole-outfit-compatibility)



| Criteria | Description |
|:---------- |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Problem | I have many pieces of clothes and sometime I don't know how to combine them to make a good outfit. |
| Solution | Use an enhanced GNN to build a fashion recommendation system. |
| Data | Polyvore data set which was collected from a popular fashion website polyvore.com. This dataset contains **21,889** outfits, in which **17,316** are for training, **1,497** for validation and 3,076 for testing. |
| Preprocess | Use inception-v3 to extract image feature, filter the categories which appear less than 100 times. The code is provided by the [official Github](https://github.com/CRIPAC-DIG/NGNN/tree/master/data) from the paper. |
| Model | Using an enhanced GNN called *Node-wise Graph Neural Network* (NGNN). The model was proposed by the research team. |
| Search | (1) **Fill-in-the-blank**: suggesting an item that matches with existing components of outfit; (2) **Compatibility prediction**: predicting the compatibility scores of given outfits. |

# Modeling

## Step 1 - Graph Construction
- First, construct a *Fashion Graph* where each **node** represents a **category** and each **edge** represents the **interaction** between two nodes.
- An outfit (e.g. sweater, short, sandals, shoulder bag) can be represented as a subgraph.
- If *two categories* have matching relation in the training dataset, there are two **directed** edges in *reverse directions* between the two corresponding nodes in *Fashion Graph*.
For example, one may select *a pair of socks* to match the shoes but less likely to select *a pair of shoes* to match socks particularly.
- To calculate the weigh of a directed edge:
$$w(n_i, n_j) = \frac{Count_{c_i,c_j}/Count_{c_j}}{\sum_k{Count_{c_i,c_k}/Count_{c_k}}}$$

$w(n_i, n_j)$: the weight of edge from $n_i$ to $n_j$.
$Count_{c_i,c_j}$: the co-occurrence frequency of category $c_i$ and $c_j$.
$Count_{c_j}$: the occurrence frequency of category $c_j$.
- The constructed Fashion Graph has 120 nodes and 12500 directed and weighted edges.

## Step 2 - NGNN
- Design a **NGNN** to model the node interactions and learn node representations.
- NGNN stands for *Node-wise Graph Neural Network*. NGNN consists of three steps.
- The first step is to *learn the initial node state*.
- The second step is to *model node interactions and update node state*.
- The third step is to *calculate the compatibility score with attention mechanism*.

### 2a - Learning Initial Node State


- The input of NGNN is features (either visual or textual features) of items.
- In NGNN, each node $n_i$ is associated with a hidden state vector $h_i$ which is dynamic. Initial state:
$$h_i^0 = tanh(r_i)$$
- $r_i$ is the representation of an item with its features $f_i$
$$r_i = W_h^if_i$$
$W_h^i$ is a linear mapping matrix for each category $c_i$.

### 2b - Modeling Node Interactions


// TODO the formula is quite complex, need to understand GNN.

### 2c - Compatibility Calculating with Attention Mechanism

- An attention layer is finally utilized to calculate the compatibility score.
- For example, a coat maybe suitable for an outfit and promote the compatibility, while it will decrease the compatibility together with summer clothes.


# Timeline

Average ~ 25 hours per week.
Targets:
- Week 1
- Have preprocessed data.
- Able to build a graph with initial state.
- Week 2
- Able to calculate the compatibility score for an outfit.
- Week 3
- Able to calculate the compatibility score for an outfit *from an UI*.
- Week 4
- Project is submitted.
- Be ready for the demo.