## RELATE

## Designed Goals of Paper:
* Scene Generation: The primary goal of the paper is to develop a model that can generate scenes and videos of multiple interacting objects. The authors aim to create a generative model that can produce visually realistic and physically plausible scenes and videos.
* Interpretable Parameterization: The paper aims to introduce a model with a physically interpretable parameter space. Instead of using uninterpretable noise vectors for scene generation, the model is designed to map these parameters to meaningful, interpretable quantities related to object appearance and spatial relationships.
## Model

*In simpler terms, the **RELATE model** represents different parts of a scene, like the background and objects in the front, using appearance and position information. It has a module called "spatial relationship" that makes sure the positions of these objects make sense, like not intersecting with each other. Then, there's a generator network that turns all of this into an actual image. RELATE learns from real images in a training setup using a GAN (Generative Adversarial Network), with a discriminator to help improve its image generation*

RELATE as shown in figure consists of two main components: An interaction module, which computes physically plausible inter-object and object background relationships, and a scene composition and rendering module, which features an interpretable parameter space factored into appearance and position vectors. The details are given next.

### 1. Physically-Interpretable Scene Composition and Rendering:

1. RELATE deals with scenes containing up to K distinct objects. It starts by sampling appearance parameters (z1, z2, ..., zK) for each individual foreground object and a background parameter (z0).
2. Appearance parameters are mapped to tensors (Ψk) using learned decoder networks. One decoder is used for the background (Ψ0) and another for the foreground objects (Ψk).
3. The tensors Ψk are defined in a spatial resolution of HxH with C as total feature channels.
4. Each foreground object is associated with a pose parameter (θk), which is geometrically interpretable. In this case, θk is assumed to be a 2D translation, applied to Ψk.
5. Appearance parameters are sampled independently, assuming that the appearance of different objects is independent.
### 2. Modeling Correlations in Scene Composition:

* RELATE departs from the assumption that the object parameters (θi) are independent. Instead, it models correlations between objects' locations and appearances and between objects and the background.
* A two-step procedure is used: First, a vector of K independently sampled poses (Θˆ) is generated. Then, a "correction" network (Γ) is used to account for the correlations. This network takes into account object interactions and enforces symmetry with respect to object order.
* The "correction" function ζ is implemented similarly to the Neural Physics Engine (NPE) and captures interactions between objects (h_s is an embedding capturing these interactions).
* The use of this scheme allows the model to work with different numbers of objects (K), which can be sampled from a defined interval.

### Ordered Scenes:

**Natural Order:** RELATE can be adapted for scenes where objects have a natural order, such as stacks of blocks. In these cases, the pose of each object $(θi)$ is conditioned on the preceding pose $(θi-1)$ using a Markovian process.

### Modeling Dynamics:

**Dynamic Predictions:** RELATE can be extended to make dynamic predictions. Initial object positions $(θk(0))$ are sampled, and they are updated incrementally based on object velocity $(vk(t))$.
**Velocity Updates:** The velocity $(vk(t+1))$ is determined by updating equations that take into account appearance parameters, initial positions, and the velocities of other objects

## Objectives
The model is trained on a dataset of N images that depict scenes with different object configurations. No additional supervision is required beyond this dataset. **While training the focus is on sum of two high fidelity losses and a structural loss.**

## High Fidelity Losses:

The high fidelity losses aim to measure how well the generated images produced by the model match real images (I) from the training dataset.
### Two specific loss components are used for high fidelity:
1. GAN Discriminator Loss ($Lgan(
ˆI, I)$) : This loss measures the realism of the generated images by contrasting them with real images using a standard GAN discriminator. GANs are a type of generative model where a discriminator network tries to distinguish between real and generated images, and the generator aims to produce images that can "fool" the discriminator.
2. Style Loss ($Lstyle(
ˆI, I)$): This loss measures the similarity in style between generated and real images. Style losses are often used to ensure that the texture and visual patterns in the generated images are similar to those in real images.
## Structural Loss:

A structural loss is introduced to encourage the model to learn a meaningful relationship between object positions and the generated images.
A position regressor network $(P)$ is trained to predict the locations of objects in the generated images $(ˆI)$. The idea is to make sure that the model can create images where the positions of objects are related to the scene content.
To simplify this task, the authors generate a modified image $(ˆI0)$ by retaining only k object (out of K) at random and minimizing the difference between the predicted position (ˇθk) and the position estimated by the position regressor P
$|
ˇθk − P(G(W(z0, zk, θk)))k|$.
Gradients are not back-propagated through θk during this process to prevent the model from collapsing to a trivial solution where all objects are positioned at the same location.
The position regressor network shares most of its weights with the discriminator network.

**Dynamic Prediction (Additional Information):**
In the case of dynamic scene predictions, a discriminator takes as input a sequence of images concatenated along the RGB dimension. This discriminator is responsible for distinguishing between real and fake sequences.
A position regressor is also used for dynamic prediction and is tasked with predicting the position of an object rendered at random with zero velocity. This is important for models that aim to predict how objects move and interact over time.

## Implementation:

* The model uses a technique called Adaptive Instance Normalization (AdaIN) to create images.
* The images have an initial size of 16x16, which is then increased to a final output size of 128x128 pixels.
* They use a popular optimization method called "Adam" to train the model for a fixed number of training cycles. They select the best model after training.
### Comparison with Other Models:

1. They compare their model's performance to other state-of-the-art methods using a measurement called the **FID(Fréchet inception distance) score**.

### Datasets:

* They test their model on four different sets of data.
* One dataset involves videos of two colored balls rolling in a bowl with different shapes.
* Two other datasets are synthetic and involve objects on tabletops and stacking blocks.
* They also have a new dataset with videos of a busy street intersection with varying numbers of cars.
* The last dataset with cars is especially interesting because it involves interactions between cars at traffic lights, like slowing down when the light is red and speeding up when it turns green.\

# Follow-Up work to look at
https://openaccess.thecvf.com/content/CVPR2021/html/Niemeyer_GIRAFFE_Representing_Scenes_As_Compositional_Generative_Neural_Feature_Fields_CVPR_2021_paper.html

https://openaccess.thecvf.com/content/CVPR2021/html/Chen_GeoSim_Realistic_Video_Simulation_via_Geometry-Aware_Composition_for_Self-Driving_CVPR_2021_paper.html

https://proceedings.neurips.cc/paper/2021/hash/4eff0720836a198b6174eecf02cbfdbf-Abstract.html

# References
github link - https://github.com/hyenal/relate/blob/main/README.md
Paper-
https://arxiv.org/pdf/2007.01272.pdf