---
# System prepended metadata

title: test image

---

# test image

![](https://i.imgur.com/C98p2h4.jpg)

![](https://i.imgur.com/hNnIhyP.gif)


---
marp: true
---

# Progress Report

Chen

03/15~03/18

-------------

# Recap of last meeting

- Integrate **PhySG**'s representation for env mapping and BRDF to **volume rendering** framework.
    - Enables extraction of **unknown arbitrary** lighting condition.
    - May omit specular term currently.
- Choose proper dataset to test the ability for the network to overfit.

-------------

# Overview

- Refactored previous code framework for current project.

- Used Blender to render a new dataset *"ball"* for toy sanity check.

- Implemented several BRDF assumptions including **diffusion, SG and SH**. 

- Implemented hash multiresolution encoding for potential acceleration of training.

- Performed experiments of variants of algorithm on dataset *ball* and *kitty*(from PhySG).

---------
# Method (diff)
- Environment map mapping: combination of $M$ SGs, i.e., $\sum_{i}^M \mu_i e^{\lambda_i(v \omega_i - 1)}$
- BRDF assumption: only diffusion, which is constant, equals to albedo
- Indirect illumination: omit.
- Shadows: omit.
- Total Rendering Equation: 

$$
L_o(\omega_o;x) = \int_{S} \mathbf{a} (\omega_i \cdot \mathbf{n})(\sum_{i}^M \mu_i e^{\lambda_i(v \omega_i - 1)}) \mathrm{d} \omega_i
$$

----------
# Baselines
- SG: add consideration for specular term, from *PhySG*.
- SH: use **Spherical Harmonic** to represent radiance, from *Plenoctree*
- NeRF: follow original *NeRF*.

----------
# Dataset - *ball*
- Use Blender to render.
- 100 views for training.
![](https://i.imgur.com/5plHhDr.png)


----------
# Dataset - *kitty*
- From *PhySG*
- Extremely hard due to complex env lighting.

![](https://i.imgur.com/5ch8NZb.png)



---------
# Core Results - *ball*
<!-- ## *ball* dataset - comparison on curve -->

![](https://i.imgur.com/vUb2hPP.png)



---------
# Core Results - *ball*
![](https://i.imgur.com/YcvVJmX.png)


---------
# Core Results - Decompose, *diff model*

![](https://i.imgur.com/2aoJfPr.png)


---------
# Core Results - Decompose, *SG model*
![](https://i.imgur.com/KaaMq6H.png)


---------
# Core Results - relighting
![](https://i.imgur.com/k1P0ecb.gif)


---------
# Core Results 
## *kitty* dataset - comparison on curve
![](https://i.imgur.com/mzF3lWa.png)


---------
# Core Results 
## *kitty* dataset - comparison on view synthesis
<!-- ![](/home/chen/Downloads/comparison_kitty.png) -->
![](https://i.imgur.com/k0K8yhb.png)


<!-- ---------
# Other Results -->

---------
# Analysis
- Our method(diffusion based BRDF + SG based env map) and SG can successfully discover object's albedo and env light in *ball* dataset.
- All volume-rendering based methods failed in producing reasonable results in *kitty*
    - We do not count in visibility information.
    - The training step is not enough to let them converge.

<!-- --------
# Limitations



---------
# Next step -->

# Quick Sync @ 0322

Chen


-------------

# Recap of last meeting

- Try other relighting to verify the result.

- Fix the lighting to ground truth to check the kitty dataset

---
# Overview
- More results: rotational relighting, normal

- Try fix the lighting.

- Implement simplified NeILF

---------
# Rotational Relighting
Result of using our trained model(analytic normal) to render relighting effects.
<!-- ![](/home/chen/rlr.gif) -->
![](https://i.imgur.com/4cfNNTL.gif)


---
# Normal Estimation - analytic normal
Analytic normal
<!-- ![](/home/chen/Downloads/normal_grad.png) -->
![](https://i.imgur.com/bU57j6f.png)


---
# Normal Estimation - MLP-predicted normal
MLP-predicted normal
<!-- ![](/home/chen/Downloads/normal_pred.png) -->
![](https://i.imgur.com/5z9OeDU.png)


This result align with observation of NerV[Srinivasan et al. 2020]

[Srinivasan et al. 2020]: NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis

---
# The result of fix lighting on kitty
(training light and viewpoint, after convergence)
<!-- ![](/home/chen/Downloads/kitty_known.png) -->
![](https://i.imgur.com/mVjaF1t.png)


Our model perform poorly even under setting of overfitting.

---
# Observation: The more complex, the worser psnr.
<!-- ![](/home/chen/Downloads/compare.png) -->
![](https://i.imgur.com/u8KX2Ff.png)

The highest line is with 1 point light. The middle line is with 6 point lights. The lowest line is with complex environment lighting.

---
# NeiLF: Does not converge yet
<!-- ![](/home/chen/Downloads/neilf.png) -->
![](https://i.imgur.com/SIyAF7a.png)

---

# Progress Report @ 0326

Chen


-------------

# Recap of last meeting

- Test whether UNISURF can produce more accurate geometry.

- Move on to multi-object scenes.

---

# UNISURF's Method

Original NeRF's Rendering Equation:

$$
\sum (1 - e^{-\sigma_i \delta_i})\prod(e^{-\sigma_j \delta_j})  c_i
$$


UNISURF's idea: use occupancy field.

$$
\sum o_i \prod(1-o_j) c_i
$$

---

# Result

<!-- ![](/home/chen/report0326/mesh2.png) -->
![](https://i.imgur.com/sqHuW31.png)



---

# Multi object scene - *3b3p* dataset

A synthetic scene with 3 balls and 3 point lights(different intensity).

<!-- ![](/home/chen/report0326/dataset1.png) -->
![](https://i.imgur.com/1yAdfgP.png)


---

# Exp 1. Unified NeRF + Lambertian + Known Light

## Method

- We directly apply method on single-object scene to current setting.
- We assume the light is known and manually render the reflectance.
- Currently we mask the background for object-centric learning

---

# Exp 1. Unified NeRF + Lambertian + Known Light
## Results - decomposition

<!-- ![](/home/chen/report0326/result1.png) -->
![](https://i.imgur.com/FNdU4PY.png)


---

# Exp 1. Unified NeRF + Lambertian + Known Light
## Results - relighting


<!-- ![](/home/chen/report0326/rlr2.gif) -->
![](https://i.imgur.com/GLrP5Bc.gif)


<!-- ---

# Exp 1. Unified NeRF + Lambertian + Known Light
## Summary and Analysis -->

---

# Exp 2. Object-Centric NeRF + Lambertian + Known Light + Known Mask

---
# Method 2.1
Represent each object as a NeRF.

We use mask to guide rendering, i.e.

$$
C(x) = \sum_k M_k(x) \sum_i T_i \alpha_i c_i
$$

---
# Result 2.1
<!-- ![](/home/chen/report0326/result2.png) -->
![](https://i.imgur.com/VDPWgzU.png)


---
# Result 2.1 - Editting 
<!-- ![](/home/chen/report0326/result3.png) -->
![](https://i.imgur.com/BrkIUko.png)


---
# Result 2.2 - Fail case

This method cannot supervise other area of object.
<!-- ![](/home/chen/report0326/result4.png) -->
![](https://i.imgur.com/U85zdyX.png)


---

# Method 2

Represent each object as a NeRF.

We follow uORF for compositional rendering.

Supervision:

- Mask supervision: 

$$\sum_{k=1}^K \sum_{i=1}^N ||\alpha_k(x_i) - M_k(x_i)||_2 $$

- Sparsity supervision:

$$\sum_{i=1}^N ||\sum_{k=1}^K \alpha_k(x_i) - 1||_2$$

---

# Result 2.2 

<!-- ![](/home/chen/report0326/result5.png) -->
![](https://i.imgur.com/ZTFRoyU.png)


---

# Result 2.2 - Editting: Moving and Relighting


<!-- ![](/home/chen/report0326/move2.gif) -->
![](https://i.imgur.com/j6jhIIV.gif)


---
# Result 2.2 - Editting: Moving and Relighting
<!-- ![](/home/chen/report0326/move.gif) -->
![](https://i.imgur.com/CZXLJvQ.gif)


---

# Exp 3. Unified NeRF + Lambertian + Unknown Light

---

## Result

<!-- ![](/home/chen/report0326/result7.png) -->
![](https://i.imgur.com/dsM8G1t.png)



---

## Visualization of Reconstructured Light

<!-- ![](/home/chen/report0326/light.png) -->
![](https://i.imgur.com/Ww6XKsc.png)


---

## Possible reasons

Near point light does not satisfy the condition of SG(which is used to model env map).


