# Chang Dick Mun's log
# Main Research Direction
- build component-wise adaptive learning rate incorporated with SGD for deep learning.
- Develop python code so that to check the efficiency of the proposed algorithm.
- Establish the convergence properties of the proposed algorithm.
- Applied the proposed algorithm into PINN.

## Motivation
- Not all of the parameters in a vector are suitable to use the same scalar of learning rate at each iteration.
- Fixed and adaptive learning rate are not a best learning rate in this case.

Goal:
- use multiple adaptive learning rate (diagonal) incorporate with SGD instead of adaptive and fixed learning rate.
- arrange all spectrums (learning rate) in matrix to find the optimal solution when the point $x$ stuck at the position at least 5 iterations.
## positive definite
$z^TQz$
- $z$ & $Q$ not necessary is positive.
- $Q$ either all positive or eigenvector is positive
- $z$ have two, so can get positive after multiply it.
# Theory and the calculation of conjugate gradient and BFGS
## Conjugate gradient
- Conjugate gradient method is an algorithm for the numerical solution of particular systems of linear equations, namely those whose matrix is positive-definite.
- The conjugate gradient method can also be used to solve unconstrained optimization problems such as energy minimization.

$d_k=-g_k+\beta_k*d_{k-1}$
$d_k$->conjugate vector
$g_k$->residual at the *k* step-> $g_k=b-Ax_k$ ($b$ as a coefficient matrix)
$d_{k-1}$-> previous conjugate vector
$\beta_k$ ->scala(constant) ->adjust direction

### Residual formula
$g_0=b-Ax_0$
- $g_0$=>negative gradient of $f$ at $x_k$,so the gradient descent method would require to move in the direction $g_0$.

$z_0=M^{-1}g_0$
- $M^{-1}$=>symmetric positive-definite and $M^-1A$ has a better condition num A.

### Fletcher-Reeves formula
$\beta_k=(r^T_{k+1})(z_{k+1}))/(r^T_kz_k)$

- less num of function evaluation
- less computational time
- less iteration
### Comparation between Conjugate gradient and Steepest descent
- $\beta_k$=0 in steepest descent in order to make them locally optimal.
- Every iteration of steepest descent methods is a bit cheaper compared to the conjugate gradient methods.

### Reference
[Conjugate gradient](https://en.wikipedia.org/wiki/Conjugate_gradient_method#As_an_iterative_method)

## BFGS
- The Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm is an iterative method for solving unconstrained nonlinear optimization problems.
- BFGS determines the descent direction by preconditioning the gradient with curvature information.
- It is a type of second-order optimization algorithm, meaning that it makes use of the second-order derivative of an objective function and belongs to a class of algorithms referred to as Quasi-Newton methods that approximate the second derivative (called the Hessian) for optimization problems where the second derivative cannot be calculated.
- The first derivative of multiple input variables may also be a vector, where each element is called a partial derivative. This vector of partial derivatives is referred to as the gradient.
- **Gradient**: Vector of partial first derivatives for multiple input variables of an objective function.

- This idea generalizes to the second-order derivatives of the many kind of inputs, which is a matrix containing the second derivatives called the Hessian matrix.
- **Hessian**: Matrix of partial second-order derivatives for multiple input variables of an objective function.

## Those formulas relate to BFGS
$d_k=-B_kg_k$ (Quasi-Newton)
$\alpha=\frac{1}{y_k^Ts_k}$
$\beta=-\frac{1}{s^TB_ks_k}$
$u=y_k$
$v=B_ks_k$

### Hessian approximation
$B_{k+1}=B_k+U_k+V_k$
$B_{k+1}=B_k+\alpha uu^T+\beta vv^T$
$B_{k+1}=B_k+(y_ky^T_k)/(y_k^Ts_k)-(B_ks_ks_k^TB_k^T)/(s_k^TB_ks_k)$ (BFGS)

- $B_k$=>approximation Hessian matrix in *k* steps
- $y_k$=> $df(x_{k+1}) - df(x_k)$
- $s_k$=> $x_{k+1} - x_k$
-
## Image filter techniques
### deep learning
Advantage - less time needed
### deep image prior
- time consuming
### classical

## Image deblurring
### Definition
Image deblurring is a type of image restoration that focuses on restoring clean images by removing distortions.

### Algorithms
There are numerous image deblurring algorithms, from classic methods that mathematically estimate the blur kernel and then reverse its effect; to more recent machine learning based methods that benefit from recent advances in machine learning and deep learning.

#### Image Deblurring techniques
- multi scale architectures
- multi-stage progressive image restoration network(MPRNet in CNN)
- prior-based blind deblurring

### Why image deblurring so difficult to solve

- A lot of knowledge in deep learning field is necessary
- A lot of data was needed??

[Image Deblurring](https://saiwa.ai/blog/image-deblurring-1/#What_is_image_deblurring)
[Deblurring and denoising](http://fph.altervista.org/dida/dott19/l4.pdf)

## Image denoising
### Definition
Image denoising is a function to remove noise from a noisy image, so as to restore the true image.

Noise: Random variation in brightness or colour information and it is frequently produced by technical limits of the image collection sensor or by improper environmental circumstances.

### techniques for image denoising
- DnCNN
- Spatial Filtering
- Transform Domain Filtering
#### Spatial Filtering
A traditional way to remove noise from image data is to employ spatial filters. Spatial filtering is the
method of choice in situations when only additive noise is present.
#### Transform Domain Filtering
The transform domain filtering can be divided according to choice of basic functions. They mainly classified in to the non-data adaptive transform and data adaptive transform.

[Image Denoising techniques](https://www.google.com/url?sa=t&source=web&rct=j&url=https://www.iosrjournals.org/iosr-jece/papers/Vol.%252011%2520Issue%25201/Version-1/L011117884.pdf&ved=2ahUKEwjRmIP1hKj8AhUH9zgGHezbBIkQFnoECAgQBg&usg=AOvVaw0dc0Vz3grpNmZVricxiR3m)
### Why image denoising so difficult to solve
- Denoising an image is a difficult task since the noise is tied to the image’s high-frequency content, such as the details. The denoised images could inevitably lose some details when the image was denoising. Hence, our goal is to strike a balance between suppressing noise as much as possible while not losing too much information.

[Different Types Noises and Image Denoising Methods](https://analyticsindiamag.com/a-guide-to-different-types-of-noises-and-image-denoising-methods/#:~:text=Denoising%20an%20image%20is%20a,not%20losing%20too%20much%20information.)

# Optimisation Theory
# Gradient Method
$d_k+1=d_k+alpha_kd_k$

$alpha_k$-> steplength
$d_k$->direction

## Steepest descent
$d_k= -g_k$
Disadvantage: When more closing to the goal, more frequently to change direction.

## Conjugate gradient
Flexure reff – 1st non linear in the world
$d_k=-g_k+\beta_k*d_{k-1}$
$beta_k$ ->scala(constant) ->adjust direction
[Fletcher reeves source](http://www.mymathlib.com/optimization/nonlinear/unconstrained/fletcher_reeves.html)

## Newton
$d_k=-H^{-1} g_k$
$H_x$ -> Hessian matrix (to determine the curvature of function)

## Quasi-Newton (FAMILY)
BFGS
- Disadvantages
- costly
- time consuming
- approximate full rank of $H_k^{-1}$

[BFGS source](https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm#Algorithm)

## Spectral gradient
Start from 1988
- Barzilai Borwein method(BB method)
- 2018 (Sim)
- Spectral gradient method
- approximate eigenvalue of the actual Hessian

Disadvantage for Spectral gradient: not accurate compare to the Quasi-Newton

## Steplength
$\alpha_k$
Method:
- Armijo line search
- Strong Wolfe

### Question
<hr/>

References here ...

# Dr Goh
## Potential vs Force field

The relationship between potential funtion $U(x,y,z)$ and the force field generated by $f$ is
$$\vec{F} = \nabla U.$$

One type of force field that is interested in physics is conservative force field. Roughly speaking, it is related to the principle of conservation of energy. A conservative force field is

- work done $W = \int \vec{F}\cdot d\vec{r}$ is path independent.
- $\vec{F}$ can be written as a $\nabla U$.

Mor info see
https://tutorial.math.lamar.edu/Classes/CalcIII/ConservativeVectorField.aspx

## Curl and Div

see https://tutorial.math.lamar.edu/Classes/CalcIII/CurlDivergence.aspx

## Refresher on Line Integral
https://tutorial.math.lamar.edu/Classes/CalcIII/LineIntegralsVectorFields.aspx

22/12/2022
- Learned about optimization, differences between search method and gradient method
- Learned optimization’s coding in python language.
- Homework: done Conjugate Gradient python code.

26/12/2022
- Understand Convolutional Neural Network (CNN)
- Understand how CNN relate to the Optimization

3/1/2023
- Present the Conjugate Gradient python code and BFGS python code.
- Understand image deblurring and denoising in CNN
- Understand the algorithm and theory of CG and BFGS

9/1/2023
- Present about DFP and CNN
- Homework: Do a part of Introduction in PD

16/1/2023
- Discuss introduction of proposal defence.
- Set a path on my Literature review. (Give some advises)

1/2/2023
- Discuss about problem statement and objective
- Homework: do objective and research methodology

8/2/2023
- Research methodology has problem, redo it
- Do ppt

15/2/2023
- present about research methodology and learning rate on literature review.
- debug python code.

23/2/2023
- draft the presentation slide.
- do research on armijo line search & Spectral gradient method (Barzilai-Borwein method).

1/3/2023
- present the research on armijo line search & Barzilai-Browein (BB) gradient method.
- Set a path/direction for preliminary result.

6/3/2023
- present the Physics-Inform Neural Networks(PINNs)
- Discuss and correct the python code of preliminary result.

23/3/2023
- show and present the priliminary result.
- Discuss and optimize the python code of priliminary result.
- Discuss and correct the presentation slide of Proposal Defence.

29/3/2023
- discuss about priliminary result's code.
- present handwriting BFGS algorithm.
- show Proposal Defence report.

5/4/2023
- Discuss and do correction on proposal defence report.
- prepare intent form of proposal defence.

10/4/2023
- Discuss and receive some suggestion for presentation slides of Proposal Defence.
- Homework: optimize presentation slides and prepare preliminary result (if can).

19/4/2023
- Discuss about detail of proposal defence.
- Prepare for mock presentation.

26/4/2023
- mock presentation.

2/5/2023
- Proposal Defence. (2pm-3pm)

12/5/2023
- do correction of proposal defence report.
- modify according the comments from proposal defence's panels.

20/5/2023
- brief about the correction of proposal defense report.
- remind supervisor and co-supervisors sign for the proposal defense's correction form.
- Homework: hand calculate for the steepest descent of the multiple variable function.

23/5/2023
-discuss about hand calculate result and recorrect it.
- Homework: do three methods of iteration method in optimization with multiple variable function.

8/6/2023
-show and present about the result of hand calculation.
-do research on the multiple damping gradient method.

16/6/2023
-show the code of multiple damping gradient method.
- Do some correction on the multiple damping gradient's python code.

22/6/2023
-complete more than 20 tested problem and applied in the optimization method.
-create an updating formula based on multiple damping gradient method.

30/6/2023
- discuss about the updating formula and modify it.

14/7/2023
- show the result of the updating formula of spectral gradient method.
- Homework: create a table and plot a line graph of those results.

21/7/2023
- show the graph result of the updating formula.
- work more tested problems.

28/7/2023
- discuss about the modified updating formula's graph result.
- learn how to create profiling graph.

11/8/2023
- show profiling graph until TP18 (DIXAANB).
- start construct abstract of the updating formula.
- prepare for presentation on 8/11/2023.

25/8/2023
- show the progress of external abstract for colloqium.
- modify the external abstract.
- gain more knowledge for optimization and improve presentation skill.

8/9/2023
- Discuss and modify external abstract.
- review some neural network for application of updating formula.
- homework: construct a bulletin for optimization.

25/9/2023
- discuss bulletin.
- discuss neural network that need to apply.
- homework: how optimization affect neural network system?

6/10/2023
- report numerical result of the updating formula of component wise adaptive learning rate.
- discuss and teach student neural network.
- homework: Study and build neural network, modify the algorithm of gradient methods, prepare presentation slides for colloqium.

20/10/2023
- show presentation slides for colloqium and have a rehearsal.
- discuss and modify the draft of conference paper.
- learn about build a neural network by using custom optimizer in tensorflow.

03/11/2023
- discuss and modify presentation slides.
- discuss stopping criteria of the proposed algorithm.

10/11/2023
- difference between Newton method and Newton-Raphson method.
- apply the updating formula in the inverse of image processing.

14/11/2023
- learn about image processing and inverse image processing.
- Homework: write a python code for inverse image processing by using linear algebra and optimization method.

17/11/23
- discuss title of the coference paper.
- modify abstract.
- learn and discuss how to apply the proposed method to image processing.

24/11/23
- discuss modified abstract, thus submit to ICMSA 2024.
- show the results of image deblurring.
- Homework: measure the similarity between ground truth image and restored image.

1/12/23
- discuss and modify the result of image processing by using iteration method in optimization.
- show the similarity of image between gauth truth and restore image.

7/12/23
- fix the error of the result of tested problems and image deblurring.
- discuss the result of the conference paper.

12/1/24
- discuss the image deblurring result.
- discuss the profiling graph linestyle.
- modify proceeding paper.

15/1/24
- discuss and modify the proceeding paper.

23/1/24
- find and fix the problem of deblurring image.
- learn how to do convergence analysis.
- start write section 1 and 2 of thesis paper.

2/2/24
- discuss convergence analysis of the proposed method.
- hoemwork: continue to learn more about convergence analysis.

1/3/24
- discuss and revise ICMSA 2024 proceeding paper.

8/3/24
- revise ICMSA proceeding paper.

22/3/24
- prepare the presentation slide for ICMSA 2024 proceeding paper.
- Discuss convergence analysis of our proposed method.

12/4/24
- check and modify presentation slide for ICMSA 2024.
- discuss convergence analysis of our proposed method.

19/4/24
- discuss and modify the proceeding paper.

26/4/24
- mock presentation for ICMSA 2024.
- fix the correction for the presentation slide.

3/5/24
- discuss and modify convergence analysis of the proposed method (DVM).

10/5/24
- discuss and modify convergence analysis of the proposed method (DVM).

31/5/24
- discuss and modify convergence analysis.
- planning future work.

4/6/24
- discuss the last part of convergence analysis.

21/6/24
- discuss work completion's summary, and the date of presentation.
- modify the summary of work completion.

28/6/2024
- prepare for work completion.
- check and modify the summary for work completion.

5/7/2024
- do some correction on theorem 1 in convergence analysis.
- start prepare the presentation slide for WCS.

12/7/2024
- check and modify the presentation slide of WCS.

19/7/2024
- check and modify the presentation slide of WCS.
- homework: continue work on thesis.

26/7/2024
- prepare for WCS.
- homework: continue work on thesis.

7/8/2024
- Work Completion Seminar.
QnA question:
- Computational time must include in image deblurring.
- explain the profiling graph (know how to construct the graph)
- log-determinant norm

9/8/2024
- discuss the correction need to do after WCS.
- continue to complete the thesis.

13/9/2024
- discuss compressed sensing.
- check and submit colloqium.
- set the date to submit intent form of thesis.

14/10/2024
- discuss compressed sensing.
- modify compressed sensing's code.

8/11/2024
-