# Diffusion + Deblurring for Medical Images

## Motive
動機：幫醫生解決醫學模糊影像問題，協助醫生診斷（diagnosis），因為現在大部分的設備有可能是白光干擾等

## Metrics

- SSIM - Structural Similarity
- PSNR - Peak Signal to Noise Ratio

## Datasets

- HyperKVASIR
- Barrett's Esophagus(?)
- BRATS
- CheXpert

## Approaches

- Deblurring
- Diffusion-based
- Deep Learning-based (GAN)

## Compare
- Traditional deblurring
- Deep-learning based (GAN)
- Diffusion based

## Papers read
- MedDeblur
- https://www.mdpi.com/2227-7390/11/1/115
- Ambiguous Medical Image Segmentation using Diffusion Models
- https://youtu.be/3gVn0-IFOKs
- https://arxiv.org/pdf/2304.04745.pdf
- GibbsDDRM
- https://arxiv.org/pdf/2301.12686v2.pdf
- Medical Image Enhancement and Deblurring
- https://www.researchgate.net/publication/342784383_Medical_Image_Enhancement_and_Deblurring
- Wavelet Transformation
- https://link.springer.com/chapter/10.1007/978-981-15-5281-6_38
- https://innovation.ox.ac.uk/licence-details/post-processing-deblurring-for-medical-imaging/
- https://pubmed.ncbi.nlm.nih.gov/17282590/
- GAN
- https://www.jsjkx.com/EN/10.11896/jsjkx.200600144
- Multi-scale CNN
- https://arxiv.org/pdf/1612.02177v2.pdf
- DDS2M: Self-Supervised Denoising Diffusion Spatio-Spectral Model for Hyperspectral Image Restoration
- Diffusion Models for Medical Anomaly Detection
- https://arxiv.org/abs/2203.04306
- pre-trained: https://www.dropbox.com/sh/up4sqmeqrlwrxkw/AACqgnSCR8p20r6aO_3_Jy94a?dl=0

- [MedSegDiff:MedicalImageSegmentationwith Diffusion Probabilistic Model](https://arxiv.org/pdf/2211.00611.pdf)
- github:https://github.com/WuJunde/MedSegDiff

## Current Research Direction
- Reduction of amount of data for training diffusion model
- Make model parameters learnable (AutoML or similar)

Diffusion Anomaly

* https://arxiv.org/abs/2305.12966
* https://arxiv.org/abs/2207.11192
* https://arxiv.org/abs/2212.01789
* https://ieeexplore.ieee.org/document/6643647
* https://www.academia.edu/3417729/Image_denoising_and_deblurring_non_convex_regularization_inverse_diffusion_and_shock_filter

## Compressing papers

- List all references, year, origin, impact factor, and a short summary
- Create a one page highlight for each paper
- Put diagrams in said highlight

Goal:
- Deblurring for medical images, specifically chest x-ray
- Improving the methodology instead of data

What to test for
- Accuracy and consistency of generated images
- Metrics: Structural Similarity, Peak Signal to Noise Ratio
- Will diffusion-based models perform better compared to other generative methods based on the above metrics?

Application:
- Assisting doctors with anomaly detection on X-Ray images
- Challenge: what to solve
- Methodology
- What is applicable to my topic?

## To do for this week

- Compile papers and references

## Skeleton for thesis

### Introduction

### Background

- Medical diagnoses
- Diffusion models for medical imaging
- yadda yadda yadda

### Related Works

- U-Net
- Diffusion Models
- Denoising Diffusion Models
- Anomaly detection

### Proposed Methodology

- Details about CheXpert
- Preprocessing methods
- Training process
- Any gotchas you had to handle (imbalance, etc.)
- Metrics

### Experiment Results

### Conclusions

## 2024/05/22

- Metrics for evaluating models: PSNR, SSIM, F1/Dice, Jacard

## 2024/05/26

Important considerations for diffusion anomaly detection
1) normal images are reasonably "normal"
2) anomalous regions are indeed anomalous

* VinDr has labeled anomalous regions for confusion matrix-related metrics (sensitivity and specificity)
* Pleural Effusion is more easily noticeable, this lung anomaly is a good starting point for a detection model
* Then train on other lung conditions

## 2024/10/17

- Heatmaps remain the primary accurate indicator of diseased regions, bounding boxes are dependent upon quality of masks generated from heatmaps

### Review
- Basic idea behind Fourier Transform: Any _periodic_ function can be expressed as a weighted sum of sines and cosines
- Higher Frequency regions contribute to finer details, but are prone to containing noise

## 2024/11/2

- Components of Diffusion Model code
- Noise scheduler (responsible for incrementally adding noise to the image)
- Timestamp embedding (positional encoding attached to input because UNet has no concept of time)
- Model architecture

To add diffusion to a model architecture, you must include a time embedding (usually using positional encoding, into the image data as a separate channel)

* We read other papers to figure out what we need to use for our own experiment and methods
* High frequency contains a lot of detail, but can contain a lot of noise
* Super resolution is a possible approach, but higher memory use will necessitate using float16
* Increasing contrast is (possibly?) another form of increasing high frequency component strength

* Look for methods utilizing Super Resolution for medical imaging, see if they also utilize deblurring
* Specificity, Recall

* Methods of increasing high frequency components: CLAHE, Deblurring, Super Resolution
* What you read from other papers goes into related works
* Find what other papers lack, and use that as your starting point
* Use ChatGPT to create a starting framework for your model
* Goal: Increase high frequency content while minimizing any resulting noise

* Methodology: Deblur + High Frequency (while keeping noise minimal) + SR
* Comparison: Objective (Metrics like PSNR, SSIM, FID, Dice/F1) and Subjective (Doctor's evaluation)
* Deadline: 2 months (\'til end of December)
* You don't have to necessarily get the author's code running if it doesn't work, take their logic and adapt it to your code

- Agenda for 2024/11/2 - 2024/11/9:
- Read papers on super resolution, and image high frequency component manipulation
- Take notes on their contributions, and other solutions they propose to the problem
- Compile notes into a short report
- Create a Python framework for models:
- Must allow straightforward insertion of various model architectures
- Loading, saving checkpoints, metric evaluation must be applicable to all models
- Hyperparameters must have a uniform interface for adaptability to many models (Dictionary?)
- Must allow straightforward visualization (or at least allow the user to add tailored visualization code easily to a module which can be called from the main script)
- Framework building
- Implement dataset loader
- Basic framework for UNet

- Agenda for 2024/11/11 - 2024/11/17
- Study UNet implementation
- Create working UNet implementation with timestep embedding
- Study noise schedulers for diffusion model
- Read papers on contrast improvement for chest x-ray images

- Agenda for 2024/11/18 - 2024/11/24
- Re-run GAN experiment results
- Prioritise finishing model training and result inference
- Note down metrics
- Notes:
- Take any common objective metric measurements from other papers used
- Methods to be compared:
- GAN (Fixed Point)
- DDIM
- DDIM + Deblur
- DDIM + Denoise
- DDIM + Deblur + Contrast/High-Freq Adjustment
- DDIM + Deblur + Contrast/High-Freq Adjustment + SR

- Agenda for 2024/11/25 - 2024/12/01
- Continue action items from last week
- Completed so far:
- Dataset class for chexpert for framework

- Agenda for 2024/12/9 - 2024/12/015
- Moving Average Sampling in Frequency Domain
- Implement Discrete Wavelet Transform (completed, python library available)
- How is the moving average calculated?
- Study guided diffusion framework
- Use ChatGPT for some assistance when needed

- GAN needs large quantities of data, while generation performance is not on par with diffusion models
- Limitations of datasets
- some datasets do not have detailed annotations (i.e. segmentation masks)

- Agenda for 2024/12/30 - 2025/01/05
- Train super resolution based on implementation provided by OpenAI's guided-diffusion repo
- Add chaining to anomaly detector model from SR output to DDIM input
- Research deblurring/denoising methods for x-ray images

- Agenda for 2025/01/13 - 2025/01/20
- SR 2x + normal DDIM generation results (done)

- 2-3 pages for conference
- Simplified content compared to thesis, should mostly cover related works, methodology
- Treat it like a proposal
- Create diagrams for your model architecture

## Today's discussion (2025/01/19)

- When discussing your experiment, place your best results in the rear
- Include medical expert's feedback, "Through some internal discussion, ..."
- Include metrics, such as PSNR with accuracy, SSIM, etc.
- Rewrite formulas, draw diagrams for your model architecture (ideally include intermediate outputs in your diagrams)
- Refer to Lina's or Chiuzu's paper for what a clear diagram should look like
- Why did you use deblurring? Factor in how noise in images (esp. high frequency) can affect image reproduction through generative models, such as diffusion models
- Remember details and your viewpoints, along with expert feedback, i.e. Why did SR not perform expectations? Why does deblurring generate better results, compared to denoising?
- Deadline: End of February

- Include engineer and domain export viewpoints
- Any questions? JUST ASK

## February goal

- Finish writing conference paper (2-3 pages long)
- Summarize findings with DDIM img2img translation
- Why does deblur improve the quality of translated images?

### First week goal (02/01 - 02/07)

- Implement classifier-free guidance for Guided Diffusion model
- Re-read papers on diffusion models
- Make use of Claude / you.com / ChatGPT Pro for interpreting math formulas

## March 1 - 7 Goals

- Generate deblurred images for entire Chexpert dataset for training deblurred model

## March 17-23

- Gather results for GAN, and DDIM with added denoising process (not deblur, denoise)
- Adversarial Autoencoder (?)
- Collect as much data as possible (easier to discard than to reacquire)

# Thesis structuring

- Introduction
- Background
- Pleural effusion
- Diagnosis and treatment
- Need for computer-assisted diagnosis
- Related works
- Computer aided detection/segmentation
- Generative models
- Early methods (VAE)
- GANs for Medical Image Segmentation
- Diffusion Models for Medical Image Segmentation
- Methodology
- Data Collection
- CheXpert
- VinDR dataset
- Data Preparation
- Label grouping
- DDIM Overview
- UNet Architecture
- Experiment
- Setup
- Results
- Performance/Quantitative Analysis
- Visual/Qualitative Analysis
- Discussion
- Conclusion
- Challenges and Limitations
- Future Work
- References

## May 3 notes

- Raise PSNR and SSIM (Overall quality)
- mIoU
- Train CFG with same parameters as original DDIM with classifier
- Theory: Drawbacks of GANs, problems addressed by DDIMs

- Paper reading:
- Pleural effusion: detection, treatment
- Models: VAE for localization, GAN for localization (Why aren't GANs used now), Diffusion Models for localization

## Paper Groups

### Diffusion Models in Medical Imaging
#### Today's reading
- ~~Denoising Diffusion Probabilistic Models (2020)~~
- ~~Denoising Diffusion Implicit Models (2020)~~
- Diffusion Models Beat GANs on Image Synthesis (2021)
- Diffusion Models for Medical Anomaly Detection (2022)
- What is Healthy? Generative Counterfactual Diffusion for Lesion Localization (2022)
- AnoFPDM: Anomaly Segmentation with Forward Process of Diffusion Models for Brain MRI (2024)
- Denoising Diffusion Models for Anomaly Localization in Medical Images (2024)

Motivation:
- Medical data tends to be harder to obtain in large quantities due to ethical and legal concerns, as well as the fragmented nature of medical databases
-

My contributions:
- The model is able to output good quality images with smaller amounts of training data
-

Action item:
- Create model architecture diagrams
- Flesh out related works section: "I propose classifier free guided DDIM for anomaly detection"
- Finish Section 3 (Methodology), Section 4 (Experiment Results)

Meeting results:
- Possible data contamination (pleural effusions occurring in tandem with other conditions)
- DDIM being overly sensitive in identifying pathological regions
- Analyse correlation between different pathologies
- Equivalence between different models to avoid apples vs oranges problem

Action item:
- Run with data provided from Dr. Yuan
- Test model with purer data (pleural effusion only, or with only a few other conditions)
- Use VinDR data for inference
- FPGAN: Increase parameter count to above 100 million (around the 110 million mark)

Action items for Friday:
- Complete diagram for CFG architecture, obtain example intermediate outputs for diagram from model
- Rewrite short paper to fit in one page

Action items for Sunday/Monday
- Read through Denoising Diffusion Model for Plug and Play Image Restoration
- ~~Write part of the experiment results section~~

Action items for Thursday 6/19
- Introduction + Background (low priority)
- ~~Polish up writing~~

- Related works (low priority)
- Elaborate more on GAN applications
- Elaborate more on computer aided diagnosis

- Methodology section
- Diffusion process:
- ~~Double check and complete math formulas~~
- ~~Classifier-free guidance section~~
- Backbone model architecture
- ~~Include explanation for attention block~~
- Add more details on each U-Net module
- Experiment section
- Qualitative analysis
- ~~Break down doctor/radiologist opinion~~
- Discussion
- Interpret results once previous sections are finished

Action Items for Saturday 6/21
- Finish writing the following sections
- Discussion (Experiments)
- Challenges (Conclusion)
- Complete diagrams and images for qualitative analysis section, taking into account doctor's opinion

Important details for thesis defense application

- ~~Absolute final deadline for application is July 15, for a defense date around the end of July. Once you pass defense, you may remain during August without worry of being expelled. All school departure procedures must be completed by the end of August.~~ End of June is your deadline, period.

- Personal deadline: ~~Thesis by Wednesday, PPT by Friday.~~ THIS WEEK

Action items for today
- ~~Run model for different noise levels and guidance scales~~

Friday Action Items
- ~~Rewrite background and related works sections~~
- ~~Background/Motivation should focus on lack of clean annotations, the greater abundance of unannotated data
- ~~Read on medical image annotation, weakly supervised methods on unlabelled data~~
- ~~Segue into autoencoders for medical image data, then GANs, THEN diffusion models~~
- ~~Read through papers in the "Medical Image Annotation" collection and "Related to DMfMAD"~~

~~- Once results are obtained for other hyperparameters, make graphs showing effects of hyperparameter changes on scores (reconstruction, predictive)
- Move all npz files to server with pleural effusion masks
- Run evaluation metrics, group for noise level change, and for guidance scale change~~

Defense Checklist:
1. 碩士學位考試申請表 (交註冊組)
2. 推薦書一份 (口試當天再簽)
3. 審定書一份 (口試當天再簽)
4. 口試評分表 (每位口試委員各一份)
5. 碩士班口試平均成績登記表
6. 口試委員聘函 (請自行寄給口試委員)
7. 文章剽竊檢測工具Turnitin之原創性比對報告一份 (口試當天交指導教授簽名)

Thursday Action Items
- ~~Rewrite stage 1 on deblurring~~ DONE
- ~~Rewrite diffusion model exposition in Methodology chapter~~
- ~~Remember the context of your project, it must be systematic~~

Saturday Action Items
- ~~Go through U-Net section, rewrite if necessary, make sure it connects to the rest of the thesis and your framework~~

Sunday Action Items
- ~~Ready LabelImg usage instructions~~
- ~~Rewrite U-Net section, adding more details about input and output shape, use Chiuzu's paper as a reference in formatting and style~~
- ~~Rewrite U-Net components (contractive, bottleneck, expansive)~~
- ~~Rewrite Residual Block, Attention Block, Embeddings~~

Tuesday Action Items
- ~~Write python script to convert bounding boxes to mask for mIoU calculation~~
- ~~Rewatch meeting recording, note down expert feedback~~
- ~~Register for defense on July 24~~

Wednesday Action Items
- Revisit Experiments section
- ~~Simplify dataset explanation~~
- ~~Rewrite Challenges and Limitations section in Conclusion~~

Thursday Action Items
- ~~Detail deblur model training procedure~~
- Mention about discussions with the medical expert on the need for privacy through the use of edge models
- Apply the above content into Introduction and Background
- ~~Limitation: Due to comptutational constraints, we run deblurring and the conditional editing models separately.~~
- https://arxiv.org/html/2503.06537v1 (For primer on traditional deblurring methods)

Friday Action Items
- ~~Finish rewriting Dataset~~
- ~~Finish rewriting Evaluation Metrics~~
- ~~Finish writing Objective Analysis of Results~~

Saturday Action Items
- ~~Prepare figures for visual analysis~~
- ~~Create separate figures for each model, the grid for each model should have input, output, heatmap, and mask overlay.~~
- Retain figures that highlight extraneous features.
- ~~Remove CheXlocalize as we are no longer using CheXpert results~~
- ~~Write Discussions section~~
- ~~Detail specifics about how the small input image size limited the impact deblurring might have~~
- ~~Strengthening conditioning through other means~~
- ~~Add more to background and motivation chapter~~
- ~~On-premise ML models for local processing of image data~~
- ~~Add summarized version to Introduction and Abstract~~

# FREE AT LAST, FREE AT LAST, THANK GOD ALMIGHTY I AM FREE AT LAST
Revision action items:
- ~~Use figures in abstract to highlight model strengths~~
- ~~Rewrite segments to better describe your methodology, and reflect your slides~~
- ~~Add AUROC plot to drive the point home~~
- ~~Improve explanation for why CheXpert results were bad, highlight focal and diffuse pathologies in your explanation~~
- ~~Rewrite related works to make paper explanation clearer, reflecting your slides~~

Other items:
- Tidy up code and documentation, figure generation scripts (make sure ALL figures are reproducible), no need to include DiffPIR
- Refactor evaluation and figure generation scripts, ideally convert to class
- Make sure it reproduces your figures consistently
- Arrange files and folders like your senior's files for final project
- ~~PPT~~
- ~~Recordings~~
- ~~Thesis~~
- ~~Turnitin report~~
- ~~Source code (just git clone, my guy)~~
- ~~Get Turnitin similarity report after revision~~
- ~~Get template for IEEE on Saturday, copy contents into IEEE template~~

孫宏民
https://drive.google.com/drive/u/1/folders/1atMXS3z1dnSAud39qDaRsTipFYsnhSCS

許富皓
https://drive.google.com/drive/u/1/folders/1pZYmDP1wyYpynmpkywd0YKz5B-M1gHdB

黃育綸
https://drive.google.com/drive/u/1/folders/1VqBN_PQsL-vGRwCEiBO72sY54pYOKDJQ

[Thesis Defense Recording](https://youtu.be/XLEFNym1s40)