# Using Mask R-CNN for predicting budding spots in *S. cerevisiae*

Marit Verheij 4791517, Matthijs Westra 5886058, Minouk Noordsij 4788338, Ryan Cui 5114640

## Introduction

<!--budding yeast, data where she wants to segment the budding spot and we use the method from the paper which segments nuclei so we have to adjust the model because the budding spot have a different shape -->

In this blog post, we embark on an in-depth exploration of the identification process of budding spots in _Saccharomyces cerevisiae_, better known as budding yeast, utilizing the methods of the paper _An Ensemble Learning and Slice Fusion Strategy for Three-Dimensional Nuclei Instance Segmentation_ by Wu et al [1]. The goal is to be able to find the find the budding spot on the budding yeast and to obtain a mask for the spot. The method that has been used to reproduce this is Mask R-CNN.
<!-- In this paper we will show and discuss... -->

## Biological background

Eukaryotic cells divide by the processes of mitosis, duplication of the nucleus, and cytokinesis, division of the cell. Between the two poles of the cell, the mitotic spindle is formed, consisting of microtubules. The mitotic spindle specifies where the cell divides, which is usually the plane of the center of the cell. This process is called a symmetric division and results in two daughter cells of equal size [2].

_Saccharomyces cerevisiae_, better known as budding yeast, is eukaryotic, but has an asymmetric cell division, making it an interesting model organism to study asymmetric cell division in eukaryotes [3]. An important part of this type of division is cell polarization: the (assymmetric) organization of components in a cell into different regions [4]. Before division, budding yeast cells have polarized growth towards a cortical site on the cell. This causes a bud to emerge, which will grow until eventually the parent cell and the bud divide into separate cells [5].

One molecule in particular that is important for this budding, is the small GTPase Cdc42. Due to temporal and spatial signals in the cell, Cdc42 polarizes at a cortical site, which starts the bud growing process [4].

By labelling Cdc42 with a fluorescent dye in _Saccharomyces cerevisiae_, the emergence of the bud can be imaged using fluorescence microscopy, resulting in images such as in figure 1. Within image analysis, a main problem is to be able to differentiate a certain object of interest from its background. There are some heuristic methods to do this, like thresholding, but these methods do not always result in reliable results. In recent times machine learing and deep learning methods have proven to be able to solve this problem. One such method is Mask R-CNN.


_**Figure 1**: Fluorescent microscopy images from a_ Saccharomyces cerevisiae _cell, with Cdc42 labelled._

## Data

<!-- Need to check time between frames -->
The data we have received from Marieke Glazenburg, are timelapses of 3D images of individual _Saccharomyces cerevisiae_ cells, where Cdc42 is labelled. These timelapses are in the form of one TIF file per cell. In these timelapses, cells go from an unpolarized state to a polarized state. In the timelapses, every frame is taken 20 seconds apart, there are 55 timesteps in total, and each timepoint consists of 57 z-stacks: 2D images that make up the 3D image _(fig. 2, left)_. Next to this, we also received manually segmented masks for the cells to use as training data _(fig. 2, center and right)_.


_**Figure 2**: Left: Schematic drawing of a_ Saccharomyces cerevisiae _cell. Each ring indicates one z-stack, and green indicates the location of Cdc42 polarization. Center: one z-stack of a_ Saccharomyces cerevisiae _cell, with Cdc42 labelled in white. Right: manually segmented mask for the same cell as seen in center image._

## Method: Mask R-CNN
The method that has been used by the paper is: EMR-CNN, which stands for Ensemble Mask R-CNN. This is a technique that is used for object detection in 3D volumes. It operates by dividing the 3D volume into 2D slices and applying an ensemble of 2D object detectors to each slice.
Each detector in the ensemble generates a set of detection results for each slice, which includes a segmentation mask, an object centroid, and a confidence score. These detection results are then fused together using an ensemble 2D fusion method to create a set of fused 2D detection results.
This approach allows the EMR-CNN to leverage the strengths of multiple 2D object detectors to achieve robust and accurate object detection in 3D volumes.

Since the provided code does not align closely with our intended objectives that is proposed by the external supervisor, an alternative method sourced from a different origin is being chosen and used. This source was in the reference of the paper that the paper has used for training.

Therefore, the platform that has been chosen is Detectron2 [6]. Detectron2 is an open-source platform for object detection and segmentation, that is developed by Facebook AI Research (FAIR). This is a second-generation platform that is rewriten from the original Detectron adding several enhancements and new features.
Detectron2 now includes all models from the original Detectron, such as Faster R-CNN, Mask R-CNN, RetinaNet, and DensePose and on top of that, has models like Cascade R-CNN, Panoptic FPN, and TensorMask. Detectron2 also supports a variety of tasks related to object detection, including object detection with boxes, instance segmentation masks, human pose prediction, semantic segmentation, and panoptic segmentation.

The method that has been used for the reproduction is the Mask R-CNN, which is used for 2D object detection and segmentation.

Mask R-CNN is a computer vision model that can identify objects in images and also draw precise outlines around them. It first looks at the image and proposes regions where it thinks objects might be. Hereafter, it then studies these regions to understand what the object could be and refines the box around the object. Finally, it draws an exact outline (or mask) around the object.

In our case, we have employed the Detectron2 framework [7], modifying its code to cater to our specific requirements of applying it to budding yeast. The initial phase involved the creation of a unique dataset, where in each budding spot is distinctly visible with a segmentation mask. This approach ensures that the training model can effectively recognize the various shapes and locations of the budding spots on the budding yeasts. Subsequently, the model was trained using this dataset. Hereafter, the training data has been trained and after the training tested.

<!-- How many data and stuff and maybe rewrite if necessary -->

<!-- ### Parameters
#TODO
Wat blaat over hoeveelheid images used for training

#TODO
Wat blaat over hyper parameters?

### Environment
The environment that has been used is Google Collaboration and the code is written in the programming language Python in Python Notebook.
-->

### Pipeline

In this section we will describe all steps that need to be taken to obtain our final result.

#### Converting TIF files to PNG
Mask R-CNN from Detectron2 uses PNG files to train. However, our dataset consists of TIF files. Therefore, first we need to convert our TIF files to PNG files. Originally we obtained images from 121 yeast cells, each having around 50 timesteps, with around 50 z-stacks, resulting in 121x50x50 is more than 300.000 PNG files. We decided to not use every datapoint, due to time constraints and the fact that most images are very similar to each other. Therefore we decided to train our model on around 800 PNG files, where we took entire z-stacks from around 100 cells at 4 different timepoints, making sure to only take images with a valid mask. This should not impact performance as the model can also learn from the negative space on the images.

#### Model creation
To create our model, we used a Python Notebook (ipynb file). This ipynb file and our training and test data are all in a Google Drive. By mounting the Drive we are able to run on a Google Compute Engine GPU. After mounting, Detectron2 is installed, and we convert our data into a readable format for Detectron2. Next, 10 randomly chosen annotated samples are visualized, allowing the user to ensure the data is correctly annotated. Then it is time to train Detectron2 on the data. For this we used COCO Instance Segmentation with Mask R-CNN on our training dataset.

#### Prediction of budding spots
After the creation of our model, we now have a predictor of masks for budding spots in budding yeast. Marieke indicated that she wants a model where she can input TIF files of yeast cells, and obtain TIF files for masks for these cells. For this, we created a new Python Notebook, which uses our already created model. After initializing, the last step is to actually use our prediction on data. Our code does this by predicting a mask on every slice of the TIF file, and combining these masks into a final TIF file.

## Results

### Annotated data

To ensure the training data that will be used by the model is annotated correctly our pipeline visualizes 10 randomly chosen annotated samples (pairs of microscopy and mask images) (fig. 3). Using this visualization the user can decide to continue or change their training data.


_**Figure 3:** Example of annotated sample. Left: microscopy image. Right: mask image._

### Test set

After training, our model also visualizes the generated masks on a random selection of images from our test set (fig. 4). As seen on the first three rows of figure 4, the model is able to accurately identify where the budding spot is on a yeast cell, or not detect a spot if there is not one present. The model is not always perfect however, on the bottom row is an example of a cell where there is no clear budding spot, where the model did find a budding spot. One other thing to note, is that the model generated a smaller mask for the second yeast cell than was manually segmented.





_**Figure 4:** Examples of budding spot recognition, each row is one yeast cell. Left: manually segmented mask, overlayed with generated mask. Center: generated mask on microscopy image. Right: original microscopy image._

## Verification

<!-- #TODO
Talking about the accuracy, false negative and false positives. -->

The verification of our Mask R-CNN model was conducted by analyzing its accuracy, false negatives, and false positives plots. False Negative is also known as a Type II error. It occurs when we fail to predict an event that does happen. False Positive is also known as a Type I error. It occurs when we predict an event that doesn’t actually happen.
These three aspects are shown in Figure 5.

Initially, the accuracy of the model shows a steady increase in accuracy over time can be seen, indicating a consistent improvement in its ability to correctly identify and segment budding spots.<br/>
For the False Negative, a decline can be seen until roughly 200 samples, after that it satures around 0.15.<br/>
In the Positive Negative figure, a notable decline in false positives can observed, which indicates that the model is performing well.
_**Figure 5:** Plots for the accuracy, false negatives, and false positives._

## Discussion and conclusion

<!-- #TODO
Something about that not the full paper was used
#TODO
Something about the shortcomings
#TODO
Conclusion and discussions -->

During the project, we have faced some challenges. One of the challenges faced, was the beginning where the method described by the paper was not suitable for the problem that we had. This has led to a search for a new method.
Another challenge faced during this project was the large volume of data. We had to make a decision to train our model on around 800 PNG files due to time constraints and the fact that most images were very similar to each other. Despite this, the model was able to learn effectively from the available data. The use of Google Compute Engine GPU reduced the training time and made the process more efficient.

In conclusion, the project successfully implemented a Mask R-CNN model using Detectron2 for the detection and segmentation of budding spots in budding yeast. The model was trained on a unique dataset created specifically for this project and was able to effectively recognize the various shapes and locations of the budding spots. The project also developed a pipeline for converting TIF files to PNG, creating the model, and predicting budding spots. The results demonstrated that the Mask R-CNN model was effective in detecting and segmenting budding spots in budding yeast. From the verification process, graphical analysis is shown which confirms the model’s capability to produce precise segmentation masks.

## Contributions

| Name | Contributions |
| -------- | -------- |
| Marit Verheij |Made poster for the poster presentation, wrote biological background, pipeline and results for the blog post, created verification figure|
| Minouk Noordsij |Developed data conversion script, wrote ipynb file for training the model (partly) and for running the model on tif files, contact with external supervisor|
| Matthijs Westra |Attempted setting up an environment to reproduce the original paper, wrote ipynb file for training the model (partly), trained the model|
| Ryan Cui |Wrote the introduction, method, verification and conclusion for the blogpost and tried working on the method on the in the paper which didin't work |


_**Figure 6:** Group 55 attending poster presentation_

## :book: References

1. L. Wu, A. Chen, P. Salama, K. W. Dunn and E. J. Delp, "An Ensemble Learning and Slice Fusion Strategy for Three-Dimensional Nuclei Instance Segmentation," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA, 2022, pp. 1883-1893, doi: 10.1109/CVPRW56347.2022.00205.
2. Alberts B, Johnson A, Lewis J, et al. Molecular Biology of the Cell. 4th edition. New York: Garland Science; 2002. Cytokinesis. Available from: https://www.ncbi.nlm.nih.gov/books/NBK26831/
3. Bi E, Park HO. Cell polarization and cytokinesis in budding yeast. Genetics. 2012 Jun;191(2):347-87. doi: 10.1534/genetics.111.132886. PMID: 22701052; PMCID: PMC3374305.
4. Vicente-Manzanares M, Sánchez-Madrid F. Cell polarization: a comparative cell biology and immunological view. Dev Immunol. 2000;7(2-4):51-65. doi: 10.1155/2000/70801. PMID: 11097201; PMCID: PMC2276057.
5. Bi E, Park HO. Cell polarization and cytokinesis in budding yeast. Genetics. 2012 Jun;191(2):347-87. doi: 10.1534/genetics.111.132886. PMID: 22701052; PMCID: PMC3374305.
6. Detectron2: A PyTorch-based modular object detection library https://ai.meta.com/blog/-detectron2-a-pytorch-based-modular-object-detection-library-/
7. Detectron2 https://github.com/facebookresearch/detectron2?tab=readme-ov-file