###### tags: `projects`

# ML4ML

## Datasets

See [spreadsheet](https://docs.google.com/spreadsheets/d/1DtYB-O0CjoPQcxQq_N6kVgeHV91Hmz9ac1HlhfX95og/edit#gid=1836493523).

## Training, Validation, Testing

Currently on full brains. However, it might be useful to have densely traced blocks:

1. Randomly select $k$ points in volume that are within $\theta$ threshold to any consensus skeleton.
Randomly in different neuron parts (distal, middle, proximal to nucleus).
2. Crop box around those points to get $k$ validation ROIs.
3. Trace densely in those ROIs in the validation and testing brain.
4. Evaluate reconstruction accuracy peridocically (during training) on those validation ROIs.

Splits are documented in the [spreadsheet](https://docs.google.com/spreadsheets/d/1DtYB-O0CjoPQcxQq_N6kVgeHV91Hmz9ac1HlhfX95og/edit#gid=1836493523).

### Benchmark Dataset (to send to JHU, 2020-01-15)

We are currently creating a benchmark dataset, which will allow us to get more reliable numbers on the quality of our methods, needed both during method development and to compare different approaches.

In particular, we randomly selected 10 cubes with a side-length of 100μm: 5 cubes from brain 2018-10-01 (validation) and 5 cubes from 2018-08-01 (testing). In the coming weeks, we will have tracers densely annotate those cubes, i.e., every neurite will be traced as close as possible to the signal within each cube. The dense reconstruction is needed in order to get reliable numbers on false positive detections. We will make those cubes available to you as soon as the tracing is finished, together with evaluation scripts measuring two aspects of the predictions: (1) foreground-background accuracy and (2) topological correctness of the final skeleton reconstruction.

In summary, we are using brains 2018-07-02 and 2018-12-01 for training, 2018-10-01 for validation, and for 2018-08-01 testing. We suggest that you use only 2018-07-02 and 2018-12-01 to train machine learning algorithms, 2018-10-01 to tune hyper-parameters, and keep 2018-08-01 to report final numbers.

## Experiments

Setups documented in this [spreadsheet](https://docs.google.com/spreadsheets/d/1avnwhgIKB_kpUzJ2oyjnYciB61OCyubUFtKc4fwrQoQ/edit#gid=0).

### Foreground Prediction Networks

1. Train distance transform and binary mask
2. Evaluate on validation dataset:
1. Find local maxima
2. Compute error metrics:
1. Scores in prediction of matched skeleton nodes (higher is better, best 1).
2. Ratio of TP/FP, where a TP is a found maximum within threshold distance to skeleton (higher is better). Ideally on fully traced validation ROIs. Until we have them, evaluate only within area used for training (where gradients were allowed to be non-zero).

## Deployment

### Foreground Prediction

Would be great to pass the FG predictions already on to the tracers:
1. This should be softmask network.
2. Radious of mask around skeleton should be smaller than `test13`.