# Object fingerprinting and tracking

# Attendees
* Scott Hosking
* Ben Evans
* Andrew Fleming
* Ellie Bowler
* Krisina Ulicna
* Evangeline Corcoran
* Oliver Strickson ([e-mail])
* Marjan Famili
* Nathan Day ([e-mail])
* Alan Lowe
* Bea Costa Gomes

## Agenda for Workshop - Tuesday 19th July 2022, BAS, Cambridge

10:30 - 10:45 Arrivals and coffee (~~Icebreaker~~)

10:45 - 10.50 Introduction

10:50 - 12:00 Use-case presentations (Meeting room 4)
* Brief (5-10 minute) presentations on each proposed use-case and its idiosyncrasies from a tracking perspective (please bring 3-4 slides if you have a use-case)
* Complete pro-forma assessing similarities/differences/technical challenges

12:00 - 13:00 Lunch (~~provided, Icebreaker~~ BYO)

13:00 - 14:00 Discussion of development objectives and roadmap (~~Seminar room 1~~)
* Complete pro-forma and next steps

14:00 - 14:30 Flexible time inc. coffee if desired

14:30 Finish

## Use cases (inc. url to image if possible)

1. **Tracking of icebergs in the Southern Ocean.**
Data challenge:
Icebergs are segmented from satellite radar imagery using a Dirichlet process mixture model. Objects tend to have high solidty and range over 5 orders of magnitude in size. We have a proof of concept detection approach and want to use the segmentations to track icebergs as they drift away from their source, melt, fragment and eventually disappear. Max. observation frequency ~ every 4 days. Tracking duration = years.

Science questions:
This helps us to understand ice flux from various outlet glaciers as well as patterns of freshwater inputs to oceans, persistence of bergs once calved and also berg motion to inform hazard mapping.

2. **Dynamic plant phenotyping**
Time-series of 2D images of whole oilseed plants taken daily over the major growth period from 5 angles (4 sideview, 1 topview) with the goal being to extract information on how the abundance and distribution of different plant structures (e.g. leaves, flowers, seed pods) change as the plant grows and under exposure to different environmental or management conditions. Currently use a patch classification approach using the Turing developed MapReader framework, which facillitates measurement of the relative amount of each plant structure with potential to isolate individual structures based on patch clustering. Methods for identifying an individual seed pod or flower and tracking it over time are lacking.

3. **Medical imaging**

4. **Cancer and stem cell tracking.**
Time-lapse microscopy images of cells with heterogeneous morphology. Currently using convolutional neural networks (e.g. U-Net) to segment images, but would like to incorporate cell shape (morphology) features into `btrack` tracking update to enable tracking of heterogenous cell mixtures (e.g. cancer models), or where individual cell morphology may change over time (e.g. stem cells).

5. **Ice floe tracking**
Similar to icebergs but with different physical controls and scales.
6. **Wildlife tracking**
Time-lapse sentinel-2 imagery of walrus haulouts. Some populations move significant distances over the course of the 4 month summer season. Ideally we would recognise individual groups as they appear and dissappear from different areas of the coast. The size and shape of haulouts change from image to image as they go to feeding grounds, there are also gaps in observations due to cloud cover.

## Session 1 proforma
Attendees: Ben Evans, Scott Hosking, Andrew Fleming, Nathan Day, Marjan Famili, Kristina Ulicna, Ellen Bowler, Evangeline Corcoran, Oliver Strickson, Alan Lowe, Bea Costa Gomez

| Use case | Object attributes | Track attributes |
| -------- | -------- | -------- |
| Cells | | |
| Icebergs |large size range, geometric differences - esp. in larger icebergs, no 'states', some info possibly available on likelihood of fragmentation | motion erratic- somtimes move a long way between observations. Wind and tide driven but sea ice holds them in place sometimes |
| Sea ice | | |
| Walrus | | |
| Seeds/leaves | | |
| Cells | | |
| | | |

**Which object attributes are shared across most uses?**
Object identity is a common theme - need to represent all cases, varying certainty.

Intracellular TB - similarities with connectivity

Macrophages similar to icebergs in that there aren't necessarily 'state' changes but morphology can change dramatically. Also true of plant organs going from bud to flower to seed.

Data handling across tiled images - boundaries and edge cases. Mosic then track or track then handle lost and found objects in each scene?

Non-uniform dT - may be desirable for macrophage case... maybe a different motion model for icebergs that also handles non-uniform dT

**Which object attributes are unique to particular problems**
Large range and heavily skewed distributions of e.g. size may be unique to iceberg case

Evie's use case unique - one object, looking at components/units that evolve through time. WOuld be good to be able to effortlessly move between representations as individuals/grouped/connected objects.

One object changing through time - plants, stem cells, ice (to some extent).

Evie also has connectivity to exploit.

**Which track attributes are shared?**

**Which track attributes are unique?**

3D tracks (Evie) - works off the bat in 3D cartesian coordinates

Iceberg tracks erratic - existing motion model not necessarily well suited to represent.

**Which attributes are not already handled by btrack?**

**Are there any attributes that are conceptually difficult/impossible to handle within the exisitng framework?**

**Beyond the use cases outlined today, are there any object/track attributes that would be desirable to cater for?**

**Any other questions we should be asking?**

Why are we interested in tracking? - Evie to retain the identity of an object through time and monitoring change in that object compared to icebergs where the spatial dynamics are important... different conceptualisations of assosciations between objects and attributes though time.

## Session 2 proforma

**How capable is btrack, as it stands, of meeting the needs of the various use cases outlined?**
3D - already capable
Visual features - could do more sophisticated matching taking account of overall distributions of features - either in pre-processing (e.g. more intelligent normalisation) or within btrack itself.

**What technical development might be needed? What sort of scale of resource/time/expertise is involved?**

**How might we think about showcasing the versatility as it develops?**

Scivision? Both datasets and models. Don't have an example of object tracking
could maybe port pipeline into scivision: e.g. https://github.com/quantumjot/BayesianTracker/blob/main/examples/example_tracking_pipeline.ipynb

**Is there 'low hanging fruit' in terms of applications/science that we, as a group, could address?**

**How might we collaborate around the tracking idea going forwards?**

Scivision could be a nice way of gathering use cases - try contributing datasets/models via scivision and use it as a way of streamlining the contribution process. https://github.com/alan-turing-institute/scivision
https://github.com/scivision-gallery

Focus on sub-issues/data problems where they are shared

e.g. the large range of sizes in icebergs (and macrophages?)

Marjan's work - SOAPS - are these the same shape? - adjacent to the tracking problem but fundamental part of being successful. Can we work together to come up with an approach that works across use cases (possibly within scivision) - possibly SOAPS.

Explore alternative trackers

Stardist interested in linking up with btrack.

Lots of ML and deep tracking algorithms coming out, but all require vast training sets. btrack maybe offers something different

Think about what data we have beyond image data. e.g. gps on bergs/walrus etc.

**What might future workshops involve?**

Small focus groups on sub-problems.

**next steps**
| Action | Person |
| -------- | -------- |
| Share a small dataset if possible (Maybe with a notebook)| all (where able to share) |
| ~~Start slack channel~~ | ~~Scott/Ben~~ |
| Fill in proformas with any relevant thoughts | All |

## Interested people

| Name | Interest | Technical |
| -------- | -------- | -------- |
| Ben Evans (BAS) | Iceberg tracking | Keen to work on adding versatility to btrack and explore available shape descriptors |
| Kristina Ulicna (UCL) | Single cell tracking | Interested in tracking challenges on other-than-biological datasets & improvements of tracking fidelity |
| Christopher Soelistyo (UCL) | Single cell tracking | Interested in the kinds of challenges faced by tracking algorithms and the solutions implemented to address them. |
| Nathan Day (UCL/Crick) | Single cell tracking | Interested in both single whole-cell tracking and tracking of intracellular objects (i.e. Mtb within macrophage cells) |

## Wish list
| Functionality | Use 1 | Use 2 |
| -------- | -------- | -------- |
| Assessment of segmentation quality/recall to inform expectation of object quality (time-series based?)|Y| |
| Better unique object identification/fingerprinting in the face of variable object representations|Y| |
| Incorporate shape descriptors in track linkage determination (tuneable weights for generality?)|Y| |
| Non-uniform time step capability and non-uniform velocity motion model|Y| |
| Ability to incorporate environmental fields (e.g. currents) to inform motion model|Y| |
| Incorporate group motion fields to improve linkage of tracks for less certain identifications|Y| |
| Allow fragmentation into arbitrary number of child objects. Possibly exploit internal structure information from parent to match shapes of children to parent. |Y| |

## Wish list ideas (where to start?)
### Assessment of segmentation quality/recall to inform expectation of object quality
* (icebergs) Time-series assessment of bulk statistics of object geomtries (solidity, fractal dimension, count etc.) to detect abrupt non-persistent changes in the object population that are likely to be artefactual/recall related.

### Better unique object identification/fingerprinting in the face of variable object representations
* (icebergs) Improve post-processing to minimise artefacts in object geometry
* Dynamic time warping to represent distance between object geomtries (see `sktime` time series classification example)
* SOAPS (see [`dscribe`](https://singroup.github.io/dscribe/1.0.x/tutorials/descriptors/soap.html) package)
* Basic descriptors (e.g. plenty already implemented in `scikit-image`)
* Features from neural networks

### Incorporate shape descriptors in track linkage determination
* Ideally allow multiple descriptors to be used with tuneable weighting in Bayesian update step to allow for different use cases
* Each trackable object will need a fixed-length feature vector
* Need to define a Probability Density Function (PDF) that describes the probability of two objects being linked given the observed features
* Related GH issue: [#btrack/issues/108](https://github.com/quantumjot/BayesianTracker/issues/108)

### Non-uniform time step capability and non-uniform velocity motion model
* Motion model (`btrack` uses a Kalman Filter) can utilise non-uniform $dt$, however, the track linking assumes $dt=1$.

### Ability to incorporate environmental fields (e.g. currents) to inform motion model
* Control vector already implemented, not currently exposed to user
* Build a control field -- expose for user specification as spatially variable

### Incorporate group motion fields to improve linkage of tracks for less certain identifications
* (icebergs) Image velocimetry to approximate bulk motion fields

### Allow fragmentation into arbitrary number of child objects.
* (icebergs) Bergs can potentially fragment into mupltiple children simultaneously. btrack currently expects max. 2 child cells following mitosis
* `btrack` global optimization step is based on heuristics, *i.e.* proposing hypotheses based on the evidence. Are there specific hypotheses for different use cases?

## Resources
Ben Evans - Some research time can be allocated

## Funding
PDEA (Evans) - small pot to fund in-person workshops over summer 2022

## Relevant projects
https://github.com/quantumjot/BayesianTracker

Workshop 2 - 20230418

Project outline - Kristina
Chris background
Alan's 'trackathon' summary
Sanson
Hiris
Priscilla
Ania
Anyone else - updates/questions etc.

* Dataset interoperability - consolidated archive of segmentations from different use-cases?
* Geospatial capability (inc. vector-format)
* Multiple children and lineages
* Options for motion models and how to implement
* Evaluation of track quality (HOTA?)
* Most generalisable options for shape description and matching - what are people using and why?
*