---
title: "Learning Generative Models of Shape with Applications to Anatomy and the Automated Analysis of Radiographs"
author: "EPSRC Responsive Model Bid"
date: ""
---

# Summary

A wealth of applications, from the clinical analysis of anatomy to capturing actors in films and games, require representations of shape. In the established literature, approaches can be categorised into (i) mathematical approaches that uphold geometric and topological constraints; (ii) statistical approaches that quantify uncertainty; and (iii) computational approaches that focus on efficient algorithms. In many cases, one or other of these must be set aside. This is often inconviently true at scale for real data, where computational speed is prioritised over mathematical constraints and understanding of uncertainty.

---

**MAIN AIM OF RESEARCH** Establish efficient and practical computational approaches to shape analysis that incorporate understanding of uncertainty and respect mathematical constraints, using X-ray radiography and disease progession of arthritic hands as a test bed.

---
For a test case, we focus on X-ray radiography, a cheap and very widely used technique, and in particular radiographs of psioratic arthritis. Any technique that applies successfully here will transfer fabulously to many other applications. Team member W. Tillett is a consultant rheumatologist and his team holds a large set of training data (X-rays of disease progression in arthritic hands). Geometric constraints are clear (bones are incompressible and the nineteen bones of the hand identify a distinct topology). Uncertainty must be classified to indicate the quality of the prediction.

Our models will:

1. Allow complex shape constraints to be enforced directly (e.g. preserving volume or topology that avoids self-intersection)
2. Capture uncertainty in predicted shapes (e.g. can be used as a statistical prior for inverse problems)
3. Permit practical and data efficient training (e.g. requiring minimal supervision or annotation)

In the analysis of radiographs, we are primarily interested in the recovery of the detailed shape of the bones (e.g. dilation or erosion of a bone where material has been added or removed) and making accurate measurements between them (e.g. the precise distance that separates neighbouring bones).
In this scenario, (1) the topology of the bones should be preserved and can be used to resolve ambiguities in the radiograph. (2) Capturing uncertainty in the automated analysis is vital for the safe application of AI technologies in a medical setting where it allows clinicians (who are ultimately responsible) to trust the outputs and make informed and reliable decisions. (3) A practical system is necessary and should operates effectively when data is scarce and difficult to obtain (e.g. involves expensive annotation from expert clinicians).

# Scheme of Work

### WP1. Flow-based statistical shape model

Shape models based on continuous deformation of reference shapes allow mathematical properties to be preserved. There is a large literature of image registration based on flow fields and some work on stochastic flows. We will use ideas from machine learning, notably the Gaussian Process Latent Variable Model (GP LVM), to describe deformation fields and their natural variation. GP-LVMs are both computationally convenient and describe uncertainty

### WP2 Uncertainty Quantification, Calibration and Representation

The statistical shape models developed in WP1 form a basis for statistical learning informed by observations following tools from Bayesian statistics. The final result is a posterior distribution on the space of shapes fitting a radiograph or time-series of radiographs. This inverse problem is challenging to solve computationally. It also challenging to present the resulting distribution to posterior: how does a medic visualise uncertainty in the resuling anatomy?

### WP3. Lifting Shape from 2D to 3D

The capture of images in 2D is much easier (cost, availability of hardware) than 3D. A high-quality shape model for a 3D anatomy has the potential to infer shape from 2D images. We aim to develop shape models for hands in 3D and infer shape from 2D radiographs.

### WP4. Active Learning System with Clinical Prototype

The shape modelling and data analysis depend on medics annotating X-rays. We will set up a feedback mechanism, to facilitate annotation. The models that we develop in WP1--WP3 will run on raw X-rays but will feature glitches that are obvious to the medic. We will develop an active-learning system that allows medics to feed back into the shape model and for dynamic updates of the learning system to occur.

## Team

- PDRA in Maths
- PDRA in CS
- REng for software development
- Resource for clinical activities?

In the long term, through our collaborators and project partners, we will demonstrate the utility of our approach for *[x, y, z]* in a clinical setting.

---
# Below is NOT for inclusion in summary
---

# Background

## CS

- Previously statistical shape models (ASM), in particularly those based on subspace models have shown much practical utility..
- Have been extended to incorporate appearance information (AAM)..
- Now we are seeing much success with deep learning approaches - however:
- These work well on texture but not on shape (e.g. paper Adam looked at for reading group)
- How to integrate the two?
- Need for generative models to be used in medical and safety situations - challenging for some circumstances but achievable for medical imaging
- Problems:
- Carefully curated construction **(font approach fixes this - and Ieva's Alignment work)**
- Remember that correspondence finding is a really difficult but important problem - again, lots of impacts beyond what we are doing here..
- Constraints: **(flow approach fixes this?)**
- Volume preservation
- Self-intersection
- Collapse, conformity - e.g. flow gets around monotonicity constraint for 3D meshes..
- Smooth interpolants
- Structure preservation
- Intersection of the domains of representation: curves, meshes, images, voxels **(advances to snake approach)**

- Approaches need to consider the specific cases when limited training data - expensive to obtain
- Active learning approach (no difference between labelling and inference really - should always feedback new information)
- Need to thing about the GUI for this sort of thing - actually talk to domain experts who don’t know about learning (ties in with curated model construction)
- End-to-end learning - call this something else? Make the best of both worlds - possibly include work from Chris and Matteo?
- Inverse rendering can now be used in an inverse problems setting to help with the lifting
- Generative model can demonstrate what we have learned - convince people to let us use it!
- Get proper uncertainty quantification to be safe
- How to represent the uncertainty to clinicians? (Maybe videos of draws so things swim around when they are uncertain?)

## Maths

- Nice text from Adwaye and Tony
- Should focus on how current theory models posed to overcome the problems above
- e.g. diffeo-morphisms to preserve topology and energy in the flow fields to overcome the registration/alignment issue
- Enumerate how we capture the nice properties
- Discussions on the limitations to practical implementations
- e.g. how to learn from data, how to perform tractable computations

# Proposal

We are going to fix the problems above to capture the nice properties from the maths side whilst preserving the pragmatic algorithmic properties from CS background and extending them with nice probabilistic ML properties.

Notes:
- Include illustrations from Adwaye's thesis (and Adam's work for 3D side of things)
- Include discussion of clinical relevance and mapping to tasks
- Maybe a pre-emptive discussion of the impact of the work (from a theory viewpoint) to other problems..

# Scheme of Work

\begin{itemize}
\item[WP1] \textbf{SDE Flow-based GP-LVM Shape Model}
\item[WP2] \textbf{Lifting Shape from 2D to 3D}
\item[WP3] \textbf{Uncertainty Quantification, Calibration and Representation}
\item[WP4] \textbf{Active Learning System with Clinical Prototype}
\end{itemize}