# Using G-bundles to define filter-dependent parallel transport

[](https://hackmd.io/6CjS6IAkQ9Cx43ZK5HNAbA)

This note is logged here: https://airtable.com/tblQstp0ROtvmmYQW/viwTPuzhq5l8cKTb4?blocks=bipb5MNwaJR8Aui5O

## The idea

For simplicity, we'll describe the idea in terms of isometries.

Given a measured metric sensor space $X$, a model space $Y$, and an isometric chart $\phi: B_r(x)\to Y$ for some $x\in X$, the set of all isometric charts from $B_r(x)\to Y$ can be identified with the group $G=\textrm{Isom}(Y)$ via the function $\psi_\phi: g\mapsto g\cdot \phi$, i.e. the function that associates the group element $g$ with the $g$-translate of the chart $\phi$.

Suppose we have a filter $f: Y\to \mathbb{R}$, and a stimulus $s: X\to \mathbb{R}$ (**did we fix terminology for the thing that causes a sensor to have a value (i.e. image is to pixel as ?????? is to abstract sensor?**)

We can then use the function $\psi_\phi$ to produce a function on $G$ via

$$g\mapsto\int_{z\in B_r(x)} s(z)f(\psi_\phi(g)(z))dz$$

This function measures the degree to which the group element $g$ aligns the pullback of the filter $f$ with under the chart $\psi_\phi(g)$ with the restriction of the stimulus to $B_r(x)$.

We can do a number of different things with this.

One thing is creating a generalized definition of a convolutional layer.

We begin by choosing a locally compatible family of charts (via the developing map or some other means) on $X$, so that $B_r(x)$ has a chart $\phi_x$ for each $x\in X$ and if $B_r(x_1)$ and $B_r(x_2)$ overlap then $\phi(x_1)$ and $\phi(x_2)$ agree on the overlap.

Given such a structure, each filter $f$ can be used to define a node $n_f$ in the convolutional layer identifiable with $X\times G$. For each stimilus $s$, $n_f(s):X\times G\to \mathbb{R}$ is the function described above that associates to $(x,g)$ the extent to which $s|_{B_r(x)}$ agrees with the pullback under $g\cdot \phi_x$ of $f$.

## A simple case to try to code up

Of course we don't need to deal with the whole group $G$ here- we can extend or restrict structure as we like. Probably this simplest case to code up is to take $G$ to be the rotation group $O(2)\subset \textrm{Isom}(\mathbb{R}^2)$, let $Y=\mathbb{R}^2$, and let $X$ be pixel space, and let $\phi_x$ be a ball of fixed radius $r$ in pixel space for each $r$ (either in the taxi-cab or euclidean metric) and embedding it in $\mathbb{R}^2$ in the obvious way to give charts.

It would be interesting to look at images containing a single curvy line and look at a filter $f$ consisting of a single vertical line.

The recipe above should give a node in a convolutional layer that takes an image of a curvy line $s$ and maps it to a function $n_f(s)$ on $X\times G$ such that

$$\max_{g\in G} n_f(s)(x,g)$$

a pair of points corresponding to plus and minus the tangent to the line if $x$ lies on the line (and $n_f(s)(x,g)$ will evaluate to zero for all $g$ if $x$ lies greater than $r$ from any point on the line).

This would be pretty cool- it already suggests a way of straightening out distorted images that we know came form $\mathbb{R}^2$.