{%hackmd theme-dark %}
###### tags: `Approved`

Approved by:
- yotam

Design model Converter from channel first to channel last
===
**(ONNX and Image data format background at the end)**

## Motivation
This task is part of the `ONNX` support Milestone ([Asana card](https://app.asana.com/0/1199886922241039/1199886922241051)).
Our final goal is to be able to import `ONNX` models to our system, and `Pytorch` models through that.

I propose here to create a tool, hopefully open sourced in the future, to convert between `keras` `channel-first` models to `keras` `channel-last` models, that tool is the missing piece in our flow to import an ONNX model to our system.

## Convert from channel first to channel last

Right now we know how to convert from an `ONNX` model to a `keras` model with an open sourced lib called [onnx2keras](https://github.com/nerox8664/onnx2keras)
But as `ONNX` graphs are ordered in a way called `channel first`, `onnx2keras` is creating the keras model in the same way using a special condition.
Our system works in the default way of `keras` called `channel last` thus, after running `onnx2keras` we need to convert between the result `channel first` model to a `channel last` model

Consider the following flow for a Pytorch user:

Pytorch->ONNX: torch.onnx.export

ONNX->Tensorleap: Webapp.ImportOnnxModel

Tensorleap->onnx2keras: onnx_to_keras
onnx2keras->Tensorleap: Keras ChannelFirst model

Tensorleap->kerasconverter(new): convert_channels_first_to_last
kerasconverter(new)->Tensorleap: Keras ChannelLast model

## Converter Design
First of, the Converter will be a python package and will have a simple api similar to `onnx2keras` package.

- I suggest calling the package `kerasconverter`

The usage will be:

from kerasconverter import convert_channels_first_to_last
from kerasconverter import convert_channels_last_to_first

That endpoint will get a keras model and return a keras model

def  convert_channels_first_to_last(ch_first_model: tf.keras.Model) ->
tf.keras.Model 
def  convert_channels_last_to_first(ch_first_model: tf.keras.Model) ->
tf.keras.Model 

As our system is expecting channel last in its analysis, we cant stick to a simple approches of wrapping the `first-channel` model in Transpose layers.
- I suggest here to run over the keras computation graph and create another model holding a converted version of each layer from `channel-first` to `channel-last`.
- `Special layers` will be `Tranpose`, `Permute` and `Reshape` where we will have to not only create the layer in `channels-first` mode but also change its inner parameters to work in a `channels-last` model
- While scanning the model we will keep a boolean flag `critical_area` that will be set to `True` after visting a convolution node and be changed to `False` after visting a non spatial layer like `Dense` or `Flatten`.
`critical_area` flag is important for converting the Special layers mentioned above, as not always it is valid to change its inner parameters.

## ONNX background
The Open Neural Network Exchange (ONNX) provides a common representation of the computation graph thus, it decouples the computation graph and its runtime framework.
For us, adding support on importing it will mean supporting almost all Deep learning packages.

## Image data format (channel location) background
Color images have height, width, and color channel dimensions.
When represented as three-dimensional arrays, the channel dimension for the image data is last by default, but may be moved to be the first dimension, often for performance-tuning reasons.

(3,360,720) --- Channels first
(360,720,3) --- Channels last
Where 3 here comes from RGB (coloured image).

`Tensorflow` works as `channel-last` by default while `ONNX` and `Pytorch` are working in`channel-first` mode by default.