# Integrating Caffe2 on Android
###### tags: `android` `deep learning` `caffe2`

## Pre-Requirement
* Android Studio

* Please install Android SDK and NDK packages. (see [Android Native Development Kit (NDK)](https://hackmd.io/WO-nZYsfR1icmYbWy3jFrQ))

* You may also want to import OpenCV library in your native code. (see [How to Import OpenCV in Android NDK](https://hackmd.io/pdIcTgr7RuKGlYjwaGDwhw))

* Download Facebook Caffe2 demo APP, AICamera [(link)](https://github.com/caffe2/AICamera/tree/master/app)

    git clone --recursive https://github.com/caffe2/AICamera.git

* clone the repository in recursive mode
* open it by Android Studio and follow the error messages to install all dependencied needed.

:::info
I just take the prebuild caffe2 library from AICamera, but you can try to build caffe2 and your own library (see [Install Caffe2](https://caffe2.ai/docs/getting-started.html?platform=mac&configuration=prebuilt)).
:::

## Caffe to Caffe2
* If your model is Caffe model, you have to transfer the model to Caffe2.

* Use the *caffe_translator.py* script to transfer your model. (see [Caffe to Caffe2](https://caffe2.ai/docs/caffe-migration.html))

    > python -m caffe_translator.py deploy.prototxt pretrained.caffemodel

:::danger
* If your model works unnormally, you may encounted with some problems by tranlating. If so, please refer to the trouble shooting section.
* If you want to use modified caffe_translator, remove -m.
:::

## Demo APP - Caffe2Demo
### Create a New Project
* Enable C++ support

* Enable Exception and RTTI Support

### Import Caffe2 Libraries
* Copy the following folders (AICamera/app/src/main/cpp/) into YourProject/app/src/main/cpp/
* caffe2
* Elgen
* googl
* class.h


* Copy the following files (AICamera/app/src/main/jniLibs/armeabi-v7a) into YourProject/app/src/main/jniLibs/armeabi-v7a (find your corresponsed ABI, if it is not existed, create one)
* libCaffe2_CPU.a
* libCAFFE2_NNPACK.a
* libCAFFE2_PTHREADPOOL.a
* libcpufeatures.a
* libglog.so
* libgnustl_shared.so
* libprotobuf-lite.a
* libprotobuf.a

:::danger
We should build this file from source code on our environment. However, I still can not figure out how to build caffe2 on PC.
:::


* Modify build.gradle(Module: app) to specify the c++ standard library and specify the ABI filter.

    defaultConfig {
        applicationId "com.jd.caffe2demo"
        minSdkVersion 22
        targetSdkVersion 25
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
        externalNativeBuild {
            cmake {
                cppFlags "-frtti -fexceptions -std=c++11" // new line
            }
        }
        ndk {
            // Specifies the ABI configurations of your native
            // libraries Gradle should build and package with your APK.
            abiFilters  'armeabi-v7a' // new line
        }
    }

* Modify CMakeList.txt :
* set the library and its properties.

        include(AndroidNdkModules)
        android_ndk_import_module_cpufeatures()

        add_library(
            caffe2
            STATIC
            IMPORTED
            )
        set_target_properties(
            caffe2
            PROPERTIES IMPORTED_LOCATION
            ${CMAKE_CURRENT_LIST_DIR}/src/main/jniLibs/${ANDROID_ABI}/libCaffe2_CPU.a
            )
        add_library(
            thread_pool
            STATIC
            IMPORTED
            )
        set_target_properties(
            thread_pool
            PROPERTIES IMPORTED_LOCATION
            ${CMAKE_CURRENT_LIST_DIR}/src/main/jniLibs/${ANDROID_ABI}/libCAFFE2_PTHREADPOOL.a
            )
        add_library(
            glog
            SHARED
            IMPORTED
            )
        set_target_properties(
            glog
            PROPERTIES IMPORTED_LOCATION
            ${CMAKE_CURRENT_LIST_DIR}/src/main/jniLibs/${ANDROID_ABI}/libglog.so
            )

        add_library(
            protobuf
            SHARED
            IMPORTED
            )
        set_target_properties(
            protobuf
            PROPERTIES IMPORTED_LOCATION
            ${CMAKE_CURRENT_LIST_DIR}/src/main/jniLibs/${ANDROID_ABI}/libprotobuf.a
            )

        add_library(
            NNPACK
            STATIC
            IMPORTED
            )
        set_target_properties(
            NNPACK
            PROPERTIES IMPORTED_LOCATION
            ${CMAKE_CURRENT_LIST_DIR}/src/main/jniLibs/${ANDROID_ABI}/libCAFFE2_NNPACK.a
            )

* link all the library we use

        target_link_libraries(
                       native-lib
                       -Wl,--whole-archive
                       caffe2
                       -Wl,--no-whole-archive
                       NNPACK
                       thread_pool
                       glog
                       protobuf
                       cpufeatures
                       ${log-lib}
                       ${android-lib})

* Rebuild the project and it is supposed to generate several C++ libraries in your cpp folder.


* Then, we can include Caffe2 libraries in our native code (native-lib.cpp).

    // native-lib.cpp
    #include <caffe2/core/predictor.h>
    #include <caffe2/core/operator.h>
    #include <caffe2/core/timer.h>
    #include "caffe2/core/init.h"

### Load Model
* Put our model, *init_net.pb* and *predict_net.pb* ,in /app/src/main/assets folder (create one if not exists).

* Since we need to load the model from asset folder, we need to include the asset manager in native code.

    // nativ-lib.cpp
    #include <android/asset_manager.h>
    #include <android/asset_manager_jni.h>

* Declare our model and predictor.

    // native-lib.cpp
    static caffe2::NetDef _initNet, _predictNet;
    static caffe2::Predictor *_predictor;

* define the native function

    // native-lib.cpp
    // A function to load the NetDefs from protobufs.
    void loadToNetDef(AAssetManager* mgr, caffe2::NetDef* net, const char *filename) {
        AAsset* asset = AAssetManager_open(mgr, filename, AASSET_MODE_BUFFER);
        assert(asset != nullptr);
        const void *data = AAsset_getBuffer(asset);
        assert(data != nullptr);
        off_t len = AAsset_getLength(asset);
        assert(len != 0);
        if (!net->ParseFromArray(data, len)) {
            alog("Couldn't parse net from data.\n");
        }
        AAsset_close(asset);
    }

    extern "C"
    void
    Java_com_jd_caffe2demo_MainActivity_initCaffe2(
            JNIEnv* env,
            jobject /* this */,
            jobject assetManager) {
        AAssetManager *mgr = AAssetManager_fromJava(env, assetManager);
        //load file
        loadToNetDef(mgr, &_initNet, "init_net.pb"); // need to match the file name.
        loadToNetDef(mgr, &_predictNet,"predict_net.pb");
        _predictor = new caffe2::Predictor(_initNet, _predictNet);
    }

* declare the native function in java

    // Caffe2.java
    // declare native functions
    public native void initCaffe2(AssetManager mgr);

* call init from java

    // MainActivity.java
    private AssetManager mgr;
    
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        // initial model
        mgr = getResources().getAssets();
        new SetUpNeuralNetwork().execute();
    }
    
    // use async thread loading the model to avoid UI timeout error
    @SuppressLint("StaticFieldLeak")
    private class SetUpNeuralNetwork extends AsyncTask<Void, Void, Void> {
        @Override
        protected Void doInBackground(Void[] v) {
            try {
                Caffe2.initCaffe2(mgr);
            } catch (Exception e) {
                Log.d("Caffe2Demo", "Couldn't load neural network.");
            }
            return null;
        }
    }

### Model Prediction

// nativ-lib.cpp
extern "C" JNIEXPORT void
JNICALL
Java_com_jd_caffe2demo_MainActivity_Caffe2Process(
    JNIEnv *env,
    jobject /* this */) {

    // image preprocessing
    // ...
    // ...
    
    // caffe2 process
    /*
    caffe2::TensorCPU input;
    input.Resize(std::vector<int>({1, Channel, IMG_INPUT_WIDTH, IMG_INPUT_HEIGHT}));
    memcpy(input.mutable_data<float>(), input_img, MAX_DATA_SIZE* sizeof(float)); // MAX_DATA_SIZE = Channel*IMG_INPUT_WIDTH*IMG_INPUT_HEIGHT

    caffe2::Predictor::TensorVector input_vec{&input};
    caffe2::Predictor::TensorVector output_vec;
    
    //run network
    _predictor->run(input_vec, &output_vec);
    */
}

// Caffe2.java
public native void Caffe2Process();

// MainActivity.java
@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    
    // new line
    btn_proc = findViewById(R.id.btn_proc);
    btn_proc.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View v) {
            // Example of a call to a native method
            Caffe2Process();
        }
    });

    // initial model
    mgr = getResources().getAssets();
    new SetUpNeuralNetwork().execute();
}

## Caffe2 GPU
“Currently Caffe2 is optimized for ARM CPUs with NEON (basically any ARM CPU since 2012). Perhaps surprisingly, ARM CPUs outperform the on-board GPUs (our NNPACK ARM CPU implementation outperforms Apple’s MPSCNNConvolution on devices older than iPhone 6s). There are other advantages to offloading compute onto the GPU/DSP, and it’s an active work in progress to expose these in Caffe2.”

* Support List (maybe..)
“Adreno (TM) 540”, “Adreno (TM) 530”, “Adreno (TM) 510”, “Adreno (TM) 430”, “Adreno (TM) 418”, “Mali-G71”, “Mali-T880”, "NVIDIA Tegra"

## Trouble Shooting
### Linking CXX shared library Failed (Build Fail)
* Description : the app would show errors at run time.

* Error Message

    Build command failed.
    Error while executing process /home/andy/Android/Sdk/cmake/3.6.4111459/bin/cmake with arguments {--build /home/andy/Documents/AICamera/app/.externalNativeBuild/cmake/debug/armeabi-v7a --target native-lib}
    [1/4] Building C object CMakeFiles/cpufeatures.dir/home/andy/Android/Sdk/ndk-    bundle/sources/android/cpufeatures/cpu-features.c.o
    [2/4] Linking C static library libcpufeatures.a
    [3/4] Building CXX object CMakeFiles/native-    lib.dir/src/main/cpp/native-lib.cpp.o
    [4/4] Linking CXX shared library ../../../../build/intermediates/cmake/debug/obj/armeabi-v7a/libnative-lib.so
    FAILED: : && /home/andy/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++  --target=armv7-none-linux-androideabi --gcc-toolchain=/home/andy/Android/Sdk/ndk-bundle/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 --sysroot=/home/andy/Android/Sdk/ndk-bundle/sysroot -fPIC -isystem /home/andy/Android/Sdk/ndk-bundle/sysroot/usr/include/arm-linux-androideabi -D__ANDROID_API__=22 -g -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb -Wa,--noexecstack -Wformat -Werror=format-security -std=c++11 -frtti -fexceptions -std=c++11 -O0 -fno-limit-debug-info  -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libatomic.a -nostdlib++ --sysroot /home/andy/Android/Sdk/ndk-bundle/platforms/android-22/arch-arm -Wl,--build-id -Wl,--warn-shared-textrel -Wl,--fatal-warnings -Wl,--fix-cortex-a8 -Wl,--exclude-libs,libunwind.a -L/home/andy/Android/Sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -Wl,--no-undefined -Wl,-z,noexecstack -Qunused-arguments -Wl,-z,relro -Wl,-z,now -shared -Wl,-soname,libnative-lib.so -o ../../../../build/intermediates/cmake/debug/obj/armeabi-v7a/libnative-lib.so CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o -Wl,--whole-archive ../../../../src/main/jniLibs/armeabi-v7a/libCaffe2_CPU.a -Wl,--no-whole-archive ../../../../src/main/jniLibs/armeabi-v7a/libCAFFE2_NNPACK.a ../../../../src/main/jniLibs/armeabi-v7a/libCAFFE2_PTHREADPOOL.a ../../../../src/main/jniLibs/armeabi-v7a/libglog.so ../../../../src/main/jniLibs/armeabi-v7a/libprotobuf.a libcpufeatures.a /home/andy/Android/Sdk/ndk-bundle/platforms/android-22/arch-arm/usr/lib/liblog.so /home/andy/Android/Sdk/ndk-bundle/platforms/android-22/arch-arm/usr/lib/libandroid.so -ldl -latomic -lm "/home/andy/Android/Sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a/libc++_static.a" "/home/andy/Android/Sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a/libc++abi.a" "/home/andy/Android/Sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a/libunwind.a" "-ldl" && :
    ../../../../src/main/jniLibs/armeabi-v7a/libCaffe2_CPU.a(nnpack_ops.cc.o):nnpack_ops.cc:function std::_Sp_counted_deleter<void*, void (*)(void*), std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&): error: undefined reference to 'std::type_info::operator==(std::type_info const&) const'
    ../../../../src/main/jniLibs/armeabi-v7a/libCaffe2_CPU.a(nnpack_ops.cc.o):nnpack_ops.cc:function std::_Sp_counted_deleter<void*, caffe2::Tensor<caffe2::CPUContext>::raw_mutable_data(caffe2::TypeMeta const&)::{lambda(void*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&): error: undefined reference to 'std::type_info::operator==(std::type_info const&) const'
    ../../../../src/main/jniLibs/armeabi-v7a/libCaffe2_CPU.a(nnpack_ops.cc.o):nnpack_ops.cc:function void caffe2::TypeMeta::_CopyNotAllowed<caffe2::Tensor<caffe2::CPUContext> >(void const*, void*, unsigned int): error: undefined reference to 'std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, int)'
    ../../../../src/main/jniLibs/armeabi-v7a/libCaffe2_CPU.a(nnpack_ops.cc.o):nnpack_ops.cc:function void caffe2::TypeMeta::_CopyNotAllowed<caffe2::Tensor<caffe2::CPUContext> >(void const*, void*, unsigned int): error: undefined reference to 'std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, int)'
    ...
    ...
    ...
    ...
    clang++: error: linker command failed with exit code 1 (use -v to see invocation)
    ninja: build stopped: subcommand failed.

* Cause : ndk cannot successfully link the standard library of C++. The key point is all std libraries have undefined reference error.

* Solution : It seems a bug from ndk (17.0.4) itself (or Android Studio). After downgrading the ndk version to 14.1, the ndk should find the std sucessfully. (wait for ndk update)

* Reference :
* https://groups.google.com/forum/#!topic/android-ndk/3iKT-kLEGpY
* https://github.com/caffe2/AICamera/issues/55

### Stuck at Initiating Stage of Predictor (Caffe to Caffe2 Issue)
* Description : The app would be stuck while initiating without any error message (just hangs).

    alog("done.");
    alog("Instantiating predictor...");
    _predictor = new caffe2::Predictor(_initNet, _predictNet);

* Error Message : It would not show any error on android logcat (maybe it need to be activate?). Since it would have the same behavior on local computer, we can test our model by the sample code of its original tuorial ([Loading_Pretrained_Models.ipynb](https://github.com/caffe2/tutorials/blob/master/Loading_Pretrained_Models.ipynb)). Then, it shows the following error message while iniating.

    Traceback (most recent call last):
  File "/Users/yilinchen/Desktop/caffemodel for caffe2/test.py", line 173, in <module>
    p = workspace.Predictor(init_net, predict_net)
  File "/anaconda3/envs/caffe2py3/lib/python3.6/site-packages/caffe2/python/workspace.py", line 157, in Predictor
    return C.Predictor(StringifyProto(init_net), StringifyProto(predict_net))
    RuntimeError: [enforce fail at conv_pool_op_base.h:179] dilation_[dim] == 1. 2 vs 1. When group is used, dilation should not be set at the same time. 
    [Finished in 1.7s with exit code 1]

* Cause : Unknown. Maybe caffe2 has not support this kind of structure yet.

* Solution (Temporarily) : edit the **deploy.prototxt** by deleting all dilations and modifying pad to be 1.

* Reference : consult with Kang Yan

### Stuck at Prediting Stage of Predictor (Caffe to Caffe2 Issue)
* Description : The app would be stuck at predicting stage of the predictor. It seems there is something wrong with the model while inferring.

    _predictor->run(input_vec, &output_vec);

* Error Message : The below error massages is also from the test on local computer.

    Traceback (most recent call last):
  File "/Users/yilinchen/Desktop/caffemodel for caffe2/test.py", line 176, in <module>
    results = p.run({'data': img})
    RuntimeError: [enforce fail at conv_transpose_op_impl.h:24] filter.ndim() == 4. filter must be 4D tensor Error from operator: 
    input: "conv7_sep" input: "conv7_w" output: "conv7" type: "ConvTranspose" arg { name: "stride" i: 4 } arg { name: "pad" i: 0 } arg { name: "kernel" i: 4 } arg { name: "order" s: "NCHW" }
    [Finished in 2.4s with exit code 1]

* Cause :
* At a glance from the error message, the problem might be caused by the dimesion problem. The input dimesion is not matched in layer 7. However, since the model is fine in caffe, it is possibly caused by caffe to caffe2 translating.

* We can add the below line in **caffe_translator.py** and execute it again to dig more information.

    def ConvertTensorProtosToInitNet(net_params, input_name):
    """Takes the net_params returned from TranslateModel, and wrap it as an
    init net that contain GivenTensorFill.
    This is a very simple feature that only works with float tensors, and is
    only intended to be used in an environment where you want a single
    initialization file - for more complex cases, use a db to store the
    parameters.
    """
    init_net = caffe2_pb2.NetDef()
    for tensor in net_params.protos:
        print(tensor.name, list(tensor.dims)) # <--- add this line
        if len(tensor.float_data) == 0:
            raise RuntimeError(
                "Only float tensors are supported in this util.")
        op = core.CreateOperator(
            "GivenTensorFill", [], [tensor.name],
            arg=[
                utils.MakeArgument("shape", list(tensor.dims)),
                utils.MakeArgument("values", tensor.float_data)])
        init_net.op.extend([op])
    init_net.op.extend([core.CreateOperator("ConstantFill", [], [input_name], shape=[1])])
    return init_net

* From the output, we can find that the dimension of *conv7_w* has been changed from 4 to 1.

        conv7_sep_b [256]
        conv7_w [256, 256, 4, 4]
        conv7_w [256]
        conv7_b [256]

* We also found that it would only happen on layer 7, the only one layer that do the scale operation. We guess there is something wrong of Scale operation while translating. Then we found that the belowing code

        @TranslatorRegistry.Register("Scale")
        def TranslateScale(layer, pretrained_blobs, is_test, **kwargs):
            mul_op = BaseTranslate(layer, "Mul")
            scale_param = layer.scale_param
            AddArgument(mul_op, "axis", scale_param.axis)
            AddArgument(mul_op, "broadcast", True)
            if len(mul_op.input) == 1:
                # the scale parameter is in pretrained blobs
                if scale_param.num_axes != 1:
                    raise RuntimeError("This path has not been verified yet.")

                output = mul_op.output[0] #<- problem

                mul_op_param = output + '_w'
                mul_op.input.append(mul_op_param)

* Since *mul_op.output[0]* is a constant string *conv7*, we would generate a same name as *conv7_w* and cover the original *conv7_w*.

* Solution 1 :
* Instead of using *mul_op.output[0]* as name, we use layer name. Change the problem line as below

        #output = mul_op.output[0]
        output = layer.name

:::danger
Not sure why caffe2 do this, and do not know whether there is any influence on the model after doing the change.
:::

* Solution 2 :
* Modify model structure without affect the model. Split *conv_7*
* From

* To

* In *deploy.prototxt*

        layer {
          name: "conv7/scale"
          type: "Scale"
          bottom: "conv7"
          top: "conv7_scale"
          scale_param {
            filler {
              value: 1.0
            }
            bias_term: true
            bias_filler {
          value: 0.0
            }
          }
        }
        layer {
          name: "conv7/relu"
          type: "ReLU"
          bottom: "conv7_scale"
          top: "conv7_relu"
        }

        layer {
          name: "decoder/concat"
          type: "Concat"
          bottom: "conv3_1/decoder"
          bottom: "conv7_relu"
          top: "decoder/concat"
        }

* Reference : https://github.com/caffe2/caffe2/issues/468
### Output Values of a Layer are NaN (Caffe to Caffe2 Issue)
* Description : the output values of the network are NaN

* Error Messages : no error, just not correct

* Cause :
* look into the weights and biases in each layer, we found that there are some values of weights and biases are NaN or Inf, -Inf.

* aftrer discussing with Kang Yan, we ++guess++ the problem might be caused by merging BatchNorm in Convolution layer when building the caffe model.

* the only possible operation that can generate NaN is BatchNorm (ex: divided by 0)

* caffe may not raise an issue if a value is NaN (protection technique?).

* Solution : Use another model that separate BatchNorm from Convolution layer.

### Softmax (Caffe to Caffe2 Issue)
* Caution : Caffe2 does not support uses to indicate a specific axis to do the Softmax. It would do a globle Softmax.

## Reference
* [在Android手机上运行Caffe2](https://zhuanlan.zhihu.com/p/33814408)
* [Caffe2 使用预训练模型进行预测](http://www.hugoli.com/2017/09/15/caffe2-%E4%BD%BF%E7%94%A8%E9%A2%84%E8%AE%AD%E7%BB%83%E6%A8%A1%E5%9E%8B%E8%BF%9B%E8%A1%8C%E9%A2%84%E6%B5%8B/)
* [What is Caffe2?](https://caffe2.ai/docs/caffe-migration.html)