# Integrating MACE on Android using CMake in Android Studio
###### tags: `android` `mace` `deep learning`
## Pre-Requirement
* Android Studio

* Please install Android SDK and NDK packages. (see [Android Native Development Kit (NDK)](https://hackmd.io/WO-nZYsfR1icmYbWy3jFrQ))

* You may also want to import OpenCV library in your native code. (see [How to Import OpenCV in Android NDK](https://hackmd.io/pdIcTgr7RuKGlYjwaGDwhw))

* Build MACE (see [How to Build MACE on Mac](https://hackmd.io/d1V8KrzpT7GxWKCu2k2RZQ))

## Demo APP - MaceDemo
### Create a New Project
* Enable C++ support

* Enable Exception and RTTI Support

### Import MACE Libraries
* Copy the folder (/mace/builds/include) to /app/src/main/cpp/

* Copy /mace/builds/lib/arm64-v8a(or other ABIs)/libncnn.a to /app/src/main/cpp/jniLibs/arm64-v8a (if jniLibs folder does not exist, create one)

* Modify CmakeList.txt in Android Studio
* set the library path and its properties.

        include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include)

        add_library(
            libmace
            STATIC
            IMPORTED
            )

        set_target_properties(
            libmace
            PROPERTIES IMPORTED_LOCATION
            ${CMAKE_CURRENT_LIST_DIR}/src/main/jniLibs/${ANDROID_ABI}/libmace.a
        )

* link all the library we need

        target_link_libraries( # Specifies the target library.
                       native-lib

                       # Links the target library to the log library
                       # included in the NDK.
                       libmace
                       ${android-lib}
                       ${log-lib} )

* Modify the build.gradle(Module:app)

    defaultConfig {
        applicationId "com.jd.ncnndemo"
        minSdkVersion 23
        targetSdkVersion 27
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
        externalNativeBuild {
            cmake {
                cppFlags "-frtti -fexceptions -fopenmp" // new line
            }
        }
        ndk {
            // Specifies the ABI configurations of your native
            // libraries Gradle should build and package with your APK.

            abiFilters  'arm64-v8a' // new line
        }
    }

* Rebuild the project and it is supposed to generate several C++ libraries in your cpp folder.


* Then, we can include MACE libraries in our native code (native-lib.cpp).

    // mace
    // Include the headers
    #include "mace/public/mace.h"
    #include "mace/public/mace_runtime.h"

### Load Model
* Here, we use the model that is converted in code format. (see [How to Build MACE on Mac - Caffe to MACE](https://hackmd.io/d1V8KrzpT7GxWKCu2k2RZQ))

    # in yaml file
    model_graph_format: code
    model_data_format: code

* Copy the header files ***mace_engine_factory.h*** and ***model.h*** (/mace/builds/your_model_folder/include/mace/public) into /app/src/main/cpp/include/mace/public

* Copy /mace/builds/your_model_folder/model/arm64-v8a (or other ABIs )/your_model.a to /app/src/main/cpp/jniLibs/arm64-v8a

* Modify CmakeList.txt in Android Studio
* set properties

        add_library(
            model_0_48
            STATIC
            IMPORTED
            )

        set_target_properties(
            model_0_48
            PROPERTIES IMPORTED_LOCATION
            ${CMAKE_CURRENT_LIST_DIR}/src/main/jniLibs/${ANDROID_ABI}/model_0_48.a
            )

* link the library

        target_link_libraries( # Specifies the target library.
                       native-lib

                       # Links the target library to the log library
                       # included in the NDK.
                       libmace
                       model_0_48 # new line
                       ${android-lib}
                       ${log-lib} )

* Rebuild the project and it is supposed to generate your model as a static library in your cpp folder.


* Then, we can include our model for further usages in our native code (native-lib.cpp).

    #include "mace/public/model_0_48.h"
    #include "mace/public/mace_engine_factory.h"

* define several structures or classes for initialization

    // native-lib.cpp
    
    struct ModelInfo {
        std::string input_name;
        std::string output_name;
        std::vector<int64_t> input_shape;
        std::vector<int64_t> output_shape;
    };


    struct MaceContext {
        std::shared_ptr<mace::MaceEngine> engine;
        std::shared_ptr<mace::KVStorageFactory> storage_factory;
        std::string model_name;
        mace::DeviceType device_type = mace::DeviceType::CPU;
        std::map<std::string, ModelInfo> model_infos = {
                {"model_0_48", {"data", "prob",
                  {1, 480, 480, 3}, {1, 480, 480, 2}}}
        };
    };
    
    mace::DeviceType ParseDeviceType(const std::string &device) {
        if (device.compare("CPU") == 0) {
            return mace::DeviceType::CPU;
        } else if (device.compare("GPU") == 0) {
            return mace::DeviceType::GPU;
        } else if (device.compare("HEXAGON") == 0) {
            return mace::DeviceType::HEXAGON;
        } else {
            return mace::DeviceType::CPU;
        }
    }

    MaceContext& GetMaceContext() {
        // stay for the app's life time, only initialize once
        static auto *mace_context = new MaceContext;

        return *mace_context;
    }

* define the model initialzation function

    // native-lib.cpp
    
    JNIEXPORT jint JNICALL
    Java_com_jd_macedemo_MACE_CreateEngine(
            JNIEnv *env, jclass thisObj, jstring model_name_str, jstring device) {
        MaceContext &mace_context = GetMaceContext();
        //  parse model name
        const char *model_name_ptr = env->GetStringUTFChars(model_name_str, nullptr);
        if (model_name_ptr == nullptr) return JNI_ERR;
        mace_context.model_name.assign(model_name_ptr);
        env->ReleaseStringUTFChars(model_name_str, model_name_ptr);

        //  load model input and output name
        auto model_info_iter =
                mace_context.model_infos.find(mace_context.model_name);
        if (model_info_iter == mace_context.model_infos.end()) {
            alog("Invalid model name: %s", mace_context.model_name.c_str());
            return JNI_ERR;
        }

        std::vector<std::string> input_names = {model_info_iter->second.input_name};
        std::vector<std::string> output_names = {model_info_iter->second.output_name};

        // get device
        const char *device_ptr = env->GetStringUTFChars(device, nullptr);
        if (device_ptr == nullptr) return JNI_ERR;
        mace_context.device_type = ParseDeviceType(device_ptr);
        env->ReleaseStringUTFChars(device, device_ptr);

        alog("device: %d", mace_context.device_type);

        mace::MaceStatus status;
        // openmp
        status = mace::SetOpenMPThreadPolicy(2, static_cast<mace::CPUAffinityPolicy>(0));

        alog("openmp result: %d", status);
        mace::MaceStatus create_engine_status =
                mace::CreateMaceEngineFromCode(mace_context.model_name,
                                               std::string(),
                                               input_names,
                                               output_names,
                                               mace_context.device_type,
                                               &mace_context.engine);

        alog("create result: %d", create_engine_status);

        return create_engine_status == mace::MaceStatus::MACE_SUCCESS ? JNI_OK : JNI_ERR;
    }

### Model Prediction

// native-lib.cpp

JNIEXPORT void JNICALL
Java_com_jd_macedemo_MACE_MaceProcess(
        JNIEnv *env,
jobject /* this */) {

    // image preprocessing
    // ...
    // ...

    // mace process
    /*
    MaceContext &mace_context = GetMaceContext();

    //  prepare input and output
    auto model_info_iter =
            mace_context.model_infos.find(mace_context.model_name);
    if (model_info_iter == mace_context.model_infos.end()) {
        alog("Invalid model name: %s", mace_context.model_name.c_str());
        return;
    }

    const ModelInfo &model_info = model_info_iter->second;
    const std::string &input_name = model_info.input_name;
    const std::string &output_name = model_info.output_name;
    const std::vector<int64_t> &input_shape = model_info.input_shape;
    const std::vector<int64_t> &output_shape = model_info.output_shape;
    const int64_t input_size =
            std::accumulate(input_shape.begin(), input_shape.end(), 1,
                            std::multiplies<int64_t>());
    const int64_t output_size =
            std::accumulate(output_shape.begin(), output_shape.end(), 1,
                            std::multiplies<int64_t>());

    // construct input
    std::map<std::string, mace::MaceTensor> inputs;
    auto buffer_in = std::shared_ptr<float>(new float[input_size], std::default_delete<float[]>());
    std::copy_n(input_img, input_size, buffer_in.get());
    inputs[input_name] = mace::MaceTensor(input_shape, buffer_in);


    // construct output
    std::map<std::string, mace::MaceTensor> outputs;
    auto buffer_out = std::shared_ptr<float>(new float[output_size], std::default_delete<float[]>());
    outputs[output_name] = mace::MaceTensor(output_shape, buffer_out);

    // run model
    mace_context.engine->Run(inputs, &outputs);
    */
}

:::danger
be careful that the input and output format are both NHWC, which means it would be like this
* input : [RGBRGBRGBRGBRGB......]
* output : [C1C2C3C1C2C3C1C2C3......]
:::

## Java Native Library (MACE.java)

// MACE.java
    
public class MACE {
    // Used to load the 'native-lib' library on application startup.
    static {
        System.loadLibrary("native-lib");
    }

    /*
     * A native method that is implemented by the 'native-lib' native library,
     * which is packaged with this application.
     */
    public static native int CreateEngine(String model, String device);
    public static native void MaceProcess();
}

## MainActivity.java

// MainActivity.java

/**
     * A native method that is implemented by the 'native-lib' native library,
     * which is packaged with this application.
     */
    public static native int CreateEngine(String model, String device);
    public native void MaceProcess(); // new line

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        // Example of a call to a native method
        CreateEngine("model_0_48", "CPU");
        MaceProcess(); // new line
    }

## Use GPU Version Library (Optional)
* load specific opencl library for your phone (optional, see [How to Build MACE on Mac - MACE GPU Build for specific SoC](https://hackmd.io/d1V8KrzpT7GxWKCu2k2RZQ))

    // native-lib.cpp
    // CreateEngine
    
     //  gpu
    mace::SetGPUHints(
            static_cast<mace::GPUPerfHint>(3),
            static_cast<mace::GPUPriorityHint>(3));

    std::vector<std::string> opencl_binary_paths = {"file:///android_asset/opencl/arm64-v8a/model_0_48_GPU_compiled_opencl_kernel.SM-G950U.msm8998.bin"};
    mace::SetOpenCLBinaryPaths(opencl_binary_paths);
    const std::string opencl_parameter_file = "file:///android_asset/opencl/arm64-v8a/model_0_48_GPU_tuned_opencl_parameter.SM-G950U.msm8998.bin";
    mace::SetOpenCLParameterPath(opencl_parameter_file);

* load cache to avoid loading opencl eveytimes

    // native-lib.cpp
    // CreateEngine
     //  opencl cache
    const char *kernel_path_ptr = "/storage/emulated/0/";
    if (kernel_path_ptr == nullptr) return JNI_ERR;
    const std::string kernel_file_path(kernel_path_ptr);
    mace_context.storage_factory.reset(
            new mace::FileStorageFactory(kernel_file_path));
    mace::SetKVStorageFactory(mace_context.storage_factory);

* remember to modify the engine device

    CreateEngine("model_0_48", "GPU"); // cpu to gpu

## Trouble Shooting
### no type named 'shared_ptr' in namespace 'std'
* Error Message :

    Build command failed.
    Error while executing process /Users/yilinchen/Library/Android/sdk/cmake/3.6.4111459/bin/cmake with arguments {--build /Users/yilinchen/Projects/MaceDemo/app/.externalNativeBuild/cmake/debug/arm64-v8a --target native-lib}
    [1/2] Building CXX object CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o
    FAILED: /Users/yilinchen/Library/Android/sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++  --target=aarch64-none-linux-android --gcc-toolchain=/Users/yilinchen/Library/Android/sdk/ndk-bundle/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64 --sysroot=/Users/yilinchen/Library/Android/sdk/ndk-bundle/sysroot  -Dnative_lib_EXPORTS -I../../../../src/main/cpp/include -isystem /Users/yilinchen/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.9/include -isystem /Users/yilinchen/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.9/libs/arm64-v8a/include -isystem /Users/yilinchen/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.9/include/backward -isystem /Users/yilinchen/Library/Android/sdk/ndk-bundle/sysroot/usr/include/aarch64-linux-android -D__ANDROID_API__=23 -g -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -Wa,--noexecstack -Wformat -Werror=format-security  -frtti -fexceptions -fopenmp -O0 -fno-limit-debug-info  -fPIC -MD -MT CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o -MF CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o.d -o CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o -c /Users/yilinchen/Projects/MaceDemo/app/src/main/cpp/native-lib.cpp
    In file included from /Users/yilinchen/Projects/MaceDemo/app/src/main/cpp/native-lib.cpp:6:
    In file included from ../../../../src/main/cpp/include/mace/public/mace.h:21:
    In file included from /Users/yilinchen/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.9/include/cstdint:35:
    /Users/yilinchen/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.9/include/bits/c++0x_warning.h:32:2: error: This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
    #error This file requires compiler and library support for the \
     ^
    In file included from /Users/yilinchen/Projects/MaceDemo/app/src/main/cpp/native-lib.cpp:6:
    ../../../../src/main/cpp/include/mace/public/mace.h:49:34: error: a space is required between consecutive right angle brackets (use '> >')
      std::vector<std::vector<int64_t>> output_shape;
                                     ^~
                                     > >
    ../../../../src/main/cpp/include/mace/public/mace.h:83:28: error: no type named 'shared_ptr' in namespace 'std'
                          std::shared_ptr<float> data);
                          ~~~~~^
    ../../../../src/main/cpp/include/mace/public/mace.h:83:38: error: expected ')'
                          std::shared_ptr<float> data);
                                         ^
    ../../../../src/main/cpp/include/mace/public/mace.h:82:22: note: to match this '('
      explicit MaceTensor(const std::vector<int64_t> &shape,
                         ^
    ../../../../src/main/cpp/include/mace/public/mace.h:86:31: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
      MaceTensor(const MaceTensor &&other);
                                  ^
    ../../../../src/main/cpp/include/mace/public/mace.h:88:42: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
      MaceTensor &operator=(const MaceTensor &&other);
                                             ^
    ../../../../src/main/cpp/include/mace/public/mace.h:92:14: error: no type named 'shared_ptr' in namespace 'std'
      const std::shared_ptr<float> data() const;

* Solution : add a cppFlag '-std=c++11' in build.gradle(Module:app)

    defaultConfig {
        applicationId "com.jd.macedemo"
        minSdkVersion 23
        targetSdkVersion 27
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
        externalNativeBuild {
            cmake {
                cppFlags "-frtti -fexceptions -fopenmp -std=c++11" // new line
            }
        }
        ndk {
            // Specifies the ABI configurations of your native
            // libraries Gradle should build and package with your APK.

            abiFilters  'arm64-v8a'
        }
    }

### library *libcdsprpc.so* not found
* It would happend when you try to build 'armeabi-v7a'. I seems we should import *libhexagon_controller.so* as well, however, the cpu should match to hexagon.

* Solution : commend the below line in */mace/libmace/BUILD*

    -DMACE_ENABLE_HEXAGON