owned this note
owned this note
Published
Linked with GitHub
# Integrating MACE on Android using CMake in Android Studio
###### tags: `android` `mace` `deep learning`
## Pre-Requirement
* Android Studio
* Please install Android SDK and NDK packages. (see [Android Native Development Kit (NDK)](https://hackmd.io/WO-nZYsfR1icmYbWy3jFrQ))
* You may also want to import OpenCV library in your native code. (see [How to Import OpenCV in Android NDK](https://hackmd.io/pdIcTgr7RuKGlYjwaGDwhw))
* Build MACE (see [How to Build MACE on Mac](https://hackmd.io/d1V8KrzpT7GxWKCu2k2RZQ))
## Demo APP - MaceDemo
### Create a New Project
* Enable C++ support
![](https://i.imgur.com/U8oIGFW.png)
* Enable Exception and RTTI Support
![](https://i.imgur.com/tAocgnd.png)
### Import MACE Libraries
* Copy the folder (/mace/builds/include) to /app/src/main/cpp/
* Copy /mace/builds/lib/arm64-v8a(or other ABIs)/libncnn.a to /app/src/main/cpp/jniLibs/arm64-v8a (if jniLibs folder does not exist, create one)
* Modify CmakeList.txt in Android Studio
* set the library path and its properties.
```
include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include)
add_library(
libmace
STATIC
IMPORTED
)
set_target_properties(
libmace
PROPERTIES IMPORTED_LOCATION
${CMAKE_CURRENT_LIST_DIR}/src/main/jniLibs/${ANDROID_ABI}/libmace.a
)
```
* link all the library we need
```
target_link_libraries( # Specifies the target library.
native-lib
# Links the target library to the log library
# included in the NDK.
libmace
${android-lib}
${log-lib} )
```
* Modify the build.gradle(Module:app)
```
defaultConfig {
applicationId "com.jd.ncnndemo"
minSdkVersion 23
targetSdkVersion 27
versionCode 1
versionName "1.0"
testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
externalNativeBuild {
cmake {
cppFlags "-frtti -fexceptions -fopenmp" // new line
}
}
ndk {
// Specifies the ABI configurations of your native
// libraries Gradle should build and package with your APK.
abiFilters 'arm64-v8a' // new line
}
}
```
* Rebuild the project and it is supposed to generate several C++ libraries in your cpp folder.
![](https://i.imgur.com/kZKb1ks.png)
* Then, we can include MACE libraries in our native code (native-lib.cpp).
```cpp=
// mace
// Include the headers
#include "mace/public/mace.h"
#include "mace/public/mace_runtime.h"
```
### Load Model
* Here, we use the model that is converted in code format. (see [How to Build MACE on Mac - Caffe to MACE](https://hackmd.io/d1V8KrzpT7GxWKCu2k2RZQ))
```yaml=
# in yaml file
model_graph_format: code
model_data_format: code
```
* Copy the header files ***mace_engine_factory.h*** and ***model.h*** (/mace/builds/your_model_folder/include/mace/public) into /app/src/main/cpp/include/mace/public
* Copy /mace/builds/your_model_folder/model/arm64-v8a (or other ABIs )/your_model.a to /app/src/main/cpp/jniLibs/arm64-v8a
* Modify CmakeList.txt in Android Studio
* set properties
```
add_library(
model_0_48
STATIC
IMPORTED
)
set_target_properties(
model_0_48
PROPERTIES IMPORTED_LOCATION
${CMAKE_CURRENT_LIST_DIR}/src/main/jniLibs/${ANDROID_ABI}/model_0_48.a
)
```
* link the library
```
target_link_libraries( # Specifies the target library.
native-lib
# Links the target library to the log library
# included in the NDK.
libmace
model_0_48 # new line
${android-lib}
${log-lib} )
```
* Rebuild the project and it is supposed to generate your model as a static library in your cpp folder.
![](https://i.imgur.com/ADxzW87.png)
* Then, we can include our model for further usages in our native code (native-lib.cpp).
```cpp=
#include "mace/public/model_0_48.h"
#include "mace/public/mace_engine_factory.h"
```
* define several structures or classes for initialization
```cpp=
// native-lib.cpp
struct ModelInfo {
std::string input_name;
std::string output_name;
std::vector<int64_t> input_shape;
std::vector<int64_t> output_shape;
};
struct MaceContext {
std::shared_ptr<mace::MaceEngine> engine;
std::shared_ptr<mace::KVStorageFactory> storage_factory;
std::string model_name;
mace::DeviceType device_type = mace::DeviceType::CPU;
std::map<std::string, ModelInfo> model_infos = {
{"model_0_48", {"data", "prob",
{1, 480, 480, 3}, {1, 480, 480, 2}}}
};
};
mace::DeviceType ParseDeviceType(const std::string &device) {
if (device.compare("CPU") == 0) {
return mace::DeviceType::CPU;
} else if (device.compare("GPU") == 0) {
return mace::DeviceType::GPU;
} else if (device.compare("HEXAGON") == 0) {
return mace::DeviceType::HEXAGON;
} else {
return mace::DeviceType::CPU;
}
}
MaceContext& GetMaceContext() {
// stay for the app's life time, only initialize once
static auto *mace_context = new MaceContext;
return *mace_context;
}
```
* define the model initialzation function
```cpp=
// native-lib.cpp
JNIEXPORT jint JNICALL
Java_com_jd_macedemo_MACE_CreateEngine(
JNIEnv *env, jclass thisObj, jstring model_name_str, jstring device) {
MaceContext &mace_context = GetMaceContext();
// parse model name
const char *model_name_ptr = env->GetStringUTFChars(model_name_str, nullptr);
if (model_name_ptr == nullptr) return JNI_ERR;
mace_context.model_name.assign(model_name_ptr);
env->ReleaseStringUTFChars(model_name_str, model_name_ptr);
// load model input and output name
auto model_info_iter =
mace_context.model_infos.find(mace_context.model_name);
if (model_info_iter == mace_context.model_infos.end()) {
alog("Invalid model name: %s", mace_context.model_name.c_str());
return JNI_ERR;
}
std::vector<std::string> input_names = {model_info_iter->second.input_name};
std::vector<std::string> output_names = {model_info_iter->second.output_name};
// get device
const char *device_ptr = env->GetStringUTFChars(device, nullptr);
if (device_ptr == nullptr) return JNI_ERR;
mace_context.device_type = ParseDeviceType(device_ptr);
env->ReleaseStringUTFChars(device, device_ptr);
alog("device: %d", mace_context.device_type);
mace::MaceStatus status;
// openmp
status = mace::SetOpenMPThreadPolicy(2, static_cast<mace::CPUAffinityPolicy>(0));
alog("openmp result: %d", status);
mace::MaceStatus create_engine_status =
mace::CreateMaceEngineFromCode(mace_context.model_name,
std::string(),
input_names,
output_names,
mace_context.device_type,
&mace_context.engine);
alog("create result: %d", create_engine_status);
return create_engine_status == mace::MaceStatus::MACE_SUCCESS ? JNI_OK : JNI_ERR;
}
```
### Model Prediction
``` cpp=
// native-lib.cpp
JNIEXPORT void JNICALL
Java_com_jd_macedemo_MACE_MaceProcess(
JNIEnv *env,
jobject /* this */) {
// image preprocessing
// ...
// ...
// mace process
/*
MaceContext &mace_context = GetMaceContext();
// prepare input and output
auto model_info_iter =
mace_context.model_infos.find(mace_context.model_name);
if (model_info_iter == mace_context.model_infos.end()) {
alog("Invalid model name: %s", mace_context.model_name.c_str());
return;
}
const ModelInfo &model_info = model_info_iter->second;
const std::string &input_name = model_info.input_name;
const std::string &output_name = model_info.output_name;
const std::vector<int64_t> &input_shape = model_info.input_shape;
const std::vector<int64_t> &output_shape = model_info.output_shape;
const int64_t input_size =
std::accumulate(input_shape.begin(), input_shape.end(), 1,
std::multiplies<int64_t>());
const int64_t output_size =
std::accumulate(output_shape.begin(), output_shape.end(), 1,
std::multiplies<int64_t>());
// construct input
std::map<std::string, mace::MaceTensor> inputs;
auto buffer_in = std::shared_ptr<float>(new float[input_size], std::default_delete<float[]>());
std::copy_n(input_img, input_size, buffer_in.get());
inputs[input_name] = mace::MaceTensor(input_shape, buffer_in);
// construct output
std::map<std::string, mace::MaceTensor> outputs;
auto buffer_out = std::shared_ptr<float>(new float[output_size], std::default_delete<float[]>());
outputs[output_name] = mace::MaceTensor(output_shape, buffer_out);
// run model
mace_context.engine->Run(inputs, &outputs);
*/
}
```
:::danger
be careful that the input and output format are both NHWC, which means it would be like this
* input : [RGBRGBRGBRGBRGB......]
* output : [C1C2C3C1C2C3C1C2C3......]
:::
## Java Native Library (MACE.java)
```java=
// MACE.java
public class MACE {
// Used to load the 'native-lib' library on application startup.
static {
System.loadLibrary("native-lib");
}
/*
* A native method that is implemented by the 'native-lib' native library,
* which is packaged with this application.
*/
public static native int CreateEngine(String model, String device);
public static native void MaceProcess();
}
```
## MainActivity.java
```java=
// MainActivity.java
/**
* A native method that is implemented by the 'native-lib' native library,
* which is packaged with this application.
*/
public static native int CreateEngine(String model, String device);
public native void MaceProcess(); // new line
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
// Example of a call to a native method
CreateEngine("model_0_48", "CPU");
MaceProcess(); // new line
}
```
## Use GPU Version Library (Optional)
* load specific opencl library for your phone (optional, see [How to Build MACE on Mac - MACE GPU Build for specific SoC](https://hackmd.io/d1V8KrzpT7GxWKCu2k2RZQ))
```cpp=
// native-lib.cpp
// CreateEngine
// gpu
mace::SetGPUHints(
static_cast<mace::GPUPerfHint>(3),
static_cast<mace::GPUPriorityHint>(3));
std::vector<std::string> opencl_binary_paths = {"file:///android_asset/opencl/arm64-v8a/model_0_48_GPU_compiled_opencl_kernel.SM-G950U.msm8998.bin"};
mace::SetOpenCLBinaryPaths(opencl_binary_paths);
const std::string opencl_parameter_file = "file:///android_asset/opencl/arm64-v8a/model_0_48_GPU_tuned_opencl_parameter.SM-G950U.msm8998.bin";
mace::SetOpenCLParameterPath(opencl_parameter_file);
```
* load cache to avoid loading opencl eveytimes
```cpp
// native-lib.cpp
// CreateEngine
// opencl cache
const char *kernel_path_ptr = "/storage/emulated/0/";
if (kernel_path_ptr == nullptr) return JNI_ERR;
const std::string kernel_file_path(kernel_path_ptr);
mace_context.storage_factory.reset(
new mace::FileStorageFactory(kernel_file_path));
mace::SetKVStorageFactory(mace_context.storage_factory);
```
* remember to modify the engine device
```java=
CreateEngine("model_0_48", "GPU"); // cpu to gpu
```
## Trouble Shooting
### no type named 'shared_ptr' in namespace 'std'
* Error Message :
```
Build command failed.
Error while executing process /Users/yilinchen/Library/Android/sdk/cmake/3.6.4111459/bin/cmake with arguments {--build /Users/yilinchen/Projects/MaceDemo/app/.externalNativeBuild/cmake/debug/arm64-v8a --target native-lib}
[1/2] Building CXX object CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o
FAILED: /Users/yilinchen/Library/Android/sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ --target=aarch64-none-linux-android --gcc-toolchain=/Users/yilinchen/Library/Android/sdk/ndk-bundle/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64 --sysroot=/Users/yilinchen/Library/Android/sdk/ndk-bundle/sysroot -Dnative_lib_EXPORTS -I../../../../src/main/cpp/include -isystem /Users/yilinchen/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.9/include -isystem /Users/yilinchen/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.9/libs/arm64-v8a/include -isystem /Users/yilinchen/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.9/include/backward -isystem /Users/yilinchen/Library/Android/sdk/ndk-bundle/sysroot/usr/include/aarch64-linux-android -D__ANDROID_API__=23 -g -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -Wa,--noexecstack -Wformat -Werror=format-security -frtti -fexceptions -fopenmp -O0 -fno-limit-debug-info -fPIC -MD -MT CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o -MF CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o.d -o CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o -c /Users/yilinchen/Projects/MaceDemo/app/src/main/cpp/native-lib.cpp
In file included from /Users/yilinchen/Projects/MaceDemo/app/src/main/cpp/native-lib.cpp:6:
In file included from ../../../../src/main/cpp/include/mace/public/mace.h:21:
In file included from /Users/yilinchen/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.9/include/cstdint:35:
/Users/yilinchen/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.9/include/bits/c++0x_warning.h:32:2: error: This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support for the \
^
In file included from /Users/yilinchen/Projects/MaceDemo/app/src/main/cpp/native-lib.cpp:6:
../../../../src/main/cpp/include/mace/public/mace.h:49:34: error: a space is required between consecutive right angle brackets (use '> >')
std::vector<std::vector<int64_t>> output_shape;
^~
> >
../../../../src/main/cpp/include/mace/public/mace.h:83:28: error: no type named 'shared_ptr' in namespace 'std'
std::shared_ptr<float> data);
~~~~~^
../../../../src/main/cpp/include/mace/public/mace.h:83:38: error: expected ')'
std::shared_ptr<float> data);
^
../../../../src/main/cpp/include/mace/public/mace.h:82:22: note: to match this '('
explicit MaceTensor(const std::vector<int64_t> &shape,
^
../../../../src/main/cpp/include/mace/public/mace.h:86:31: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
MaceTensor(const MaceTensor &&other);
^
../../../../src/main/cpp/include/mace/public/mace.h:88:42: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
MaceTensor &operator=(const MaceTensor &&other);
^
../../../../src/main/cpp/include/mace/public/mace.h:92:14: error: no type named 'shared_ptr' in namespace 'std'
const std::shared_ptr<float> data() const;
```
* Solution : add a cppFlag '-std=c++11' in build.gradle(Module:app)
```
defaultConfig {
applicationId "com.jd.macedemo"
minSdkVersion 23
targetSdkVersion 27
versionCode 1
versionName "1.0"
testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
externalNativeBuild {
cmake {
cppFlags "-frtti -fexceptions -fopenmp -std=c++11" // new line
}
}
ndk {
// Specifies the ABI configurations of your native
// libraries Gradle should build and package with your APK.
abiFilters 'arm64-v8a'
}
}
```
### library *libcdsprpc.so* not found
* It would happend when you try to build 'armeabi-v7a'. I seems we should import *libhexagon_controller.so* as well, however, the cpu should match to hexagon.
* Solution : commend the below line in */mace/libmace/BUILD*
```
-DMACE_ENABLE_HEXAGON
```