Technical notes‎ > ‎

Tutorial - part 1: Using OpenCV Nonfree Module (SIFT, SURF) in Android NDK Projects

posted May 22, 2013, 7:58 PM by Robert Wang   [ updated Jun 4, 2016, 11:03 PM ]
Keywords: OpenCV; Android; NDK; Native program; nonfree module; SIFT; SURF; feature detection.

Updated: 
  • 2013-10-25 I tested the latest version of OpenCV: Version 2.4.6 for Android. The compilation script provided in this tutorial for the libnonfree.so and the native Android application still work perfectly.
  • 2014-02-12 I tested OpenCV-2.4.8 and OpenCV for Android 2.4.8. The tutorial still works perfectly. ( I used OpenCV-2.4.8, android-ndk-r9c-windows-x86_64 Android NDK, adt-bundle-windows-x86_64-20131030 Android SDK).  I also added example project for OpenCV 2.4.8 (basically the same as the previous package). Download here.
  • 2014-02-26 Part 2 of this tutorial is added. Tutorial Part 2: Use OpenCV Nonfree module (SIFT, SURF) in Android Applications via JNI
  • 2015-02-23 In the latest version of OpenCV, the Nonfree module has been moved to opencl_contrib repo, and the Nonfree module requires to link to xfeature2d module.
  • 2015-03-18, for those who working on newer OpenCV, such as OpenCV-2.4.10, you can check this link (Kornel's answer to a Stack Overflow question), Kornel has a good summary of the steps mentioned here. Thanks Kornel!
-----------------------------------------------------------------------

Notice: This article talks about how to use Nonfree module of OpenCV (SURF and SIFT) with Android JNI and NDK. But if you don't want to bother to use JNI,  you could also use JAVA code to call these functions, just as you did for other modules in OpenCV. Thanks ArthurT for the comments:
"Thank you very much for all the detailed explanation. Maybe what you would need to state more clearly is that for people only wanting to use SIFT and/or SURF via Java without caring about the JNI side (as I did), they need to copy-paste libnonfree.so and libopencv_java.so into their jniLibs folder under the relevant architecture folder (armeabi, armeabi-v7a, mips or x86). libopencv_java needs to be replaced if it is already in the folder(s). After this, static library loading with: System.loadLibrary("opencv_java"); System.loadLibrary("nonfree"); simply works and the job is done."
-----------------------------------------------------------------------
OpenCV provides a good implementation of both the SIFT and SURF keypoint detection and feature description algorithms. The following figures demonstrate SIFT keypoints detection using SIFT algorithm built in OpenCV library.

         

However, due to the well-known patent issues, SIFT and SURF algorithms are categorized into nonfree module and not included in the release package of OpenCV for Android. There are a couple of ways to build nonfree module for Android native project. If you search online, you can also find tutorials showing you how to build your own OpenCV for Android. But, we don't want to spent a whole day or so to rebuild the whole OpenCV library just because we need one missing module. This tutorial will show you a quick way to build your nonfree module (SIFT and SURF) in OpenCV for Android NDK project. You can follow this tutorial to set up your own project, or you can download the attached package for a quick start.

My development environment is set up as follows:
- cygwin (I am using Windows 7)
- android-ndk-r8c (install path: D:\Android_dev\android-ndk-r8c)
- OpenCV-2.4.5-android-sdk (install path: D:\CV_dev\OpenCV-2.4.5-android-sdk)
- OpenCV 2.4.5 (install path: D:\CV_dev\opencv2.4.5)
We actually only need to copy a few files from OpenCV source code to OpenCV-android-sdk. The need files will be included in the attachment, so you don't need to install OpenCV 2.4.5 package if you don't need other functions from the full version OpenCV source code. A ready-to-go example NDK project package are attached in the attachment for your convenience. Feel free to modify it and use it in your own project. 

How to build?

OK. Let's start! (BTW, I will explain how you can do it from scratch. If you use the package I provided, you can skip all the steps here. ) We will first build libnonfree.so, and then show how to use it to build an Android native application.
Step 1: create project, prepare files needed. 
Copy the nonfree folder from opencv2.4.5\modules\nonfree\include\opencv2\ to OpenCV-2.4.5-android-sdk\sdk\native\jni\include\opencv2\. We only need two files: nonfree.hpp and features2d.hpp; delete other unused files. 

Create a folder to hold our new project for libnonfree.so. Here, I call it nonfree_opencv_android. Create  jni folder under nonfree_opencv_androidCopy the following files from opencv2.4.5\modules\nonfree\src to jni folder:
nonfree_init.cpp, precomp.cpp (since OpenCV-2.4.8, no such file; it's OK, we don't need it), precomp.hpp, sift.cpp, surf.cpp

Step 2, build libnonfree.so
create Android.mk and Application.mk scripts. This Android.mk is used to build libnonfree.so, which includes implementation of both SIFT and SURF.

Aplication.mk

APP_ABI := armeabi 
#APP_ABI += armeabi-v7a # you can do either armeabi or armeabi-v7a, steps are the same.
APP_STL := gnustl_static
APP_CPPFLAGS := -frtti -fexceptions
APP_PLATFORM := android-15

Android.mk

LOCAL_PATH:= $(call my-dir)
include $(CLEAR_VARS)
OPENCV_INSTALL_MODULES:=on
OPENCV_CAMERA_MODULES:=off
include /cygdrive/d/CV_dev/OpenCV-2.4.5-android-sdk/sdk/native/jni/OpenCV.mk

LOCAL_C_INCLUDES:= /cygdrive/d/CV_dev/OpenCV-2.4.5-android-sdk/sdk/native/jni/include
LOCAL_MODULE    := nonfree
LOCAL_CFLAGS    := -Werror -O3 -ffast-math
LOCAL_LDLIBS    += -llog

# for 2.4.8, delete the line precomp.cpp \
LOCAL_SRC_FILES := nonfree_init.cpp \
precomp.cpp \
sift.cpp \
surf.cpp
include $(BUILD_SHARED_LIBRARY)
Note: The paths to OpenCV.mk and LOCAL_C_INCLUDES are set according to my installation paths. Please modify based on your setup.

Start cygwin, cd into the project folder, nonfree_opencv_android in my case. Type ndk-build to build the libnonfree.so
Since we are creating a customized library out of OpenCV library, so you need to modify the a few lines in the header files and the cpp files. you can easily fix them by checking the compile error messages.

Since many friends have issues with these fixes. I provide the fix steps here for OpenCV-2.4.8:
 1. In jni/precomp.hpp, delete "#include "cvconfig.h"
 2. in jni/precomp.hpp, delete (or comment out) 
    #  include "opencv2/nonfree/ocl.hpp"
    #  include "opencv2/ocl/private/util.hpp"
 3. Give access permissions to libopencv_java.so under obj/local/armeabi
    chmod 777 libopencv_java.so
 4. There might also be some errors caused by the lack of reading permission of a few .a static library copied to obj folder. You can fix them by simply using chmod to give the files the right permission.

$ cd sift_opencv_android
$ ndk-build
Cygwin         : Generating dependency file converter script
Compile++ thumb  : nonfree <= nonfree_init.cpp
Compile++ thumb  : nonfree <= precomp.cpp
Compile++ thumb  : nonfree <= sift.cpp
Compile++ thumb  : nonfree <= surf.cpp
Prebuilt       : libgnustl_static.a <= <NDK>/sources/cxx-stl/gnu-libstdc++/4.6/libs/armeabi/
Prebuilt       : libopencv_java.so <= /cygdrive/d/CV_dev/OpenCV-2.4.5-android-sdk/sdk/native/jni/../libs/armeabi/
SharedLibrary  : libnonfree.so
d:/android_dev/android-ndk-r8c/toolchains/arm-linux-androideabi-4.6/prebuilt/windows/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld.exe: error: cannot open ./obj/local/armeabi/libgnustl_static.a: Permission denied
[a lot of similar errors...I ignored them here.]

$ cd obj/local/armeabi
$ chmod 777 libgnustl_static.a 
$ chmod 777 libopencv_java.so
$ cd ../../../
$ ndk-build
SharedLibrary  : libnonfree.so
Install        : libnonfree.so => libs/armeabi/libnonfree.so
Install        : libopencv_java.so => libs/armeabi/libopencv_java.so
So far, you have got libnonfree.so along with libopencv_java.so in nonfree_opencv_android/libs/armeabi folder. You can easily build any SIFT or SURF applications using those libraries. If you want to use SIFT and SURF in JAVA code in your Android application, you only need to write JNI interfaces for the functions you want to use.

[Updated 2/26/2014]
Since many friends asked questions about building Android applications using JNI interface, I wrote another tutorial (you can consider it as the Part 2 of this tutorial) to show how to build an Android application using the OpenCV nonfree module. You can read it from herehttps://sites.google.com/site/wghsite/technical-notes/opencv_nonfree_android_jni_demo

Step 3: build SIFT test program using libnonfree.so and libopencv_java.so
Create a project folder call sift_opencv_android. Create a jni folder inside the project folder. Then copy libnonfree.so and libopencv_java.so into jni. (Of course, you can also leave these two .so files where they are, but you have to make sure you set the library path correctly.)
Create a sift_test.cpp in jniI wrote a very simple SIFT test program. It basically reads an image and detects the keypoints, then extracts feature descriptors, finally draws the keypoints to an output image.

test_sift.cpp

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/nonfree/features2d.hpp>
#include <opencv2/nonfree/nonfree.hpp>
#include <iostream>

using namespace cv;
using namespace std;

int main( int argc, char** argv )
{
if( argc != 3)
{
cout <<" Usage: sift input_image output_image" << endl;
return -1;
}
//cv::initModule_nonfree();
//cout <<"initModule_nonfree() called" << endl;

Mat image;
image = imread(argv[1], CV_LOAD_IMAGE_COLOR); 
if(! image.data )
{
cout <<  "Could not open or find the image" << std::endl ;
return -1;
}

vector<KeyPoint> keypoints;
Mat descriptors;

// Create a SIFT keypoint detector.
SiftFeatureDetector detector;
detector.detect(image, keypoints);
cout << "Detected " << (int) keypoints.size() << " keypoints" <<endl;

// Compute feature description.
detector.compute(image,keypoints, descriptors);
cout << "Computed feature."<<endl;

// Store description to "descriptors.des".
FileStorage fs;
fs.open("descriptors.des", FileStorage::WRITE);
cout << "Opened file to store the features."<<endl;
fs << "descriptors" << descriptors;
cout << "Finished writing file."<<endl;
fs.release();
cout << "Released file."<<endl;

// Show keypoints in the output image.
Mat outputImg;
Scalar keypointColor = Scalar(255, 0, 0);
drawKeypoints(image, keypoints, outputImg, keypointColor, DrawMatchesFlags::DRAW_RICH_KEYPOINTS);
cout << "Drew keypoints in output image file."<<endl;

#ifdef WIN32
namedWindow("Output image", CV_WINDOW_AUTOSIZE );
imshow("Output image", outputImg);
waitKey(0);
#endif
cout << "Generate the output image."<<endl;
imwrite(argv[2], outputImg);

cout << "Done."<<endl;
return 0;
}

Create Android.mk and Application.mk inside jni.

Application.mk

APP_ABI := armeabi 
#APP_ABI += armeabi-v7a
APP_STL := gnustl_static
APP_CPPFLAGS := -frtti -fexceptions
APP_PLATFORM := android-15

Android.mk

include $(CLEAR_VARS)
LOCAL_MODULE    := sift_prebuilt
LOCAL_SRC_FILES := libnonfree.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_MODULE    := opencv_java_prebuilt
LOCAL_SRC_FILES := libopencv_java.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_C_INCLUDES:= /cygdrive/d/CV_dev/OpenCV-2.4.5-android-sdk/sdk/native/jni/include
LOCAL_MODULE    := test_sift
LOCAL_CFLAGS    := -Werror -O3 -ffast-math
LOCAL_LDLIBS    += -llog -ldl 
LOCAL_SHARED_LIBRARIES := sift_prebuilt opencv_java_prebuilt
LOCAL_SRC_FILES := test_sift.cpp
include $(BUILD_EXECUTABLE)

Then use cygwin, enter sift_opencv_android folder. Use ndk-build to compile the code. You will meet "permission denied" errors again. Fix them using chmod. You finally will get executable file test_sift in libs folder.
$ ndk-build
Install        : libopencv_java.so => libs/armeabi/libopencv_java.so
Install        : libnonfree.so => libs/armeabi/libnonfree.so
Executable     : test_sift
Install        : test_sift => libs/armeabi/test_sift

Step 4: test your program on an Android device.
Use "adb push" tools to upload test_sift, libnonfree.so, libopencv_java.so along with a test image to your device. Use "adb shell" login your device. Before you run your program, make sure you add the current folder containing our two libraries to LD_LIBRARY_PATH. The following is a bash script to help you run your program. On my device, I did the experiment in /data/local/sift directory, if this directory doesn't work for you, choose other directories that you have permission to read/write.

run.sh

echo "push the executable into the device"
adb shell "rm -r /data/local/sift"
adb shell "mkdir -p /data/local/sift"
adb push ./libs/armeabi /data/local/sift/
adb push ./img/img1.jpg /data/local/sift/

echo "start SIFT computation"
echo "                        "
adb shell "cd /data/local/sift && LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data/local/sift&& chmod 777 ./test_sift &&./test_sift img1.jpg img1_result_output.jpg"
echo "                        "

echo "fetch the result image back."
adb pull /data/local/sift/img1_result_output.jpg ./img
We are almost done! If you follow the above steps, I believe you have already successfully run the SIFT program on an Android device. 

Conclusion
There are also other alternative ways to build your own project with nonfree module, such as you can have a standalone but fat application, or you can have an application with a single libnonfree.so that contains libopencv_java.so inside, and so on. They can be built using the similar way as mentioned above, and you only need to modify your Android.mk script. I, personally, suggest the one with  app+libnonfree.so+libopencv_java.so, since this one provides you the best flexibility and generates a small application. 

By compiling our own nonfree module, we are able to modify SIFT or SURF source code whatever you want, for example, to modify the parameters used in SIFT detector (those are predefined as static variables in sift.cpp). You can create a more flexible interface to call SIFT/SURF functions. Moreover, if you want to integrate some new detectors or feature descriptors into feature detection library, you can easily insert your functions into this nonfree module and reuse some of the existing function interfaces. 

About the attachments:
1. The package is created for your reference. It is your responsibility to use it in a legal and correct way.
2. You can download the Android NDK project package I created from the attachments. The path variables are set based on my configuration. Before you use it, please modify the Android.mk script and make sure the OpenCV path is set correctly according to your setup. Then, you are good to go. I also provided a script for you to easily test the program. 
3. A libnonfree.so and a libopencv_java.so are also provided for your reference, so that you can directly test your application with SIFT/SURF. But again, you use them at your own risks.



Disqus comments