Developer Guide

Tutorial: how to use CraftAR On-Device Image Recognition for Android

icon Date Developer Guide

icon Date Dec 22, 2020

icon author Catchoom Team

icon folder

This article applies only to the On-Device Image Recognition SDK.

In order to add on-device collections to your Android app, it is necessary to:

  1. create an on-device collection in CraftAR;
  2. generate a collection bundle for your SDK version (On-device Image Recognition SDK or Augmented Reality SDK v4 +;
  3. download the bundle of that collection from CraftAR and add it to the app;
  4. you are ready to go and start using the on-device collection with the SDK

Once the collection is added to the device, you can start using the on-device Image Recognition SDK to perform visual search queries on the device. You can find more details for the first step in the tutorial about how to create an on-device collection in CraftAR. And for the second and third steps, please read the tutorial about how to Manage on-device collections for the Android SDKs.

An Image Recognition app using the native On-device Android Image Recognition SDK can be implemented by following two steps. First, set up the Activity/Fragment and then run image recognition to get the results for each item that is recognized.

If you want to see an example that implements On-device Image Recognition, take a look at the open source samples available in our Github repository.

1. Set up the SDK in your app

Loading collections into memory

Once the collection is added to the local database, we can set the collection that will be used for on-device image recognition using the  CraftAROnDeviceIR instance. Loading collections can take a few seconds, depending on the amount of images to load. The SDK provides progress feedback for this process as well.

To load the collection into the device memory we will use the  setCollection() method. This call is also asynchronous. To receive the callbacks, you will need an  SetCollectionListener . You can implement these listeners in your android  Activity.

Note that this could also be done in the same CraftARActivity where you perform the recognition process.

Extending CraftARActivity and implementing CraftARSearchResponseHandler

The activity where the image recognition is running must extend from  CraftARActivity. If you want to place the Camera view in a Fragment, you can do it, but ensure that its parent activity extends  CraftARActivity. (Check the examples in github to see how to implement the camera view inside a fragment).

The image recognition requests are asynchronous. To receive the search responses, you need a  CraftARSearchResponseHandler. In this case, and for simplicity, we will implement the interface in the same activity.

Set up the camera capture

The  CraftARSDK class manages the camera capture. Once the view is loaded, get the instance of the  CraftARSDK and start the camera capture.

If the camera initialization fails, the method  onCameraOpenFailed() will trigger. This can happen for multiple reasons, but it basically means that the camera could not be accessed. Sometimes, restarting the camera works. Sometimes you will have to restart the device. Note that this can also happen if another application is using the camera (for example, the flashlight), or if you haven’t added the necessary permissions in the AndroidManifest

Prepare the SDK to process searches

The  CraftARSDK class also manages the search processes by sending search events to the SearchController, in this case the  CraftAROnDeviceIR instance. The  CraftAROnDeviceIR instance performs visual searches in the collection of images that is previously set for this app. You can find more details about setting the bundles in the separate tutorial about how to manage collection bundles with the On-device Image Recognition SDK.

We suggest to set the  CraftAROnDeviceIR instance as the  searchController. The  searchController is the object that will manage the  singleShotSearch()startFinder() and  stopFinder() calls of the  CraftARSDK. We will also set our Activity as the  CraftARSearchResposneHandler to receive the responses from the search process.

2. Implementing the searches and parsing the results

Once you are ready with the previous steps, it’s time to add code to start scanning the real world.

Using the  CraftARSDK class we can search the on-device image database in two modes: Finder Mode or Single Shot Mode.

Option A. Use Single Shot Mode to take a single picture

Call  singleShotSearch to perform a search with a single image. By calling this method, the SDK triggers the still image capture from the camera and forwards the captured image to the search controller (in this case, the  CraftAROnDeviceIR instance)

A common approach consists in triggering the search when the user performs an action on the UI (for example, clicking on a button). Ensure that you have initialized everything before allowing the user to perform the singleShotSearch, otherwise it will fail.

The response to a query triggers the searchResults callback with the results available in an array that is ready to parse. If the query failed, the searchFailed() callback will be triggered. A query can fail for multiple reasons, but the most common one is when the query image does not have enough details. If you point to a completely uniform surface (i.e a white wall), you will receive this error.

Option B. Use Finder Mode for continuous scanning

Call  startFinder to start searching continuously without user intervention. This method forwards the camera frames to the search controller (in this case, the  CraftAROnDeviceIR instance)

For every frame that is processed, the SDK generates a query and searches for matches in the local collection. The response to a query produces a callback to  searchResults with the results available in an array that is ready to parse.

Related Posts in Developer Guide