Developer Guide

Tutorial: CraftAR On-device Image Recognition for iOS

icon Date Developer Guide

icon Date Dec 22, 2020

icon author Catchoom Team

icon folder

This article applies only to the On-device Image Recognition SDK.

In order to add on-device collections to your iOS app, it is necessary to:

  1. create an on-device collection in CraftAR;
  2. generate a collection bundle for your SDK version (On-device Image Recognition SDK or Augmented Reality SDK v4 +;
  3. download the bundle of that collection from CraftAR and add it to the app;
  4. you are ready to go and start using the on-device collection with the SDK

Once the collection is added to the device, you can start using the on-device Image Recognition SDK to perform visual search queries on the device. You can find more details for the first step in the tutorial about how to create an on-device collection in CraftAR. And for the second and third steps, please read the tutorial about how to Manage on-device collections for the iOS SDKs.

An Image Recognition app using the native On-device iOS Image Recognition SDK can be implemented by following three steps: first, set up the  UIVIewController, then load the collection for on-device Image Recognition and, finally run image recognition to get the results for each item that is recognized.

If you want to see an example that implements On-device Image Recognition, take a look at the open source samples available in our Github repository

1. Set up the SDK in your  UIViewController

Adopt  CraftARSDKProtocol and  SearchProtocol in your  UIViewController

Set up the camera capture

The  CraftARSDK_IR class manages the camera capture. Once the view is loaded, get the instance of the  CraftARSDK_IR and start the camera capture with a  UIView to show the video feed from the camera.

Loading collections into memory

Once the collection is added to the local database, we can set the collection that will be used for on-device image recognition using the  CraftAROnDeviceIR instance. Loading collections can take a few seconds which depend on the amount of images. The SDK provides progress feedback for this process as well.

Prepare the SDK to process searches

The  CraftARSDK_IR class also manages the search processes by sending search events to the  searchControllerDelegate, in this case the  CraftAROnDeviceIR. The  CraftAROnDeviceIR class performs visual searches in the collection of images that is previously set for this app. You can find more details about setting the bundles in the separate tutorial about how to manage collection bundles with the On-device Image Recognition SDK.

Once the  VideoCapture is ready, it performs a callback to  didStartCapture. We suggest to set the  CraftAROnDeviceIR's search controller instance as the  searchControllerDelegate, and the  UIViewController as the delegate of the  CraftAROnDeviceIR to receive the responses from the search process.

2. Implementing the searches and parsing the results

Once the ViewController has the necessary protocols and instance of  CraftAROnDeviceIR, it’s time to add code to start scanning the real world.

Using the  CraftARSDK_IR class, we can search the on-device image database in two modes: Finder Mode or Single Shot Mode.

Option A. Use Single Shot Mode to take a single picture

Call  singleShotSearch to perform a search with a single image. By calling this method, the SDK triggers the still image capture from the camera and forwards the captured image to the  CraftAROnDeviceIR.

A common approach consists in triggering the search when the user performs an action on the UI:

The response to a query produces a callback to  didGetSearchResults with the results available in an array that is ready to parse.

Option B. Use Finder Mode for continuous scanning

Call  startFinder to start searching continuously without user intervention. This method forwards the camera frames to the  CraftAROnDeviceIR instance.

For every frame that is processed, the SDK generates a query and searches for matches in the local collection.

The response to a query produces a callback to  didGetSearchResults with the results available in an array that is ready to parse.

Related Posts in Developer Guide