Tutorial on Using Machine Learning on Android with TFLite

Richards
By -Richards
8:18 PM
0

Related Post

Hello, back again with the Machine Learning Series on Android. In the previous article, we discussed various ways to implement machine learning/AI in Android applications. In this blog, we will practice directly one of the frameworks mentioned, namely TensorFlow Lite (TFLite).

TensorFlow Lite is a lightweight and efficient version of the Tensorflow framework that Machine Learning Developers often use to develop and deploy models. It is designed in such a way that it allows us to run the model on devices with limited resources, such as mobile phones and embedded systems.

Some examples of applications that use TensorFlow Lite are Gmail, Google Assistant, Google Nest, and Shazam. Of course the names are familiar, right?

For those who are used to being involved in the Android world and don't want to miss out on the hype of AI technology, you certainly don't just want to be a user, right? Of course you want to know how to implement machine learning on Android.

So, in this tutorial, we will create an object classification application using TFLite. Interestingly, you don't need to know a deep understanding of machine learning to follow this tutorial. Curious? Come on, take a look at the steps and create your first AI application!


Project Preparation


Tutorial on Using Machine Learning on Android with TFLite

So that you can focus on practical implementation of TFLite, we have provided a starter project that contains basic camera functions using CameraX, a library from Android Jetpack to display the camera directly in the application more easily.

In fact, there are many ways to get images in the application, such as selecting an image from the gallery or utilizing the built-in camera using Camera Intent. However, here we use CameraX so we can analyze images in real-time. Apart from that, in CameraX there is also ImageAnalysis which we can integrate with TFLite.

Okay, now please directly download or clone the following initial project:


Please open the project using Android Studio. If we pay attention, there are several additional codes in it as follows.

  • CameraX library dependencies in build.gradle.kts.
  • PreviewView from CameraX and TextView in activity_main.xml.
  • Defining Camera permissions in AndroidManifest.xml.
  • The process of requesting Camera permission, initializing the ProcessCameraProvider to display the camera, and implementing ViewBinding in the MainActivity.

The following is how the application from the starter project looks when it is run.


After granting camera access, the app can display the camera directly.

TFLite Model Preparation


If you already have a TensorFlow Model, first convert it to a TensorFlow Lite Model with the .tflite extension. As a result, it is smaller, more efficient, and supports the TensorFlow Lite framework.

If you don't have your own model, you can use TensorFlow Lite Model Maker to carry out transfer learning. Transfer learning is an approach to using previously trained models to perform certain tasks. This is used as a starting point to create a new model with a similar task.

If you don't have a previous machine learning background, you can also use TensorFlow Hub to look for patented model references. So, in this exercise we will use the following model to perform object detection.


Please download the model and import it into Android Studio by right clicking on the app folder and selecting New → Other → TensorFlow Lite Model. From here, you will know that Android Studio is actually good friends with TFLite. The proof is that there is a direct import feature in Android Studio.

Tutorial on Using Machine Learning on Android with TFLite


Then, select the file that was downloaded in the Model location and tick all the options.

Tutorial on Using Machine Learning on Android with TFLite

Click Finish and see now the file is in the app/ml folder. You can also read the metadata of the file thanks to the previously added library.

Tutorial on Using Machine Learning on Android with TFLite

This file can later be accessed like an asset by adding configuration to build.gradle.kts.

android {
   ...
   buildFeatures {
       viewBinding = true
       mlModelBinding = true
   }
}

OK, now that the model is ready, it's time to add the main library from TensorFlow Lite for machine learning processing. Since the object classification falls into the task-vision category, add the following dependency to build.gradle.kts.

dependencies {
   ...
   implementation("org.tensorflow:tensorflow-lite-gpu:2.13.0")
   implementation("org.tensorflow:tensorflow-lite-task-vision:0.4.4")
}

Don't forget to do Sync Now every time you add a new library to the Gradle configuration to download it.


Create TFLite Helper


After all the preparations are complete, we will create a helper class to make it easier for us to group the functions used to carry out the ML process. Additionally, the code becomes neater and cleaner with this separation.

First, create a new class by right-clicking on the package name and selecting new → Kotlin Class/File. Then add parameters and listeners as follows.

class ImageClassifierHelper(
   var threshold: Float = 0.1f,
   var maxResults: Int = 3,
   var numThreads: Int = 4,
   val modelName: String = "mobilenet_v1_1.0_224_quantized_1_metadata_1.tflite",
   val context: Context,
   val imageClassifierListener: ClassifierListener?
) {
   private var imageClassifier: ImageClassifier? = null
 
   interface ClassifierListener {
       fun onError(error: String)
       fun onResults(
           results: List<Classifications>?,
           inferenceTime: Long
       )
   }
}

The listener here functions to notify the main class when the process is successful or failed. If the process is successful, the onResults function is called. However if the process fails, the onError function is called. This is what is called a callback.

Next, create a new function called setupImageClassifier and call it inside init so that it is called every time this class is created.

init {
   setupImageClassifier()
}
 
fun setupImageClassifier() {
   val optionsBuilder = ImageClassifier.ImageClassifierOptions.builder()
       .setScoreThreshold(threshold)
       .setMaxResults(maxResults)
 
   val baseOptionsBuilder = BaseOptions.builder().setNumThreads(numThreads)
   optionsBuilder.setBaseOptions(baseOptionsBuilder.build())
 
   try {
       imageClassifier =
           ImageClassifier.createFromFileAndOptions(context, modelName, optionsBuilder.build())
   } catch (e: IllegalStateException) {
       imageClassifierListener?.onError(
           "Image classifier failed to initialize. See error logs for details"
       )
       Log.e("ImageClassifierHelper", "TFLite failed to load model with error: " + e.message)
   }
}

Here we set up some configurations on TensorFlow Lite before processing. The following is the function used.

  • setScoreThreshold: Determines the minimum threshold for the accuracy of the results displayed. 0.1 means 10%.
  • setMaxResults: Determines the maximum limit on the number of results displayed.
  • setNumThreads: Specifies the number of threads used to perform ML processing.
  • createFromFileAndOptions: Creates an ImageClassifier based on previously defined model file assets and options.

Notes:

Actually, there is also a baseOptionsBuilder.useGpu() function if you want to determine that the processing is done on the GPU, instead of the CPU. However, you need to add the tensorflow-lite-gpu library.

Let's continue writing the code. Still in the ImageClassifierHelper file, create a new function to perform classification processing as follows.

fun classify(image: ImageProxy) {
   if (imageClassifier == null) {
       setupImageClassifier()
   }
   val bitmapBuffer = Bitmap.createBitmap(
       image.width,
       image.height,
       Bitmap.Config.ARGB_8888
   )
   image.use { bitmapBuffer.copyPixelsFromBuffer(image.planes[0].buffer) }
   image.close()
 
   var inferenceTime = SystemClock.uptimeMillis()
   val imageProcessor = ImageProcessor.Builder().build()
   val tensorImage = imageProcessor.process(TensorImage.fromBitmap(bitmapBuffer))
 
   val imageProcessingOptions = ImageProcessingOptions.builder()
       .setOrientation(getOrientationFromRotation(image.imageInfo.rotationDegrees))
       .build()
 
   val results = imageClassifier?.classify(tensorImage, imageProcessingOptions)
   inferenceTime = SystemClock.uptimeMillis() - inferenceTime
   imageClassifierListener?.onResults(
       results,
       inferenceTime
   )
}
private fun getOrientationFromRotation(rotation: Int): ImageProcessingOptions.Orientation {
   return when (rotation) {
       Surface.ROTATION_270 -> ImageProcessingOptions.Orientation.BOTTOM_RIGHT
       Surface.ROTATION_180 -> ImageProcessingOptions.Orientation.RIGHT_BOTTOM
       Surface.ROTATION_90 -> ImageProcessingOptions.Orientation.TOP_LEFT
       else -> ImageProcessingOptions.Orientation.RIGHT_TOP
   }
}

At the beginning, we first check whether the ImageClassifier instance is still null or not. If not, we continue to change the image buffer to Bitmap with the createBitmap function. After that, create a TensorImage from the Bitmap using the process function.

After that, we also prepare ImageProcessingOptions to set the orientation of the input image according to the image in the model. This is necessary so that the results provided are accurate.

As a note, because the function provided is setOrientation, we need to create a new function to convert rotationDegress to orientation as in the example above.

Once both objects are ready, you can call the classify function to start the classification process. Apart from that, you also create an inferenceTime variable to calculate the time needed for processing.

Finally, we save the resulting data via the imageClassifierListener?.onResults function.

As a result, the entire code in ImageClassifierHelper is as follows.

class ImageClassifierHelper(
   var threshold: Float = 0.1f,
   var maxResults: Int = 3,
   var numThreads: Int = 4,
   val modelName: String = "mobilenet_v1_1.0_224_quantized_1_metadata_1.tflite",
   val context: Context,
   val imageClassifierListener: ClassifierListener?
) {
   private var imageClassifier: ImageClassifier? = null
 
   init {
       setupImageClassifier()
   }
 
   fun setupImageClassifier() {
       val optionsBuilder = ImageClassifier.ImageClassifierOptions.builder()
           .setScoreThreshold(threshold)
           .setMaxResults(maxResults)
       val baseOptionsBuilder = BaseOptions.builder().setNumThreads(numThreads)
       optionsBuilder.setBaseOptions(baseOptionsBuilder.build())
       try {
           imageClassifier =
               ImageClassifier.createFromFileAndOptions(context, modelName, optionsBuilder.build())
       } catch (e: IllegalStateException) {
           imageClassifierListener?.onError(
               "Image classifier failed to initialize. See error logs for details"
           )
           Log.e("ImageClassifierHelper", "TFLite failed to load model with error: " + e.message)
       }
   }
 
   fun classify(image: ImageProxy) {
       if (imageClassifier == null) {
           setupImageClassifier()
       }
       val bitmapBuffer = Bitmap.createBitmap(
           image.width,
           image.height,
           Bitmap.Config.ARGB_8888
       )
       image.use { bitmapBuffer.copyPixelsFromBuffer(image.planes[0].buffer) }
       image.close()
 
       var inferenceTime = SystemClock.uptimeMillis()
       val imageProcessor = ImageProcessor.Builder().build()
       val tensorImage = imageProcessor.process(TensorImage.fromBitmap(bitmapBuffer))
 
       val imageProcessingOptions = ImageProcessingOptions.builder()
           .setOrientation(getOrientationFromRotation(image.imageInfo.rotationDegrees))
           .build()
 
       val results = imageClassifier?.classify(tensorImage, imageProcessingOptions)
       inferenceTime = SystemClock.uptimeMillis() - inferenceTime
       imageClassifierListener?.onResults(
           results,
           inferenceTime
       )
   }
   private fun getOrientationFromRotation(rotation: Int): ImageProcessingOptions.Orientation {
       return when (rotation) {
           Surface.ROTATION_270 -> ImageProcessingOptions.Orientation.BOTTOM_RIGHT
           Surface.ROTATION_180 -> ImageProcessingOptions.Orientation.RIGHT_BOTTOM
           Surface.ROTATION_90 -> ImageProcessingOptions.Orientation.TOP_LEFT
           else -> ImageProcessingOptions.Orientation.RIGHT_TOP
       }
   }
   interface ClassifierListener {
       fun onError(error: String)
       fun onResults(
           results: List<Classifications>?,
           inferenceTime: Long
       )
   }
}

CameraX integration with TFLite


Once the helper class is ready, we will integrate ImageClassifier with CameraX by using ImageAnalysis as a UseCase in the last argument of the bindToLifecycle function.

class MainActivity : AppCompatActivity() {
 
   private lateinit var binding: ActivityMainBinding
   private lateinit var imageClassifierHelper: ImageClassifierHelper
 
   ...
 
   private fun startCamera() {
       imageClassifierHelper =
           ImageClassifierHelper(
               context = this,
               imageClassifierListener = object : ImageClassifierHelper.ClassifierListener {
                   override fun onError(error: String) {
                       runOnUiThread {
                           Toast.makeText(this@MainActivity, error, Toast.LENGTH_SHORT).show()
                       }
                   }
                   override fun onResults(results: List<Classifications>?, inferenceTime: Long) {
                       runOnUiThread {
                           results?.let { it ->
                               if (it.isNotEmpty() && it[0].categories.isNotEmpty()) {
                                   println(it)
                                   val sortedCategories =
                                       it[0].categories.sortedByDescending { it?.score }
                                   val displayResult =
                                       sortedCategories.joinToString("\n") {
                                           "${it.label} " + NumberFormat.getPercentInstance().format(it.score).trim()
                                       }
                                   binding.tvResult.text = displayResult
                               }
                           }
                       }
                   }
               })
       val cameraProviderFuture = ProcessCameraProvider.getInstance(this)
       cameraProviderFuture.addListener({
           val cameraProvider: ProcessCameraProvider = cameraProviderFuture.get()
           val preview = Preview.Builder()
               .setTargetAspectRatio(AspectRatio.RATIO_4_3)
               .build()
               .also {
                   it.setSurfaceProvider(binding.viewFinder.surfaceProvider)
               }
           val imageAnalyzer =
               ImageAnalysis.Builder()
                   .setTargetAspectRatio(AspectRatio.RATIO_4_3)
                   .setTargetRotation(binding.viewFinder.display.rotation)
                   .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
                   .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
                   .build()
                   .also {
                       it.setAnalyzer(Executors.newSingleThreadExecutor()) { image ->
                           imageClassifierHelper.classify(image)
                       }
                   }
           try {
               cameraProvider.unbindAll()
               cameraProvider.bindToLifecycle(
                   this,
                   CameraSelector.DEFAULT_BACK_CAMERA,
                   preview,
                   imageAnalyzer
               )
           } catch (exc: Exception) {
               Toast.makeText(this, "Gagal memunculkan kamera.", Toast.LENGTH_SHORT).show()
           }
       }, ContextCompat.getMainExecutor(this))
   }
}

First, we initialize the ImageClassifierHelper class with Context and ClassifierListener arguments. For other reasons, you can change it or create settings for these settings if you want. However, in this exercise we use the default argument to keep it simple.

In the ClassifierListener callback, we display a Toast when an error occurs. Meanwhile, if successful, we display the results in a TextView. Don't forget that the data displayed is sorted first from the highest score (read: accuracy) and converted into a percentage.

Don't forget to use the runOnUiThread block because basically we cannot update the UI from the background thread. For this reason, it needs to be converted into a UI thread first using this block.

Then when initializing ImageAnalysis, you set the following things.

  • setTargetAspectRatio: Sets the target aspect ratio of the image. In this case, the aspect ratio is set as 4:3 to match the TFLite model.
  • setTargetRotation: Sets the rotation of the image target based on the view rotation associated with the viewFinder. This ensures that the analyzed image matches the screen orientation.
  • setBackpressureStrategy: Sets a strategy to overcome delay problems in image processing. STRATEGY_KEEP_ONLY_LATEST allows analysis to use only the most recent image if previous images have not yet finished processing.
  • setOutputImageFormat: Sets the output image format produced by ImageAnalysis to RGBA 8888. This is an image format that contains red, green, blue, and alpha color components with 8 bits per component.
  • setAnalyzer: Here, we set up image analysis by running a single executor that will run a lambda function that will accept an image as input and then call the classify method of the imageClassifierHelper object to classify the image.

Once ImageAnalysis is ready, attach the object to the fourth argument of the bindToLifecycle function, which is a feature for adding certain use cases to CameraX, such as the image analysis above.

OK, the application for analysis is ready. Let's run the application on the device and see the results.

With code that is not too complex, you can create your own application that applies machine learning for object classification. It can be seen that the resulting data is three in number and the threshold is above 10% as defined in the configuration.

To see the finished results of the project, you can see it at the following link. 


If you follow this tutorial seriously, you will have the basic ability to apply TensorFlow Lite for all types of processing. Because basically all types of vision processing have the same basis. So, if you want to change it to text detection or object detection, it won't take long.


If you want to learn more about CameraX or the basics of creating other applications, you can also study in the Android Intermediate Application Development Learning class.

The hope is that by having this basis, you can create various kinds of applications as solutions to problems around you by utilizing AI. If you have any application ideas, please write them in the comments column, OK! Get excited and make your AI application a reality!!
Tags:

You Might Like

Show more
Tutorials

Post a Comment

0Comments

Post a Comment (0)

#buttons=(Ok, Go it!) #days=(20)

Our website uses cookies to enhance your experience. Learn more
Ok, Go it!
Share to other apps
Copy Post Link