Speed ​​up Tensorflow lite using your android device’s GPU

If you want to use AI, you have to use GPU.

When using an AI model on Android, by default it will run only on the CPU.
If your device has a GPU (Graphics Processing Unit), you can significantly speed up processing by using the GPU .

For example, when running Yolov8 on a Google Pixel 7a, it runs in
0.9 seconds on the CPU
0.2 seconds on the GPU

GPUDelegate

To use the device’s GPU, set TensorFlowLite’s GPUDelegate
. Just add GPUDelegate to the options when initializing the TensorFlowLite Interpreter.

(By the way, there is also NNAPIDelegate (NeuralNetworkAPI), but this was no different from the CPU in my environment (Google Pixel 7a).)

Method

Installing the required libraries

build.gradle

implementation("org.tensorflow:tensorflow-lite:+")
implementation("org.tensorflow:tensorflow-lite-gpu:+")
implementation("org.tensorflow:tensorflow-lite-gpu-api:+")
implementation("org.tensorflow:tensorflow-lite-gpu-delegate-plugin:+")
implementation("org.tensorflow:tensorflow-lite-support:+")

It seems that the four libraries from the top may crash if the versions do not match.
If you set it to :+, the latest version will be installed, so it’s OK.

GPUDelegate settings

module import

import org.tensorflow.lite.Interpreter
import org.tensorflow.lite.gpu.CompatibilityList
import org.tensorflow.lite.gpu.GpuDelegate

Delegate settings.

Set GPUDelegate in the model’s Interpreter options.

val compatList = CompatibilityList()

val options = Interpreter.Options().apply{
    if(compatList.isDelegateSupportedOnThisDevice){
        // if the device has a supported GPU, add the GPU delegate
        val delegateOptions = compatList.bestOptionsForThisDevice
        this.addDelegate(GpuDelegate(delegateOptions))
    } else {
        // if the GPU is not supported, run on 4 threads
        this.setNumThreads(4)
    }
}

interpreter = Interpreter(model, options)

The model will now run using the GPU if your device has one available.

🐣

I’m a freelance engineer.
Work consultation
Please feel free to contact us with a brief development description.
rockyshikoku@gmail.com

I am creating applications using machine learning and AR technology.

I send machine learning / AR related information.

GitHub

Twitter
Medium