Performance optimization

Optimize Video Mobile SDK performance with GPU acceleration, frame skipping, and memory management.

This guide explains performance optimization strategies for the Video Mobile SDK, including hardware acceleration, frame rate management, and memory efficiency.

GPU vs CPU processing

The SDK supports both GPU and CPU inference. GPU processing is significantly faster on most devices.

Configure Android GPU

Configure your app to use GPU:

import photos.eyeq.dynamic.DynamicProcessor

// GPU acceleration (recommended)
App.dynamicProcessor.init(useGpu = true)

// CPU fallback
App.dynamicProcessor.init(useGpu = false)

Check GPU support (Android)

Before enabling GPU, verify device support:

import photos.eyeq.dynamic.DynamicProcessor

fun initializeWithFallback() {
    val gpuSupported = App.dynamicProcessor.isDelegateSupported()
    
    if (gpuSupported) {
        App.dynamicProcessor.init(useGpu = true)
    } else {
        // Fall back to CPU
        App.dynamicProcessor.init(useGpu = false)
    }
}

Available delegates (Android)

Use delegates where necessary in Android:

Delegate	Description	Best for
`GPU`	OpenGL ES GPU acceleration	Most devices
`NNAPI`	Android Neural Networks API	Devices with dedicated NPU
`CPU`	Standard CPU processing	Fallback option

iOS GPU behavior

iOS automatically uses GPU acceleration through Metal when available. No configuration is required.

Frame skipping reduces processing load by reusing corrections from previous frames. This trades some quality for better performance on lower-end devices or high frame rate video. Higher values can also improve video quality by reducing flicker, so experiment with different values.

Android frame skipping

Use setDeflickerParams to configure frame skipping:

import photos.eyeq.dynamic.DynamicProcessor

// Parameters: frames to skip, curveAvg (0-1), imgAvg (0-1)

// Process all frames (highest quality)
App.dynamicProcessor.setDeflickerParams(frames = 0, curveAvg = 0.08f, imgAvg = 0.9f)

// Skip every other frame (balanced)
App.dynamicProcessor.setDeflickerParams(frames = 1, curveAvg = 0.08f, imgAvg = 0.9f)

// Skip 2 frames between processed frames (performance mode)
App.dynamicProcessor.setDeflickerParams(frames = 2, curveAvg = 0.08f, imgAvg = 0.9f)

Parameter	Description	Recommended value
`frames`	Number of frames to skip between inferences	0-2
`curveAvg`	Curve averaging for deflicker (0-1)	0.08
`imgAvg`	Image averaging for deflicker (0-1)	0.9

When switching videos, reset the deflicker state:

App.dynamicProcessor.resetDeflicker()

iOS frame skipping

Use setParams to configure frame skipping:

// Parameters: deflickerCurve, deflickerImage, skip

// Process all frames (highest quality)
pfcDynamic.setParams(deflickerCurve: 0.08, deflickerImage: 0.9, skip: 0)

// Skip every other frame (balanced)
pfcDynamic.setParams(deflickerCurve: 0.08, deflickerImage: 0.9, skip: 1)

// Skip 2 frames between processed frames (performance mode)
pfcDynamic.setParams(deflickerCurve: 0.08, deflickerImage: 0.9, skip: 2)

Parameter	Description	Recommended value
`deflickerCurve`	Curve averaging for deflicker	0.08
`deflickerImage`	Image averaging for deflicker	0.9
`skip`	Number of frames to skip between inferences	0-2

To reset the deflicker state:

pfcDynamic.resetDeflicker()

Memory management

Efficient memory handling is critical for video processing, where frames are processed continuously.

Android bitmap handling

Recycle bitmaps when they're no longer needed to prevent memory leaks:

import android.graphics.Bitmap
import photos.eyeq.dynamic.DynamicProcessor

class ImageProcessor {
    private var lastResult: Bitmap? = null
    
    fun processFrame(input: Bitmap, strength: Float): Bitmap {
        // Recycle previous result before creating new one
        lastResult?.recycle()
        
        // Process the image
        lastResult = App.dynamicProcessor.processImage(input, strength)
        return lastResult!!
    }
    
    fun release() {
        lastResult?.recycle()
        lastResult = null
    }
}

iOS memory management

Use autoreleasepool for batch processing to manage memory efficiently:

func processFrames(images: [CGImage], strength: Float) -> [CGImage] {
    var results: [CGImage] = []
    
    for image in images {
        autoreleasepool {
            if let processed = try? pfcDynamic.processImage(image, strength: strength) {
                results.append(processed)
            }
        }
    }
    
    return results
}

Performance benchmarking

Use the SDK's built-in benchmarking tools to identify bottlenecks and optimize your implementation.

Android benchmarking

The SDK provides benchmarking listeners for profiling. Implement EyeQBenchListener:

import photos.eyeq.dynamic.bench.EyeQBenchListener
import photos.eyeq.dynamic.bench.BenchDynamic
import photos.eyeq.dynamic.bench.BenchEgl
import android.util.Log

class PerformanceMonitor : EyeQBenchListener {
    
    override fun onDynamic(bench: BenchDynamic) {
        Log.d("Perf", """
            Processing times:
            - Scale: ${bench.scaleTime}ms
            - Read from bitmap: ${bench.readFromBitmap}ms
            - Normalize: ${bench.normalizeTime}ms
            - Inference: ${bench.inferenceTime}ms
            - Adjust: ${bench.adjustTime}ms
        """.trimIndent())
    }
    
    override fun onEgl(bench: BenchEgl) {
        Log.d("Perf", """
            EGL times:
            - Texture: ${bench.textureTime}ms
            - Render: ${bench.renderTime}ms
            - Read: ${bench.readTime}ms
            - Release: ${bench.releaseTime}ms
        """.trimIndent())
    }
    
    override fun onDeflicker(consumeTime: Long) {
        Log.d("Perf", "Deflicker time: ${consumeTime}ms")
    }
}

// Attach listener
App.dynamicProcessor.setBenchListener(PerformanceMonitor())

Performance tips

Quick reference for common performance optimizations.

Do

Apply the following tips to improve performance:

Use GPU acceleration when available. Use the isDelegateSupported() method to check on Android.
Use processImagePreview for preview rendering (faster, scaled down).
Recycle bitmaps on Android after use.
Use frame skipping for video on slower devices.
Process on background threads.
Reset deflicker state when switching video sources.

Don't

Avoid the following to see better performance:

Process full-resolution images for preview.
Block the main thread during processing.
Create new bitmaps for every frame without recycling.
Ignore GPU support checks on older devices.

VIDEO-SDK Version 1.0.0.23 built from aa5eef97017e23db1d3051b079500606825ef474 on 5-6-2023.