Creating Frame Processor Plugins

Creating a Frame Processor Plugin for Android

The Frame Processor Plugin API is built to be as extensible as possible, which allows you to create custom Frame Processor Plugins. In this guide we will create a custom QR Code Scanner Plugin which can be used from JS.

Android Frame Processor Plugins can be written in either Java, Kotlin or C++ (JNI).

Mostly automatic setup

1. Run Vision Camera Plugin Builder CLI

npx vision-camera-plugin-builder android

info

The CLI will ask you for the path to project's Android Manifest file, name of the plugin (e.g. QRCodeFrameProcessor), name of the exposed method (e.g. scanQRCodes) and language you want to use for plugin development (Java or Kotlin). For reference see the CLI's docs.

2. Register the package in MainApplication.java

        @Override
        protected List<ReactPackage> getPackages() {
          @SuppressWarnings("UnnecessaryLocalVariable")
          List<ReactPackage> packages = new PackageList(this).getPackages();
          ...
          packages.add(new QRCodeFrameProcessorPluginPackage()); // <- add
          return packages;
        }

Manual setup

Java

1. Open your Project in Android Studio
2. Create a Java source file, for the QR Code Plugin this will be called QRCodeFrameProcessorPlugin.java.
3. Add the following code:

import androidx.camera.core.ImageProxy;
import com.mrousavy.camera.frameprocessor.FrameProcessorPlugin;

public class QRCodeFrameProcessorPlugin extends FrameProcessorPlugin {

  @Override
  public Object callback(ImageProxy image, Object[] params) {
    // code goes here
    return null;
  }

  QRCodeFrameProcessorPlugin() {
    super("scanQRCodes");
  }
}

note

The JS function name will be equal to the name you pass to the super(...) call (with a __ prefix). Make sure it is unique across other Frame Processor Plugins.

4. Implement your Frame Processing. See the Example Plugin (Java) for reference.
5. Create a new Java file which registers the Frame Processor Plugin in a React Package, for the QR Code Scanner plugin this file will be called QRCodeFrameProcessorPluginPackage.java:

import com.facebook.react.ReactPackage;
import com.facebook.react.bridge.NativeModule;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.uimanager.ViewManager;
import com.mrousavy.camera.frameprocessor.FrameProcessorPlugin;
import javax.annotation.Nonnull;

public class QRCodeFrameProcessorPluginPackage implements ReactPackage {
  @NonNull
  @Override
  public List<NativeModule> createNativeModules(@NonNull ReactApplicationContext reactContext) {
    FrameProcessorPlugin.register(new QRCodeFrameProcessorPlugin());
    return Collections.emptyList();
  }

  @Nonnull
  @Override
  public List<ViewManager> createViewManagers(@Nonnull ReactApplicationContext reactContext) {
    return Collections.emptyList();
  }
}

6. Register the package in MainApplication.java

        @Override
        protected List<ReactPackage> getPackages() {
          @SuppressWarnings("UnnecessaryLocalVariable")
          List<ReactPackage> packages = new PackageList(this).getPackages();
          ...
          packages.add(new QRCodeFrameProcessorPluginPackage()); // <- add
          return packages;
        }

Kotlin

1. Open your Project in Android Studio
2. Create a Kotlin source file, for the QR Code Plugin this will be called QRCodeFrameProcessorPlugin.kt.
3. Add the following code:

import androidx.camera.core.ImageProxy
import com.mrousavy.camera.frameprocessor.FrameProcessorPlugin

class ExampleFrameProcessorPluginKotlin: FrameProcessorPlugin("scanQRCodes") {

  override fun callback(image: ImageProxy, params: Array<Any>): Any? {
    // code goes here
    return null
  }
}

note

The JS function name will be equal to the name you pass to the FrameProcessorPlugin(...) call (with a __ prefix). Make sure it is unique across other Frame Processor Plugins.

4. Implement your Frame Processing. See the Example Plugin (Java) for reference.
5. Create a new Kotlin file which registers the Frame Processor Plugin in a React Package, for the QR Code Scanner plugin this file will be called QRCodeFrameProcessorPluginPackage.kt:

import com.facebook.react.ReactPackage
import com.facebook.react.bridge.NativeModule
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.uimanager.ViewManager
import com.mrousavy.camera.frameprocessor.FrameProcessorPlugin

class QRCodeFrameProcessorPluginPackage : ReactPackage {
  override fun createNativeModules(reactContext: ReactApplicationContext): List<NativeModule> {
    FrameProcessorPlugin.register(ExampleFrameProcessorPluginKotlin())
    return emptyList()
  }

  override fun createViewManagers(reactContext: ReactApplicationContext): List<ViewManager<*, *>> {
    return emptyList()
  }
}

6. Register the package in MainApplication.java

        @Override
        protected List<ReactPackage> getPackages() {
          @SuppressWarnings("UnnecessaryLocalVariable")
          List<ReactPackage> packages = new PackageList(this).getPackages();
          ...
          packages.add(new QRCodeFrameProcessorPluginPackage()); // <- add
          return packages;
        }

🚀 Next section: Finish creating your Frame Processor Plugin (or add iOS support to your Frame Processor Plugin)