Models

The Live Sense models provide an interface for interacting with the underlying machine learning (ML) models used for object detection.

Note

Object Detection/Recognition describes the combination of object classification, identifying objects of interest that exist in an image, and localization by identifying where the object is located in the image. An example is a Live Sense model that identifies a vehicle in the image and provides a bounding box of where the vehicle was identified.

The core function of every model is to take in an RGB image and return a list of detections found within the image.

Each Live Sense model has a set of object classes that it is able to detect. It is up to the developer to determine which objects, and therefore models, are relevant to their use case.

Model Types and Detection Classes

The following table describes the models and their available classes:

Category Model Name Label Recommended Confidence
Road Basics Roadbasics pedestrian 45
bicycle 50
car 50
motorcycle 50
truck 50
traffic-light 50
stop-sign 50
Road Alerts RoadAlertsCVModel brake-light-on 80
TrafficLightCVModel green 75
red 75
yellow 75
Road Hazards cone_barrier cone 60
strip-stand 60
cylindrical-barrier 60
delineator-barrier 60
pothole pothole 80
signage roadworks-going-on 60
road-closed 55
speed_bump_object speedbump 55
Road Signs Roadsigns maximum-speed-limit-[5-130] 60

Models Available in Beta Mode

The models below are available for beta testing:

Category Model Name Label Recommended Confidence
Road Alerts TrafficLightCVModel green 75
red 75
yellow 75
Road Hazards bridge_tunnel bridge 75
tunnel 75
height_restriction_signs height-restriction-sign-Xft-Yin 60

Confidence Configuration

Live Sense SDK allows to configure the confidence values of the models as well as the individual classes within each model.

Update the Confidence of a Model

Updates the confidence of all the classes available in the specified model to the provided value.

List<Recognition> recognizeImage(Bitmap bitmap, int sensorOrientation, float minimumConfidence)

For a model, the minimumConfidence here can be used from the Recommended Confidence mentioned in the table described in the Model Types and Detection Classes section.

Update the Confidence of a Class

Updates the confidence of the specified class to the provided value. This supersedes the confidence configured for the parent model.

void addClassMinConfidence(String classLabel, float classMinConfidence)

The classLabel and classMinConfidence can be used from the Label, and respective Recommended Confidence values mentioned in the table described in the Model Types and Detection Classes section.

Limitations

Using a lower confidence threshold will result in more detections, but an overall lower accuracy. This can sometimes cause the labels to show false positives. For this reason, applications should keep their confidence threshold at or above 0.6 to maintain reasonably accurate output.

Threading

The Live Sense models are not thread-safe and should be both initialized and executed on the same thread. Utilizing multiple threads for the same model instance may result in unexpected behavior.

Models may be executed in parallel, but each model instance can only handle one image at a time. Executing a model before the previous call has completed will result in an exception.

Model Execution Options

Live Sense SDK supports multiple instances for executing the Live Sense models. Exact support depends upon the device and model being used.

Note

If a configuration cannot be used as-is, the SDK will fall back to the next best supported configuration.

Object Recognition

A Live Sense Recognition describes the following properties of a detected object:

  • class - What object was detected. For more information, see Model Types and Detection Classes.
  • location - Where the object was found in the image frame.
  • confidence score - A number between 0 and 1 that indicates confidence that the object was correctly detected.

NNEngine

NNEngine represents the library that will be used to execute the ML model. The following libraries are supported:

  • TensorFlow Lite (TFLite)
  • Snapdragon Neural Processing Engine (SNPE)

TFLite is the default NNEngine and can run on most any device meeting the minimum requirements.

SNPE support in Live Sense SDK requires a device with a QC Snapdragon processor and OpenCL support. If SNPE is not available for the device or current model, the SDK will fall back to TFLite.

Note

You must download and include the SNPE SDK (version 1.27.1 or compatible) in your application to use the SNPE engine option. SNPE SDK is optional and not required for other Live Sense functionality. For download details, see the Qualcomm developer site.

Engine Runtime

Each NNEngine may have multiple runtimes available. If none are specified, then the SDK will select the best runtime available for the current NNEngine.

Engine Runtime Support

CPU GPU
TFLite Y N
SNPE N Y

Model Runtime Support

TFLite/CPU SNPE/GPU
Road Basics Y Y
Road Alerts (Brake light) Y Y
Traffic Light Status Y N
Road Signs Y N
Road Hazards Y N
// Setting Road Basics to run on GPU (will only work on Snapdragon)
RoadBasicsModel.Options rbOptions = new RoadBasicsModel.Options();
rbOptions.setNnEngine(NNEngine.SNPE);
rbOptions.setRuntime(EngineRuntime.GPU);
RoadBasicsModel roadBasicsModel = new RoadBasicsModel(rbOptions);

// Road Basics variant on CPU
RoadBasicsModel.Options rbOptions = new RoadBasicsModel.Options();
rbOptions.setNnEngine(NNEngine.TFLITE);
rbOptions.setRuntime(EngineRuntime.CPU);
rbOptions.setNumThreads(4); // Manually set based on available processor resources, normally 2-4
RoadBasicsModel roadBasicsModel = new RoadBasicsModel(rbOptions);

Regions

To provide better detections, some models will allow selection of a region. If the desired region is supported, an ML model better tuned for that region will be used in place of the default one.

RoadSignsModel allows this selection through the countryCode option and supports the following region/country codes:

  • global - Default region if desired one is not available
  • bra - Brazil
  • oce - Oceania
  • sea - South-east Asia (such as Singapore or Thailand)

Split Inference

Split inference involves modification of the source image shape by a model to improve detection output. What this option does depends upon the model, and this option should be disabled if large modifications to the image are made before passing it into the model.

results matching ""

    No results matching ""