Models
The Live Sense models provide an interface for interacting with the underlying machine learning (ML) models used to detect objects.
Note
Object Detection/Recognition describes the combination of object classification, identifying objects of interest that exist in an image, and localization by identifying where the object is located in the image. An example is a Live Sense model that identifies a vehicle in the image and provides a bounding box of where the vehicle was identified.
The core function of every model is to take in an RGB image and return a list of detections found within the image.
Each Live Sense model has a set of object classes that it is able to detect. It is up to the developer to determine which objects, and therefore models, are relevant to their use case.
Model types and detection classes
Category | Model Name | Label | Recommended Confidence |
Road Basics | RoadBasicsModel (day) | pedestrian | 47 |
| | rider | 47 |
| | car | 46 |
| | bicycle | 45 |
| | motorcycle | 47 |
| | truck | 45 |
| | traffic-light | 47 |
| | bus | 50 |
Road Hazards | Cones & Barriers | rectangular-barrier | 60 |
| | strip-stand | 60 |
| | cylindrical-barrier | 60 |
| | delineator-barrier | 60 |
| | cone | 60 |
| Pothole | pothole | 80 |
Road Signs | RoadSignsModel | DS_SpeedLimit[5-130] | 55 |
| | DS_AdvisorySpeedLimit[10-85] | 55 |
| | DS_Stop | 55 |
| | DS_DoNotEnter | 55 |
| | DS_NoAccess | 55 |
| | DS_StartConstruction | 55 |
| | DS_Yield | 55 |
| | DS_PedestrianCrossing | 55 |
| | DS_SpeedBumps | 55 |
| | DS_StopHereOnRed | 55 |
| | DS_NoLeftTurn | 55 |
| | DS_NoRightOrLeftTurn | 55 |
| | DS_NoUTurn | 55 |
| | DS_NoUTurnOrLeftTurn | 55 |
| | DS_ParkingRestriction | 55 |
Road Signs (TRT) | RoadSignsModel | DS_SpeedLimit[5-130] | 60 |
| | DS_SpeedLimit[5-130] | 60 |
| | DS_Stop | 60 |
Brake Light | BrakeLight | car-brake-light-on | 45 |
Note
For the Road Signs model, only the label prefix is given above.
See the Road Signs section below for the full label list with sample images
Models available in beta mode
The models below are available for beta testing:
Category | Model Name | Label | Recommended Confidence |
Road Basics | Traffic Light Status | traffic-light-green | 75 |
| | traffic-light-red | 75 |
| | traffic-light-yellow | 75 |
| RoadBasicsNightModel | pedestrian | 45 |
| | motorcycle | 40 |
| | car | 40 |
| | truck | 40 |
| | bus | 40 |
Lane Detection | Road Lanes | lane | N/A |
Road Hazards | speed_bump_object | cross-walk | 55 |
| speed_bump_signage | speed-bump-sign | 55 |
| bridge_tunnel | bridge | 75 |
| | tunnel | 75 |
| height_restriction_signs | height-restriction-sign-Xft-Yin | 60 |
Confidence configuration
Live Sense SDK for Linux allows you to configure the confidence values of the models.
Update the confidence of a model
You may set the minimum confidence threshold for all classes available in a specified model using the model classes' respective options struct. The options are set during model initialization and may be updated via a setter method.
ls::RoadBasicsModelOptions roadBasicsOptions;
roadBasicsOptions.roadBasicsConfidenceThresh = 0.45;
roadBasicsOptions.trafficLightConfidenceThresh = 0.75;
std::vector<ls::RoadHazardsModelOptions> roadHazardsModelOptions = {
{ls::RoadHazardsModelType::RH_CONE_CONSTRUCTION, 0.6},
{ls::RoadHazardsModelType::RH_POTHOLES, 0.6},
{ls::RoadHazardsModelType::RH_SPEED_BUMP_OBJECT, 0.55},
{ls::RoadHazardsModelType::RH_SPEED_BUMP_SIGNAGE, 0.55},
{ls::RoadHazardsModelType::RH_BRIDGE_TUNNEL, 0.75},
{ls::RoadHazardsModelType::RH_HEIGHT_RESTRICTION_SIGNS, 0.6}};
ls::RoadSignsModelOptions roadSignsOptions;
roadSignsOptions.roadSignsConfidenceThresh = 0.55;
ls::BrakeLightModelOptions brakeLightOptions;
brakeLightOptions.brakeLightConfidenceThresh = 0.8;
For a model, the confidence threshold here can be used from the 'Recommended Confidence' described in the Model Types and Detection Classes table.
Limitations
Using a lower confidence threshold will result in more detections, but an overall lower accuracy. This can sometimes cause the labels to show false positives. For this reason, applications should keep their confidence threshold at or above 0.6
to maintain reasonably accurate output.
Threading
The Live Sense models are not thread-safe and should be both initialized and executed on the same thread. Utilizing multiple threads for the same model instance may result in unexpected behavior.
Models may be executed in parallel, but each model instance can only handle one image at a time. Executing a model before the previous call has completed will result in an exception.
Model execution options
Live Sense SDK for Linux supports multiple instances for executing the Live Sense models. Exact support depends upon the device and model being used.
Object recognition
A Live Sense Recognition
describes the following properties of a detected object:
- class - What object was detected. For more information, see Model Types and Detection Classes.
- location - Where the object was found in the image frame.
- confidence score - A number between 0 and 1 that indicates confidence that the object was correctly detected.
Model types
The following libraries are supported:
TensorFlow Lite (TFLite)
TensorRT
Coral Edge TPU
Engine runtime support
| CPU | GPU | TPU |
TensorFlow Lite | Y | Y | N |
TensorRT | N | Y | N |
Coral Edge TPU | N | N | Y |
For the TensorFlow Lite variant to load a model on GPU, you must set the appropriate model option to true
.
ls::ModelOptions mOptions;
mOptions.setUseGpuDelegate(true);
Note
TensorFlow Lite's Linux-based OpenGL ES GPU support is still experimental. Not all Live Sense models may be supported on all GPUs.
Model runtime support
| TFLite fp32 | TFLite fp16 | TFLite INT8/Quant | TensorRT fp32 | TensorRT fp16 | Edge TPU INT8 |
Road Basics | Y | Y | Y | Y | Y | Y |
Road Basics Night | Y | Y | Y | N | N | Y |
Road Signs | Y | Y | Y | Y | Y | Y |
Cones & Barriers | Y | Y | Y | Y | Y | Y |
Pothole | Y | Y | Y | Y | Y | Y |
Speed Bumps | Y | Y | N | N | N | N |
Speed Bumps Signs | Y | Y | Y | N | N | Y |
Bridges & Tunnels | Y | Y | Y | N | N | Y |
Height Restriction Signs | Y | Y | Y | N | N | N |
Road Lanes | Y | Y | Y | Y | Y | N |
Brake Light | Y | Y | Y | Y | Y | Y |
By default, the SDK will run fp32
variant. You can enable fp16
or INT8
by setting the respective variable to TRUE
in the model options. Ensure that only one variant is set to TRUE
at a time.
ls::ModelOptions mOptions;
mOptions.setRunInFp16(true);
mOptions.setRunInInt8(true);
Distance and position estimation
Note
This feature is in Beta, the returned distance and relative position values may be inaccurate.
All of the object detection models listed above can provide an estimated distance and position relative to the camera's point of view for each detected object.
The output values for the distance
and relativePosition
properties of a detected object are as follows:
- Height: Distance of the detection from the ground.
- Lateral: Distance of the detection relative to the center of the camera's view. This value may be positive or negative with a negative value signifying that the detection is towards the left side of the camera's view and vice versa.
- Depth: Distance that the device needs to travel forward in a straight line so that the actual distance is equal to the lateral distance.
- Distance: Actual distance of the detection from the camera's point of view.
All distance values are provided in meters and are used in the following features of Live Sense SDK
- Time-to-collision alerts
- Relative velocity estimation
Road signs
Group | Label | Sample Image |
SpeedLimit | DS_SpeedLimitXX_White_Circle_00 |  |
| DS_SpeedLimitXX_White_Circle_01 |  |
| DS_SpeedLimitXX_White_Rectangle_01 |  |
AdvisorySpeedLimit | DS_AdvisorySpeedLimitXX_Yellow_Rectangle_00 |  |
| DS_AdvisorySpeedLimitXX_Yellow_Rectangle_01 |  |
| DS_AdvisorySpeedLimitXX_Yellow_Rectangle_03 |  |
Stop | DS_Stop_Red_Octogon_00 |  |
DoNotEnter | DS_DoNotEnter_Red_Circle_00 |  |
| DS_DoNotEnter_White_Rectangle_00 |  |
| DS_DoNotEnter_White_Rectangle_01 |  |
NoAccess | DS_NoAccess_White_Circle_00 |  |
StartConstruction | DS_StartConstruction_White_Triangle_00 |  |
Yield | DS_Yield_White_Triangle_00 |  |
PedstrianCrossing | DS_PedestrianCrossing_Blue_Rectangle_00 |  |
| DS_PedestrianCrossing_White_Triangle_00 |  |
| DS_PedestrianCrossing_Yellow_Circle_00 |  |
| DS_PedestrianCrossing_Yellow_Diamond_01 |  |
| DS_PedestrianCrossing_Yellow_Triangle_00 |  |
SpeedBumps | DS_SpeedBumps_Blue_Rectangle_00 |  |
| DS_SpeedBumps_White_Triangle_00 |  |
| DS_SpeedBumps_Yellow_Diamond_01 |  |
StopHereOnRed | DS_StopHereOnRed_White_Rectangle_00 |  |
NoLeftTurn | DS_NoLeftTurn_White_Circle_00 |  |
| DS_NoLeftTurn_White_Rectangle_02 |  |
NoRightOrLeftTurn | DS_NoRightOrLeftTurn_White_Rectangle_00 |  |
| DS_NoRightOrLeftTurn_White_Circle_00 |  |
NoUTurn | DS_NoUTurn_White_Circle_00 |  |
NoUTurnOrLeftTurn | DS_NoUTurnOrLeftTurn_White_Rectangle_00 |  |
ParkingRestriction | DS_ParkingRestriction_Blue_Circle_00 |  |
| DS_ParkingRestriction_White_Circle_01 |  |
| DS_ParkingRestriction_White_Rectangle_03 |  |