Doorbell ML Application

Description

These applications are designed for doorbell systems, leveraging on-device machine learning to detect the presence of a person within the camera’s field of view. Upon detection, the system captures and processes images to provide visual context such as high-resolution stills or a sequence of preroll images leading up to the event. This use case combines features from multiple SDK sample applications, including jpeg_preroll and image_stitching, to deliver a comprehensive visual snapshot that enhances detection reliability and situational awareness.

Use Case Specific Details

Adapting Pipeline for Custom Object Detection Models

This person detection pipeline can be adapted to work with custom object detection models. However, certain validation steps and potential modifications are required to ensure compatibility.

Prerequisites for Model Compatibility

Before adapting this pipeline for another object detection model, you must verify the following:

1. Model Format Requirements

  • Your object detection model should be in .tflite format

  • The model should produce similar output tensor structure (bounding boxes, confidence scores)

2. Vela Compiler Compatibility Check

Step 1: Analyze Original Model

  1. Load your object_detection_model.tflite file in Netron

  2. Document the output tensors:

    • Tensor names

    • Tensor identifiers/indexes

    • Quantization parameters (scale and offset values)

    • Tensor dimensions

Step 2: Compile with Vela

  1. Pass your model through the Vela compiler to generate model_vela.bin or model_vela.tflite

  2. Analyze the Vela-compiled model in Netron using the same steps as above

Step 3: Compare Outputs Compare the following between original and Vela-compiled models:

  • Output tensor indexes/identifiers: Verify if they remain in the same order

  • Quantization parameters: Check if scale and offset values are preserved

  • Tensor dimensions: Ensure dimensions match your expected output format

Pipeline Adaptation Process

Case 1: No Changes Required

If the Vela compilation preserves:

  • βœ… Output tensor indexes in the same order

  • βœ… Same quantization scale and offset values

Result: You can proceed with the existing pipeline without modifications.

Case 2: Modifications Required

If the Vela compilation changes:

  • ❌ Output tensor index order

  • ❌ Quantization parameters

Required Actions: Modify the pipeline code as described below.

Code Modifications

If your model’s output tensor indexes change after Vela compilation, you need to update the tensor parameter assignments in uc_person_detection.c:

Location: detection_post_process function

Original Code:

g_box1_params = &g_all_tens_params[0];
g_box2_params = &g_all_tens_params[1];
g_cls_params  = &g_all_tens_params[2];

Modified Code: Update the array indexes according to your Vela-compiled model’s output tensor identifiers:

// Example: If your model_vela output has different tensor order
g_box1_params = &g_all_tens_params[X];  // Replace X with actual index from Netron
g_box2_params = &g_all_tens_params[Y];  // Replace Y with actual index from Netron
g_cls_params  = &g_all_tens_params[Z];  // Replace Z with actual index from Netron

Running the Application using VS Code Extension

Windows note: Ensure the USB drivers are installed for streaming. See the Zadig steps in
SR110 Build and Flash with VS Code.

  1. In VS Code, open Video Streamer from the Synaptics sidebar.

    Video Streamer

  2. For logging output, click SERIAL MONITOR and connect to the DAP logger port on J14.

    • To make it easier to identify, ensure only J14 is plugged in (not J13).

    • The logger port is not guaranteed to be consistent across OSes. As a starting point:

      • Windows: try the lower-numbered J14 COM port first.

      • Linux/macOS: try the higher-numbered J14 port first.

    • If you do not see logs after a reset, switch to the other J14 port.

  3. Doorbell application automatically connects to Video Streamer upon reset.

  4. On person detection, the video streamer opens with the captured frame and preroll context images.