Software Architecture

Node Firmware

// Audio sampling configuration
#define SAMPLE_RATE 44100
#define SAMPLE_BITS 16
#define CHANNELS 1

// I2S microphone setup
void setup_audio() {
    i2s_config_t i2s_config = {
        .mode = I2S_MODE_MASTER | I2S_MODE_RX,
        .sample_rate = SAMPLE_RATE,
        .bits_per_sample = SAMPLE_BITS,
        .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
        .communication_format = I2S_COMM_FORMAT_STAND_I2S,
        .intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
        .dma_buf_count = 8,
        .dma_buf_len = 1024,
        .use_apll = false
    };
    // Initialize I2S peripheral
    i2s_driver_install(I2S_NUM_0, &i2s_config, 0, NULL);
}

Machine Learning Pipeline

# TensorFlow model conversion
import tensorflow as tf

def convert_to_tflite(saved_model_path, output_path):
    # Convert the model
    converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path)
    
    # Enable optimizations
    converter.optimizations = [tf.lite.Optimize.DEFAULT]
    converter.target_spec.supported_types = [tf.float16]
    
    # Convert to 8-bit integers
    converter.target_spec.supported_ops = [
        tf.lite.OpsSet.TFLITE_BUILTINS_INT8
    ]
    converter.inference_input_type = tf.int8
    converter.inference_output_type = tf.int8
    
    # Convert the model
    tflite_model = converter.convert()
    
    # Save the model
    with open(output_path, 'wb') as f:
        f.write(tflite_model)

ML Development Platform: Edge Impulse

For the development and deployment of the machine learning models, particularly for sound event classification and feature extraction, this project will leverage Edge Impulse. Edge Impulse is an end-to-end development platform specifically designed for creating and deploying machine learning models on resource-constrained edge devices like the ESP32-S3.

Key Characteristics & Benefits for this Project:

• End-to-End Workflow: Covers data collection, signal pre-processing (DSP), model training, and on-device deployment, streamlining the development process.
• Audio Optimised: Provides robust support for audio data, including built-in DSP blocks for feature extraction (e.g., spectrograms, MFCCs) crucial for acoustic event detection.
• Efficient Model Deployment: Facilitates the creation of highly optimised models (e.g., using TensorFlow Lite for Microcontrollers with quantization) suitable for the ESP32-S3's limited resources.
• C++ Library Export: Allows for the export of the entire Impulse (DSP code + ML model + inference SDK) as a self-contained C++ library, simplifying integration into the ESP32-S3 firmware.
• Rapid Prototyping: Enables quick iteration on different DSP parameters and model architectures via the Edge Impulse Studio.

Leveraging Edge Impulse:

The Edge Impulse platform will be utilised as follows:

• Data Ingestion & Management: Uploading and managing datasets of acoustic samples for training and testing.
• Impulse Design: Using the Edge Impulse Studio to visually design the ML pipeline. This will involve selecting appropriate DSP blocks for audio feature extraction and choosing/configuring a suitable learning block (e.g., a neural network classifier).
• Model Training & Validation: Training the sound classification model within the platform and evaluating its performance.
• Deployment: Exporting the trained Impulse as a C++ library. This library will then be integrated into the ESP32-S3 Sense firmware. The firmware will use the Edge Impulse SDK functions (included in the library) to run the DSP and inference locally, enabling on-device sound event classification.

Server Architecture

API Endpoints

Local Network Communication