0. Group Assignment

Project Overview

In our FabAcademy group assignment, we implemented voice-controlled interaction between two independent IoT devices:

1. Hongtai's AI Smart Glasses Kit – Equipped with voice recognition and supports both English and Chinese commands.
2. Feng Lei's LED Marquee System – An LED setup controllable via the MQTT protocol.

Core Objective:
To verify the feasibility of communication between IoT devices using voice commands like “turn on/off the light” (or the Chinese equivalents “开灯/关灯”) to remotely control the LED marquee.

Communication Protocol

To enable communication between the two systems, we used MQTT (Message Queuing Telemetry Transport), a lightweight messaging protocol ideal for IoT due to its efficiency and ease of use.

MQTT Broker

A public MQTT broker was used as the intermediary between the smart glasses and the LED marquee, enabling them to send and receive messages.

• Broker Address: broker.emqx.io
• Port: 1883

Communication Steps

1. Voice Command Recognition:
   The AI glasses listen for specific voice commands.

2. Command Interpretation:
   After recognition, the glasses determine the action to perform:

   • If the command is “turn on the light”, they send an MQTT message to switch on the LEDs.
   • If the command is “turn off the light”, they send a message to switch them off.

3. MQTT Messaging:
   The glasses publish a message to a designated MQTT topic.

4. LED Marquee Response:
   The marquee subscribes to the same topic and reacts accordingly.

MQTT Topic Structure

We used the following MQTT topic to manage LED control:

• Topic: fablab/chaihuo/led/control/#

This topic is used by the glasses to publish commands and by the marquee to subscribe and respond.

System Architecture

Component Interaction Diagram

Hardware Configuration

AI Glasses Side

• Main Board: XIAO ESP32S3 Sense
• Microphone: Built-in PDM Microphone
• Interaction: Capacitive touch sensor (GPIO7)

LED Marquee Side

• Main Board: XIAO ESP32C3
• Actuator: LED Array

Goal: Use the smart glasses to send commands over Wi-Fi and MQTT to control the LEDs on the marquee.

Core Implementation

AI Glasses Code Highlights

1. Base Initialization

   • Initialize network stack (esp_netif_init)
   • Create default event loop (esp_event_loop_create_default)
   • Initialize NVS flash (nvs_flash_init) with retry logic
   • Launch console REPL (console_cmd_init)

2. Register Feature Modules

   • Wi-Fi control commands
   • TTS/STT (Text-to-Speech/Speech-to-Text) modules

3. Touch Button Initialization

   • Install touch element library
   • Configure touch button (Channel 7, Sensitivity 0.1)
   • Register button callback button_handler
   • Start touch detection

4. MQTT Client Initialization

   • Configure broker (mqtt://broker.emqx.io)
   • Start MQTT client and register event listeners

#include <esp_log.h>
#include <esp_system.h>
#include <nvs_flash.h>
#include <sys/param.h>
#include <string.h>
#include <driver/i2c.h>

#include "freertos/FreeRTOS.h"
#include "freertos/task.h"

#include "esp_netif.h"
#include "nvs_flash.h"
#include "esp_wifi.h"
#include "esp_event.h"

#include "esp_timer.h"
#include "mqtt_client.h"

#include "console_wifi.h"
#include "console_tts.h"
#include "console_stt.h"
#include "console_itt.h"

#include "touch_element/touch_button.h"

#define TAG "main"

static esp_mqtt_client_handle_t client = NULL;

static touch_button_handle_t button_handle;

static char result[4096] = { 0 };

static void button_handler(touch_button_handle_t out_handle, touch_button_message_t *out_message, void *arg)
{
    (void)out_handle; // Unused
    if (out_message->event == TOUCH_BUTTON_EVT_ON_PRESS)
    {
        console_cmd_stt_rec(5);
        console_cmd_stt_get_result(result, sizeof(result));
        ESP_LOGI(TAG, "result: %s\n", result);
        if(strstr(result, "开灯")!= NULL)
        {
            ESP_LOGI(TAG, "turn on the light");
            esp_mqtt_client_publish(client, "fablab/chaihuo/led/control/", "ON", 0, 0, 0);
        }

        if (strstr(result, "关灯")!= NULL)
        {
            ESP_LOGI(TAG, "turn off the light");
            esp_mqtt_client_publish(client, "fablab/chaihuo/led/control/", "OFF", 0, 0, 0);
        }

        if (strstr(result, "on") != NULL && strstr(result, "light") != NULL)
        {
            ESP_LOGI(TAG, "turn on the light");
            esp_mqtt_client_publish(client, "fablab/chaihuo/led/control/", "ON", 0, 0, 0);
        }
        if (strstr(result, "off") != NULL && strstr(result, "light") != NULL)
        {
            ESP_LOGI(TAG, "turn off the light");
            esp_mqtt_client_publish(client, "fablab/chaihuo/led/control/", "OFF", 0, 0, 0);
        }
    }
    else if (out_message->event == TOUCH_BUTTON_EVT_ON_RELEASE)
    {
        ESP_LOGI(TAG, "Button[%d] Release", (int)arg);
    }
    else if (out_message->event == TOUCH_BUTTON_EVT_ON_LONGPRESS)
    {
        ESP_LOGI(TAG, "Button[%d] LongPress", (int)arg);
    }
}

static void mqtt_event_handler(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data)
{
    esp_mqtt_event_handle_t event = event_data;
    switch (event->event_id)
        {
            case MQTT_EVENT_CONNECTED:
                ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED");
                break;
            case MQTT_EVENT_DISCONNECTED:
                ESP_LOGI(TAG, "MQTT_EVENT_DISCONNECTED");
                break;
            case MQTT_EVENT_DATA:
                ESP_LOGI(TAG, "MQTT_EVENT_DATA");
                break;
            default:
                break;
        }
}

static void mqtt_app_start(void)
{
    const esp_mqtt_client_config_t mqtt_cfg = {
    .broker.address.uri = "mqtt://broker.emqx.io",
    .credentials.client_id = "fablab_chaihuo_glasses",
};

    client = esp_mqtt_client_init(&mqtt_cfg);
    esp_mqtt_client_register_event(client, ESP_EVENT_ANY_ID, mqtt_event_handler, NULL);
    esp_mqtt_client_start(client);
}

void app_main(void)
{
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    esp_err_t ret = nvs_flash_init(); // Initialize NVS
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND)
    {
        ESP_ERROR_CHECK(nvs_flash_erase());
        ret = nvs_flash_init();
    }
    // Initialize console REPL
    ESP_ERROR_CHECK(console_cmd_init());

    ESP_ERROR_CHECK(console_cmd_wifi_register());
    ESP_ERROR_CHECK(console_cmd_tts_register());
    ESP_ERROR_CHECK(console_cmd_stt_register());
    ESP_ERROR_CHECK(console_cmd_itt_register());

    /* Initialize Touch Element library */
    touch_elem_global_config_t global_config = TOUCH_ELEM_GLOBAL_DEFAULT_CONFIG();
    ESP_ERROR_CHECK(touch_element_install(&global_config));
    ESP_LOGI(TAG, "Touch element library installed");

    touch_button_global_config_t button_global_config = TOUCH_BUTTON_GLOBAL_DEFAULT_CONFIG();
    ESP_ERROR_CHECK(touch_button_install(&button_global_config));
    ESP_LOGI(TAG, "Touch button installed");

    touch_button_config_t button_config = { .channel_num = TOUCH_PAD_NUM7, .channel_sens = 0.1F };

    ESP_ERROR_CHECK(touch_button_create(&button_config, &button_handle));
    ESP_ERROR_CHECK(touch_button_set_dispatch_method(button_handle, TOUCH_ELEM_DISP_CALLBACK));
    /* Subscribe touch button events (On Press, On Release, On LongPress) */
    ESP_ERROR_CHECK(touch_button_subscribe_event(button_handle, TOUCH_ELEM_EVENT_ON_PRESS, TOUCH_PAD_NUM7));
    ESP_ERROR_CHECK(touch_button_set_callback(button_handle, button_handler));

    touch_element_start();

    ESP_LOGI(TAG, "System initialized");
    mqtt_app_start();

    // start console REPL
    ESP_ERROR_CHECK(console_cmd_start());
}

Marquee Code Highlights

1. System Setup

   • Initialize Serial (baud rate: 115200)
   • Configure LED pins
   • Connect to Wi-Fi (SSID: ASUS, Password: admin856)

2. MQTT Client Setup

   • Set MQTT server to broker.emqx.io:1883
   • Register message callback

3. Main Loop

   • Maintain MQTT connection (auto-reconnect if lost)
   • Listen to topic fablab/chaihuo/led/control/

4. Command Processing

   • Parse incoming MQTT messages
   • If message = ON → turn on LEDs (LOW signal)
   • If message = OFF → turn off LEDs (HIGH signal)

#include <WiFi.h>
#include <PubSubClient.h>

// Wi-Fi configuration
const char* ssid = "yourwifi";             // Set your Wi-Fi name
const char* password = "yourpassword";     // Set your Wi-Fi password

// MQTT Broker configuration
const char* mqtt_server = "broker.emqx.io"; // EMQX broker address
const int mqtt_port = 1883;                 // EMQX MQTT port

// LED pin definition
const int LED_PINS[] = {D0, D1, D2, D3, D4, D5}; // Pins for custom expansion board
const int LED_COUNT = 6;

// Define MQTT topic
#define LED_CONTROL_TOPIC "fablab/chaihuo/led/control/" // LED control topic

// Create WiFi and MQTT client objects
WiFiClient espClient;
PubSubClient client(espClient);

// Function declarations
void setupWiFi();
void reconnect();
void callback(char* topic, byte* message, unsigned int length);
void testLEDs();

void setup() {
    Serial.begin(115200);
    delay(3000); // Delay a few seconds to allow Serial Monitor to start
    
    Serial.println("\n\n=== LED Marquee Control System Starting ===");
    
    // Initialize all LED pins
    for(int i = 0; i < LED_COUNT; i++) {
        pinMode(LED_PINS[i], OUTPUT);
        digitalWrite(LED_PINS[i], LOW);  // Turn off all LEDs initially
    }
    
    // Connect to Wi-Fi
    setupWiFi();
    
    // Set MQTT server and callback
    client.setServer(mqtt_server, mqtt_port);
    client.setCallback(callback);
    
    // Test LED status
    testLEDs();
    
    Serial.println("System initialized, waiting for MQTT control commands...");
}

void loop() {
    // Check MQTT connection
    if (!client.connected()) {
        reconnect();
    }
    client.loop(); // Handle MQTT messages
    
    // Check for Serial commands (for testing)
    if (Serial.available() > 0) {
        String command = Serial.readStringUntil('\n');
        command.trim();
        command.toLowerCase();
        
        if (command == "on") {
            Serial.println("Turn on all LEDs via Serial");
            for (int i = 0; i < LED_COUNT; i++) {
                digitalWrite(LED_PINS[i], HIGH);
            }
        } 
        else if (command == "off") {
            Serial.println("Turn off all LEDs via Serial");
            for (int i = 0; i < LED_COUNT; i++) {
                digitalWrite(LED_PINS[i], LOW);
            }
        }
        else if (command == "test") {
            Serial.println("Test all LEDs");
            testLEDs();
        }
    }
}

// Function to connect to Wi-Fi
void setupWiFi() {
    delay(10);
    Serial.println();
    Serial.print("Connecting to Wi-Fi: ");
    Serial.println(ssid);
    
    WiFi.begin(ssid, password);
    
    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }
    
    Serial.println("");
    Serial.print("Wi-Fi connected! IP address: ");
    Serial.println(WiFi.localIP());
}

// Function to reconnect to MQTT
void reconnect() {
    // Keep trying until reconnected
    while (!client.connected()) {
        Serial.print("Attempting MQTT connection...");
        // Use a random client ID to avoid conflicts
        String clientId = "fablab_chaihuo_led_";
        clientId += String(random(0xffff), HEX);
        
        // Try to connect
        if (client.connect(clientId.c_str())) {
            Serial.println("Connected");
            // Subscribe to LED control topic
            client.subscribe(LED_CONTROL_TOPIC);
            Serial.print("Subscribed to topic: ");
            Serial.println(LED_CONTROL_TOPIC);
        } else {
            Serial.print("Connection failed, rc=");
            Serial.print(client.state());
            Serial.println(" retrying in 2 seconds");
            // Wait 2 seconds before retrying
            delay(2000);
        }
    }
}

// Callback function when a message is received
void callback(char* topic, byte* message, unsigned int length) {
    Serial.println("\n====== MQTT Message Received ======");
    Serial.print("Topic: ");
    Serial.println(topic);
    Serial.print("Length: ");
    Serial.println(length);
    Serial.print("Payload: ");
    
    // Print raw bytes and build string
    String messageStr;
    for (int i = 0; i < length; i++) {
        messageStr += (char)message[i];
        Serial.print((char)message[i]);
        Serial.print("[");
        Serial.print((int)message[i]);
        Serial.print("] ");
    }
    Serial.println();
    Serial.print("Message content: ");
    Serial.println(messageStr);
    
    // Trim and convert to lowercase for comparison
    messageStr.trim();
    messageStr.toLowerCase();
    
    // Flexible command matching
    if (messageStr == "on" || messageStr.indexOf("on") >= 0) {
        Serial.println("Recognized ON command");
        // Turn on LEDs using HIGH
        for(int i = 0; i < LED_COUNT; i++) {
            digitalWrite(LED_PINS[i], HIGH);
            delay(50); // Short delay for visibility
        }
        Serial.println("All LEDs turned on");
    } 
    else if (messageStr == "off" || messageStr.indexOf("off") >= 0) {
        Serial.println("Recognized OFF command");
        // Turn off LEDs using LOW
        for(int i = 0; i < LED_COUNT; i++) {
            digitalWrite(LED_PINS[i], LOW);
            delay(50); // Short delay for visibility
        }
        Serial.println("All LEDs turned off");
    }
    else if (messageStr == "test" || messageStr.indexOf("test") >= 0) {
        Serial.println("Recognized TEST command");
        testLEDs();
    }
}

// Function to test the LEDs
void testLEDs() {
    Serial.println("Testing LEDs");
    
    // Ensure all LEDs are initially off
    for(int i = 0; i < LED_COUNT; i++) {
        digitalWrite(LED_PINS[i], LOW);
    }
    delay(500);
    
    // Turn on each LED one by one
    Serial.println("Lighting LEDs one by one");
    for(int i = 0; i < LED_COUNT; i++) {
        digitalWrite(LED_PINS[i], HIGH);
        delay(200);
    }
    delay(1000);
    
    // Turn off each LED one by one
    Serial.println("Turning off LEDs one by one");
    for(int i = 0; i < LED_COUNT; i++) {
        digitalWrite(LED_PINS[i], LOW);
        delay(200);
    }
    
    // Blink all LEDs
    Serial.println("Blinking all LEDs");
    for(int j = 0; j < 3; j++) {
        // All on
        for(int i = 0; i < LED_COUNT; i++) {
            digitalWrite(LED_PINS[i], HIGH);
        }
        delay(300);
        
        // All off
        for(int i = 0; i < LED_COUNT; i++) {
            digitalWrite(LED_PINS[i], LOW);
        }
        delay(300);
    }
    
    Serial.println("LED test completed");
}

PubSubClient Installation

1. Install PubSubClient Library in Arduino IDE:
   • Open Arduino IDE → Tools → Manage Libraries...
   • Search "PubSubClient" by Nick O'Leary → Click Install
2. Verify Installation:
   • Restart Arduino IDE
   • Compile your sketch to ensure it works without errors

Testing LED Control with MQTTX

1. Launch MQTTX App
   Download MQTTX if not installed.

2. Create a New Connection

   • Name: Any (e.g., "LED Test")
   • Host: broker.emqx.io
   • Port: 1883

3. Connect to the Broker

4. Publish a Message

   • Topic: fablab/chaihuo/led/control/
   • Message: on, off, or test

5. Observe LED Behavior

   • on → All LEDs light up
     

Sending "on" via MQTTX will light up all the LEDs on the marquee development board.

• off → All LEDs turn off
  

Sending "off" via MQTTX will turn off all the LEDs on the marquee development board.

• test → LEDs blink sequentially and all together

6. Speech Recognition Integration

Print the recognized content.

• Saying “开灯” / “turn on the light” → sends ON
• Saying “关灯” / “turn off the light” → sends OFF

end the corresponding message to the designated topic.

7. (Optional) Subscribe to Status Topic
   • Topic: fablab/chaihuo/led/status to receive state updates

Testing Notes

1. Ensure ESP32 is Properly Flashed

   • Check Serial Monitor for Wi-Fi/MQTT connection logs

2. Topic Matching

   • Ensure topic in MQTTX matches the code exactly

3. Case Sensitivity

   • Messages are case-sensitive (on/off lowercase only)

4. Network Access

   • Ensure both ESP32 and MQTTX host have internet access

Troubleshooting

• Serial Monitor Debugging

  • Open Serial Monitor at 115200 baud rate
  • Check for received messages or errors

• MQTT Connection

  • Ensure the device is connected
  • Watch if reconnect() keeps retrying

• LED Pins

  • Verify pin configuration and wiring
  • LEDs use active-low control

• Local MQTT Option

  • Use a local Mosquitto broker if public broker is unstable

Final Demo – AI Glasses Driving the Marquee

After completing development, we tested the smart glasses and the LED marquee together. The left side shows Feng Lei’s board and MQTTX running on his PC. The middle shows Hongtai’s glasses and laptop.

Test the two devices together side by side.

The on-site test went smoothly. Hongtai successfully controlled the LED lights on Feng Lei’s marquee development board by issuing voice commands to the AI smart glasses.

Results:

Test Scenario	Expected Result	Actual Result
Voice: “开灯” (Turn on)	LEDs turned on	✅ Success
Voice: “关灯” (Turn off)	LEDs turned off	✅ Success

Project Outcomes

Successfully implemented the following:

• Mixed-language voice recognition (Chinese + English)
• Real-time cross-device control via MQTT

Conclusion

This project demonstrates how MQTT can enable communication between different systems. By integrating voice control with the LED marquee, we showcased a practical application of IoT technology.

References

1. EMQX
2. Seeed Studio XIAO ESP32C3 Official Documentation
3. Seeed Studio XIAO ESP32S3 Official Documentation