W15 | Interface and Application Programming

For the group assignment, we compared two interface tools: Python + Guizero and Lovable. With Guizero we built a desktop interface in Thonny with two buttons to control an LED — turning it on and off. We started with a basic version and then improved it by adding colors to the buttons and updating the interface text. With Lovable we generated a hand tracking interface using a single prompt. The result was a professional-looking web app that detected hand movements in real time, showing a skeleton overlay on the joints and details about each finger — including whether they were extended or curled, their bend angle, and pinch value. We also found a problem: the interface sometimes detected the thumb incorrectly, reading it as closed when it was extended and vice versa.

For this week's assignment, I wanted to explore an idea I had considered for Pambu during the early stages of the project: equipping the robot with a camera that could detect children's emotions in real time — happy 😊, sad 😢, or angry 😠 — so it could trigger different emotional regulation responses automatically. However, this idea was eventually set aside; Pambu's final design uses push buttons instead for emotion input.

For this week, I decided to revisit that original concept and explore what tools and approaches could realistically support it — not to implement it in the final project, but to understand what would be involved. This led me to two different experiments, both using my XIAO ESP32-S3 board and its RGB LED as a way to test communication between a UI and embedded hardware.

First sketch / idea of my final project

The first tool I explored was Lovable, an AI-powered web application generator that lets you describe an interface in natural language and generates a fully functional web app from it. Since it has a limited number of free tokens, I needed to be strategic with my prompts — so I used ChatGPT first to help me craft a clear, detailed prompt before loading it into Lovable.

My first prompt targeted three-emotion detection using the laptop camera, sending serial commands to the XIAO ESP32-S3 to light the RGB LED in different colors:

The intended flow was straightforward: the browser would access the laptop camera and run a facial emotion recognition model in real time. Each time an emotion was detected, the interface would send a command through Web Serial to the XIAO ESP32-S3 — and the board would respond by changing the RGB LED color accordingly. The emoji and text label on screen would update simultaneously, creating a synchronized response between the interface and the hardware.

The problem was that no browser-compatible pre-trained model for three-emotion detection could be found. Without it, the interface had nothing to run. This attempt could not move forward. 😔

Since the first attempt failed, I went back to ChatGPT to find an alternative. It was clear that for three-emotion detection there were no browser-compatible pre-trained models available, so I changed the goal: instead of detecting three emotions, I focused on a single detection — whether the user is smiling or not. For this, ChatGPT suggested MediaPipe Face Landmarker, a browser-compatible solution that loads automatically without requiring local model files.

I crafted a new prompt:

Lovable generated both the web interface and the Arduino sketch — the code for the board is displayed directly inside the app, ready to copy into Arduino IDE. The result was a web app called Smile Detector with a live camera feed on the left and the emotion status on the right, plus a "Connect to XIAO ESP32S3" button to establish the serial connection from the browser.

The Smile Detector works through two components that communicate via Web Serial: the browser-based interface generated by Lovable, and the XIAO ESP32-S3 running the Arduino sketch.

On the browser side, MediaPipe Face Landmarker analyzes the webcam feed frame by frame, detecting facial landmarks and calculating whether the user is smiling. When the detection state changes — from no smile to smile, or vice versa — the interface sends a serial command to the board. Commands are only sent on state changes to avoid flooding the serial port with repeated messages.

Two commands are used:

On the board side, the Arduino sketch waits for incoming commands through the serial connection. When it receives RAINBOW, it starts cycling through colors smoothly — gradually shifting the hue and writing the corresponding values to the RGB LED on pins D7, D8, D9. When it receives OFF, it stops and turns the LED off. Since my accessory board uses a common anode RGB LED, the color values are handled with inverted logic internally inside the writeRGB function — so the rest of the code stays clean and readable.

void loop() {
  if (Serial.available()) {
    String cmd = Serial.readStringUntil('\n');
    cmd.trim();
    if (cmd == "RAINBOW") rainbow = true;
    else if (cmd == "OFF") {
      rainbow = false;
      writeRGB(0, 0, 0);
    }
  }
  if (rainbow) {
    hsvToRgb(hue, ...); // converts hue to RGB for smooth color cycling
    hue += 0.6f;        // advances the color each loop
    delay(15);          // controls the speed of the effect
  }
}

The loop checks for incoming commands — RAINBOW starts the color cycle, OFF stops it and turns the LED off

For the second experiment I took a completely different approach — no camera, no AI detection. Instead, I built a simple desktop interface using Python in Thonny that lets the user manually trigger emotion states through buttons. Each button sends a command through serial to the XIAO ESP32-S3, which responds by changing the RGB LED color.

To build the interface I used two libraries that need to be installed before running the script:

The interface has four buttons: Happy 😊, Sad 😢, Angry 😠, and OFF — each with a color matching the emotion it represents. When a button is pressed, Python sends a single character through serial to the board: 3 for happy (green), 1 for sad (blue), 2 for angry (red), and 4 to turn the LED off. On the board side, an Arduino sketch listens for the incoming character and sets the RGB LED to the corresponding color on pins D7, D8, D9.

The Mood RGB Controller works through two components: a Python script running in Thonny that creates the interface, and the XIAO ESP32-S3 running the Arduino sketch.

When the user clicks a button, the Python script sends a single character through the serial connection to the board:

• 3 → Happy 😊 → LED turns green
• 1 → Sad 😢 → LED turns blue
• 2 → Angry 😠 → LED turns red
• 4 → OFF → LED turns off

The board listens for incoming characters and responds immediately by changing the LED color. The connection is established at the start of the script and stays open as long as the app is running. When the app is closed, the connection closes automatically. Unlike the Smile Detector, there is no automatic detection here — the user decides which emotion to send. This makes the interface simpler and more direct, and it worked without any issues from the first try.

Problem — Common anode vs common cathode LED

The code generated by Lovable assumed a common cathode LED, but my accessory board uses a common anode LED — which means the logic is inverted: 0 means fully on and 255 means fully off. The generated code also had an inconsistency: the COMMON_ANODE flag was set to false, but the OFF command was already written as writeRGB(255, 255, 255) — which is actually the correct behavior for common anode. The colors worked, but the code was contradicting itself.

✅ Fix

Two corrections were made to the Arduino sketch:

• Changed COMMON_ANODE false to COMMON_ANODE true
• Changed the OFF command from writeRGB(255, 255, 255) to writeRGB(0, 0, 0), letting the function handle the inversion internally

The same logic applied to the Mood RGB Controller sketch — all color values are written with inverted logic to account for the common anode configuration.