17. Wildcard Week

Machine Vision Facial Expression Interaction for Jelamp

Project Title: Facial Expression Recognition for Jelamp Using ESP32-S3 Camera and Edge Impulse

What I Made

I made a small interactive machine vision system for Jelamp. The system uses a camera to capture the user's face, runs a trained machine learning model on the ESP32-S3, and changes the NeoPixel LED color based on the detected facial expression.

User face
  |
  v
ESP32-S3 camera
  |
  v
Edge Impulse facial expression model
  |
  v
Expression classification
  |
  v
NeoPixel LED color response

Example Responses

Why I Chose This

Jelamp is designed to be more than a normal desk lamp. It is an expressive robotic lamp that can move, sense, and respond to the user. For Wildcard Week, I wanted to add a visual interaction feature. By using a camera and machine learning model, Jelamp can detect simple facial expressions and respond with light.

This makes the lamp feel more alive and interactive.

How This Meets the Assignment Requirement

This project meets the Wildcard Week requirement because it uses a digital process not covered in my other assignments:

• I used machine vision to capture image data.
• I used Edge Impulse to train a machine learning model.
• I deployed the trained model to an embedded microcontroller.
• I connected the digital classification result to a physical output: NeoPixel LEDs.
• I documented the process so that it can be reproduced.

Data collection -> Model training -> Embedded deployment -> Physical interaction

System Overview

+------------------------+
|       User Face        |
+-----------+------------+
            |
            v
+------------------------+
| ESP32-S3 Camera Board  |
| Captures image input   |
+-----------+------------+
            |
            v
+------------------------+
|   Edge Impulse Model   |
| Classifies expression  |
+-----------+------------+
            |
            v
+------------------------+
| ESP32-S3 Logic         |
| Chooses LED response   |
+-----------+------------+
            |
            v
+------------------------+
| NeoPixel LED Ring      |
| Shows color feedback   |
+------------------------+

Materials and Components

Step-by-Step Process

Happy
Neutral
Sad / Surprised

Happy
Not Happy

Prediction: happy
Confidence: 0.86
Inference time: ___ ms

Red
Green
Blue
White
Rainbow

If expression is Happy:
  show warm yellow or rainbow animation

If expression is Neutral:
  show soft white or calm blue

If expression is Sad or Surprised:
  show blue or purple breathing light

Problems and Solutions

Final Result

The final system can use an ESP32-S3 camera to classify simple facial expressions and change the NeoPixel LED color based on the result. This demonstrates how machine vision can be embedded into a physical interactive object.

For my final project, this feature can become part of Jelamp's emotional response system. Jelamp can use visual input to understand the user's expression and respond through light, movement, or both.

Minimum Viable Version

If time is limited, the project can still meet the assignment requirement with two expression classes:

Happy
Not Happy

The output behavior can be:

Happy -> yellow NeoPixel
Not Happy -> blue NeoPixel

This still demonstrates:

• Image data collection
• Machine learning model training
• Embedded deployment
• Physical LED output
• Reproducible documentation

Files to Include

To make the project reproducible, I should include:

• Edge Impulse project screenshots
• Dataset examples
• Model training result
• Confusion matrix screenshot
• Arduino code
• Wiring diagram
• Photos of the setup
• Short demo video
• Final testing results

Reflection

This week helped me understand how machine learning can be used as part of a physical interactive product. Instead of only controlling LEDs manually, I trained a model that allows the lamp to respond to visual input.

The most important lesson was that the quality of the dataset strongly affects the quality of the result. Good lighting, clear labels, and enough examples are important for a reliable model.

This experiment is useful for Jelamp because it adds a new layer of interaction. In the future, I can combine facial expression detection with servo movement so the lamp can respond with both motion and light.