Back to Weekly Assignments

Week 17: Wildcard Week — Machine Learning

Assignment

Design and produce something with a digital process not covered in another assignment, documenting requirements met and everything necessary to reproduce it.

Project: Rock–Paper–Scissors hand-gesture recognition using the Seeed XIAO ESP32S3 Sense onboard camera and a machine-learning image classifier (Edge Impulse).

Show a hand gesture in front of the camera → the board classifies rock, paper, or scissors and prints the result on the Serial Monitor.

Checklist — requirements met

RequirementHow this assignment meets it
Document workflow(s) and process(es) Step-by-step workflow below: data collection → Edge Impulse training → Arduino deployment → testing.
Process not covered in other assignments On-device computer vision + ML inference on ESP32S3 Sense. Other weeks used color sensor (Week 9), ESP-NOW (Week 11), and rule-based servo logic — not camera-based gesture classification.
Problems encountered and fixes Documented in Problems & fixes section.
Original design files and source code Gesture_RPS_week17.ino + Edge Impulse project export (see Reproduction).
Hero shot of the result Hero shot — working setup with live classification.

Why this is Wildcard (not covered elsewhere)

  • Week 9 — TCS34725 color sensor (I²C), not camera vision.
  • Week 11 — ESP-NOW wireless + web server; input was buttons, not ML gestures.
  • Week 15 — interface/programming survey; no trained model deployed on MCU.
  • This week — OV2640 camera + Edge Impulse CNN classifier running on XIAO ESP32S3 Sense (digital process: ML model training + embedded inference).

Hardware

  • Seeed Studio XIAO ESP32S3 Sense (camera + digital microphone + IMU on one board)
  • USB-C cable
  • Plain background + stable lighting for training and inference
XIAO ESP32-S3 Sense board XIAO ESP32S3 Sense front with camera

Workflow & process

Step 1 — Setup board & camera

  1. Arduino IDE 2.x → Boards Manager → install esp32 by Espressif.
  2. Select board: XIAO_ESP32S3 (or Seeed XIAO ESP32S3 Sense).
  3. Enable PSRAM and correct partition scheme for camera + model.
  4. Test camera with Seeed / Edge Impulse firmware example; confirm frames in Serial or EI Studio.

Step 2 — Collect training data (Edge Impulse)

  1. Create project at Edge Impulse Studio.
  2. Connect XIAO ESP32S3 Sense via edge-impulse-arduino data forwarder or upload images.
  3. Collect ≥50 samples per class: rock, paper, scissors.
  4. Keep hand distance and background consistent; vary only the gesture shape.

Step 3 — Train image classifier

  1. Impulse design: Image → Image processing (96×96) → Classification (3 classes).
  2. Train model; target accuracy >85% on validation set.
  3. Deployment → Arduino library → download and install in Arduino/libraries/.

Step 4 — Deploy & test on device

  1. Merge Edge Impulse exported setup() / loop() with project sketch.
  2. Open Serial Monitor (115200 baud); show each gesture and verify correct label.
  3. Debounce predictions (≥800 ms) to avoid repeated triggers on one pose.

Source code

Repository sketch (links to Edge Impulse library after export):

// Core logic — map highest class score above threshold
String gesture = "unknown";
float threshold = 0.70;
if (scissors > threshold && scissors > rock && scissors > paper) gesture = "scissors";
else if (rock > threshold && rock > paper && rock > scissors) gesture = "rock";
else if (paper > threshold && paper > rock && paper > scissors) gesture = "paper";
Serial.println(gesture);

Problems encountered & fixes

  • Camera failed to init — Enabled PSRAM in board settings; used Sense variant pin map, not generic ESP32-S3.
  • Low accuracy / confused paper vs scissors — Added more training images with fingers clearly separated; kept same distance from camera.
  • Unstable labels (flickering) — Added debounce: only accept a new gesture after 800 ms and score >0.70.
  • Model too large for flash — Reduced input to 96×96 and used Edge Impulse EON Compiler / int8 quantization.
  • Poor results under yellow light — Re-collected samples under white daylight; avoided mixed color temperature.

Hero shot

Final working setup: XIAO ESP32S3 Sense detecting a scissors gesture; result shown on Serial Monitor.

Hero shot — XIAO ESP32S3 Sense gesture recognition setup

Reproduction — what you need

ItemDetails
BoardSeeed XIAO ESP32S3 Sense
SoftwareArduino IDE 2.x, esp32 core ≥3.0, Edge Impulse Studio account
SourceGesture_RPS_week17.ino + Edge Impulse Arduino library export
Labelsrock, paper, scissors (3 classes)
Baud rate115200

Useful links