An Autonomous Puppeteering System

A.k.a. Pitou would be proud 🐱🧡


Final presentation video 🎬
The puppet on its stage ✨

πŸ’‘ Project Idea

This project is an electronic puppeteering system inspired by Neferpitou’s Nen-based manipulation from Hunter x Hunter. It explores themes of control, autonomy, and possession through a controller entity that governs the movement of a skeletal puppet.

Inspiration image for the puppeteering system
Img shows Neferpitou's Nen Puppets- Terpsichora and Puppeteering respectively. Img source: Pinterest

The system operates autonomously through pre-programmed motion sequences. I'm thinking movements like slow head tilt - pause - sudden jerk, arm lifts as if resisting then drops. Repetitive, clinical movements. Each sequence is loopable, time-based and/or triggered by presence or touch.

From here 2 things can be explored:

  1. The puppeteer is programmed to control the puppet.
  2. The puppet moves as if possessed, while the puppeteers digits/hand move out of sync which sell the whole eerie part and acts as a visual representation of invisible electronic control.

🧩 Project Architecture Overview

This project consists of two primary entities:

  1. Controller Entity (Puppeteer Unit)
  2. Skeletal Puppet (Actuated Entity)
System breakdown: electronics, power, inputs, and outputs.
Microntroller parts and fabrication process for different parts.

Rather than directly mirroring user input, the system filters and transforms signals through timing logic, motion constraints, and behavioural state switching. This creates movement that feels delayed, resistant, and semi-autonomous rather than fully obedient.

πŸŒ€ Design Evolution

The project went through a bunch of iterations before landing on its final form. Each one changed how you interact with the puppet.

πŸ”Ή Trying out stuff

First idea: puppet reacting to music. A sound sensor drove the servos...technically worked, but it felt random, not intentional (Β¬_Β¬)

πŸ”Ή Figuring out the UI

Switched to voice commands, so the puppet can actually perform: wave, bow, dance πŸ’ƒ

πŸ”Ή Prototyping

3D printed a model from GrabCAD and tied strings to two dowel sticks to figure out the movement range + no. of threads


Then swapped my hands for servo motors. Thankfully this worked *^____^*

πŸ”— System Architecture

User (voice / buttons)
   ↓
Web Interface
   ↓
WiFi (WebSocket, port 81)
   ↓
XIAO ESP32-S3
   ↓  I2C
PCA9685 Servo Driver
   ↓
5 Γ— SG90 Servos (head, wrists, knees)
   ↓
Strings & Pulleys
   ↓
Puppet Motion 🎭

The ESP32 creates its own WiFi network. The browser connects to it, sends a command like {"command": "wave"}, and the ESP32 tells the PCA9685 which servos to move.

πŸ› οΈ Digital Fabrication

3D printing, laser cutting, PCB milling, sanding & finishing

πŸ”Ή The Stage (Laser cutting)

Theatre modelled in SolidWorks with slot joints, ready for the laser cutter

πŸ”Ή The Puppet + Pulleys (3D printing)

Articulated puppet, 3D printed in PLA. Strings tied at the head and wrists

Custom pulleys were designed and 3D printed to sit on the SG90 horns and wind the strings.

πŸ”Ή Servo Layout

Mapped out the 7 servos on cardboard first: head, 2 elbows, 2 wrists, 2 knees

In the final version the elbows were dropped, leaving 5 active servos: head, both wrists and both knees.

⚑ Electronics

Custom PCB designed in EasyEDA for the XIAO ESP32-S3 with a 4-pin header for the PCA9685 (SDA, SCL, VCC, GND).

Milling the board πŸŒ€

XIAO ESP32-S3 soldered on

😭 Things That Went Wrong

Failure 01: Mirrored PCB πŸͺž
Milled the whole board...only to find the design was mirrored. The XIAO had to sit upside down (β•₯﹏β•₯)

Upside-down XIAO, held together with tape and hope

Failure 02: Lifted Copper Trace
During soldering, a copper trace peeled off the board. RIP PCB #2.

Failure 03: Missing Antenna πŸ“Ά
Spent hours debugging WiFi before realising...the antenna wasn't plugged in. (ΰ²₯﹏ΰ²₯)

The tiny antenna that cost me several hours of my life

πŸ’» Programming

Written in Arduino IDE. Libraries: Adafruit PWM Servo Driver, WebSockets (Markus Sattler), ArduinoJson.

Key parts of the program:
  • β€’ ESP32 runs as a WiFi access point (PuppetControl)
  • β€’ WebSocket server on port 81 receives JSON commands
  • β€’ Each joint has 3 positions: limp (0Β°), neutral (90Β°), full (180Β°)
  • β€’ smoothMove() steps the servo 1Β° at a time so motion isn't jerky
  • β€’ Combo poses: stand, limp, wave, bow, nod, dance
  • β€’ Several phrasings per command (e.g. "raise left wrist", "left wrist up", "lift left wrist") so voice input is more forgiving

πŸ”Ή Channel map (final version)

JointPCA9685 channel
Right wristCH2
Left wristCH13
Left kneeCH15
Right kneeCH0
HeadCH5
void smoothMove(int ch, int target, int stepDelay = 10) {
  int current = currentAngles[ch];
  int step = (target > current) ? 1 : -1;
  while (current != target) {
    current += step;
    setServo(ch, current);
    delay(stepDelay);
  }
}

πŸ“± Interface

Final Puppet Controller interface
Final Puppet Controller UI: tap to speak, or use the quick command buttons. The active joints light up on the diagram

The browser listens to your voice, converts it to text and sends it to the ESP32. Say "raise right wrist" and...it does 🫑

The full write-up of the interface (voice input, buttons, WebSocket messages) is in Week 14 Β· Interface & Application.

🧾 Bill of Materials

  • β€’ Seeed XIAO ESP32-S3 + antenna Γ— 1
  • β€’ PCA9685 16-ch servo driver Γ— 1
  • β€’ SG90 micro servos Γ— 7
  • β€’ FR1 copper board (custom PCB) Γ— 1
  • β€’ Acrylic sheet (stage)
  • β€’ PLA filament (puppet + pulleys)
  • β€’ Thread / strings
  • β€’ 5V external power supply
  • β€’ Fairy lights (stage lighting ✨)

πŸ’­ Reflection

βœ… What worked

  • β€’ Voice + button control over WiFi
  • β€’ Smooth servo motion with the limp/neutral/full system
  • β€’ The stage turned out sooo pretty

❌ What didn't

  • β€’ PCB #1 mirrored, PCB #2 lost a trace
  • β€’ String tension needs constant re-tuning

πŸš€ Future improvements

  • β€’ The autonomous "possessed" mode from my original idea πŸ‘€
  • β€’ Puppeteer hand moving out of sync with the puppet
  • β€’ Hiding the electronics inside the stage

πŸ“ Design Files

Final Puppet Controller Code (.ino)
Puppet Controller Interface (.html)
Puppet Controller v2 (.ino)
Puppet Controller v1 (.ino)
1 Servo Test (.ino)
2 Servo Test (.ino)
SG90 Pulley (.3mf)
PCB Gerber Files (.zip)
PCB Schematic, EasyEDA (.json)