An Autonomous Puppeteering System

Project Idea

This project is an electronic puppeteering system inspired by Neferpitou’s Nen-based manipulation from Hunter x Hunter. It explores themes of control, autonomy, and possession through a controller entity that governs the movement of a skeletal puppet.

The system operates autonomously through pre-programmed motion sequences. I'm thinking movements like slow head tilt - pause - sudden jerk, arm lifts as if resisting then drops. Repetitive, clinical movements. Each sequence is loopable, time-based and/or triggered by presence or touch.

From here 2 things can be explored:

1. The puppeteer is programmed to control the puppet.
2. The puppet moves as if possessed, while the puppeteers digits/hand move out of sync which sell the whole eerie part and acts as a visual representation of invisible electronic control.

Project Architecture Overview:

This project consists of two primary entities:

1. Controller Entity (Puppeteer Unit)
2. Skeletal Puppet (Actuated Entity)

System Structure:

At its core, the system is built around a custom electronics architecture:

1. Microcontroller: XIAO (RP2040) with custom PCB design
2. Power Supply: LiPo battery with voltage regulation
3. Actuation: Servo motors driving articulated joints
4. Optional Inputs: Environmental sensors or manual interface devices
5. Outputs: Movement, light feedback, and possible sound elements

Rather than directly mirroring user input, the system filters and transforms signals through timing logic, motion constraints, and behavioural state switching. This creates movement that feels delayed, resistant, and semi-autonomous rather than fully obedient.
The behaviour-state framework allows the puppet to transition between idle, active, disturbed, and looping motion patterns, reinforcing themes of mediated control and constrained autonomy. The controller entity houses the electronics, power system, and logic. The puppet receives processed signals and translates them into constrained physical motion.

Control Logic:

The project does not rely solely on fixed choreography. Instead, it operates through a flexible control framework that may include:

· Sensor-driven responses (light, sound, proximity)
· Manual input (joystick or button interface)
· Hybrid interaction models

Physical Fabrication:

· Controller Entity: Laser-cut or 3D-printed enclosure housing electronics
· Puppet Entity: Abstract skeletal structure fabricated through 3D printing and optional molding/casting
· Threads: Led strips/physical strings connecting both entities as a symbolic representation of control

Design Evolution

The project underwent multiple iterations before arriving at its final form. Each iteration explored a different method of interaction and helped define the final experience.

Trying out stuff

The initial concept explored the idea of a puppet responding to music and sound. A sound sensor was used to detect volume levels and drive servo movements. While technically feasible, the interaction lacked intentionality and expressive control.

Figuring out the UI

I wanted to explore voice-based commands. This allowed more direct interaction and gave the puppet performative qualities. Commands such as wave, bow and dance could be triggered through a browser interface.

Protyping:

I 3D printed a model from Grabcad and tied some strings to figure out the movement range, type, the no. of threads etc. So first I tried using two dowel sticks to actually use it as a puppet.

Then later I tried it with servo motors. Thankfully this worked.*^____^*

Multiple prototypes were developed to test movement range, cable tension, pulley placement and mechanical reliability.

System Architecture

The system receives user commands through a browser-based interface. These commands are transmitted wirelessly to the ESP32-S3, which interprets the input and sends motion commands to the PCA9685 servo driver. The servos actuate a system of strings and pulleys, generating expressive puppet movements.

Digital Fabrication

Laser cutting was used to fabricate the puppet body and stage structure.
Custom pulleys and mounting components were designed and 3D printed to support the cable-driven mechanism.
Extensive sanding and finishing were performed to improve fit and assembly quality.
A custom PCB was designed using EasyEDA to integrate the ESP32-S3, power distribution and communication interfaces.

Electronics Failures

Programming

The ESP32-S3 hosts a WebSocket server that receives commands from a browser interface and translates them into coordinated servo movements.

Interface Development

A browser-based control interface was developed to issue commands such as wave, bow, dance and posture changes.

Assembly

Testing & Validation

Testing focused on motion quality, cable tension, servo calibration and wireless responsiveness.

Final Result

The final prototype successfully demonstrates a gesture-controlled animatronic puppet capable of expressive performance through custom electronics and digitally fabricated mechanisms.

Bill of Materials

Reflection

What Worked

What Didn't

Future Improvements