Final Project: Patchlet

My final project was a wireless and wearable MIDI controller bracelet, called patchlet. It sends generic MIDI CC messages based on the pitch and roll. Those could then be mapped to control parameters in light software, but also synthesizers, visual art software, etc.. All design files and code to reproduce it can be downloaded below.

Digital Files

• Code
  • code.zip The dongle sub-project is an ESP-IDF project and the remote sub-project is a PlatformIO project.
• PCB designs (KiCAD)
  • pcb.zip
  • pcb-dongle.zip
• CAD (FreeCAD), .step, and .dxf files of all 2D and 3D designs (except circuit and PCB designs)
  • housing.zip
  • housing-dongle.zip

The Patchlet

This project is a wearable, wireless MIDI interface in form of a bracelet. Based on the positioning of the arm, values of user-configured control parameters can be changed. This enables light operators or artists to interact with their media setups while moving thus being away from their personal computer.

The system consists of a remote and an USB dongle. The remote contains an ESP32S3 retrieving data from a 9-DoF sensor (with 3 degrees from per sensor: Gyroscope, Accelerometer/IMU, Magnetometer). The measurement data are converted to bytes and sent to the dongle. The dongle registers itself as a USB-MIDI device and forwards retrieved data in form of MIDI packets to the personal computer it is attached to. It is then up to the user to decide how to use this generic MIDI controller.

For integrating the BNO085 sensor, the corresponding breakout board by Adafruit is used due to time constraints. The sensor provides an SH2 interface introducing virtual sensors. Those are an abstraction of real sensors whose data are combined using sensor fusion. More details can be found in the input-devices week (9).

As a power supply, a re-chargeable vape battery was used. The battery is connected to a switch, so the remote can be turned off. I added heat sinks to both micro-controllers so that in case heat development would be significant, that would not be a bottleneck in the development.

Due to the availability of specific libraries I programmed the remote using PlatformIO and the Arduino framework and the dongle using the ESP-IDF.

Bill of Materials (BoM)

The material costs for the final project therefore are 39.37€. The item with the largest cost, the Adafruit BNO085 board containing the gyroscope, can be substituted to directly use this item of 3.84€. This way, the cost per piece could be decreased to roughly 21€.

Besides that, soldering material (including soldering tin, flux, etc.) is needed.

Prior Work & Literature

Wearables

• Imogen Heap using MiMU gloves: https://youtu.be/3QtklTXbKUQ?feature=shared&t=554
  • inputs:
    • pitch, yaw, roll, specific positions that can be reached,
    • acceleration,
    • position of the single fingers,
    • position of all fingers (e.g. when making a fist).
  • applications:
    • Fist: recording audio input (voice), can be looped.
    • Another gesture: deleting the loop.
    • Finger bending: each finger activates an effect, hand position changes parameter.
    • air drumming movements: wrist flick, slap
    • modulating parameters, such as panning
  • There is a software for the PC included for user configuration of the device.
• Modular body bend sensors, using velostat
• Hand muscle trainer

Other glove projects

• https://hackaday.io/project/8455-power-glove-universal-hid
• https://www.youtube.com/watch?v=NjTWAy4-lLI

Communication & MIDI

• modular MIDI sequencer

Programming (click for details)

A detailed documentation of the firmware development process can be found at the documentation page of the system-integration week (15).

A gyroscope was set up for reading pitch and roll of the glove. The sensor data are transmitted from the remote to the dongle using ESP-NOW. The dongle then converts everything to MIDI CC messages. The remote firmware was developed using PlatformIO and the Arduino environment. The dongle firmware was developed using the ESP-IDF. Reasons are discussed at the system-integration week (15) page.

Retrieving Data From BNO085 9-DoF Sensor

The video below shows the remote controller (ESP32S3) reading data from the BNO085 sensor and outputs it onto the serial console. A documentation of the development process can be found at the system-integration week (15) page.

Check the system-integration week (15) page or the original files to see the

1// code code code (see above).

Communication Between Remote & Dongle

The video below shows the remote controller (ESP32S3) sending the retrieved data to the dongle (ESP32S3). A documentation of the development process can be found at the system-integration week (15) page.

Check the system-integration week (15) page or the original files to see the

1// code code code (see above).

Dongle Converting Data to MIDI

The video below shows the remote controller (ESP32S3) sending the retrieved data to the dongle (ESP32S3) which again converts those data to MIDI messages and forwards them to a synthesizer software (Cardinal). A documentation of the development process can be found at the system-integration week (15) page.

Check the system-integration week (15) page or the original files to see the

1// code code code (see above).

Fabrication

Electronics (click for details)

In the electronics design week (6) I learned how to use KiCad for designing PCBs. In the electronics production week (8) I learned how to fabricate a PCB using a CNC mill for isolation milling. In the system integration week (15) I designed and fabricated PCBs for the remote and the dongle.

The design for the dongle does only contain a single ESP32. There are no further components contained. The purpose of the dongle design is to be able to fabricate a plate the dongle ESP32 can be mounted onto so it can be put into the packaging.

Packaging (click for details)

3D and 2D design was taught in multiple weeks. Most about CAD I learned during the computer-controlled cutting week (3), the computer-controlled machining week (7), and mechanical & machine design week (12). As a CAD software I used FreeCAD. With FreeCAD I designed the housings for the PCBs, as well as the wristbands to mount the remote onto the forearm. Below you can see the most relevant features of the remote and dongle packaging. Details can be read on the page on the system integration week (15).

Remote

The PCB was integrated into the packaging by introducing radii at the bottom of the upper hull. The PCB was layed into the upper hull from below and was then held in place by mounting the bottom.

Wristband

The remote is added to the arm using wristbands. A wristband consists of laser-cut polyester strips, sewn velcro tape and wooden laser-cut brackets for easier handling.

The wristbands were lasered from the polyester sheets that were ordered for the wildcard week (16). For instructions on how to use the laser cutter, the computer-controlled cutting week (3) can be checked. The exact cutter parameters for cutting polyester can be checked in the wildcard week documentation.

Dongle

The dongle packaging is a simplified version of the remote packaging. The PCB is mounted in the same way. The bottom and the upper hull are connected using M2 wooden screws.

Assembly

Putting the antenna inside the case led to messages not being transmitted. Therefore, it needed to be mounted at the outside. To make everything disassembleable, the antenna was mounted using velcro tape as well. For that, one part of the tape needed to be mounted on the housing’s surface. I used double-sided tape by tesa. All other tapes available at the lab were not sticky enough not to be accidentally torn off when removing the antenna.

The dongle was assembled by laying the PCB into the top of the housing and fastening the bottom and the top of the housing using four M2 screws. The remote was assembled by threading the wristbands into the bottom of the remote, laying the PCB and the battery into the top of the housing, sliding top and bottom together, and fastening them using one M2 screw.

Evaluation

The product was evaluated by

• shaking and dropping it to check if any parts were loose,
• connecting the dongle to the PC and mapping the MIDI controls to a synthesizer plug-in + using the controller for manipulating interactive video & lighting setups and re-iterating on where it did not feel good from the perspective of a user.

Implications

Implications were documented in week 18.

Answered Questions

In week 18 part of our assignment was to state questions we would need to answer to finish the final project. Below the questions and answers to them can be seen.

• On which inputs should the vibration motor be programmed to be feedback?
  • Not answered, since the vibration motor was not included
• How is the cut pattern for cutting the glove look and how should it be attached to a hand?
  • This was not answered, but it was found that a normal hand-like pattern from polyester is not elastic at all and therefore makes it impossible to function as a hand overlay for mounting bend sensors.
• How is the PCB integrated with the glove?
  • The PCB would be integrated with the glove by mounting ribbon cables coming from the PCB using snap fasteners that are sewn onto the glove.
• How is the IMU integrated with the ESP-IDF?
  • It is not. Instead of the ESP-IDF, the Arduino environment was used for programming the remote containing the Adafruit BNO085 board.
• How to built a sensor for detecting pressure at the fingertips using velostat? Is this even possible?
  • Building pressure sensors was excluded from the scope of the project during development.
• How are the sensor data processed? (Acceleration peaks, noisy bend sensor signals)
  • As of now, there is not much data processing implemented by me. I only use events and values provided by the Adafruit board.
• How are the bend sensors calibrated?
  • No bend sensors were used so they did not need to be calibrated.

Further Work

There are many ideas to be implemented. Some of them are listed below.

• Second controller for the other hand.
• NOTE_ON message on acceleration peaks.
• Relative position of hands to each other as CC input.
• Glove-like overlay for the hand, including
  • (velostat) bend sensors on the fingers,
    • http://iainmccurdy.org/diy/bendsensor/bendsensor.html
    • https://www.instructables.com/Fabric-bend-sensor/
  • pressure sensors for drumming,
  • WS2812B LEDs
    • attached to the back of the hand to indicate pitch and roll
    • attached to index, middle, ring fingers to indicate finger bends
  • PCB on fabric: either embroidered or sewn, just like the Lilypad
  • vibration motor for feedback
• Adding OSC mode to use controller as OSC controller. Alternatively, a converter can be created using e.g. puredata.
• Case for everything to carry around.
• Improve feel of wristband. It is not yet suitable for all wrists.
• Replace Adafruit IMU breakout board with the actual SMD part.
• Power management
  • Add temperature monitoring.
  • Add voltage monitoring.
  • Research again about industry-grade battery management.
• Calculate position from acceleration values to have it as a control input.
• Laser TOF sensor to measure distance of palm from floor, send as CC message. Only turn on if palm is turned down.

Further links

• https://www.digikey.com/en/products/detail/gct/FFC3B11-04-T/10657221
• https://www.digikey.com/en/products/detail/molex/0982670701/3469873
• https://i.ytimg.com/vi/7ZRpyZp4gq8/maxresdefault.jpg
• https://duckduckgo.com/?q=lilypad+arduino
• https://fabricademy.fabcloud.io/handbook/

Bend Sensors

In the end, I did not include any bend sensors, but I started building them. For that I used velostat, a material that changes resistance when it is compressed. For making bend sensors using velostat and fabric, this tutorial could be considered. Additional resources include

• Post about making drum pads
• Playlist on making velostat-based sensors

  

  

For measuring the changing resistance one could use a wheatstone bridge or a voltage divider. I decided to use a voltage divider for the first iteration. The voltage above the variable resistance, according to the voltage devider rule, is \[V_{out} = V_{CC} \frac{R_{var}}{R_{fixed} + R_{var}}\] where \(R_{var}\) is the resistance of the velostat-strip and \(R_{fixed}\) is the value of the other resistor in the voltage divider. While the range of \(R_{var}\) is given, \(R_{fixed}\) can be chosen by the designer to maximize the range of \(V_{out}\). The reason for this is to use the full range of the ADC on the micro-controller so the impact of the quantization noise is minimized. Another point to consider was that the measured voltage should not be too close to the boundary of the ADC’s range. Otherwise, the ADC might saturate and the measured signal would be distorted. Following this objective, the voltage divider would need to be chosen so that the measured voltage is centered around the center of the ADC’s range.

The question now was how to choose \(R_{fixed}\). I chatted with a language model (GPT 4) to quickly gain an overview (the chat). The following I understood from that. The objective to maximize the voltage range could be expressed as \[\Delta V_{out} = V_{CC} \cdot \left( \frac{R_{max}}{R_{fixed} + R_{max}} - \frac{R_{min}}{R_{fixed} + R_{max}} \right).\] where \(R_{max}\) and \(R_{min}\) are the boundaries of \(R_{var}\). The term \[R_{fixed} = \sqrt{R_{max}\cdot R_{min}}\] is a rule of thumb for the estimated maximum. Due to time constraints, it was only verified graphically below. The \(R_{fixed}\) value to center \(V_{out}\) can be calculated by calculating the voltage divider factor to be \(0.5\), if \(R_{var}\) is centered. Setting \(R_{var}\) to the center of the resistance range, i.e. \(R_{var} = 0.5\Delta R + R_{min}\) with \(\Delta R = R_{max} - R_{min}\) leads to \[V_{out} = V_{CC} \frac{0.5\Delta R + R_{min}}{R_{fixed} + 0.5\Delta R + R_{min}}\]. Requiring the factor term of resistances to be \(0.5\) leads to \[R_{fixed} = 0.5(R_{max} + R_{min}).\] This presents two options to choose the fixed resistance. Below, these are visualized.

Since the exact resistance range of the velostat sensor is dependent on the size of the velostat strip, the exact position of the conductor contacts, etc. and can therefore vary from piece to piece. Therefore, I would choose to use the second proposed estimate, the arithmetic mean of the range boundaries.

Attaching Cables to Fabric

WS2812 LEDs

In the output-devices week (10), I designed a PCB for integrating WS2812B LEDs and set them up to indicate pitch and roll from the gyroscope. The idea was to mount the LEDs onto the glove later on.

Vibration Motor

To have some haptic feedback, I researched on how to integrate a vibration motor in a circuit. Details can be read in the output-devices week (10) page.

Richard gave me the hint: there are different types of vibration motors, coin-style and linear vibration. The problem with coin-style vibration motors is that they have a longer step response. In the entertainment industry where ibration motors are used for feedback they use linear motors.