Final Project: Patchlet

Table of Contents

My final project was a wireless and wearable MIDI controller bracelet, called patchlet. It sends generic MIDI CC messages based on the pitch and roll. Those could then be mapped to control parameters in light software, but also synthesizers, visual art software, etc.. All design files and code can be downloaded at the bottom of this page to be able to reproduce it.

The Patchlet

An important part of music culture events is not only the music, but also visual art as well as the light scenery. My idea was to have wearable, wireless interface in form of a glove or a bracelet including several sensors (accelerometers, vector magnetometer, etc.). The device could then send control signals to a music or light component, e.g. via OSC or MIDI. This would enable light operators, musicians and visual artists to change parameters in their setup while moving and/or being away from their personal computer.

This sketch shows a first idea of a glove-like MIDI controller.
Possible sensors to include were a gyroscope for detecting pitch and roll of the hand and pressure (plus maybe bend) sensors at some fingertips. That would enable the user to use finger drumming and bending to send MIDI signals. The scope was later narrowed down to only include the gyroscope.

Alternative ideas for potential final projects are listed at the final project proposal page.

BOM

Item Price/Unit (€) Quantity Where from
XIAO ESP32S3 6.09 2 AliExpress
ADAFRUIT 9-DOF ORIENTATION IMU F 21.85 1 Digikey
Pin sockets SMD vertical, 1x20, (actually needed per piece: 1x7 (2x) + 1x6 (2x)) 1.99 1.5 Conrad
Pin headers THT vertical, 1x40, (actually needed per piece: 2 1x7, 2 1x6) 0.46 1.6 Conrad
DIN A3 Plywood, 3mm thick 2.50 0.005 Amazon
Prusament PETG, Urban Grey 27.99 0.055 Prusa
M2x8 mm wood screws (300 pack, actually needed: 5) 5.99 0.1 Amazon
USB C to USB A cable 1.22 1 Amazon
FR1 sheet 0.53 0.5 Amazon
Velcro tape for sewing, 16mm x 3m (hooks + loops) 6.99 0.2 Amazon
Polyester 2.20 0.17 Makema
Sewing thread 4.30 less than one roll Makema
Jeans needle 0.91 0.1 Amazon

The material costs for the final project therefore are 39.37€. The item with the largest cost, the Adafruit BNO085 board containing the gyroscope, can be substituted to directly use this item of 3.84€. This way, the cost per piece could be decreased to roughly 21€.

Besides that, soldering material (including soldering tin, flux, etc.) is needed.

Prior Work & Literature

Wearables

  • Imogen Heap using MiMU gloves: https://youtu.be/3QtklTXbKUQ?feature=shared&t=554
    • In the example, she uses them to modulate e.g. panning
    • The inputs are
      • pitch, yaw, roll, specific positions that can be reached,
      • acceleration,
      • position of the single fingers,
      • and the position of all fingers (e.g. when making a fist).
    • There is a software included with the controller so that different things can be mapped.
    • She uses the glove in the following ways
      • Fist: recording audio input (voice), can be looped.
      • Another gesture: deleting the loop.
      • Finger bending: each finger activates an effect, hand position changes parameter.
    • The MiMU glove can also be used for air drumming. Different types of drumming movements are supported.
      • wrist flick
      • slap
  • Modular body bend sensors, using velostat
  • Hand muscle trainer

Other glove projects

Communication & MIDI

Fabrication

Programming

In the input-devices week (9), I set up a gyroscope for reading pitch and roll of the glove. In the networking-and-communication week (11), I set up the ESP32S3 to send MIDI signals to the laptop via USB-MIDI. Together with my fellow student Niclas, I set up two ESP32S3 boards to communicate with each other.

In the system integration week (15), I integrated the gyroscope with the MIDI communication. Initially, I developed entirely using the ESP-IDF. However, I faced reliability issues with the SPI interface for the BNO085 sensor using its library for the ESP-IDF. It was not possible to use the I2C interface in the BNO085 library for the ESP-IDF. Therefore, I switched to using the Arduino environment for the remote to enable communication with the sensor via I2C. The dongle I kept programming using the ESP-IDF for USB-MIDI support, which again was not available in the Arduino environment. ESP-NOW was used to transmit pitch and roll of the BNO085 sensor. Finally, I designed MIDI behavior where pitch controls CC messages (only when above a threshold) and roll triggers NOTE_ON/OFF messages. The dongle could then be registered as a MIDI controller and mapped in an arbitrary digital audio workstation for controlling filter cut-off, pitch, etc.

Electronics

In the electronics design week (6) I learned how to use KiCad for designing PCBs. In the electronics production week (8) I learned how to fabricate a PCB using a CNC mill for isolation milling. In the system integration week (15) I designed and fabricated PCBs for the remote and the dongle. Below one can see the final setup without housing.

The final setup without housing.

Packaging

3D and 2D design was taught in multiple weeks. Most about CAD I learned during the computer-controlled cutting week (3), the computer-controlled machining week (7), and mechanical & machine design week (12). As a CAD software I used FreeCAD. With FreeCAD I designed the housings for the PCBs, as well as the wristbands to mount the remote onto the forearm. Details can be read on the page on the system integration week (15).

CAD for the remote housing.
CAD for the dongle housing.
Wristband.

Assembly

During the design process of electronics and packaging much thought went into how to assemble everything. Details are documented on the page about the system integration week (15).

The assembled project.

Evaluation

The product was evaluated by

  • shaking and dropping it to check if any parts were loose,
  • connecting the dongle to the PC and mapping the MIDI controls to a synthesizer plug-in.
  • using the controller for performing visual art and re-iterating on where it did not feel good from the perspective of a user.

Implications

Implications were documented in week 18.

Answered Questions

In week 18 part of our assignment was to state questions we would need to answer to finish the final project. Below the questions and answers to them can be seen.

  • On which inputs should the vibration motor be programmed to be feedback?
    • Not answered, since the vibration motor was not included
  • How is the cut pattern for cutting the glove look and how should it be attached to a hand?
    • This was not answered, but it was found that a normal hand-like pattern from polyester is not elastic at all and therefore makes it impossible to function as a hand overlay for mounting bend sensors.
  • How is the PCB integrated with the glove?
    • The PCB would be integrated with the glove by mounting ribbon cables coming from the PCB using snap fasteners that are sewn onto the glove.
  • How is the IMU integrated with the ESP-IDF?
    • It is not. Instead of the ESP-IDF, the Arduino environment was used for programming the remote containing the Adafruit BNO085 board.
  • How to built a sensor for detecting pressure at the fingertips using velostat? Is this even possible?
    • Building pressure sensors was excluded from the scope of the project during development.
  • How are the sensor data processed? (Acceleration peaks, noisy bend sensor signals)
    • As of now, there is not much data processing implemented by me. I only use events and values provided by the Adafruit board.
  • How are the bend sensors calibrated?
    • No bend sensors were used so they did not need to be calibrated.

Further Work

There are many ideas to be implemented. Some of them are listed below.

  • Second controller for the other hand.
  • NOTE_ON message on acceleration peaks.
  • Relative position of hands to each other as CC input.
  • Glove-like overlay for the hand, including
  • Adding OSC mode to use controller as OSC controller. Alternatively, a converter can be created using e.g. puredata.
  • Case for everything to carry around.
  • Improve feel of wristband. It is not yet suitable for all wrists.
  • Replace Adafruit IMU breakout board with the actual SMD part.
  • Power management
    • Add temperature monitoring.
    • Add voltage monitoring.
    • Research again about industry-grade battery management.
  • Calculate position from acceleration values to have it as a control input.
  • Laser TOF sensor to measure distance of palm from floor, send as CC message. Only turn on if palm is turned down.

Further links

Bend Sensors

In the end, I did not include any bend sensors, but I started building them. For that I used velostat, a material that changes resistance when it is compressed. For making bend sensors using velostat and fabric, this tutorial could be considered. Additional resources include

For measuring the changing resistance one could use a wheatstone bridge or a voltage divider. I decided to use a voltage divider for the first iteration. The voltage above the variable resistance, according to the voltage devider rule, is \[V_{out} = V_{CC} \frac{R_{var}}{R_{fixed} + R_{var}}\] where \(R_{var}\) is the resistance of the velostat-strip and \(R_{fixed}\) is the value of the other resistor in the voltage divider. While the range of \(R_{var}\) is given, \(R_{fixed}\) can be chosen by the designer to maximize the range of \(V_{out}\). The reason for this is to use the full range of the ADC on the micro-controller so the impact of the quantization noise is minimized. Another point to consider was that the measured voltage should not be too close to the boundary of the ADC’s range. Otherwise, the ADC might saturate and the measured signal would be distorted. Following this objective, the voltage divider would need to be chosen so that the measured voltage is centered around the center of the ADC’s range.

The question now was how to choose \(R_{fixed}\). I chatted with a language model (GPT 4) to quickly gain an overview (the chat). The following I understood from that. The objective to maximize the voltage range could be expressed as \[\Delta V_{out} = V_{CC} \cdot \left( \frac{R_{max}}{R_{fixed} + R_{max}} - \frac{R_{min}}{R_{fixed} + R_{max}} \right).\] where \(R_{max}\) and \(R_{min}\) are the boundaries of \(R_{var}\). The term \[R_{fixed} = \sqrt{R_{max}\cdot R_{min}}\] is a rule of thumb for the estimated maximum. Due to time constraints, it was only verified graphically below. The \(R_{fixed}\) value to center \(V_{out}\) can be calculated by calculating the voltage divider factor to be \(0.5\), if \(R_{var}\) is centered. Setting \(R_{var}\) to the center of the resistance range, i.e. \(R_{var} = 0.5\Delta R + R_{min}\) with \(\Delta R = R_{max} - R_{min}\) leads to \[V_{out} = V_{CC} \frac{0.5\Delta R + R_{min}}{R_{fixed} + 0.5\Delta R + R_{min}}\]. Requiring the factor term of resistances to be \(0.5\) leads to \[R_{fixed} = 0.5(R_{max} + R_{min}).\] This presents two options to choose the fixed resistance. Below, these are visualized.

Visualization of the above equations using GeoGebra to verify that the first proposed estimate is indeed an estimate for the maximum of the voltage range and to check if the second proposed estimate varies significantly from the first one.

Since the exact resistance range of the velostat sensor is dependent on the size of the velostat strip, the exact position of the conductor contacts, etc. and can therefore vary from piece to piece. Therefore, I would choose to use the second proposed estimate, the arithmetic mean of the range boundaries.

Attach Cables to Fabric

WS2812 LEDs

In the output-devices week (10), I designed a PCB for integrating WS2812B LEDs and set them up to indicate pitch and roll from the gyroscope. The idea was to mount the LEDs onto the glove later on.

Vibration Motor

To have some haptic feedback, I researched on how to integrate a vibration motor in a circuit. Details can be read in the output-devices week (10) page.

Schematic for integrating a DC motor.

Richard gave me the hint: there are different types of vibration motors, coin-style and linear vibration. The problem with coin-style vibration motors is that they have a longer step response. In the entertainment industry where ibration motors are used for feedback they use linear motors.

Digital Files