Final Project: Patchlet

My final project was a wireless and wearable MIDI controller bracelet, called patchlet. It sends generic MIDI CC messages based on the pitch and roll. Those could then be mapped to control parameters in light software, but also synthesizers, visual art software, etc.. All design files and code can be downloaded at the bottom of this page to be able to reproduce it.

The Patchlet

An important part of music culture events is not only the music, but also visual art as well as the light scenery. My idea was to have wearable, wireless interface in form of a glove or a bracelet including several sensors (accelerometers, vector magnetometer, etc.). The device could then send control signals to a music or light component, e.g. via OSC or MIDI. This would enable light operators, musicians and visual artists to change parameters in their setup while moving and/or being away from their personal computer.

Alternative ideas for potential final projects are listed at the final project proposal page.

BOM

The material costs for the final project therefore are 39.27€. The item with the largest cost, the Adafruit BNO085 board containing the gyroscope, can be substituted to directly use this item of 3.84€. This way, the cost per piece could be decreased to roughly 21€.

Beside that, soldering material (including soldering tin, flux, etc.) is needed.

Prior Work & Literature

Wearables

• Imogen Heap using MiMU gloves: https://youtu.be/3QtklTXbKUQ?feature=shared&t=554
  • In the example, she uses them to modulate e.g. panning
  • The inputs are
    • pitch, yaw, roll, specific positions that can be reached,
    • acceleration,
    • position of the single fingers,
    • and the position of all fingers (e.g. when making a fist).
  • There is a software included with the controller so that different things can be mapped.
  • She uses the glove in the following ways
    • Fist: recording audio input (voice), can be looped.
    • Another gesture: deleting the loop.
    • Finger bending: each finger activates an effect, hand position changes parameter.
  • The MiMU glove can also be used for air drumming. Different types of drumming movements are supported.
    • wrist flick
    • slap
• Modular body bend sensors, using velostat
• Hand muscle trainer

Other glove projects

• https://hackaday.io/project/8455-power-glove-universal-hid
• https://www.youtube.com/watch?v=NjTWAy4-lLI

Communication & MIDI

• modular MIDI sequencer

Programming

In the input-devices week (9), I set up a gyroscope for reading pitch and roll of the glove.

In the networking-and-communication week (11), I set up the ESP32S3 to send MIDI signals to the laptop via USB-MIDI. Together with my fellow student Niclas, I set up two ESP32S3 boards to communicate with each other. Both can be seen in the two videos below.

At that point I was only using the ESP-IDF for development, because I did not find a USB(-MIDI) library that worked with the ESP32S3 available for the Arduino environment. The only ESP-IDF library available for the BNO085 board only implemented an SPI interface for communication with the sensor board. The developers argued that the I2C communication at the side of the Adafruit board behaves faulty sometimes, so they did not even implement it. However, when I started integrating the gyroscope with the MIDI communication, the SPI communication with the sensor board sometimes did not work.

The measured data are packed into the following struct.

1typedef struct __attribute__((packed)) {
2    uint8_t pitch;
3    uint8_t roll;
4} MIDIData_t;

The struct is packed. That means that data are written directly after another in the memory and there is no padding added. It is explained in this forum post as well. I did this to have complete control over how the structure is assembled on bit-level. The reason for that is that Niclas and I had issues with correctly casting received packets. I do not know if the bit-level position of the values in the struct were playing a role or if the problem was something else, but it worked. Sending the values from the remote to the dongle can be seen below.

I then programmed the dongle to process the pitch as usual CC messages, but the roll as NOTE_ON/NOTE_OFF messages. Furthermore, the remote only sends MIDI data for the gyroscope’s pitch if the value is larger than 20. The reason is how MIDI controllers are mapped to software parameters. It is done by telling the software to listen to any MIDI input and then only change the one knob to be mapped to a parameter. To be able to either send pitch CC messages or send NOTE_ON/OFF messages, the user needs to be able to send only one of type of message at a time. With the current implementation the user just needs to hold the controller so the front faces down to deactivate sending pitch values. This, however, is not optimal, since it is not possible for the user to send values below 20. One could, however adapt the range of the CC value so that the current value 20 would correspond to a new value of 1. The code can be seen below.

 1void midi_send_data(MIDIData_t data)
 2{
 3    // Use roll as a switch.
 4    // ..when the gyroscope is turned downwards, a note-on message is sent. When it is turned upwards, a note-off message is sent.
 5    const int roll_threshold = 63;
 6    // ..introduce hysteresis to prevent oscillations of states.
 7    const int roll_thresh_upper = roll_threshold + 10;
 8    const int roll_thresh_lower = roll_threshold - 10;
 9    if (data.roll > roll_thresh_upper && !midi_roll_last_msg_was_note_on){
10        uint8_t note_on_roll[3] = {
11            NOTE_ON | CHANNEL,
12            MIDI_ROLL_NOTE_NUM,
13            120, // Velocity. 
14        }; 
15        // ..ensure, NOTE_ON and NOTE_OFF can only occur alternating.
16        midi_roll_last_msg_was_note_on = true;
17        #ifdef USB
18        tud_midi_stream_write(CABLE_NUM, note_on_roll, 3);
19        #endif
20        ESP_LOGE(TAG, "NOTE_ON");
21    } else if (data.roll < roll_thresh_lower && midi_roll_last_msg_was_note_on) {
22        uint8_t note_off_roll[3] = {
23            NOTE_OFF | CHANNEL,
24            MIDI_ROLL_NOTE_NUM,
25            120, // Velocity. 
26        }; 
27        midi_roll_last_msg_was_note_on = false;
28        #ifdef USB
29        tud_midi_stream_write(CABLE_NUM, note_off_roll, 3);
30        #endif
31        ESP_LOGE(TAG, "NOTE_OFF");
32    }
33
34    // Use pitch as knob.
35    // ..only activate if certain threshold is exceeded.
36    uint8_t pitch_inv = 127 - data.pitch;
37    const int pitch_thresh = 20; 
38    if (pitch_inv > pitch_thresh){ 
39        uint8_t cc_msg_pitch[3] = { 
40            CONTROL_CHANGE | CHANNEL,
41            MIDI_PITCH_CONTROL_NUM, // Index. Also known as "control number". CC 16-19 indicates a "generic control".
42            pitch_inv, // Data. Also known as "control value". Encodes the "position" of the "knob".
43        };
44        #ifdef USB
45        tud_midi_stream_write(CABLE_NUM, cc_msg_pitch, 3); 
46        #endif
47    }
48}

The below video shows the controller operating the cutoff-value of a low-pass filter which again is applied to a distorted sine oscillator.

PCB Fabrication

In the electronics design week (6) I learned how to use KiCad for designing PCBs. In the electronics production week (8) I learned how to fabricate a PCB using a CNC mill for isolation milling. The pictures below show the development process of the electronics of the project.

Problems

System Integration & Assembly

Designing the Housing

3D and 2D design was taught in multiple weeks. Most about CAD I learned during the computer-controlled cutting week (3), the computer-controlled machining week (7), and mechanical & machine design week (12). As a CAD software I used FreeCAD. With FreeCAD I designed the housings for the PCBs, as well as the wristbands to mount the remote onto the forearm. Below the different features of the housing is explained. The design is parametric, but rather to document specific dimensions than to make it adjustable. Some parameters can be adjusted, be careful though. There are two FreeCAD projects, one for the dongle and one for the remote. Each project consists of multiple files. In each project, the file called d.FCStd contains a spreadsheet with the corresponding parameters. There, most dimensions are expressed as a sum of an actual dimension and a tolerance associated with it. This way, the dimensions and tolerances can be understood.

Old Designs

Fabricating the Wristband

To mount the PCB onto the forearm, two wristbands were fabricated. These were then threaded through the pockets in the bottom part of the housing.

Problems

Evaluation

The product was evaluated by

• shaking and dropping it to check if any parts were loose,
• connecting the dongle to the PC and mapping the MIDI controls to a synthesizer plug-in.
• using the controller for performing visual art and re-iterating on where it did not feel good from the perspective of a user.

Implications

Implications were documented in week 18.

Answered Questions

In week 18 part of our assignment was to state questions we would need to answer to finish the final project. Below the questions and answers to them can be seen.

• On which inputs should the vibration motor be programmed to be feedback?
  • Not answered, since the vibration motor was not included
• How is the cut pattern for cutting the glove look and how should it be attached to a hand?
  • This was not answered, but it was found that a normal hand-like pattern from polyester is not elastic at all and therefore makes it impossible to function as a hand overlay for mounting bend sensors.
• How is the PCB integrated with the glove?
  • The PCB would be integrated with the glove by mounting ribbon cables coming from the PCB using snap fasteners that are sewn onto the glove.
• How is the IMU integrated with the ESP-IDF?
  • It is not. Instead of the ESP-IDF, the Arduino environment was used for programming the remote containing the Adafruit BNO085 board.
• How to built a sensor for detecting pressure at the fingertips using velostat? Is this even possible?
  • Building pressure sensors was excluded from the scope of the project during development.
• How are the sensor data processed? (Acceleration peaks, noisy bend sensor signals)
  • As of now, there is not much data processing implemented by me. I only use events and values provided by the Adafruit board.
• How are the bend sensors calibrated?
  • No bend sensors were used so they did not need to be calibrated.

Further Work

There are many ideas to be implemented. Some of them are listed below.

• Second controller for the other hand.
• NOTE_ON message on acceleration peaks.
• Relative position of hands to each other as CC input.
• Glove-like overlay for the hand, including
  • (velostat) bend sensors on the fingers,
    • http://iainmccurdy.org/diy/bendsensor/bendsensor.html
    • https://www.instructables.com/Fabric-bend-sensor/
  • pressure sensors for drumming,
  • WS2812B LEDs
    • attached to the back of the hand to indicate pitch and roll
    • attached to index, middle, ring fingers to indicate finger bends
  • PCB on fabric: either embroidered or sewn, just like the Lilypad
  • vibration motor for feedback
• Adding OSC mode to use controller as OSC controller. Alternatively, a converter can be created using e.g. puredata.
• Case for everything to carry around.
• Improve feel of wristband. It is not yet suitable for all wrists.
• Replace Adafruit IMU breakout board with the actual SMD part.
• Power management
  • Add temperature monitoring.
  • Add voltage monitoring.
  • Research again about industry-grade battery management.
• Calculate position from acceleration values to have it as a control input.
• Laser TOF sensor to measure distance of palm from floor, send as CC message. Only turn on if palm is turned down.

Further links

• https://www.digikey.com/en/products/detail/gct/FFC3B11-04-T/10657221
• https://www.digikey.com/en/products/detail/molex/0982670701/3469873
• https://i.ytimg.com/vi/7ZRpyZp4gq8/maxresdefault.jpg
• https://duckduckgo.com/?q=lilypad+arduino
• https://fabricademy.fabcloud.io/handbook/

Bend Sensors

In the end, I did not include any bend sensors, but I started building them. For that I used velostat, a material that changes resistance when it is compressed. For making bend sensors using velostat and fabric, this tutorial could be considered. Additional resources include

• Post about making drum pads
• Playlist on making velostat-based sensors

  

  

For measuring the changing resistance one could use a wheatstone bridge or a voltage divider. I decided to use a voltage divider for the first iteration. The voltage above the variable resistance, according to the voltage devider rule, is \[V_{out} = V_{CC} \frac{R_{var}}{R_{fixed} + R_{var}}\] where \(R_{var}\) is the resistance of the velostat-strip and \(R_{fixed}\) is the value of the other resistor in the voltage divider. While the range of \(R_{var}\) is given, \(R_{fixed}\) can be chosen by the designer to maximize the range of \(V_{out}\). The reason for this is to use the full range of the ADC on the micro-controller so the impact of the quantization noise is minimized. Another point to consider was that the measured voltage should not be too close to the boundary of the ADC’s range. Otherwise, the ADC might saturate and the measured signal would be distorted. Following this objective, the voltage divider would need to be chosen so that the measured voltage is centered around the center of the ADC’s range.

The question now was how to choose \(R_{fixed}\). I chatted with a language model (GPT 4) to quickly gain an overview (the chat). The following I understood from that. The objective to maximize the voltage range could be expressed as \[\Delta V_{out} = V_{CC} \cdot \left( \frac{R_{max}}{R_{fixed} + R_{max}} - \frac{R_{min}}{R_{fixed} + R_{max}} \right).\] where \(R_{max}\) and \(R_{min}\) are the boundaries of \(R_{var}\). The term \[R_{fixed} = \sqrt{R_{max}\cdot R_{min}}\] is a rule of thumb for the estimated maximum. Due to time constraints, it was only verified graphically below. The \(R_{fixed}\) value to center \(V_{out}\) can be calculated by calculating the voltage divider factor to be \(0.5\), if \(R_{var}\) is centered. Setting \(R_{var}\) to the center of the resistance range, i.e. \(R_{var} = 0.5\Delta R + R_{min}\) with \(\Delta R = R_{max} - R_{min}\) leads to \[V_{out} = V_{CC} \frac{0.5\Delta R + R_{min}}{R_{fixed} + 0.5\Delta R + R_{min}}\]. Requiring the factor term of resistances to be \(0.5\) leads to \[R_{fixed} = 0.5(R_{max} + R_{min}).\] This presents two options to choose the fixed resistance. Below, these are visualized.

Since the exact resistance range of the velostat sensor is dependent on the size of the velostat strip, the exact position of the conductor contacts, etc. and can therefore vary from piece to piece. Therefore, I would choose to use the second proposed estimate, the arithmetic mean of the range boundaries.

Attach Cables to Fabric

WS2812 LEDs

In the output-devices week (10), I designed a PCB for integrating WS2812B LEDs and set them up to indicate pitch and roll from the gyroscope. The idea was to mount the LEDs onto the glove later on.

Vibration Motor

To have some haptic feedback, I researched on how to integrate a vibration motor in a circuit. Details can be read in the output-devices week (10) page.

Richard gave me the hint: there are different types of vibration motors, coin-style and linear vibration. The problem with coin-style vibration motors is that they have a longer step response. In the entertainment industry where ibration motors are used for feedback they use linear motors.

Surface finish

If it is not enough to just have a print, but to have a very smooth surface, there are different methods to achieve that. One is to use this product by Smooth-On. Another method is to use filler material (called Bondo or Spachtelmasse, in particular Feinspachtel). The procedure is as follows

• Put on safety equipment as described on the product. In my case this was ear, eye and respiratory protection as well as gloves.
• Add a little bit of hardener (included in the product) to the filler material and mix it well.
• Put a good amount onto the print so that every surface is covered. Do not put too much, as it would need to be grind away later.
• Let everything dry, roughly 24 hours.
• Grind everything to have a smooth surface wearing gloves to not put any residual fat or dirt onto the object (also wearing protection as above).
  • first with roughness 120 (not too rough, otherwise this could add gaps to the surface).
  • then with even finer roughness (like 500)
• Then, spray the part.
  • Be careful! Have the nozzle 15 to 20 cm away from piece to not put too much coating in one place.
  • Wait 15 minutes for one layer to dry.
  • Add multiple layers while grinding between adding them. Again, be careful.

Digital Files

• Code
  • code.zip The dongle sub-project is an ESP-IDF project and the remote sub-project is a PlatformIO project.
• PCB designs (KiCAD)
  • pcb.zip
  • pcb-dongle.zip
• CAD (FreeCAD), .step, and .dxf files of all 2D and 3D designs (except circuit and PCB designs)
  • housing.zip
  • housing-dongle.zip