Wildcard week
Week 15 - Documentation
Assingment
Design and produce something with a digital fabrication process (incorporating computer-aided design and manufacturing) not covered in another assignment, documenting the requirements that your assignment meets, and including everything necessary to reproduce it. Possibilities include (but are not limited to):
...
Embedded programming: machine learning
...
This week (actually, many weeks) was full of errors, but it's part of the learning process. What was interesting was getting to know all the trouble of getting a machine learning model to a microcontroller. I started the week full of expectations, setting a goal which I made related to my final project, but things went south, very very south.
My initial project was to train an ML model that could recognize music notes. Therefor, I recorded a database of 140 files containing 20 records of the “do, re, mi, fa, sol, la, si” notes.
Then I wrote a python script in Collab to train my model. You can go check the notebook here: music-notes
And I even got the model in an .h format, but I could not get pass here…
I needed a Mel-Frequency Cepstral Coefficients (MFCC) extractor to get the features from the recorded audio. Looking for a library in Arduino IDE that could help with that I couldn't find any.
So, I looked for some guidance in google and found out about Edge Impulse…
Edge Impulse is an AI platform for model's optimization and development.
So, I uploaded the music notes database to train it again and export it a Tensorflow Lite file. Tensorflow Lite is a framework for implementing machine learning models in microcontrollers.
I trained the model, I know that the results look bad, but it was the first model and I wanted to try downloading it to the MCU.
I deployed the model as a library for Arduino having the facility that Edge Impulse already gave me the option.
Building process:
Once the model is created as an Arduino library it gets downloaded automatically.
In the Arduino IDE you add the .zip library you have just downloaded.
Then, I go to examples and open the esp32_microphone_continuous file.
When I compiled the program, it did it without errors.
Nevertheless, I couldn't run it in the esp32 because I could not understand the code! The code lacked explanations and Edge Impulse didn't have a clear tutorial for it. I search online and find different YouTube videos how made it work, but I didn't have an esp32 with a microphone included, so I had to wire a separate one, for which I was unable to make the change in the code. I looked for different documentations, searched in forums like GitHub and Stack Overflow, but it was a failed search for me.
Looking for other solutions I found the youtube channel of Seeed Studios, where they presented a speech recognition project using the XIAO nRF52840 Sense.
So, I went on looking for the official Seed Studio page. Miraculously, I have one of these microcontrollers, so I thought it was going to be light after the tunnel.
I found the tutorial I needed, like sent from heaven…
I opened the Google Colab page they made for training a model. I didn't change anything as I wanted to make it work exactly as they have for then make my own model.
Unfortunately, the code kept giving errors, mostly because of updated versions of the libraries and frameworks. Again, I did everything in my coding and searching capabilities in the given time, but I couldn't make it work. I still want to make clear that I'm using the code and examples as given from Seeed Studio…
Alright then, I'm not able to train a custom model (or any given model by the developer) so I'm going to use the example code given from Seeed Studio for the Xiao I'm using.
I made no changes to the code and tried to compile it, but guess what? It was unable to compile and showed me errors regarding the Arduino tensorflow lite library. This library is not available anymore, so it is deprecated and not functional. This is the error using Seeed Ciao BLE – XIAO nRF52840 Sense with the Micro_speech example:
Ok, maybe it's my fault for jumping from the Speech Recognition documentation without seeing Getting Started…
So, I'm going to follow the initial documentation they have.
So, I'm using the program example I downloaded from their library to capture the data from the IMU sensor.
The project they introduce us here is to obtain the data of the sensor while punching and flexing so then we can train a machine learning model to help us predict our movement.
Gifs taken from:Seeed Studio
Everything great here, I, actually, made an upgrade to this part and made a python code to store the data.
I was able to train a model using their Colab example and it went well.
I could get the Tenso Flow Lite model.
Winning this battle gave me high hopes, but the war wasn't over. I ran the program from the tutorial, and it gave me the same error as the micro_speech program. Therefor, I tried the example of the example code and the same happened.
Error using Seeed Xiao BLE – XIAO nRF52840 Sense with the IMU_Classifier example:
At this point, I only wanted to download any machine learning model to a microcontroller to prove that it was possible, anything was enough for me. So, I went to most basic, a Sine predictor. This model is the “Hello World” of machine learning. After a little research I got into Eloquent Tiny ML. I installed the libraries and followed the tutorials. The goal of this program is to predict the sine function based on an input number. First, I trained a model using a Python program.
Results
I copied the output of the training process to a new file in .h format to use it as an Arduino library.
I used the code of the Sine_example from the EloquentTinyML library.
And guess what, it didn't work. Error using Seeed Xiao RP2040 with the Sine_example:
I even used Arduino Web Editor thinking it was an error of library installation, but it kept giving the same error…
And after many other things I tried to get a machine learning model to a microcontroller unit and failing in the process I could fix the sine example with the help of the Eloquent Arduino official page. I could make it work.
I run the code in an esp32 and finally got a machine learning model to a microcontroller unit.
The video shows the sine product of a random number and the prediction based on the model training. As it can be seen the prediction is very close to the real number, therefor we can conclude that the training was succesful.
Discussion
This assignment was the most stressful I did in the Fab Academy, what is ironic is that I chose what to do. I don't consider myself a beginner in machine learning, I've done a few projects and published some papers using it. Nevertheless, working with embedded machine learning was frustrating. The lack of documentation and updated scripts from the developers makes it really hard to work with it. It is possible to have embedded machine learning in microcontrollers, but there is need of better documentation and continuous updates to it so that everyone can have access to it.
Here, you can download the :