Assignment

Design, build, and connect wired or wireless node(s) with network or bus addresses

Process

For this week's assignment i chose to take on the work done in previous weeks and expand it. I will continue working with the UDP based connectivity of the OSC protocol to send messages back and forth the board and a device. I already did a bit of this on Week 9 - Embedded Programming and Week 10 - Input Devices.

For this week though, i will be using a novel device i have never worked on. On Friday i got a Microsoft Hololens 2 from my work, so the choice was clear. While i go through all the process about learning how to program on it, i will be doing things with the goal of communicating an 3D holographic AR app with the readings of a sensor on the ESP32-based Barduino board. More on this on the Week 9 intro.

The new hololens 2. State of the art AR

The idea is to set up a bidirectional communication, so that we can send information from the hololens to the microcontroller, and from the microcontroller to the hololens with OSC messages through UDP. Both devices will be on the same network so there shouldn't be many issues with communication.

From the official website: "Open Sound Control (OSC) is a protocol for communication among computers, sound synthesizers, and other multimedia devices that is optimized for modern networking technology. Bringing the benefits of modern networking technology to the world of electronic musical instruments, OSC's advantages include interoperability, accuracy, flexibility, and enhanced organization and documentation".

For the programming part i will go with Unity 3D since i'm very familiar with it and it is probably the easiest way to get something done for the hololens.

Hololens setup

The first will be to set up the hololens so that we can program for it. This step by step guide from Microsoft has everything that you need to get something going. It is really important to make sure you have all the prerequisites installed. In my case i sort of ignored the windows SDK part assuming i would already have it and i didn't.

Once everything is installed and setup, we can open Unity and create an app, or open an example from the downloaded packages, which i strongly recommend. To set up Unity in order to deploy for the hololens, here's yet another step by step guide.

The building process is similar to when building apps for iOS, where Unity builds xcode projects that you then open and deploy your game into your ipad or iphone. In this case, Unity builds a Visual Studio project that you then open on it's own and deploy on the hololens. This whole process is a bit cumbersome, but i'm sure it will get easier, and you'll be able to play it from within Unity in a few months.

But before actually building any app, we need to set the hololens into developer mode to be able to access it's file system. And then configure Visual Studio to run the app. This guide has again everything we need, step by step.

Alright

Enough of setups and initializations. Our hololens is now set up and ready to go. We've been able to open an example on Unity, and load it and run it on the hololens. Now what.

I'll focus on something simple since setting this whole thing up took more time than expected. What i'll try to do is have the ESP32 read a temperature sensor and send the values to the hololens, where i will render them. At the same time, i will use the hololens to use a 3D AR UI to control an SMD LED on the board. The diagram will be as follows:

The communications diagram

With my COVID-19-sensor-pack came a ds18b20 temperature sensor. I followed this guide to get it going with the code provided in it, which uses two libraries i hadn't seen before" OneWire.h to deal with 1 data wire sensors, and DallasTemperature.h to convert the readings into human-readable temperatures (C and F)

At the same time i reviwed the instructions to get an SMD working, just to be sure. I found this link very helpful

I started to put together previous assignments code with these 2 snippets. You can find the files at the end of the page, but here are some highlights:

Sending the temperature OSC message from the board (in loop function):
float temperatureInCelsius = sensors.getTempCByIndex(SENSOR_INDEX);

// Send the value to hololens
OSCMessage msg("/fromboard");
msg.add(temperatureInCelsius);
Udp.beginPacket(outIp, outPort);
msg.send(Udp);
Udp.endPacket();
msg.empty();

Receiving OSC messages from the hololens (in loop function):
// Receive led values from hololens UI
OSCMessage msgr;
int size = Udp.parsePacket();
if (size > 0) {
  while (size--) {
    msgr.fill(Udp.read());
  }
  if (!msgr.hasError()) {
    msgr.dispatch("/toboard_r", ledR);
    msgr.dispatch("/toboard_g", ledG);
    msgr.dispatch("/toboard_b", ledB);
    msgr.dispatch("/toboard_toggle", led);
  } else {
    OSCErrorCode error = msgr.getError();
    Serial.print("error: ");
    Serial.println(error);
  }
}

And then programming the callback:
void ledG(OSCMessage &msg) {
  g = round(msg.getFloat(0)*255.0);
  //Serial.println("G msg: " + String(g));
}

One thing i said on previous assignments is that i would change the OSC library i used for the ESP32 to one with more possibilities, the previous one i had used, while simpler, only allowed for a certain number of incoming messages. For this i moved to this one developed by Berkeley's CNMAT center (Center for New Music and Audio Technologies). You can find it here

Also, i also pledged on the previous assignments to change the OSC library on the Unity end, since for the previous assignments i had to route the messages from Unity to ESP32 through an intermediate layer programmed in VVVV. I finally used this one, but after much digging i found out the issue was not related to the OSC library, but to Windows firewall blocking the messages.

This firewall problem went away when i disabled the firewalls on windows. But alas! i cannot do that on the hololens, which is what i really want. After searching for an answer for much time, i finally found this post that, although talking about a different problem, pointed me to a section of the Unity config where you could check some boxes to allow communications in and out. You can find it in the player settings, under the capabilities section. Look for "PrivateNetworkClientServer"

This wasn't the only weird problem i had to solve. While uploading the code to the board with the Arduino IDE, i was getting this: Warning: Could not auto-detect Flash size (FlashID=0xffffff, SizeID=0xff), defaulting to 4MB. After searching for an answer, i found this post on the espressif github issues where someone points out:

"Or possibly something is connected to GPIO12 and is pulling it high on reset. This sets 1.8V mode which will cause the WROOM internal flash to brown out."

That was it, by disconnecting the cable from port GPIO12 to the data pin of the temperature sensor, i could upload the code normally.

Using the hand interaction hololens example on Unity, i basically stripped many of the assets to keep just 3 things:

  • A slider group with R,G,B values to control the SMD color.
  • A button to turn on and off the LED.
  • A text where i could display the temperature.

I then added a kind of valve that will move with the temperature reading. These are the elements of the interface before laying them out on the 'real world'

UI elements on the Unity scene

There's nothing else to do, except running the example and placing all the elements where we want them. So I laid out the board and the breadboard on the table, run the application, placed the UI elemenets, and used to UI to see both devices communicating in the magic of these 3D holograms

Sadly, obviously a 2D video doesn't transmit the feeling of seeing hovering digital holograms over real world pieces and interchanging information between them. But there's no other way to see this.

Demo

Hololens has a very handy function where when holding the + and - volume buttons for 3 seconds, you can start recording video of what you see with all the AR stuff on top of it. Here's the final video:

The Files