12.

Interface and Application Programming

Home

Assignments

group assignment
compare as many tool options as possible

individual assignment
write an application that interfaces a user with an
input and/or output device that you made

Download the files

This week the only thing to download is the source code of the examples below.

Download source code files

Table of contents

Prologue

For this week I want to have an interface which shows you the humidity of soil overtime, and when it reaches a certain level it notifies you about watering. You can then either automatically water the soil until a specific measurement is reached, or for fun you could use the gyrometer of the phone to control a servo which waters the plant. When you do that, you could have a graphic of a glass of water filling up.

I’d also like to add a different feature more related to the armchair. The input would be you sitting down on the armchair (maybe through detecting a change of light when you sit and cover an LDR?). When this happens you send a notification to an app on your phone which tells you different things depending on the time of the day. It’s the morning? Then it tells you your calendar, first call. It’s the evening? Then it reminds you to drink a glass of warm milk. App background could be dark at night, but white in the morning. It could make coffee in the morning (although I’d want it before I go and sit down). Plenty of things.

Input: soil moisture. Output: “water dem plants”.

Calibrating the conductivity of the soil

To start I wanted to prove my concept, so I used screws in water and measured the resistance the water creates between them. It is around 7kΩ. I used the 200k setting at first, then switched to 20k for a higher precision one, although it doesn’t really matter, what’s important is to see the maximum conductivity.

alt_text

Later we’ll calibrate the conductivity of a completely dry soil, then water it until the soil has absorbed as much water as possible and calibrate the maximum conductivity then.

Note //
For now I am using water so that I can take the screws out of it and have direct feedback of the conductivity rather than having to wait for days for the soil to dry.

As I’m studying the bio voltaics I know the plants generate negative charge when growing… I wonder how that affects the conductivity of the soil, if at all?

UPDATE: for the rest of the project I decided to use a photoresistor to “simulate” in a shorter time the variation of the moisture sensor. I can’t wait for 2 days for the soil to get dry.

Using the ESP32 to send the data online

We’ll have the sensor connected to the ESP32 which can either host a webpage or connect to one online. It’ll send the moisture data, and from the app we’ll be able to react to it. We’d probably want to use the map() function to map the lowest voltage possible (from the calibration of when the soil is completely dry) to 0 and the highest voltage to 100 or whatever.

In our class Edu showed how we could use HTTP requests to send an “order” to the ESP32, something like 192.168.1.125/gpio/1. I wanted to see if it was possible to do it the other way, so I checked the documentation for the Wifi library. It helped me understand the code that Edu gave more in depth, the role of the server and the client, but I couldn’t figure out how to make the server send data to the client, rather than receiving it. I checked the WiFi library on the Arduino website with the write() functions for the Client and Server classes, but I just couldn’t seem to get it to work.

Instead I decided to use WebSockets.

Adapting a project using ESP32, WebSockets and P5.js

Here I take an example I found online which reads value from an IMU (Inertial Measurement Unit) and transmits it over WebSocket to a P5.js file which rotates the model.

Reveal the section

Mastering WebSockets

Having gone through a project and understood how WebSocket work, I created a sketch from scratch

after days of it mastering me...

Now that I've went through the libraries and understand how it works, let's do it nice and clean.

Note //
Since this documentation I found a different library for WebSockets with ESP32 by Gil Maimon which seems easier, better written and also support WSS (WebSocket Secure for using with HTTPS protocol).

Why use WebSockets?

  1. It is widely used and supported (96% compatible)
  2. It is quick and easy to learn and deal with *cough*
  3. It is used with Service Workers and Push Notifications on websites (great for PWAs)
  4. It is something I wanted to learn and this was the perfect excuse for it
  5. It is fast

Here’s the WebSocket Protocol (RFC 6455) bible.

Setting WebSocketsServer up in Arduino/ESP32

It took me two days to get it to work because I downloaded a repo which had everything I wanted: an ESP32 acting as WebServer with WebSockets sending messages to an HTML page with P5.js running on it. The only problem was that it wasn’t documented and did more complex things that I need. It took in values of an IMU, then converted them and sent them to the sketch.js page which made more complicated and confusing things. It wasn’t very clean code and there’s a few things that were missing that were throwing errors. For example the webSocketEvent function was calling hexdump() which wasn’t declared anywhere on the file and I had to find it on the arduinoWebSockets Github (in the end it wasn’t used at all). I had to go through the file and cut away code which didn’t seem relevant to what I need to do - anything that had to do with BNO (the IMU). I had also used this page which set up the project from scratch but I didn’t understand how to handle webSocket on Javascript. How to see incoming messages. I got it later. By playing around.

#includes

There were a few includes for libraries for the BNO which I had to get rid of it. By looking around I found which ones were the essential ones. In the repo I downloaded there is an if statement that includes different libraries depending on the board you’re using (in Tools > Board).

#if defined(ESP8266) // if you are using an ESP8266 use these libraries
#include <ESP8266WiFi.h>
#include <ESP8266WebServer.h>
#include <ESP8266mDNS.h>
ESP8266WebServer webServer(80); // start the webserver
#else // otherwise use the ESP32 libraries
#include <WiFi.h>
#include <WebServer.h>
#include <ESPmDNS.h>
WebServer webServer(80); // start the webserver
#endif
// Libraries needed no matter the board
#include <Wire.h>
#include <DNSServer.h>
#include <WiFiClient.h>
#include <WebSocketsServer.h>
#include <Hash.h>
// and include the HTML and p5.js files we shall host
#include "index_html.h"
#include "sketch_js.h"
Note //
Note that the index.html and sketch.js files have a .h extension. It will deal with renaming and giving them the proper extensions later with WebServer.

Declaring all needed variables

const int ldrPin = 36; // photoresistor
const int ledPin = 13; // LED
// Generally, you should use "unsigned long" for variables that hold time
unsigned long previousMillis = 0;
const long interval = 1000;
// Logins for WiFi
const char* ssid = "WIFI_NAME"; // name of WiFi network
const char* password = "PASSWORD"; // the password
// create a WebSocket on port 81
WebSocketsServer webSocket = WebSocketsServer(81);

The webSocketEvent() function

This handles anything to do with webSocket events, such as when it connects, is connected, receives a message and much more. Let’s go through it, it is pretty self-explanatory (now).

void webSocketEvent(uint8_t client_num, WStype_t type, uint8_t * payload, size_t length) {
switch(type) {
    case WStype_DISCONNECTED: // Np connexion is established
      // Below we print to Serial using printf() which formats the variable with the % sign
      Serial.printf("[%u] Disconnected!\n", client_num);
      break;
    case WStype_CONNECTED: // A connexion is established
        {
            IPAddress ip = webSocket.remoteIP(client_num); // we get the IP address of the client
            Serial.printf("[%u] Connected from %d.%d.%d.%d url: %s\n", client_num, ip[0], ip[1], ip[2], ip[3], payload); // and print it to serial

            // here’s the interesting part, we send a message to the client saying “Connected”
            webSocket.sendTXT(client_num, "Connected");
        }
        break;
    case WStype_TEXT:
      // here webSocketEvent is receiving txt, which means it was sent by the client

      // IMPORTANT: payload comes as an array of bytes
      Serial.printf("[%u] %s\n", client_num, payload); // print the message (called payload)

      // send message to client
      webSocket.sendTXT(client_num, "Received");

      if (!strncmp((char *)payload, "on", length)) {
        digitalWrite(ledPin, HIGH);
      } else {
        digitalWrite(ledPin, LOW);
      }

      // send data to all connected clients
      webSocket.broadcastTXT("Hello everybody.");

      break;

    case WStype_BIN:
    case WStype_ERROR:{}
    case WStype_FRAGMENT_TEXT_START:{}
    case WStype_FRAGMENT_BIN_START:{}
    case WStype_FRAGMENT:{}
    case WStype_FRAGMENT_FIN:{}
    case WStype_PING:{}
    case WStype_PONG:{}
      break;
}

Things I’ve learned from this function alone: a lot.

The type of variables the webSocketEvent function takes

uint8_t client_num

A uint8_t is a type of unsigned integer of 8 bits, in other words a byte. The client is assigned an ID which allows the webSocket to recognise who is speaking to it.

WStype_t type

This is a custom type declared in the WebSockets.h file of the arduinoWebSockets library by Links2004. You can all of them in the code above.

uint8_t * payload

The * means it is an array of bytes. It can be a string or any kind of data you want. You have to handle the reading of it yourself.

size_t length

It is an unsigned integral type. It is an integer that is the length of the payload. It is useful to know if you need to omit an NULL termination that closes the payload array. I think it can be used as well in case payload had been declared previously with a fixed memory, and that the message sent doesn’t use the entire memory. Then you don’t need to read the entire array, you can stop before… I think?

switch(type) statement

The switch statement evaluates different “cases” - it means depending on what the type of event the webSocketEvent function is receiving, it will execute different code. If the type is WStype_CONNECTED then it prints a message to the Serial and sends a message to the client saying a connection was made.

printf()

One of the first thing we see if I learned about printf(), which formats adds and formats variable to a string by using the % symbol followed by a letter, i.e.

Serial.printf("[%u] Connected from %d.%d.%d.%d url: %s\n", client_num, ip[0], ip[1], ip[2], ip[3], payload);
  • It formats the client variable to an unsigned decimal integer with %u
  • It formats the IP address in a signed decimal integer with %d
  • It formats the payload as a string of characters with %s

webSocket.sendTXT(client_num, "Connected");

sendTXT() sends a message to the client identified with client_num. The asterix apparently means it is a pointer. I’m not exactly sure what it means. I thought it was making an array. Reading the WebSocketsServer.h, the message (below called payload) can be:

  • uint8_t * --> an array of bytes
  • const uint8_t * → a constant array of bytes (?)
  • char * -->an array of characters
  • const char * -->a constant array of characters
  • String amp; → supposedly assigns a pointer? Wtf, I have no idea.

alt_text

strncmp()

I learned about strncmp(), which compares two arrays of bytes without having to convert them to strings. It allowed me to compare the data that the client sent to on or off to switch the LED accordingly. As it is used in an if statement I thought strncmp() returned a True or False, but it doesn’t If you check the documentation, it actually returns Useful related link https://arduino.stackexchange.com/questions/55024/how-to-convert-byte-payload-to-string This to understand printf conversion types: http://www.cplusplus.com/reference/cstdio/printf/ When I was saying if (!strncmp((char *)payload, "on", length)) it would work when it the payload actually said “off”, and vice-versa… it’s because I thought it returned True or False, but it strncmp actually returns an int: http://www.cplusplus.com/reference/cstring/strncmp/ - so I added a ! before and it worked!

webSocket.broadcastTXT("message here");

It accepts the same messages as for sendTXT() the only difference being that the message is sent to any client that is connected to the server, that is why it doesn’t take a client_num parameter.

webServer_setup()

Here’s the code.

void webServer_setup() {
  webServer.on("/", handleRoot);
  webServer.on("/sketch.js", handleSketch);
  webServer.onNotFound(handleNotFound);
  webServer.begin();
  Serial.println("HTTP server started");
  webSocket.begin();
  webSocket.onEvent(webSocketEvent);
  Serial.println("WebSocket server started");
}

As the name says this setup the web server. It tells to run handleRoot when the root page is requested by a client, and run handleSketch when the sketch.js file is requested. Same with handleNotFound when 404 Error. It starts the webServer and the webSocket, and tells the webSocket to run webSocketEvent whenever something happens (whenever onEvent is triggered).

handleRoot() and handleSketch()

Here’s the code.

// For the webServer_setup()
void handleRoot() {
  webServer.send(200, "text/html", INDEX_HTML); // serves the index.html file
}
void handleSketch() {
  webServer.send_P(200, "text/javascript", SKETCH_JS);
}

As you can read, the webServer uses the send() to send to the client a file. It comes with 3 parameters:

  • code → a response code - 200 to 299 being a successful code
  • content_type → a string which explains the type of the file
  • content → the name of the file to send

When WebServer sends the sketch.js file it uses send_P() instead of send() like for the HTML file. I checked the WebServer.h documentation (see WebServer.h at line 123) and it seems the only difference is that the send() function takes the content_type as a String and send_P() takes it as PGM_P, which is a PROGMEM type. I would guess you could use send_P() for the html page as well as.

Our good ol’ setup()

You feel a bit more secure here. It runs only once and you’ve this function a few time in the past already.

void setup() {
  Serial.begin(115200);
  pinMode(ledPin, OUTPUT);
  // Connect to wifi
  WiFi.begin(ssid, password);

  Serial.println("");
  Serial.print("Connecting to WiFi");

  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
  webServer_setup();
}

Here we start the Serial, set the pinModes, connect to WiFi and run the webServer_setup() function.

loop()

Finally we have our loop.

void loop()
{
  if (WiFi.status() != WL_CONNECTED) {
    return;
  }
  webSocket.loop();
  webServer.handleClient();
  unsigned long currentMillis = millis();
  if (currentMillis - previousMillis >= interval) {

    // save the last time you blinked the LED
    previousMillis = currentMillis;
    String value = String(analogRead(ldrPin));
    // Serial.println(value);
    webSocket.broadcastTXT(value);
  }
}

In our case it checks if we are connected to the WiFi. If we aren’t exit the loop and start again. If we are then run webSocket.loop() which I guess looks for events. webServer.handleClient()... apparently “it does the needful http server stuff and calls the right callback function”. We attribute currentMillis to how long since the Arduino is running with millis(). We use it after to compare how much time has gone since the previousMillis. If it’s over a certain interval (in my case 1s), then analogRead() the photoresistor and most importantly of all… send that value to all the clients that are connected with webSocket.broadcastTXT(value). Note // For a little while I was trying to send the value with sendTXT() but the debugger said that it wasn’t declared in this scope. It seems it can only be run in webSocketEvent() ?

Setting up WebSocket in browser

Once the WebSocket server was set up on the ESP32 it was all pretty easy.

Opening a socket

In JavaScript you open a WebSocket using the IP address of the ESP32. Luckily you don’t need to remember it, or even to know it, as you can get it with JavaScript.

websocket = new WebSocket('ws://' + window.location.hostname + ':81/');

This opens a socket. You need to define what to do when the connection is opened, when it’s closed, when you receive a message or an error.

websocket.onopen = function(event) { onOpen(event) }
websocket.onclose = function(event) { onClose(event) }
websocket.onmessage = function(event) { onMessage(event) }
websocket.onerror = function(event) { console.log(event.data); }

onOpen(), onClose(), onMessage() are functions I declared later in the file. The first two simply log in the console a message, while onMessage() is the more important one.

Dealing with messages

As we saw above I sent the data in event to the onMessage function. The event parameter in function(event) are the functions onopen, onclose, etc. To access what has been received for the onmessage function you simply read event.data. Data sent through websocket is either sent as text is UTF-8 or binary (see here).

Sending messages

Sending messages with JavaScript and WebSocket is as easy as websocket.send(“your text here”);... Easy! We can send send a message like on and off and the ESP32 compares the string and does the appropriate action with it.

Can't use ADC_2 pins when using WiFi on ESP32

For some reason it could only read 4095, even if changed to other pins... but if I created a new sketch from scratch and used a simple analogRead() with none of the web server and all, it worked fine. I thought maybe the WiFi being used might affect it? I found this comment on an issue in GitHub saying “ESP32 uses the ADC2 to manage wifi functions, so if you use Wifi, you can´t use that register”, which I was using (GPIO12)

alt_text

I changed to GPIO36, which is ADC1_0 and it works! Celebrations.

Casting the message to an integer

To make sure I first cast the value into a string - with String(data) - and then into an int - with parseInt().

function onMessage(event){
  var value = String(event.data);
  value = parseInt(value);
  receivedData(value);
}

Now we have an integer to play with, let’s send it to the function that will actually do something with it… in P5.js.

Mastering P5.js

Mastering is a big word. Let’s say “starting to understand”.

Before anything, let’s understand P5.js, how it works and how we will use the data.

How P5.js works and creating a graph

It is very similar to files being run on the Arduinos or ESP32. It has a setup() and a loop() (called draw) function.

I wanted to create a graph that displays the values given by the photoresistor over WebSocket.

The setup() function

In the setup() function create a canvas with the desired size and can execute drawings that happen once.

function setup() {
  createCanvas(windowWidth, windowHeight);
  background(73,252,213);

  noStroke();
  fill(255);
  rect(2, height - 10, width-4, 2);
}

In my case I create a canvas that is the full width and height of the page.

Note //
windowWidth and windowHeight are values that exist by default, you do need to create them.

I create a background of the same color as my website.
I tell that the next components should have not stroke and be filled with white (255).
I finally create a rectangle - rect() - with the first two value being the x and y position, and the last two being the width and the height.

Note //
Using width and height in the code will use the width and height declared when creating the canvas.

The draw() function

In my case I didn’t put anything in the draw function as the bars will be created as we receive values from the WebSocket

receivedData(data) function to add bars

The code to the bars is pretty simple.

function receivedData(data) {
  if (x >= width) { x = 2; background(73,252,213); rect(2, height - 10, width-4, 2); }
  value = map(data, 0, 4095, 0, height - 10);
  rect(x, height - 12, 2, -value);
  x += 3;
}

We use map() to map the minimum and maximum value we receive from the WebSocket to 0 and the height of the window minus 10 pixels.

We then create a rectangle at the position of x and height of the window minus 12 pixels (remember that the position is from the top right of the screen). The rectangle is an arbitrary width of 2 pixels, and the y value is negative the value sent from the websocket.

Finally we add 4 to x (x += 3 is equivalent to x = x + 3). We add 3 because it is the width of a bar plus a pixel to separate them, for style.

I added a line at the start of the function which checks if x is wider than the window width, and if so it resets x to 2, creates a background to delete all the bars and recreate the rectangle at the bottom of the page.

There’s stuff missing

It’s great but I can’t zoom in, or see what the data value is… it just isn’t interactive.

Note //
I quickly realised that P5.js is more geared towards interacting with data rather than simply displaying it. I tried to find a library to create bar graphs or a line-plot and couldn’t seem to find much.

Using examples for nicer visualisations

I spent a fair amount of time looking at the examples on the P5.js website to see what was possible. There were a few examples that I found really cool:

  • Distance 2D, which uses the distance from the mouse to set the width of circles
  • Video Pixels, which uses a similar technique to above but uses a video as a background and varies the “resolution” of it depending on the mouseX.
  • Pointillism, which uses the Mouse X position to draw circles of different radius with colors picked at random, it is quite beautiful.
  • Kaleidoscope, which mirrors your drawing in multiple angles.
  • Wavemaker which creates an array of circles circling around, and uses the mouse position to create different wave directions
  • Morph, which uses vectors to create shapes and morph in between them, I thought it was quite cool for button state transitions

Using the Wavemaker example

I decided to use the Wavemaker example as I like how it resembles grass undulating in the wind. It’s quite hypnotising and relaxing, all things that I think are in a way related to my final project: a peaceful armchair to sit in in the morning or evening.

I went through the code to understand it. It is quite simple. In the draw() function there is this code.

function draw(){
  background(10, 20, 0, 10); // translucent background (creates trails)
  // make a x and y grid of ellipses
  for (let x = 0; x <= width; x = x + 30) {
  for (let y = 0; y <= height; y = y + 30) {

  // starting point of each circle depends on mouse position
  const xAngle = map(mouseX, 0, width, -4 * PI, 4 * PI, true);
  const yAngle = map(mouseY, 0, height, -4 * PI, 4 * PI, true);
  // and also varies based on the particle's location
  const angle = xAngle * (x / width) + yAngle * (y / height);
  // each particle moves in a circle
  const myX = x + 20 * cos(2 * PI * time + angle);
  const myY = y + 20 * sin(2 * PI * time + angle);

  // fill color for the circles, and no stroke
  fill(40, 200, 40);
  noStroke();
  ellipse(myX, myY, 10); // draw particle
  }
}
time += 0.01; // update time

Before the function the t variable is declared so that the value can be saved and not be initialized every time the draw() function loops.

Making it interactive

I thought it could be fun if we could use a sensor to vary the size of the circle or the thickness of them. It was surprisingly easy. In this line of the function we studied above const myX = x + 20 * cos(2 * PI * t + angle); the 20 is the radius of the circle. I replaced it with radius, a variable that, just like time, I declared outside draw(). I did the same with this line of code ellipse(myX, myY, 10); where I replaced 10 with the variable thickness.

Using the WebSocket value to vary these variables

We can only vary one of those variables, but a button could be clicked to switch between them for different effects. Just like the graph project I used the receivedData() function to deal with the WebSocket values.

function receivedData(value) {
  value = map(value, 1400, 2400, 0, 30);
  thickness = value;
}

Extra ideas

I could make a function that is called everytime onmessage is called and automatically set the min and max values of the map() to the minimum and maximum data it has received so far. That’d allow for better results and less tweaking of settings.

Actually, let’s do it

In the onMessage function I added the code below.

dataArray.push(value); // add the value in an array)

// checks every 100 data received
if (xTime > 100) {
  min = parseInt(Math.min.apply(null, dataArray.filter(function(n) { return !isNaN(n); }))); // returns the minimum value of the array
  max = parseInt(Math.max.apply(null, dataArray.filter(function(n) { return !isNaN(n); }))); // returns the maximum value of an array
  xTime = 0;
  console.log(min, max);
}

// increments by 1
xTime++;

It works! … but isn’t great. The first 100 data points nothing works because it needs to calibrate, but after that it works pretty well. I’m not sure about the efficiency of add all the data points in an array, as it’ll quickly get big, and therefore calculating the min and max value more resource extensive.

The beauty of WebSocket

The beauty of WebSocket is that the data is sent to any page that requests it. Any page, even the one you are reading this one, can run on JavaScript...

var websocket = new WebSocket(“ws://[IP-ADDRESS]:81”);

... and a connection would be made. Then with...

websocket.onmessage = function(event.data) { // do something with it }

... do whatever you want with it.

It’s great because it means on my computer I can open different page with different P5.js sketches, and treat the data differently. It allowed me to test the JavaScript code on my computer before having to upload it on the ESP32 for it to serve it to the client.

WebSocket on JavaScript

https://javascript.info/websocket

Using Progressive Web Apps (PWA)

Why use a PWA?

I’ve been interested in programming apps in the past but was a bit intimidated about learning Swift for iOS or Java for Android. I had heard about Progressive Web Apps which simply takes an HTML page and makes it downloadable on your phone, accessible offline and with time gave it more and more features for it to act as a web app, for example having access to all sensors of the phone, etc. (iOS + Safari is still very close for security reasons which is single handedly slowing down the expansion of PWAs.)

The main reason I love it, is that you can create app for every devices with just HTML, no need to learn new languages. See for your eyes below, in less than 30 mins I converted my webpage into an app by adding a simple manifest.json file with the app name, color, icons and if I want it to be fullscreen or not.

With Chrome on Mac

With Chrome on Android

Note //
PWAs have to be served over an https protocol otherwise it won't work. When I tried to access the WebSocket from the ESP32 it couldn't because it wasn't WebSocket SSL. It can't be done with the library used for this project, but after searching around, I found this ArduinoWebsockets library by Gil Maimon that can. It actually seems to be more simpe to use as well.

This article from Medium gives a good list of pros and cons of PWA. Quick summary of why I think it’s a good choice in my case.

Pros:

  • Uses HTML pages, so tools like P5.js, Three.js, A-Frame are on the table. I can host it on my Gitlab repo.
  • Cross-platform compatibility: as it’s a website it can run on any devices, even a desktop
  • Access to the devices sensors and features, such as camera, microphone, orientation, etc.
  • Use of web notifications to notify user (except for Safari in iOS)
  • No need to be registered on the Apple or Google app stores

Cons:

  • Not as energy efficient
  • Access to the devices sensors might not be as extensive as native apps (getting there though)

There are a few more cons on the article, but it seems to treat more about the marketing of apps rather than their functionalities (i.e. indexing on App stores, loyalty programs, etc.)

Note //
While doing some research in the past I had also landed on Ionic and other similar platforms which allows you to code your app in HTML and they convert into “native” code for mobile platforms. It is very similar to PWAs, and they now offer to actually make them PWAs.

These platforms integrates transitions between pages like slide, zoom, etc. for more realistic app UX.

Easy way to turn a webpage into a web app: https://dev.to/becodeorg/the-easy-way-to-turn-a-website-into-a-progressive-web-app-77g

Sensors on iOS (in this case A-Frame) needs permissions in Settings: https://github.com/aframevr/aframe/issues/3976

Playing with A-Frame and the Lenovo AR headset - resizing a DIV: https://stackoverflow.com/questions/6219031/how-can-i-resize-a-div-by-dragging-just-one-side-of-it

Have you?