Week 14 - interface and application programming

• Link to diary
• Link to assignment & assessment
• Link to recitation notes
• Progress

Heroshot:

Description

This week I am going to make an attempt to make an ESP that output values to an interface. The output should be the current state of the ESP. The ESP should have three LED's (RGB) and they way they light up, should be reflected on the interface. Ideally they can also be modified through the interface.

There are more then enough resources available on the web to achieve these step.

• Input Data on HTML Form ESP32/ESP8266 Web Server using Arduino IDE
• ESP32/ESP8266: Control Outputs with Web Server and a Physical Button Simultaneously
• Turning on and off an LED over Wi-Fi on the ESP

After a little discussion with Henk I decide that the spiral to follow this week are as follows:

#	Spiral	Description	success?
1	Find and test the code	Find the tutorial that comes closest to what I want to achieve and follow that with a factory made ESP board
2	Debugging	Get my existing self made ESP32 board to work
3	Make it work	Get the tested code on the, now, working board
4	Prove understanding	Modify and document the code I used to get it all to work and prove thereby that I understand what is happening
5	Stretch spiral	Add functions in line with my end project and make modifications that bring it closer to the end goal

Spiral 1.

So this spiral starts out by searching for a bunch of manuals and reading them so that I pick the one that will bring me as close as possible to my end goal.

In the description and research of this week you can see that I have already found a handful.

Randomnerdtutorials seems to have all the ESP tutorials that you could ever need. So I start by using their webserver introduction post.

In order to work nice and smoothly and try something new, I will use the new Arduino2.0 IDE this week. It is still in beta, but it gives the nice option to do splitscreen and quickly go from tutorial to IDE.

I follow the tutorial step for step and succeed. Indeed, as expected, not the hardest part of the Fabacademy so far...

NOTE In this example we stumble on a interesting interface dilemma: should a button display its current status, or the status you will get when you press it. In the ESP webserver in the video above the buttons show what you will get when you click them. This is confusing as the button is grey and says OFF when the LED is ON and the other way around. To mitigate this issue, the maker of this program wrote the current status above the button. But what if you would just reverse the button display/working?

How about we change that...?

Modifying the code

In order to play, learn and improve the code above, I try to make some alteration. There is some HTML in the code that I can easily modify to make these alterations.

            // Display current state, and ON/OFF buttons for GPIO 26  
            // client.println("<p>GPIO 26 - State " + output26State + "</p>");
            // If the output26State is off, it displays the ON button       
            if (output26State=="off") {
              client.println("<p><a href=\"/26/on\"><button class=\"button button2\">OFF</button></a></p>");
            } else {
              client.println("<p><a href=\"/26/off\"><button class=\"button\">ON</button></a></p>");
            } 

            // Display current state, and ON/OFF buttons for GPIO 27  
            // client.println("<p>GPIO 27 - State " + output27State + "</p>");
            // If the output27State is off, it displays the ON button       
            if (output27State=="off") {
              client.println("<p><a href=\"/27/on\"><button class=\"button button2\">OFF</button></a></p>");
            } else {
              client.println("<p><a href=\"/27/off\"><button class=\"button\">ON</button></a></p>");
            }
            client.println("</body></html>");

I don't know about you, but to me this makes much more sense.

Spiral 2 - Debugging my ESP board from week 10

For consistency, this spiral is described in Week 10

tl;dr:

• make sure you use the right voltage: 5V and not 3.3
• make sure you compile for the right board; ESP dev board, not ESP wrover module.

Spiral 3 & 4

In order to test if spiral 2 was really done I uploaded the code I made yesterday to my own board and that worked. I now have a esp32 with a running webserver. Now I want to show data on the website it presents from GVB sources.

My hunch is that there is stuff with API's that will need to happen. And something with external databases...

• From this website I get the code that I need to retrieve chuck norris jokes from an Api
• From this website I get the code to run the webserver and show something on it.

If I can combine the two.. I should have what I want for this spiral...

After carefully reading through and combining the two pieces of code, I cam up with this sketch. I added comments where I could to the sketch. But to give a quick rundown of what happens, here is the summery

• load the libraries we need: (WiFi.h, HTTPClient.h, ArduinoJson.h)
• set the web server port number to 80
• start the serial monitor at 9600
• connect to the local network
• print the IP address
• start web server
• check if a client connects to the server
• Initiate HTTP client
• Start the request to "https://api.chucknorris.io/jokes/random";
• Use HTTP GET request to get response from server
• Print parsed value on Serial Monitor
• Then display the HTML web page
• With this header: ESP32 Web Server making chuck norris jokes
• and the printout from the chucknorris api

Reloading the page gives you a new joke.

Spiral 5 - Add an external datasource

For my final project I want to visualize public transport data. In the netherlands all data of public transport is available in the public domain under a CC0 license. In order to gain access to this data you do have to sign an agreement and someone has to permit you access. YOu can access the data at https://ndovloket.nl/ after your agreement is singed and returned to the foundation.

At a later stage I might have to dive into the legal ramifications of this semi-open model and how the foundation is functioning as some form of gatekeeper, but that is for later date.

My request for access was (very) quickly approved and I now have access to all kinds of data from all public transport companies in The Netherlands. From this is need to gather the relevant data and how I can parse this into a format that shows something meaningful on my ESP webserver. I am not even 100% sure that the data of the ferries is in this dataset for example...

Finding the relevant data.

I start my search at http://data.ndovloket.nl/. There I find a bunch of .zip files in the GVB folder that, when unzipped, turn out to contain an directory with, again, two zip files; xml.zip and csv.zip. These files contain the xml and csv files with the actual data. But so far all the names, zips and directories are clearly meant for computer processing. The names and structures make little sense to the human brain.

I need to dive into the documentation and ask around to find out how I can use this data for my project. The resources I have at my disposal are:

• The NVDO google user group. Sadly this is not really available to me as the email I used to register for NVDO is not attached to a google account and therefor I can't use the google group. Again a weird design decision by NVDO.
• In the welcome email I got there are a few pointers:

In onze kennisbank is per koppelvlak beschreven hoe de bestanden in elkaar zitten, ook zijn er op Github een aantal projecten om dienstregelingen rechtstreeks in een database te laden of om te zetten naar GTFS.

• there is an extended manual available online to check how you can read realtime data: http://htmwiki.nl/#!hackathon/realtime.md Interestingly enough this is about transport company HTM (from another part of the country). Hopefully the information that is available there can be extrapolated to my local transport company.

The realtime data is actually the only data that is relevant for me. So I need to check that out

Where are the ferries?!

So as far as I understand from this document there are several datastreams available and they have exotic names:

• BISON KV6, KV15, KV17: tcp://pubsub.besteffort.ndovloket.nl:7658
• KV78Turbo: tcp://pubsub.besteffort.ndovloket.nl:7817
• NS InfoPlus: tcp://pubsub.besteffort.ndovloket.nl:7664

Then this data consists of serval parts and the first part is an envelope. On the basis of an envelope, someone can subscribe to a part of a datastream.

So, for instance, to get only the data from the GVB (the company that owns the ferries), I need to run a subscribe command with the prefix /GVB/.

All this data is divided int a bunch of ZeroMQ publish-subscribe systems. Begs the question: what is a ZeroMQ publish-subscribe system?

ZeroMQ

ZeroMQ (also spelled ØMQ, 0MQ or ZMQ) is an asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker. The library's API is designed to resemble Berkeley sockets.

ZeroMQ is developed by a large community of contributors, founded by iMatix, which holds the domain name and trademarks. There are third-party bindings for many popular programming languages.

~ Source Wikipedia ~

After this I chatted with a friend of mine and asked him for advice. Here are the relevant bits from this chat:

• xml in a zip file is gonna be a serious struggle for your ESP. in python that's like 4 lines of code. in the esp that's gonna be a bit of a monster C program. my advise would be to use a python lambda to just give json to your esp. json is very microcontroller friendly.
• ZeroMQ is an enterprise queue. every time you read the word "enterprise" you should stay away from it.
• a queue allows you to subscribe to a specific datastream, so you do not have to fetch the data at time intervals... instead what happens is that your clients subscribe to a stream and your client gets auto notified when new dtata becomes available
• i cant imagine a single use of zeromq for a microcontroller application. i am not aware of a good zeromq library for esp32. yo that's gonna be tricky
• the easiest way to do it imho is have a computer to the zmq -> json and then esp32 does json to led actuation. a raspberry pi can do that easily
• well, I adapted this little 15-line python script to your needs so that once you accomplish jumping through all the hoops, it will fetch messages from the ZMQ and print them https://0bin.net/paste/WQM1+Ncr#vDDV0Kw-hkQ4pL/4vLiY2HQjcVmc1xurOjZrL0vyjce
• to do that in the ESP32 itself will be much much trickier
• they don't seem to provide examples so it's even unclear if the data that is there is the one you need?

Here is the code that Luis made:

    import zmq
    import time

    ctx = zmq.Context()
    sock = ctx.socket(zmq.SUB)
    sock.connect("tcp://pubsub.besteffort.ndovloket.nl:7658")
    sock.subscribe("/GVB/KV15messages") # Subscribe to all topics

    print("Starting subscriber loop ...")
    while True:
        msg = sock.recv_string()
        print(f"Received: {msg}")

    sock.close()
    ctx.term()

I cycle to Waag to ask Nadieh what I should do and she is so kind to spend a good amount of time with my problem.

In the meantime I have set up a Virtual Private Server based on Debian10 and used the NDOVloket IP adres form to submit the IP address of that server. A quick test a few minutes later showed that my IP address had been whitelisted.

Nadieh modifies the code form the (one) example we find a bit and comes up with this code to run in Python3:

from gzip import GzipFile
from io import StringIO
import zmq

context = zmq.Context()

subscriber = context.socket(zmq.SUB)
subscriber.connect("tcp://pubsub.besteffort.ndovloket.nl:7658")
subscriber.setsockopt(zmq.SUBSCRIBE, b"/GVB/KV15messages") /// Notice the extra 'b' that goes here.

while True:
multipart = subscriber.recv_multipart()
    address = multipart[0]
    contents = ''.join(multipart[1:])
    try:
        contents = GzipFile('','r',0,StringIO(contents)).read()
        print('GZIP', address, contents)
    except:
        raise
        print('NOT ', address, contents)

subscriber.close()
context.term()

Two results.

• Using the example code we get information, but it is encoded gibberish and we do not know (yet) how to decode it
• Using the code we modified to link to GVB the program hangs and no data is ever returned.

This whole setup has a google group. and there is probably more information to be found there. But I do not have access (yet). When I do, I continue.

For now I am going to halt this spiral. And in the next chapter I will explain what I will do next.

NOTE I got the feedback from the maker of the NDOV loket database that there is no realtime data of the ferries in that database anyway.

Daar is geen real time data van. Nouja dat is er wel, maar dan kom je op AIS terecht :-)

So either I have to find a free source of AIS data (most of them seem payed) or just hard code the time table...

Create my own API!

As I now have my own server, I figured that I can make it serve data to my ESP as well. This data will then be a number from 1 to 3 and depending on the number an LED connected to my ESP will light up. This can be the beginning from having an external server parsing data that is used to inform my ESP(s) what they should do.

But why so difficult Watson!? When all you can also do is:

• Get all the times from GVB.nl
• Paste them in a spreadsheet
• Fireup Stein and make an account there.
• Ask Stein to create an API on the basis of your spreadsheet
• And voilá! There is your API

Note: you have to amend the URL with the name of the sheet you are using. If you Google sheets is in another language, your url end in that language. So in my case, not sheet1, but /blad1

the API I creates using stein creates a JSON file. In the setup I used above the text that was created and parsed on the ESP32 was just clear text on a website. Now I have one file containing all the data. So I somehow have to figure out how to parse a JSON file and how to show only one part of the data in that file.

JSON to ESP32

To start I create a new file with the old code so I can mix and mesh it. I follow a new tutorial I found on JSON and ESP32 and take snippets code from that tutorial to change my own original code. A lot of relevant information on JSON and Arduino/ESP32 can be found on the website of randomnerdtutorials

As JSON is my primal objective here and now, I remove all code related to the outputs and LEDs and make the sketch small and more understandable.

Side step: installing NODE RED

I know this is an unusual and maybe senseless sidestep. But after reading a bit on JSON and stuff, I decide to get my feet wet with NodeRed. And I follow this tutorial to install it on a debian VPS.

For this I use:

• debian VPS with 1024 RAM, 1CPU and 10Gb HD
• YUNOHost script to manage and install the software.
• Let's encrypt certificate
• NodeRed version 1.3.4~ynh1

After I got the installation of Node red running I continue with the RandomNerd Tutorial.

I am able in the end to create a random number on my own website...

The random number is refreshed every second.

It proves indeed a futile step for now. I am happy I got it running and will continue at a later date with NodeRED. But tho achieve what I want to do now.. it makes little sense...

JSON to ESP32 pt. 2

In the evening of Tuesday I had a lengthy conversation with my friend Luis Fernandez. He's a teacher at the HvA in Amsterdam and Artes in Arnhem. He teaches frequently on many of the topics we cover in Fabacademy and was able to help me really well with some of the things I was struggling with.

I will try to write down what he told me to my best abilities. Sadly there won't be much time left to actually try it as my class will start soon...

Still the main takeaway I learn from Luis is that setting up my own server with nodeRed was not a bad idea! He encourages me to put all the smartness in a server and keep the endnodes, the microcontrollers, as dumb as possible. He has a few reasons for this:

• It is often much harder to update code in an microcontroller that is glued to a wall or in another hard to get to place.
• the code needed to achieve something in C can be much more complex then the code you'd need on the web or on your server

So we arrive at the point where I basically need my microcontroller to do two things:

• Get an integer from a server.
• Light up a combination of LED's based on the integer.

I am now wondering what the minimal MCU is that I'd need to achieve this... An ESP8266 can already do this. But can we go even smaller?

But before we arrive there, I am going to write down some of the interesting things I learned from talking to Luis:

Deserialization; If you get a plan text file, or a file that has some structured text in it, your computer will still see it as just one block of text. This needs to be deserialized so that you go from a string to a datastructure. Luis showed me this in the in-browser Javascript parser.

JSON needs to be parsed. In the ESP you need to deserialize data. You go from a string to a datastructure. That contains keys and values. YOu need this to extract data (values) from objects (keys). In javascript this is super easy as it is meant to do that. In C we need a few more lines.

In order to continue building this amazing machine, I have a few more links to dig into:

• http://api.derfunke.net/pont/ijplein.json
• https://github.com/espressif/arduino-esp32/tree/master/libraries/HTTPClient/examples
• http://worldclockapi.com/api/json/cet/now
• https://0bin.net/paste/5LR2XAZs#NZLIydCdDHtlk8sJ78w-CxoZf2F/QItZy/6Yoh6VkCb

another way of fetting current time im esp would be to use the ntp protocol thats easier even, because the ntp library already gives you a time and date and gmt offset. as numbers that you can work with without any conversion

from Luis

Research and relevant links

• Input Data on HTML Form ESP32/ESP8266 Web Server using Arduino IDE
• ESP32/ESP8266: Control Outputs with Web Server and a Physical Button Simultaneously
• Turning on and off an LED over Wi-Fi on the ESP

What I have learned (aka, should have done differently)

I enjoyed playing with NodeRed, but it was far from my path. I should have stayed a bit more focused on what I wanted to achieve. The dependence on an external data source also shows the price of dependance: if you need to be admitted and such, you loose control over your time and possibilities.

Quentin Bolsee has a tip on getting a clock in my project. That would eliminate the need for an external webserver and wifi! Here is also an interesting instructable with the RTC.