Networking and Communications
individual assignment
design, build, and connect wired or wireless node(s) with network or bus addresses
group assignment
Send a message between two projects
This week the only thing to download is the source code of the examples below.
Download source code filesThis week Oscar set up a MQTT broker on Raspberry Pi at home and got all the students to send some data from their ESP32. David and I suscribed to each others' topic and actuated devices and LEDs depending on each others' inputs.
For my final project I am doing something very similar. Well, exactly the same, almost, so I by documenting my individual assignment, it will also document the group assignment.
For this week Oscar transformed his Raspberry Pi into a server at home and ran an MQTT broker on it.
During a live session all the students could run some code on their ESP32 or ESP8266 and send live data to the MQTT broker. We were Publishers.
We could then subscribe to other students’ data-streams and use it however we wanted to. We were Subscribers. In my case I used Arman’s data to detect when he goes above a certain threshold and switch on my LED.
It was cool but not very challenging as it was mostly just copy and pasting some code, making sure the right libraries were installed, adding your WiFi SSID and password and that was it… something more could have been done with the data being received, but I feel it’s more to do with coding rather than networking.
I wanted to set up my own MQTT broker with my own Raspberry Pi as that would teach me about running servers, connections, opening and forwarding ports on your router (safely, hopefully).
This article was advised by Oscar and seems really nicely written and covers a few subjects. It starts with installing nginx. It is a web server (similar to apache), but in our case we want to use it as a reverse proxy. A reverse proxy deals with managing the communications from the outside world with inside your home network. It does this through the RPi.
I followed the very simple instructions of the tutorial above which is just to use apt-get to install the nginx package. It should automatically launch a server but I get this error.
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
It seems a server is already being served on this port, or at least the port is taken… I know I used this Raspberry Pi in the past so maybe there are some services automatically being run on startup but I’m not sure where to check. I looked in the /etc/systemd/system folder and there’s a few, but dunno what to pick.
Note //
From what I understand systemd are services you can set to automatically launch when your RPi boots.
I searched for the error and found this article. The person uses a few commands to see what’s happening on the network, including these ones:
The sudo netstat -pan | grep “:80”
command shows that there’s apache2 using the port 80 (default non-secure http port - find a list of port use here).
Conclusion: I need to stop the apache2 service, but how? We know that 1) it’s an apache server, and 2) it automatically starts up on boot… let’s google search “rpi apache2 shutdown server”...
This article shows how to start and stop an apache server using different commands, one uses the service
command which seems encouraging (I’ve seen the command around before, I think it’s related to automatic startup script launching).
Another, more understandable, and revealing which folder to look into, command is
. I typed it in the terminal hoping it’ll stop the server… Let’s visit http://raspberrypi.local and see if the page is still being served… and it isn’t! You can confirm that by running the /etc/init.d/apache2 stop
sudo netstat -pan | grep “:80”
again and see that it’s not returning anything.
That’ll do for the moment, but I will have to find out how to delete the automatic server start script. I’m guessing it is in the service command? (maybe this?)
Note //
After checking using man service
to check what the manual of the service command I learned that “Service runs a System V init script, systemd unit, or upstart job”, but also noticed at the bottom it said: SEE ALSO systemctl, which after some research is linked to systemd which I know are scripts to run on startup… more here. To be investigated later.
Meanwhile, if I run nginx
in the CLI, boom a page is being served when visiting the URL. (Other tutorial I started to follow but forgot about.)
The main tutorial I’m following, the one from Tinkerman, mostly just brushes up on what needs to be done, which is nice, but if you get stuck you need to figure it out yourself, which is what I’ve been doing. Here he says to use Certbot with Let’s Encrypt, but I can’t seem to install Certbot, so I’m doing some research. I found this article that says: “Before you get started with setting up SSL on your Raspberry Pi, make sure that you have a domain name already set up and pointed at your IP address as an IP Address cannot have a certified SSL Certificate.”
I guess another thing to search, pointing a domain name to an IP address? Oscar mentioned no-ip.com to do that. Let’s check it out.
To do so, as mentioned above, you need to use a domain name, so I set one up with no-ip.com. It was actually pretty straight-forward. I set up an account with them and create a domain name - http://banjamanscatt.onthewifi.com/. It automatically detects the IP address I’m using to visit the server and sets it up as the IP address to link it to.
The thing is, this IP address isn’t static and might change in time, so they advise you to setup their DUC (Dynamic DNS Update Client) software which runs on the Raspberry Pi and will update No-IP when the IP address changes. They advise to install that software on a device that is always on, so it could be my MacBook, but as the Raspberry Pi will be the server, it will be on when we need it. Therefore it makes sense to install it there.
I don’t have access to it via GUI (well, I do but I need to plug it to a screen, etc… long), so instead I do on the CLI via SSH (ssh pi@raspberrypi.local
).
sudo su -
enter the password.cd /usr/local/bin
wget https://www.noip.com/client/linux/noip-duc-linux.tar.gz
tar xzf noip-duc-linux.tar.gz
cd no-ip-2.1.9-1/
make install
Note //
The step might be not needed if your router allows the Dynamic DNS server. In my case Orange does.
Ok! You now have a domain name point to your IP address and the software installed on your RPi. All you need to do now is to forward the right port to the right device on your network.
Access your router by using the standard ip address 192.168.1.1, use your logins (Orange’s Livebox has a standard password of admin, with user admin, nice and secure…), visit the Advanced tab, on the left click NAT and then Port Mapping. Click the + at the bottom right.
You can either manually type in the settings, or simply select at the top the Applications, in our case Servers, from the drop-down menu select Web Server, and it automatically fills in what ports to forward (80 & 443, non-secured and secured), you just need to put the IP address of the device on your network you want to forward to. In my case my RPi is at 192.168.1.193. The only thing I changed is allowed both TCP & UDP as protocols.
Finally in No-IP you can check if it works.
Or even better, actually visit the domain name you entered, and, tada, the NGINX page is being served.
Good, back to encrypting things.
Note //
I checked it a few days later and it didn’t have access to port 80. I cheched the IP address that banjamanscatt.onthewifi.com and it was pointing to a different IP. It seems the Dynamic Update Client (DUC) doesn’t work.
Ok, so we’ve got a domain name now.
The install of Certbot didn’t work because I’m guessing NGINX was running on port 80.
See "systemctl status nginx.service" and "journalctl -xe" for details.
invoke-rc.d: initscript nginx, action "start" failed.
I ran systemctl stop nginx.service
to stop the serving, and then tried sudo apt-get install certbot
again… and for some reason the same error came up.
I wanted to check if it was still running so did sudo netstat -pan
and noticed a line saying tcp 0 0 192.168.1.193:80 85.58.174.117:59964 FIN_WAIT2 -
, with FIN_WAIT2 where usually it is LISTEN or ESTABLISHED.
I did some google search and found an article saying “FIN_WAIT_2 seems to occur when the server has an active connection with a client and wants to shut down the TCP connection”... I still had my banjamanscatt.onthewifi.com tab opened, so I closed it and run the command again… the line is gone.
Let’s try the install again, fingers crossed it can launch a server now (not really sure why it is to be honest, but let’s see).
Still fails, and still says
Package nginx-full is not configured yet.
Package nginx-light is not installed.
Package nginx-extras is not installed.
Wait, there’s a line saying “certbot is already the newest version (0.28.0-1~deb9u2).”... which means it’s installed… good, I don’t need to care about the rest just yet then.
Oh… just find what seems to be nice instructions on the Certbot website: https://certbot.eff.org/. Let’s follow them.
Wow… that was way easier! Literally did it in under 2 mins, although I still have the same problem of Nginx running…
I did sudo systemctl disable nginx
and same for Apache and then rebooted the RPi with shutdown -r now
to see if it worked.
It seems to have worked as if I check the status for both it says inactive… good.
Ok, let’s try the commands again, ugh. sudo /usr/local/bin/certbot-auto --nginx
... It’s running and seems happy so far… Creating virtual environment… Installing Python packages…
Certbot has problem setting up the virtual environment.
We were not be able to guess the right solution from your pip output.
Consult https://certbot.eff.org/docs/install.html#problems-with-python-virtual-environment for possible solutions.
Ok… let’s visit the link and do what they say (something about creating a swap file if we are on a system with not much RAM, but I’m pretty sure that isn’t the problem - another error said
THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
pycparser==2.19 from https://www.piwheels.org/simple/pycparser/pycparser-2.19-py2.py3-none-any.whl#sha256=bc15573b7c6edd24407526dbbc7a0bd33d80d8af44231c37f58d73f56ff9cab6 (from -r /tmp/tmp.PtGtzUCKeH/letsencrypt-auto-requirements.txt (line 113)):
Expected sha256 a988718abfad80b6b157acce7bf130a30876d27603738ac39f140993246b25b3
Got bc15573b7c6edd24407526dbbc7a0bd33d80d8af44231c37f58d73f56ff9cab6
zope.component==4.6 from https://www.piwheels.org/simple/zope-component/zope.component-4.6-py2.py3-none-any.whl#sha256=74f55521dec189c08d98341edce929eba6bb2404662d1878f1b289af46f6f6a5 (from -r /tmp/tmp.PtGtzUCKeH/letsencrypt-auto-requirements.txt (line 135)):
Expected sha256 ec2afc5bbe611dcace98bb39822c122d44743d635dafc7315b9aef25097db9e6
Got 74f55521dec189c08d98341edce929eba6bb2404662d1878f1b289af46f6f6a5
Let’s replace the hashes? No idea what I am doing. I’m gonna get hacked.
File doesn’t exists, which makes sense, it’s probably created while the command is being run...
Ok… what to do now? Let’s go back to the first tutorial, whoop. Here at 5b.
I enter the command but it says -bash: certbot: command not found
… it makes sense, our lovely official certbot tutorial told us to delete it. Let’s install it again.
sudo certbot certonly --webroot -w /var/www/example -d banjamanscatt.onthewifi.com -d www.banjamanscatt.onthewifi.com
Almost there! It says getting certification, etc. but then gives an error… it might be because we are pointing to var/www/example when Nginx is serving a different location? Nginx is serving this file /var/www/html/index.nginx-debian.html - so I change the path to /var/www/html/ in the command…
Before I had a few errors, now I only have one, so we are on the right path. It’s about DNS problem: NXDOMAIN looking up A for www.banjamanscatt.onthewifi.com - check that a DNS record exists for this domain
Reading this, I realised that maybe it’s the www.banjamanscatt.onthewifi.com that gives an error (which makes sense now I write down, there’s two sub-domains). You can confirm the error because it’s for this specific domain. Let’s delete it for the command…
sudo certbot certonly --webroot -w /var/www/html -d banjamanscatt.onthewifi.com
Yes! It worked! I’m not sure if I should share all these details… although it doesn’t seem too sensitive.
We just need to configure the Nginx file to use the keys we created. Open in super user the config file.
sudo nano /etc/nginx/sites-available/default
listen 443 ssl default_server;
ssl_certificate /etc/letsencrypt/live/banjamanscatt.onthewifi.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/banjamanscatt.onthewifi.com/privkey.pem;
And that’s it! Visit https://banjamanscatt.onthewifi.com and a little padlock appears on the left to say it’s secure. Boom!
Note//
Nginx gave me quite a few errors when I was trying to stop the server, start it, get it’s status, etc. it was always complaining. This line nginx -t
helps tell you what’s wrong. I also used this a lot sudo systemctl status nginx
and changing status to start or stop, depending what you need to do.
I’m planning to have an ESP32 read values of the moisture of the soil, send it to the RPi which will host a website and display those values.
To configure the reverse proxy I went back to the original tutorial and followed what it said.
It’s pretty easy, just create a new config file for nginx sudo touch /etc/nginx/sites-available/proxy
then enter in editing mode with sudo nano /etc/nginx/sites-available/proxy
and finally copy and paste this in, replacing the domain name with the right one
server {
# general server parameters
listen 443;
server_name banjamanscatt.onthewifi.com;
access_log /var/log/nginx/ banjamanscatt.onthewifi.com.access.log;
# SSL configuration
ssl on;
ssl_certificate /etc/letsencrypt/live/ banjamanscatt.onthewifi.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/ banjamanscatt.onthewifi.com/privkey.pem;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
location /esp32/ {
# header pass through configuration
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# extra debug headers
add_header X-Forwarded-For $proxy_add_x_forwarded_for;
# actual proxying configuration
proxy_ssl_session_reuse on;
proxy_pass http://esp32.local/;
proxy_redirect default;
#proxy_redirect ^/(.*)$ /esp32/$1;
proxy_read_timeout 90;# WS Headers
# websocket support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
}
}
… finally close the file by typing Ctrl + X and then type Y to confirm the save.
That should be it!
For some reason my HTTPS protocol is failing and page isn’t loading… Actually both port 80 and 443 aren’t being redirected… I really don’t know what is happening.
One day it’s all working. One day only port 80. One day nothing. I think it’d be easier to reinstall the RPi from scratch, but it’s the easy solution. Imagined if I had loads of stuff installed on it…
Let’s try and figure this out.
Point 3 says that port 80 & 443 are open and are being redirected to the RPi where the server is running properly. The problem must be with the domain name.
NoIP is configured to point to the right IP address but somehow it is not displaying it. Is it because the DNS servers take a while to register the change? I tried using the 8.8.8.8 DNS which is from Google and should be up to date.
Again, it’s pretty easy. I’ve followed this article which is linked from the previous article.
It seems Node-RED is installed by default on the RPi, so you only need to launch it. In the command line type sudo node-red-start
.
It creates and hosts a web server with Node-RED running on port 1880. To access it type in your command line http://RASPBERRY_IP:1880/
. You’ll have something like this.
It advises to add some packages through npm (node package manager), like the node-red-admin.
The thing is it says to go to /home/pi/.node-red
which doesn’t exist on my RPi… so where is it?
I typed this in the Terminal to find it sudo find / -name node-red -type d
. It shows a list of results. The one that makes sense seems to be /usr/lib/node_modules/node-red
.
I then type the command npm install -g node-red-admin
. It gives loads of errors but it seems to work at the end.
Ok, well I ran this bash <(curl -sL
https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)
Which deletes old versions of Node-RED and Node.js. The Raspberry Pi website advises to use this way, so… yeah.
The folder we couldn’t find before, the one at /home/pi/.node-red
is now there. Good. That’s a good sign.
Note //
Actually maybe it was useless. I did this so I could runnpm install node-red-node-openweathermap
but it seems I only needed to click on Install on the Manage Palette in the hamburger menu. I thought this was where the already installed ones were, but now, it’s to actually install a package… oh well.
I also installed the node-red-dashboard (node) which allows you to easily create a UI displaying your data.
On the left is all the nodes you can ever wish for. As a test I wanted to get the weather and display, get myself familiar with Node-RED.
I dragged the openweather map node in, entered the API key I got from here, and finally chose the city and country. It updates periodically and gets a JSON object that is transmitted to the next node as msg.payload. In that JSON there’s a key called “tempc” which is the temperature in ºC.
To display that temperature I drag a text node from the dashboard package, set the value format to {{msg.payload.tempc}} (it’s already msg.payload by default).
You see at the top there’s a group which you have to add to be able to display the dashboard nodes. When you edit that group, it also tells you that you need to add a tab.
Dashboard works hierarchically as App > Tab > Groups > Nodes. You can have multiple nodes in a group, multiple groups in a tab and multiple tabs in the app. Learn more (like I did) through this video.
Just for fun I also added a gauge to display the temperature in a fun way.
Finally, I want to chart the values of the soil moisture over time. I’ll receive those values through MQTT, but I don’t have set up yet, so for now I will simply enter the value through a text box, and then submit them.
Add the text input node from the dashboard section, and double-click on it. Add it to whatever group you want it to be in. Give it a label if you want. The main thing to change here is the delay to 0. That means that the input will be only sent when press Enter or Tab, and not every x amount of time. In my case I’m changing the mode to number which means I can’t send a wrong value to the chart node. Topic isn’t essential, yet.
Add the chart node from the dashboard, and connect the output of the text input node to its input. There are some settings for this node, but they aren’t essential, so we won’t touch them (but please do).
Here’s my entire Node-RED project for now.
And what it looks like.
I mean, you don’t have many options anyway. It’s MQTT publish or subscribe. The only options or QoS or Retain, which if you are confused by what they mean, like I was, read this article made up of short sentences. Very straight to the point which is nice.
Quick resume is, QoS means Quality of Service. If set at 0 the message is sent once it may or may not be received by the broker. There isn’t any confirmation. Quality of 1 says the broker has received it, but it may have received it multiple times (duplicates). Quality of 2 says it has received and, it’s been read. Basically it’s more back and forth messages saying “yes I received it. You sure? Yes. Ok cool”.
Retain means that the broker keeps the last value it was sent as the current value. It doesn’t change until it receives a new value. That means that somebody that subscribes to a topic may read a value straight away, and doesn’t need for new ones to be sent. Good if you wanna get true or false states out of sensors (amongst other)
While sitting in front of Node-RED I needed to clarify what was going to be connected to what, when it would be triggered and what it would do. I tried to write it down, then transform into a flowchart with draw.io and finally heading back to Node-RED.
Flow 1:
Flow 2:
I’m following this tutorial for this.
It’s pretty simple. Just install the Mosquitto Broker with sudo apt-get install mosquitto mosquitto-clients
on your RPi.
We can test by typing mosquitto_sub -h localhost -t "sensor/temperature"
int the command line to subscribe to a topic, then in another terminal command line type mosquitto_pub -h localhost -t "sensor/temperature" -m
22.5
.
As you type the second command, it will show up in the first terminal window.
Cool, that was easy. Now I want to couple it with Node-RED to create a dashboard of the data.
Good. Now we have Node-RED installed, we have mosquitto working, we just need to send data from the ESP32 to the RPi.
For some reason the https:// we set in the first section isn’t working, but that’s fine, I can make the RPi and ESP32 talk through the local network for now.
This week Oscar pretty much set up something exactly like this (well, actually exactly like this) where our individual ESP32s at home would send data to a RPi he set up at home.
This is second nature by now. In Arduino go to Sketch > Include Library > Manage Libraries and add the library called PubSubClient.
#include <WiFi.h>
#include <PubSubClient.h>
// SETUP WIFI CREDENTIALS
const char* ssid = "YOUR_SSID";
const char* password = "YOUR_PWD";
// SETUP MQTT DETAILS
const char* mqtt_server = "192.168.1.63";
const char* mqtt_user = "YOUR_USER";
const char* mqtt_pass = "YOUR_OTHER_PWD";
WiFiClient espClient;
PubSubClient client(espClient);
const int ledPin = 13;
const int soilMoisturePin = 36;
const int pumpPin = 16;
void setup() {
pinMode(ledPin, OUTPUT);
Serial.begin(9600);
setup_wifi();
client.setServer(mqtt_server, 1883);
client.setCallback(callback);
pinMode(ledPin, INPUT); // don't really need to do that as it's ADC
}
void loop() {
if (!client.connected()) reconnect();
client.loop();
// Publish every minute
if (millis() % 60000 == 0) {
// READ YOUR SENSOR DATA HERE
float value = analogRead(soilMoisturePin);
// Send value as characters
char msg[50];
snprintf (msg, 50, "%f", value);
Serial.print("Publish message: ");
Serial.println(msg);
// SET THE TOPIC TO PUBLISH HERE
client.publish("soil-moisture", msg);
}
}
void callback(char* topic, byte* payload, unsigned int length) {
Serial.print("Message arrived [");
Serial.print(topic);
Serial.print("] ");
for (int i = 0; i < length; i++) {
Serial.print((char)payload[i]);
}
Serial.println();
String strPayload = String((char*)payload);
// If isWatering is true
if (strPayload == true) {
// turn pump on for 5 secs
if (millis() % 5000 == 0) {
digitalWrite(pumpPin, HIGH);
}
}
}
void reconnect() {
// Loop until we're reconnected
while (!client.connected()) {
Serial.print("Attempting MQTT connection...");
// Create a random client ID
String clientId = "ESP32-Desk";
clientId += String(random(0xffff), HEX);
// Attempt to connect
if (client.connect(clientId.c_str(), mqtt_user, mqtt_pass)) {
Serial.println("connected");
// SET THE TOPIC TO SUBSCRIBE HERE
client.subscribe("isWatering");
} else {
Serial.print("failed, rc=");
Serial.print(client.state());
Serial.println(" try again in 5 seconds");
// Wait 5 seconds before retrying
delay(5000);
}
}
}
void setup_wifi() {
delay(10);
// We start by connecting to a WiFi network
Serial.println();
Serial.print("Connecting to ");
Serial.println(ssid);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
}
Note //
Oscar also included a useful section for using MQTT with Python.
In the final project the goal is for the RPi to switch on the pump when the soil moisture is below a certain level. We could easily hard-code this in the ESP32 itself, but I would like it for the RPi to tell the ESP32 when to do it, as I also want to be able to manually trigger the watering.
That means I want to subscribe to a topic called “isWatering”, and when the value is true (or 1) then the ESP32 actuates the pump. When it changes back to false (or 0) it switches it off. It is useful because then on Node-RED I can add a button that can trigger the pump manually.
I tried and it works! The chart moves as we send data to it through the ESP32, or even this app I downloaded on my phone.
This week the only thing to download is the source code above.
Download source code files