Assignment
Week14: Networking and communication
Assignment
- individual assignment
- design, build, and connect wired or wireless node(s) with network or bus addresses
- group assignment
- send a message between two projects
Table of content
My Work
Group Assignment
send a message between two projects(link )
MQTT (ESP32 as a pubsub client - AWS IoT as a MQTT broker)
Researching for distributed machining idea (that allows participants who are in separated location to operate the separated machine for each local site of participant work altogether), I investigated how we can implement interfaces between remote location.
There are some ways to do dispatching message to remote location like WebSocket, MQTT, IFTTT.
Overview of MQTT![](../images/ESP32/ESP32_06_MQTT_sequence.jpg)
MQTT broker configuration
Select MQTT broker
Firstly, I tried to setup Heroku as it looks to be able to install in simple GUI. However, at the screen of Heroku's add-on, I got error of "the add-on is out of stock".
So, I moved to AWS IoT. It looks that Fee of AWS IoT Core is free in "free charge volume".
Names to be shared in client and MQTT broker
Same names need to be specified in client application and AWS IoT management console
Element name in client (PubSubClient) |
Name configured both in client and MQTT broker | Setting place in AWS IoT management console |
---|---|---|
device name(deviceId) | FA2020JpDevice1 | Manage > Things |
arn (*endpoint) | a2toz7cb5zl4er-ats.iot.ap-northeast-1.amazonaws.com | Manage > Things > "FA2020JpDevice1" > Interact > HTTPS |
publish topic name (*pubTopic) | fa2020jp/topic* (ex. fa2020jp/topic1, fa2020jp/topic2) |
Manage > Things > "FA2020JpDevice1" > Security > (attached certificate) > Policies > policy name("FA2020JpPolicy2")
{
  "Effect": "Allow",   "Action": "iot:Publish",   "Resource": "arn:aws:iot:ap-northeast-1:062759670692:topic/fa2020jp/topic*" } |
subscribe topic name(*subTopic) | fa2020jp/topic* (ex. fa2020jp/topic1, fa2020jp/topic2) |
Manage > Things > "FA2020JpDevice1" > Security > (attached certificate) > Policies > policy name("FA2020JpPolicy2") Note that "subscribe" in policy needs to adopted to "subscribefilter" and "receive" in policy needs to adopted to "subscripted" topic name
{
  "Effect": "Allow",   "Action": [     "iot:Subscribe"   ],   "Resource": [     "arn:aws:iot:ap-northeast-1:062759670692:topicfilter/fa2020jp/topic*"   ] }, {   "Effect": "Allow",   "Action": [     "iot:Receive"   ],   "Resource": [     "arn:aws:iot:ap-northeast-1:062759670692:topic/fa2020jp/topic*"   ] } |
SSL certification for MQTT
Before embedding SSL certification files and key file, it's possible to check by openssl command
If above command works without error, the other setting like attachment of policy or naming (endpoint, device name or topic name might be wrong). In my case, failed by following reason.
- Check you use the right certification that is attached
- Check every content in certification or key file is embedded in program (you need to include new line code (\n) both for content and begin/end string.
- policy did not allow "subscribefilter" or "subscribe" > change rule name
- Check if "activate" the policy in AWS management console.
- If the client application cannot subscribe the topic, PubSubClient disconnects the MQ connection. So, if the MQ connect is repeated, it's possible to be failed by incorrect subscription definition in policy.
Policy
For debugging, I attached a policy allowing everything as follows. If this works, something is wrong in policy setting.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iot:*", "Resource": "*" } ] }
MQTT Pubsub client
Component and connection
![](../images/assignment/week14/w14_composition_sample_mqtt.png)
Source code: mqtt_aws_v1.ino
Check Network connection
For connecting ESP32 to internet over Wifi, you need to do some steps (Tips of Wifi and SSL configuration).
Check port
MQTT default port is 8883. Instead, it looks to be able to assign 443 (it seems to require additional configuration as my application is failed just by setting 443 port)
Changed max packet size configuration in PubSubClient.h
At PubSubClient.h in library, Change #define MQTT_MAX_PACKET_SIZE 128
to #define MQTT_MAX_PACKET_SIZE 512
Experiment for publish to topic1 and topic2, then subscribe only from topic1
Overview of MQTT![](../images/ESP32/ESP32_07_MQTT_different_topic_896x_q85.jpg)
Above experiment was done by tithering Wifi network of my android phone.
After above experiment, I changed my WIfi router at home and that supports 2.4Ghz Wifi network. In that environment, MQTT over AWS IoT performance is quite quick. Even with writing serial monitor both in publisher and subscriber, send publish - receive message costs under 40 msec
17:57:59.422 -> Publishing message to topic1 fa2020jp/topic1 17:57:59.422 -> {“channel”: {“1":{“tone”:“28"}}} 17:57:59.422 -> Publishing message to topic2 fa2020jp/topic2 17:57:59.422 -> {“channel”: {“2":{“tone”:“29"}}} 17:57:59.422 -> Published. 17:57:59.460 -> Received. topic=fa2020jp/topic1 17:57:59.460 -> {“channel”: {“1":{“tone”:“28"}}}
MQTT in work
Make program as a client template (with adding JSON function)
For easier implementation as a group project that can connect devices over MQTT in remote location, I made this like a template.
As a coding for other members, I specified some point to update in comment like:
- Set your ssid and password
- Configure your topic (or for your use to publish/receive message)
- Specify your topic name to subscribe in connectAWSIoT()
- Write your logic on receiving message in mqttCallback()
- Write your logic for publishing message in mqttLoop()
![](../images/ESP32/ESP32_08_mqtt_aws_v2.jpg)
Also, I added JSON function using ArduinoJson library. It would be useful to retrieve necessary value from byte message in JSON format.
mqtt_aws_v2.ino(Gitlab, clean up security information)
For detail documentation of environmental setup and updating point in source in template, please see Group assignment page of networking and communications
MQTT (Node-RED - AWS IoT - ESP32)
Connect to Node for aws-iot
![](../images/assignment/week14/w14_compositionmqtt_aws_Node-RED.png)
I installed and setup Node-RED on RaspberryPi, then connected it to MQTT Broker.
This image is the connection between dESP32 and subscriber over MQTT.
Flow in Node-RED
![](../images/NodeRED/NodeRED_M_02_01_pubsub_flow.jpg)
For accessing to MQTT Broker on AWS IoT, I made a simple flow that publishes and subscribes to a topic.
If you do not have node to access to AWS IOT, use node-red-contrib-aws-iot
. You can see simple procedure to install it in the page of Node-RED Tips: Add Node for "AWS IoT"
![](../images/NodeRED/NodeRED_M_02_02_set_Message.png)
In function node, "set message" from injected timestamp as a json format
![](../images/NodeRED/NodeRED_M_02_03_json_.png)
In Json node, convert javascript object to json text
"payload" is pre-determined element to handle message content between nodes in Node-RED.
Security configuration for accessing to MQTT broker on AWS IoT
![](../images/NodeRED/NodeRED_M_02_04_aws-iot-out.png)
aws-iot-out" node is for setting to publish message to MQTT.
At property screen, set:
- "Server" (same as "Endpoint" of MQTT Broker oo AWS IoT)
- "Topic" (topic name should follow policy set configured on AWS IoT)
- "QoS" (0 or 1. In my environment, both worked. If MQTT connection is bad, it's considerable to change this to 0 to 1)
![](../images/NodeRED/NodeRED_M_02_05_aws-iot-out_properties.png)
"connection" tab in detail property of server, set:
- "Port" as 8833 (default)
- "Client ID" should be okay with blank (this depends on your policy in MQTT broker. In my case, if the device name(client id here) is duplicated with other devices, MQTT is repeated to be disconnected from MQTT broker after 3 seconds MQTT connect. Then I changed "statement" block in MQTT broker's policy to allow any client device like:
"Statement": [ { "Effect": "Allow", "Action": [ "iot:Connect" ], "Resource": [ "arn:aws:iot:ap-northeast-1:062759670692:client/*" ] },
![](../images/NodeRED/NodeRED_M_02_06_aws-iot-out_security.png)
"connection" tab in detail property of server, set file path of:
- private key
- certification
- CA root certification
Experiment: ESP32 and Node-RED over MQTT AWS IoT broker
ESP32 and Node-RED over MQTT AWS IoT brokerExperiment: multiple ESP32 boards(with the other students in remote place) and Node-RED over MQTT
![](../images/assignment/week14/w14_composition_messages_between_projects.png)
Then I did experiment with the other students (from Fablab Kamakura) as group assignment (send a message between two project). For detail procedure, please see group assignment page link )
![](../images/assignment/week14/w14_NodeRED_mqtt_flow_fa2020jp_experiement.png)
For experiment to publish message from Node-RED to ESP32 over MQTT, changed injected string charactors on Node-RED.
![](../images/assignment/week14/mqtt_aws_v3_test_k2.jpg)
At first, we had error to connect to MQTT (connect and disconnect is repeated). The client ID of MQTT (device ID) cannot be the same when connecting to MQTT at the same time.
After we corrected device ID for each local ESP32 (ex. ID10: Oguri-san, ID11: Kimura-san, ID1: Me), it worked and we could publish/subscribe message from ESP32, RaspPi to multiple ESP32 at remote location.
mqtt_aws_v3.ino(Gitlab, clean up security information)
For avoiding duplicated device ID of MQTT client, I changed the source code and use mac address of the device as follows.
WiFi.macAddress(mac_addr);
sprintf(deviceId, "%02X:%02X:%02X:%02X:%02X:%02X", mac_addr[0], mac_addr[1], mac_addr[2], mac_addr[3], mac_addr[4], mac_addr[5]);)
I2C (ATtiny3216 input board - OLED display)
For my final project, I want to display the status that is input by tactile switches because it will change the mode of sound. Showing the values in display would also help debugging. I used 0.96 inches 128x64 dots OLED display
I designed and fabricated components for this
![](../images/final/input/f06_07_input-board-sw.HEIC_896x-q85.jpg)
![](../images/final/input/f06_13_input_switch_back.jpg)
Input board for switches(ver.1.1)
- 4 switches and necessary circuit for Vcc, GND and data with pullup resister
- Designs of board (link to the input device page for final project)
![](../images/final/input/f06_06_input-board.HEIC_896x-q85.jpg)
Input board(ver.1.0)
- Control input signals, supply power from FTDI cable to pins and dispaly it to OLED display by I2C
- Designs of board (link to the input device page for final project)
![](../images/final/input/f06_23_input_connection.HEIC_896x-q85.jpg)
Then I connected the boards by cables and programmed to dispaly the switch status programmed in input board.
![](../images/final/networking/f09_OLED.jpg)
Test code for this is final_Input_brd_v0.30_i2c.ino
I also documented simple code example or added the link to video for experiment to show switches status changes in final project page.
References
- MQTT
- MQTT Broker
- AWS IoT
- AWS IoT - Core Tutorial
- AWS IoT Developper Guide
- AWS IoTとRaspberry PiではじめるIoT超入門
- AWSで遊ぶ 〜M5StickC (ESP32) とAWS IoTのMQTT双方向通信〜
- RSA鍵、証明書のファイルフォーマットについて
- CA Certificates for Server Authentication
- AWS IoT - 接続関連の問題の診断
- 基本的な AWS IoT Core ポリシー変数
- Publish/Subscribe Policy Examples
- AWS IoT MQTT クライアントでデバイスの MQTT メッセージを確認する
- ESP32でAWS IoTに繋いでThing Shadowを弄る(2) トラブルシューティング編"
- MQTT - PubSubClient
- Github - knolleary/pubsubclient
- ESP32でAWS IoTに繋いでThing Shadowを弄る(★)
- ESP32活用① ESP32とブラウザでお話しする(8)MQTTで通信
- Arduino JSON
Lessons Learned
- MQTT is quite powerful infrastructure for networking multiple sensor device in terms of easy implementation and performance.
- Node-RED provides fine user interface that allows to combine with MQTT and operate remote devices.
Files
- ESP32 client code: mqtt_aws_v3.ino(Gitlab, clean up security information)
- Node-RED flow definition: mqtt_aws_flows_fa2020jp_experiment.json