Skip to content

Data Broker

Adafruit IO Adafruit IO
Adafruit IO Adafruit IO Arduino Library

Locus Pocus Data Broker - Adafruit IO + Arduino

A stream of actionable location events can be made available as a data source for use in applications, but there needs to be a reliable way to distribute the data. The distribution is often handled by a "data broker," which specializes in collecting and publishing streams of data from different sources. Other applications can then subscribe to the published data streams. So, an application could subscribe to a stream of location-entered events for the person with the device & App.

Presuming a stream of location events, such as entering a location or leaving a location is available, the data needs to be reliably distributed as a feed of event messages. For IoT and smart device applications, a standard messaging protocol for data exchange is Message Queuing Telemetry Transport (MQTT).

The primary application functionality is to:

  • Set up a data broker service
  • Set up a data feed that can be subscribed to
  • Connect to the data broker, check subscribed feed(s) in real time for events
  • Make an application response to the events

Research

I considered several different data broker services. There are a variety that have been used in previous projects.

  • Adafruit IO - provides a free-tier MQTT service with specific library code for connecting to the serivce
  • HiveMQ Cloud - provides a free-tier MQTT service, can use general Arduino MQTT libraries
  • Home Assistant - self-hosted service - can set up your own Home Assistant broker, often using a small compute platform such as Raspberry Pi

What Others Have Done Beforehand

In considering potential data broker services, I reviewed the data brokers used in some of the previous related projects.

Project Data Broker
Modern Weasley Clock Adafruit IO
Weasley Clock Home Assistant
Where'sLy Clock Project HiveMQ Cloud
Magic-Clock Home Assistant

Data Broker Selection for Locus Pocus

In general, for managing privacy, I would gravitate toward the self-hosted Home Assistant broker. But the additional setup and management time was not on the cards. I selected Adafruit IO as the data broker - it provides a free usage tier and specific libraries. Also, my global evaluator (Iván Sánchez Milara) had created a tutorial on using the service, and I thought it was an interesting connection: Sending events to ESP8266 using Adafruit IO and IFTT .

Locus Pocus Data Broker Setup with Adafruit IO

Setup with Adafruit IO is fairly straightforward. An account with Adafruit is needed to get started. I already had an account with Adafruit, and that provides access to the free tier of Adafruit IO.

Moving to the Adafruit IO Dashboard, you can see information about status, data feeds and usage.

Adafruit IO Dashboard

Adafruit IO Dashboard

In order to manage feeds, you select the Feeds View. This provides an overview of available feeds, activity, and enables creation of new feeds.

Adafruit IO Feeds

Adafruit IO Feeds

Creating a new data feed is straightfoward. Use the "New Feed" button and provide a name for the feed, with optional description.

Adafruit IO Create New Feed

Adafruit IO Create New Feed

The Feed dashboard provides information on feed activity. It enables you to view recent data events, as well as to download historical feed data. If you have numerical data, such as sensor readings, the graph will display the values over time. For non-numerical data, such as for Locus Pocus, the values for each event are listed below the graph area.

Adafruit IO Feed Data

Adafruit IO Feed Data

An important part of this for setup is that you can directly add data to the feed from the interface, which was very helpful for testing.

Adafruit IO Feed Data

Adafruit IO Adding Feed Data

Connecting to Location Activity Data Provider - IFTTT

Adafruit IO has an integration available for receiving data from IFTTT - the selected locaton data provider for Locus Pocus. Adafruit IO has a dashboard for setting up integrations, called "Power-Ups". IFTTT connectivity was configured by selecting the IFTTT power-up on the Adafruit IO side, providing IFTTT account detail, and confirming the connection while logged in to the IFTTT account. Setting up the integration made the IFTTT data available via an Adafruit IO feed.

Adafruit IO IFTTT Integration

Adafruit IO IFTTT Integration

Locus Pocus Application Connection to Adafruit IO

The Locus Pocus system uses the Adafruit IO Arduino library to connect with and use the data broker service. This generally involves 2 main steps

  • Configuring the connection - specifying credentials for WiFi access and Adafruit account
  • Using the data service - connecting to the Adafruit IO and accesing data feeds

Configuring Adafruit.io

In order to receive data feeds from Adafruit IO, 2 connections are needed.

  • Connect to WiFi - for general internet / cloud network access
  • Connect to Adafruit IO Service - to access Adafruit IO data feeds

In order to connect to WiFi, it is necessary to provide WiFi credentials to your local network. In order to connect to Adafruit, it is necessary to provide Adafruit IO credentials - these are your username and a special key for data access, which is available in your Adafruit IO account dashboard.

The ArduinoAdafruit IO Arduino library provides a set of code examples. The examples use a common pattern, where configuration / credentials are entered in a "config.h" file, which is then imported into the application code.

The config.h file does several primary things:

  • Specifies WiFi access credentials - ssid and password for the WiFi access point
  • Specifies Adafruit IO access credentials - username and access key for connecting to Adafruit IO services
  • Configures WiFi library setup for the type of embedded computing board you are using
  • Created the "io" object - which is the main way of interacting with Adafruit IO in the application code

The example config.h file for the "adafruitio_01_subscribe" example is shown below.

config.h
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
/************************ Adafruit IO Config *******************************/

// visit io.adafruit.com if you need to create an account,
// or if you need your Adafruit IO key.
#define IO_USERNAME "your_username"
#define IO_KEY "your_key"

/******************************* WIFI **************************************/

// the AdafruitIO_WiFi client will work with the following boards:
//   - HUZZAH ESP8266 Breakout -> https://www.adafruit.com/products/2471
//   - Feather HUZZAH ESP8266 -> https://www.adafruit.com/products/2821
//   - Feather HUZZAH ESP32 -> https://www.adafruit.com/product/3405
//   - Feather M0 WiFi -> https://www.adafruit.com/products/3010
//   - Feather WICED -> https://www.adafruit.com/products/3056
//   - Adafruit PyPortal -> https://www.adafruit.com/product/4116
//   - Adafruit Metro M4 Express AirLift Lite ->
//   https://www.adafruit.com/product/4000
//   - Adafruit AirLift Breakout -> https://www.adafruit.com/product/4201
//   - Adafruit AirLift Shield -> https://www.adafruit.com/product/4285
//   - Adafruit AirLift FeatherWing -> https://www.adafruit.com/product/4264

#define WIFI_SSID "your_ssid"
#define WIFI_PASS "your_pass"

// uncomment the following line if you are using airlift
// #define USE_AIRLIFT

// uncomment the following line if you are using winc1500
// #define USE_WINC1500

// uncomment the following line if you are using mrk1010 or nano 33 iot
// #define ARDUINO_SAMD_MKR1010

// comment out the following lines if you are using fona or ethernet
#include "AdafruitIO_WiFi.h"

#if defined(USE_AIRLIFT) || defined(ADAFRUIT_METRO_M4_AIRLIFT_LITE) ||         \
    defined(ADAFRUIT_PYPORTAL)
// Configure the pins used for the ESP32 connection
#if !defined(SPIWIFI_SS) // if the wifi definition isnt in the board variant
// Don't change the names of these #define's! they match the variant ones
#define SPIWIFI SPI
#define SPIWIFI_SS 10 // Chip select pin
#define NINA_ACK 9    // a.k.a BUSY or READY pin
#define NINA_RESETN 6 // Reset pin
#define NINA_GPIO0 -1 // Not connected
#endif
AdafruitIO_WiFi io(IO_USERNAME, IO_KEY, WIFI_SSID, WIFI_PASS, SPIWIFI_SS,
                   NINA_ACK, NINA_RESETN, NINA_GPIO0, &SPIWIFI);
#else
AdafruitIO_WiFi io(IO_USERNAME, IO_KEY, WIFI_SSID, WIFI_PASS);
#endif
/******************************* FONA **************************************/

// the AdafruitIO_FONA client will work with the following boards:
//   - Feather 32u4 FONA -> https://www.adafruit.com/product/3027

// uncomment the following two lines for 32u4 FONA,
// and comment out the AdafruitIO_WiFi client in the WIFI section
// #include "AdafruitIO_FONA.h"
// AdafruitIO_FONA io(IO_USERNAME, IO_KEY);

/**************************** ETHERNET ************************************/

// the AdafruitIO_Ethernet client will work with the following boards:
//   - Ethernet FeatherWing -> https://www.adafruit.com/products/3201

// uncomment the following two lines for ethernet,
// and comment out the AdafruitIO_WiFi client in the WIFI section
// #include "AdafruitIO_Ethernet.h"
// AdafruitIO_Ethernet io(IO_USERNAME, IO_KEY);

Connecting / Using Adafruit IO Data Broker Service

Once configured, the application code can connect to Adafruit IO in order to receive data feeds. Here, I hightlight the main touchpoints in the application code for interacting with Adafruit IO. The detailed application code is documented in the Primary Control section.

The following code excepts show the following main steps for connecting and using the Adafruit IO service. The code presumes the overall Adafruit IO service object - "io" - has been set up in config.h, as noted above. The steps are:

  • Clock feed is set up
  • MQTT connection is set up
  • MQTT connection is started
  • Message handler is set up for what to do when data is received on the feed
    • function to handle messages is defined later on
  • Application setup waits until there is both a WiFi connection and an Adafruit IO service connection
  • Application contiually checks for feed data activity
locuspocus-primary.ino - Adafruit IO Service Excepts
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
...

// set up the 'clock' data feed
AdafruitIO_Feed *clockdata = io.feed("clock");
...

// start MQTT connection to io.adafruit.com
io.connect();
...

// set up a message handler for the count feed.
// the handleMessage function (defined below)
// will be called whenever a message is
// received from adafruit io.
clockdata->onMessage(handleMessage);
...

// wait for an MQTT connection
while(io.mqttStatus() < AIO_CONNECTED) {
...
}
...


void loop() {
... 

  // io.run(); is required for all sketches.
  // it should always be present at the top of your loop
  // function. it keeps the client connected to
  // io.adafruit.com, and processes any incoming data.
  io.run();
... 
}

// this function is called whenever a 'clock' message
// is received from Adafruit IO. it was attached to
// the clock feed in the setup() function above.
void handleMessage(AdafruitIO_Data *data) {
  // Get broker data as string and process message
  sendClockMessage(data->value());
}
...