14. Interface and application programming

This week we had to make a user interface so a user could interact with the development boards we made in previous weeks. Processing is a nice program we can use to create user interfaces on the computer and interact with peripherals using serial communication, UDP, TCP, HTTP or a range of other protocols. I decided to use UDP as the means of communication with the board this week. The board would act as both the UDP server and the wifi network.

View the group assignment here

The Program's Functionality

The user on the computer can do a right or left click and a corresponding number (0 or 1 respectively) will be displayed on the tft screen attached to the esp microcontroller.

Arduino Code

The arduino code can be broken down as folows:

• Lines 1-4 the libraries for drawing on the screen on my board and the libraries for wifi and UDP networking are imported. Each of these is required for what I'm trying to do here.

  The wifi (Wifi.h) library will be used to actually create a network from the ESP32. It is known as an "access point". This means we will be using the ESP32 as a router (like one you have at home). Laptops, like my own, will then be able to connect to it and there will be connectivity with all other things connected to it. In our case we will have my laptop connected to it and the ESP32 itself.

  Next we import the WifiUdp library. This library provides the framework to allow the laptop to speak to the ESP32 using the UDP protoocl. The distinction between UDP and some of its alternatives is that UDP does not require an "acknowledgment" of a sent packet. This means if Tom sends a message to Sally with UDP, Tom never gets confirmation that his message was received. This allows for higher performance (sending more packets/data per second at the cost of actually knowing what was received or not. I chose it here to test with because my video game system could potentially use UDP.

  The Adafruit_GFX and Adafruit_ST7735 are used to communicate properly with the screen attached to the ESP32.

              
                #include <WiFi.h>
                #include <WiFiUdp.h>
                #include <Adafruit_GFX.h>    // Core graphics library
                #include <Adafruit_ST7735.h> // Hardware-specific library for ST7735
              
            

• 6-11 sets the appropritate pins to communicate with the screen. If these pins don't match the actual physical layout then nothing will nbe displayed on the board. In week 9 I specify the purpose of each pin on the screen.

              
          #define TFT_CS        9
          #define TFT_RST        46 // Or set to -1 and connect to Arduino RESET pin
          #define TFT_DC         3
  
          #define TFT_MOSI 10  // Data out
          #define TFT_SCLK 11  // Clock out
              
            

• 13-14 create the objects to draw to the screen as well as the SPI class to communicate with it. An SPI class is explicitly passed to use the hardware SPI for higher refresh rates (although probably not really needed in the current use case). This is also discussed in week 9.

              
                SPIClass *spi = new SPIClass(HSPI);
                Adafruit_ST7735 tft = Adafruit_ST7735(spi, TFT_CS, TFT_DC, TFT_RST);
              
            

• 16-17 set the credentials for when the local network on the esp32 will be created leveraging the Wifi library we included above. This will allow the laptop to connect to the ESP32's access point.

  
              const char* ssid = "ESP32_AP";        // SSID of the Access Point
              const char* password = "12345678";    // Password of the Access Point
            

• Line 19 sets the port for which there will be UDP communication. Read more about ports here. HTTP uses port 80 and HTTPS uses port 443 by default for example, but you can explicitly set ports for your specific task.

  
              unsigned int localPort = 5005;      // local port to listen on
            

• 123 create the UDP object which will later be used.

              
                WiFiUDP Udp;
              
            

• 25-44 starts the screen communication and UDP server. First the SPI class begins communication on the screens data pin(s). Then the sceen is initialized with the appropriate size with initR. Then the screen is rotated to be in landscape (0 is portrait, 1 is landscape) because that is the orientation of our screen. Then we use fillScreen to fill the screen with black. After that we begin Serial communication with Serial.begin. This is done for debugging purposes so we can read and make sure everything looks good. Finally the wifi access point (what makes our ESP32 a "router") is initialized and we print the IP of our ESP32 to later be used by the laptop. We need the IP of the ESP so we can know where to send UDP data. After that we use UDP.begin with the port specified above to begin "listening" for data being sent. This will allow us to later receive data and display things on the screen according to what we've received from the laptop.

              
                void setup() {
                  spi->begin(TFT_SCLK, -1, TFT_MOSI, TFT_CS);
                  tft.initR(INITR_MINI160x80);  // Init ST7735S mini display
                  tft.setRotation(1);
                  tft.fillScreen(ST77XX_BLACK);
                  Serial.begin(9600);
  
                  // Set up the Wi-Fi in Access Point mode
                  Serial.println("Setting up AP...");
                  WiFi.softAP(ssid, password);
  
                  IPAddress IP = WiFi.softAPIP();
                  Serial.print("AP IP address: ");
                  Serial.println(IP);
  
                  // start UDP
                  Udp.begin(localPort);
                  Serial.print("Listening on port ");
                  Serial.println(localPort);
                }            
              
            

• 46-68 polls for UDP packets. Polling basically means the ESP is constantly asking "is there new data?". If there is then it will act accordingly, otherwise keep asking the same question. If a UDP packet is received from our laptop (or anyone else for that matter) it checks the value. If it is a 1 or a 0 it will be printed to the TFT screen.

            
              void loop() {
                int packetSize = Udp.parsePacket();
                if (packetSize) {
                  // read the packet into packetBuffer
                  int len = Udp.read(packetBuffer, 255);
                  if (len > 0) {
                    packetBuffer[len] = 0;
                  }
                  Serial.print("Received packet: ");
                  Serial.println(packetBuffer);
  
                    tft.setTextColor(ST77XX_BLACK, ST77XX_WHITE);
                    tft.setTextSize(1);
                    tft.setCursor(0, 0);
                  if (strcmp(packetBuffer, "1") == 0) {
                    tft.println("Received 1");
                    // Add your action for 1 here
                  } else if (strcmp(packetBuffer, "0") == 0) {
                    tft.println("Received 0");
                    // Add your action for 0 here
                  }
                }
              }
            
          

Processing Code

The processing code can be broken down as follows. This is run on my laptop.

• line 1 imports the UDP library. This will be used to send UDP packets from the laptop to the ESP32.

  
              import hypermedia.net.*;
  
            

• 6-10 sets up the UDP class and the size of the interface (200x200). It does not listen on the UDP port because it will only send, not receive data. Basically you can think of this as a one way pipe. Messages are sent from the laptop but it does not receive.

  
            void setup() {
              size(200, 200);  // set the size of the window
              udp = new UDP(this, 6000);  // create a new UDP object
              udp.listen(false);  // disable listening (client mode)
            }
  
            

• 12-16 draws the actual interface which is just a box asking the user to right or left click in it. The text() function call lets the user know how to interact with the interface. That's good UX!

  
              void draw() {
                background(255);  // set background color to white
                fill(0);  // set fill color to black
                text("Click to send 1 or 0", 50, height/2);  // display instructions
              }
  
            

• 18-22 is the callback function for mouse clicks and sends data over UDP, 1 corresponding to a left click, 0 for a right click. A "callback function" is a function that gets run whenever a certain event happens. In our case, whenever a mouse click happens, the code in this function will also get run.

              
                void mousePressed() {
                  String message = (mouseButton == LEFT) ? "1" : "0";  // send "1" for left click and "0" for any other click
                  udp.send(message, "192.168.4.1", 5005);  // replace with your Arduino's IP address and port
                  println("Mouse pressed: " + message);  // print the message to the console
                }
  
              
            

Concluding Thoughts

This worked pretty consistently but if I really wanted to ensure that the screen changed appropriately to every mouse click I would've used a protocol that gives an acknowledgement like TCP or HTTP (whcih is built on top of TCP). This would allow the laptop to know when data was not received and give the opportunity to resend it.

Code files

code for this week can be found here