14. Interface and Application Programming - Week 14¶
14.1. Assignments¶
Interface and Application Programming
individual assignment
write an application that interfaces a user with an
input &/or output device that you made group assignment
compare as many tool options as possible
14.2. Group Assignment¶
I tried out the tooling with Rust and Slint as a UI-Framework. For the communication with my dev-board I used the TinyUSB library from the esp-idf and set up my board as a CDC-ACM Serial device.
And for debugging I used the UNIX commands cat and echo and for convenience ‘minicom’ to easily read, write and reconnect to the USB Serial device.
14.2.1. General¶
There are many approaches for UI-Development but most of it resolves to using a library/framework (because implementing everything from scratch would be very time-consuming). And there are many of them.
The kinds of UI basically break down in 3 different groups:
Web-Based
using a browser to render your content
Native
using a native operating system application to render your content without a browser in-between
CLI (command-line interface)/text-based
just using the command-line and text as a UI (they can also be beautiful e.g.: btop)
And also there are many different approaches the libraries/frameworks take to develop a UI, I’m going to explain only the approach my framework uses.
The Slint framework uses a DSL (Domain-Specific Language) just for creating the UI, that means you have a whole programming Language where the UI is created by ‘programming’ it. So the objects you create in there and the order you call them actually influences the UI and layout of elements in it. But the actual application logic is written in another language.
To communicate, the DSL then needs to be compiled (for the UI itself) and also transpiled (compiled into another language) to provide an interface the application logic can interact with.
This is done because:
In bigger projects where many people work on it simultaneously, you want to split the work so that different people can work parallel without interfering other peoples work and a DSL is a pretty effective way doing this. Here you only initialy need to design an interface the 2 parts communicate with and then the rest can be developed parallel without interference. Also the back-end/frontend can be extended/fixed (almost) without breaking the other part.
Compiling the DSL for different operating systems and visual styles is easier/more reliably then with most other frameworks because this can just be added by changing the DSL compiler. E.g. Slint can be compiled for Windows, Mac, Linux, Web and even Android and in the future also IOS.
For the application logic I used rust and used a simple Serial-Library to send and read data via USB. This USB device (in CDC-ACM mode) is then just a serial devices reading and sending characters over the interface that is then found in the operating system (e.g. in linux under /dev/ttyACM0) where you can read/write those characters.
14.2.2. Understanding the USB Protocol¶
Note
Generated with deepseek-r1:14b (self-hosted) with following prompt:
Can you explain the Protocol behind USB?
The Universal Serial Bus (USB) protocol is a fundamental technology that enables communication between devices and computers. Here’s a detailed breakdown of its key components and functions:
History and Purpose:
Developed in the mid-1990s to replace older interfaces like parallel and serial ports.
Designed to be fast, versatile, and user-friendly, supporting a wide range of devices.
Key Components:
Host: Typically the computer or laptop that acts as the central controller.
Device: The peripheral connected to the host, such as a printer or keyboard.
Bus: The physical medium (cable) through which data is transmitted between devices.
Data Transfer Mechanisms:
Packets: Data is sent in packets, each containing information about its origin and destination.
Addresses: Assigned by the host to identify devices on the bus, facilitating communication.
Transfer Types:
Control Transfers: Used for device configuration and management.
Bulk Transfers: For large data transfers, like files or images.
Interrupt Transfers: Real-time communication, essential for devices like keyboards and mice.
Isochronous Transfers: Guaranteed bandwidth for time-sensitive data, such as audio or video.
Descriptors:
Endpoints: Channels for communication between the host and device, each with specific roles (e.g., input/output).
Configurations: Define how a device is set up and operates.
Power Management:
Devices can draw power from the host, with mechanisms in place to prevent overloading.
Power requirements are negotiated during connection setup.
Error Handling:
Protocols exist for detecting and recovering from data loss or corruption, including retransmission mechanisms.
Physical Aspects:
Cables and connectors adhere to specific standards (e.g., USB-A, USB-C) ensuring compatibility and reliable communication.
USB Versions:
USB 1.0/2.0/3.0: Offer different data transfer rates (e.g., USB 3.0 supports SuperSpeed).
Devices negotiate the highest supported speed during connection.
Hubs and Root Hubs:
Hub: Allows multiple devices to connect to a single USB port.
Root Hub: Located on the host, manages connections to other hubs and devices.
14.2.3. Setup¶
14.2.3.1. Rust + Slint¶
For Rust I used my already used rust-setup (I basically downloaded rustup with my package-manager and the normal rustc compiler and rest of the default toolchain from rustup ).
To get started I used this template, which has the needed structure for slint and a simple counter to learn the concepts.
14.2.3.2. TinyUSB + ESP-IDF¶
14.3. Individual Assignment¶
I wanted to control the LEDs via USB and also display the touch value, when I press the touchpads on my device.
14.3.1. USB Device Setup¶
First I setup an USB Serial device with this example.
Here the TinyUSB Code:
/*
* SPDX-FileCopyrightText: 2022-2024 Espressif Systems (Shanghai) CO LTD
*
* SPDX-License-Identifier: Unlicense OR CC0-1.0
*/
#include "driver/gpio.h"
#include "driver/touch_sensor.h"
#include "driver/touch_sensor_common.h"
#include "esp_log.h"
#include "freertos/FreeRTOS.h"
#include "freertos/queue.h"
#include "freertos/task.h"
#include "led_strip.h"
#include "portmacro.h"
#include "sdkconfig.h"
#include "tinyusb.h"
#include "tusb_cdc_acm.h"
#include "tusb_console.h"
#include <stdint.h>
#include <stdio.h>
static const char *TAG = "example";
static uint8_t rx_buf[CONFIG_TINYUSB_CDC_RX_BUFSIZE + 1];
#define TOUCH_DEFAULT_LEVEL 50000
#define TOUCH_MAX_LEVEL 800000
#define NUM_OF_TOUCHPADS 2
#define NUM_OF_LEDS 6
static const touch_pad_t button[NUM_OF_TOUCHPADS] = {
TOUCH_PAD_NUM1,
TOUCH_PAD_NUM2,
};
#define LED_V0 1
#define LED_V1 2
#define LED_V2 3
#define LED_G0 6
#define LED_G1 5
#define LED_G2 4
static const char *TOUCH_TAG = "CoolKeyboard";
static led_strip_handle_t led_strip;
union Leds {
struct {
uint8_t id;
uint8_t r;
uint8_t g;
uint8_t b;
} value;
uint32_t raw;
};
/**
* @brief Application Queue
*/
static QueueHandle_t app_queue;
typedef struct {
uint8_t buf[CONFIG_TINYUSB_CDC_RX_BUFSIZE + 1]; // Data buffer
size_t buf_len; // Number of bytes received
uint8_t itf; // Index of CDC device interface
} app_message_t;
int init_device() {
// TOUCH PAD INIT
touch_pad_init();
for (int i = 0; i < NUM_OF_TOUCHPADS; i++) {
touch_pad_config(button[i]);
}
touch_pad_set_measurement_interval(TOUCH_PAD_SLEEP_CYCLE_DEFAULT);
touch_pad_set_charge_discharge_times(TOUCH_PAD_MEASURE_CYCLE_DEFAULT);
touch_pad_set_voltage(TOUCH_PAD_HIGH_VOLTAGE_THRESHOLD,
TOUCH_PAD_LOW_VOLTAGE_THRESHOLD,
TOUCH_PAD_ATTEN_VOLTAGE_THRESHOLD);
touch_pad_set_idle_channel_connect(TOUCH_PAD_IDLE_CH_CONNECT_DEFAULT);
for (int i = 0; i < NUM_OF_TOUCHPADS; i++) {
touch_pad_set_cnt_mode(button[i], TOUCH_PAD_SLOPE_DEFAULT,
TOUCH_PAD_TIE_OPT_DEFAULT);
}
/* Denoise setting at TouchSensor 0. */
touch_pad_denoise_t denoise = {
/* The bits to be cancelled are determined according to the noise level.
* */
.grade = TOUCH_PAD_DENOISE_BIT4,
.cap_level = TOUCH_PAD_DENOISE_CAP_L4,
};
touch_pad_denoise_set_config(&denoise);
touch_pad_denoise_enable();
ESP_LOGI(TOUCH_TAG, "Denoise function init");
/* Enable touch sensor clock. Work mode is "timer trigger". */
touch_pad_set_fsm_mode(TOUCH_FSM_MODE_TIMER);
touch_pad_fsm_start();
// END TOUCHPAD
// LED STRIP INIT
led_strip_config_t strip_config = {
.strip_gpio_num = 3,
.max_leds = NUM_OF_LEDS, // at least one LED on board
};
led_strip_spi_config_t spi_config = {
.spi_bus = SPI2_HOST,
.flags.with_dma = true,
};
ESP_ERROR_CHECK(
led_strip_new_spi_device(&strip_config, &spi_config, &led_strip));
led_strip_clear(led_strip);
// END LED STRIP
return 1;
}
/**
* @brief CDC device RX callback
*
* CDC device signals, that new data were received
*
* @param[in] itf CDC device index
* @param[in] event CDC event type
*/
void tinyusb_cdc_rx_callback(int itf, cdcacm_event_t *event) {
/* initialization */
size_t rx_size = 0;
/* read */
esp_err_t ret =
tinyusb_cdcacm_read(itf, rx_buf, CONFIG_TINYUSB_CDC_RX_BUFSIZE, &rx_size);
if (ret == ESP_OK) {
app_message_t tx_msg = {
.buf_len = rx_size,
.itf = itf,
};
// copy from buffer to local object
memcpy(tx_msg.buf, rx_buf, rx_size);
// send local object to a queue used in another thread
xQueueSend(app_queue, &tx_msg, 0);
} else {
ESP_LOGE(TAG, "Read Error");
}
}
/**
* @brief CDC device line change callback
*
* CDC device signals, that the DTR, RTS states changed, not needed
*
* @param[in] itf CDC device index
* @param[in] event CDC event type
*/
void tinyusb_cdc_line_state_changed_callback(int itf, cdcacm_event_t *event) {
int dtr = event->line_state_changed_data.dtr;
int rts = event->line_state_changed_data.rts;
ESP_LOGI(TAG, "Line state changed on channel %d: DTR:%d, RTS:%d", itf, dtr,
rts);
}
void app_main(void) {
// Create FreeRTOS queue to receive the messages send via serial
app_queue = xQueueCreate(5, sizeof(app_message_t));
assert(app_queue);
app_message_t msg;
ESP_LOGI(TAG, "USB initialization");
// initialise the TinyUSB Library
const tinyusb_config_t tusb_cfg = {
.device_descriptor = NULL,
.string_descriptor = NULL,
.external_phy = false,
#if (TUD_OPT_HIGH_SPEED)
.fs_configuration_descriptor = NULL,
.hs_configuration_descriptor = NULL,
.qualifier_descriptor = NULL,
#else
.configuration_descriptor = NULL,
#endif // TUD_OPT_HIGH_SPEED
};
// install the usb driver, removing the default one used for flashing the XIAO
ESP_ERROR_CHECK(tinyusb_driver_install(&tusb_cfg));
// initialise the CDC-ACM device that should be used on USB
tinyusb_config_cdcacm_t acm_cfg = {
.usb_dev = TINYUSB_USBDEV_0,
.cdc_port = TINYUSB_CDC_ACM_0,
.rx_unread_buf_sz = 64,
.callback_rx = NULL,
.callback_rx =
&tinyusb_cdc_rx_callback, // the first way to register a callback
.callback_rx_wanted_char = NULL,
.callback_line_state_changed = NULL,
.callback_line_coding_changed = NULL};
ESP_ERROR_CHECK(tusb_cdc_acm_init(&acm_cfg));
// register a callback for the line change, not needed
ESP_ERROR_CHECK(tinyusb_cdcacm_register_callback(
TINYUSB_CDC_ACM_0, CDC_EVENT_LINE_STATE_CHANGED,
&tinyusb_cdc_line_state_changed_callback));
// if more USB CDC-ACM devices exist on the host
#if (CONFIG_TINYUSB_CDC_COUNT > 1)
acm_cfg.cdc_port = TINYUSB_CDC_ACM_1;
ESP_ERROR_CHECK(tusb_cdc_acm_init(&acm_cfg));
#endif
ESP_LOGI(TAG, "USB initialization DONE");
// initialise led and touch
init_device();
union Touch {
uint32_t touch[NUM_OF_TOUCHPADS];
uint8_t raw[4 * NUM_OF_TOUCHPADS];
};
while (1) {
// read the touch data
union Touch touch_data = {.touch = {0, 0}};
uint32_t sum = 0;
for (int i = 0; i < NUM_OF_TOUCHPADS; i++) {
ESP_ERROR_CHECK(touch_pad_read_raw_data(button[i], &touch_data.touch[i]));
sum += touch_data.touch[i];
}
// write those bytes to the queue before sending them
if (sum > NUM_OF_TOUCHPADS * TOUCH_DEFAULT_LEVEL) {
int bytes =
tinyusb_cdcacm_write_queue(0, touch_data.raw, sizeof(union Touch));
ESP_LOGI(TAG, "%d bytes queued!", bytes);
// the flush then sends the bytes to the host
esp_err_t err = tinyusb_cdcacm_write_flush(0, 50);
if (err != ESP_OK) {
ESP_LOGE(TAG, "CDC ACM write flush error: %s", esp_err_to_name(err));
}
}
// received send data from the host stored in the queue
if (xQueueReceive(app_queue, &msg, 100 / portTICK_PERIOD_MS)) {
if (msg.buf_len) {
union Leds leds[NUM_OF_LEDS] = {};
/* Print received data*/
ESP_LOGI(TAG, "Data from channel %d:", msg.itf);
ESP_LOG_BUFFER_HEXDUMP(TAG, msg.buf, msg.buf_len, ESP_LOG_INFO);
// and then just set the pixel because only those bytes are send
led_strip_set_pixel(led_strip, msg.buf[0], msg.buf[1], msg.buf[2],
msg.buf[3]);
led_strip_refresh(led_strip);
}
}
}
}
Here I had some problems first when sending because it waited at the QUEUE and I needed to change to blocking time: 100 / portTICK_PERIOD_MS which reduced the blocking time so it is interactive.
14.3.2. USB Rust Setup¶
As a first Approach I tried the library nusb And got the code working to find my device, but then I couldn’t find out, why I can’t send data via serial…
println!("Gathering USB Devices...");
let mut devices = list_devices().expect("Error Receiving Devices!");
for device in devices {
println!("ID: {}:{}", device.vendor_id(), device.product_id());
}
let esp_devices = devices.find(|device| device.vendor_id() == 12346);
let keyboard: nusb::DeviceInfo = match esp_devices {
Some(k) => {
println!(
"Found keyboard: \n product_string: {} \n ID: {}:{}",
k.product_string().expect("No Product String avaibible"),
k.vendor_id(),
k.product_id()
);
k
}
None => {
println!("No keyboard found...");
std::process::exit(-1);
}
};
let keyboard_dev = keyboard.open()?;
let kb_interface = keyboard_dev.detach_and_claim_interface(0)?;
loop {
let buf = RequestBuffer::new(512);
let received = block_on(kb_interface.bulk_in(0x80, buf)).into_result();
dbg!(received);
}
… but I noticed, that this library is not used for CDC-ACM devices but for all other USB devices (or maybe there was also an option to use it but I couldn’t find it ._.).
Then I tried an explicit Serial Library: serialport which worked flawlessly (after learning, how to use ;] ).
To find (currently only the first) USB Device, I used:
use serialport::{available_ports, SerialPort, SerialPortType};
fn main() {
// get currently available serial ports
let devices = available_ports().expect("Unable to gather Serial Ports");
let keyboard_dev = devices
.iter()
// using a Lambda-function I print out every found device and information from it
.find(|&device| match device.port_type.clone() {
SerialPortType::UsbPort(usb) => {
println!("Found USB serial device '{}'", device.port_name);
dbg!(usb);
true
}
_ => false,
}).expect("No USB Serial Device found!");
// try to connect to the serial port
let mut keyboard = serialport::new(&keyboard_dev.port_name, 115200).open()?;
}
Note
To find out the Baudrate (here 115200) I looked in the sdkconfig- file for ‘BAUD’ and found some parameter like: “CONFIG_CONSOLE_UART_BAUDRATE=115200”.
Then when its open, I could read the values and also write some back:
let id: i8: 0;
let id_prev: i8: 0;
loop {
let mut buf: [u8; 4 * NUM_OF_KEYS] = [0, 0, 0, 0, 0, 0, 0, 0];
// read from the serial device
match keyboard.read(&mut buf) {
Ok(l) => {
println!("Read {l} bytes:");
dbg!(buf);
// convert them to uint32
let touch_keys: [u32; NUM_OF_KEYS] = [
u32::from_ne_bytes(buf[..4].try_into().expect("Could not convert to u32")),
u32::from_ne_bytes(buf[4..8].try_into().expect("Could not convert to u32")),
];
// print touch values
println!("Actual touch values: {}; {}", touch_keys[0], touch_keys[1]);
}
Err(_) => {}
};
// used protocol with the id constantly changed for testing
let led_on: [u8; 4] = [id, 0, 255, 255];
let led_off: [u8; 4] = [id_prev, 0, 0, 0];
// write to serial interface
match keyboard.write(&led_on) {
Ok(l) => println!("Wrote {l} bytes to turn ON {id}!"),
Err(_) => {}
};
// like above, actually send it
keyboard.flush();
// turn of last led
match keyboard.write(&led_off) {
Ok(l) => println!("Wrote {l} bytes to turn OFF {id_prev}!"),
Err(_) => {}
};
keyboard.flush();
// just go through the leds in a cycle
id_prev = id;
id = (id + 1) % 6;
sleep(time::Duration::from_millis(100));
}
This loop continuously reads the key presses and writes new values for the LEDs to switch around.
14.3.3. Rust Slint UI¶
A simple ui:
And ths DSL - Slint code
import { Button, VerticalBox, HorizontalBox, ComboBox, Slider } from "std-widgets.slint";
component LedsGrid inherits GridLayout {
property <int> led_count: 6;
property <int> board_count: 2;
min-height: 100px;
// callback function used to execute code in rust
callback toggle_led(id: int, state: bool);
// BEGIN Buttons for the leds
Button {
text: self.state ? "LED 0 ON" : "LED 0 OFF";
row: 0;
col: 0;
property <int> id: 0;
property <bool> state: false;
// the 'clicked' function executed when button is pressed
clicked => {
self.state = !self.state;
// call the toggle callback function to execute the rust code
root.toggle_led(self.id, self.state);
}
}
Button {
text: self.state ? "LED 1 ON" : "LED 1 OFF";
row: 0;
col: 1;
property <int> id: 1;
property <bool> state: false;
clicked => {
self.state = !self.state;
root.toggle_led(self.id, self.state);
}
}
Button {
text: self.state ? "LED 2 ON" : "LED 2 OFF";
row: 1;
col: 0;
property <int> id: 2;
property <bool> state: false;
clicked => {
self.state = !self.state;
root.toggle_led(self.id, self.state);
}
}
Button {
text: self.state ? "LED 3 ON" : "LED 3 OFF";
row: 1;
col: 1;
property <int> id: 3;
property <bool> state: false;
clicked => {
self.state = !self.state;
root.toggle_led(self.id, self.state);
}
}
Button {
text: self.state ? "LED 4 ON" : "LED 4 OFF";
row: 2;
col: 0;
property <int> id: 4;
property <bool> state: false;
clicked => {
self.state = !self.state;
root.toggle_led(self.id, self.state);
}
}
Button {
text: self.state ? "LED 5 ON" : "LED 5 OFF";
row: 2;
col: 1;
property <int> id: 5;
property <bool> state: false;
clicked => {
self.state = !self.state;
root.toggle_led(self.id, self.state);
}
}
// END Buttons
}
component Piano inherits HorizontalBox {
min-height: 50px;
in property <bool> key0: false;
in property <bool> key1: false;
// BEGIN Rectangles to visualise key presses
Rectangle {
background: key0 ? green : black;
border-color: white;
border-width: 1px;
}
Rectangle {
background: key1 ? green : black;
border-color: white;
border-width: 1px;
}
// END Rectangles to visualise key presses
}
export component AppWindow inherits Window {
in-out property <int> counter: 42;
in property <bool> key0: false;
in property <bool> key1: false;
min-height: 500px;
// global callback to use in RUST connects to the callback in the led grid
callback toggle_led <=> led_grid.toggle_led;
VerticalBox {
// just show the device (forgot to implement to show the actual used one xD)
HorizontalBox {
Text {
text: "USB Device:";
}
ComboBox {
model: ["/dev/ttyACM0"];
width: 50%;
}
}
Text {
text: "LEDs:";
}
// add the led grid
led_grid := LedsGrid { }
Text {
text: "Piano Keys";
}
// add the piano keys, connecting the input variables defined in the root to use in rust
piano := Piano {
key0 <=> key0;
key1 <=> key1;
}
}
}
And here the integration with rust:
use core::time;
use std::{
error::Error,
rc::Rc,
sync::{Arc, Mutex},
thread::{self, sleep},
};
use serialport::{available_ports, SerialPort, SerialPortType};
// compile slint and get modules to work with them and be able to call the functions of the UI
slint::include_modules!();
use std::convert::TryFrom;
const NUM_OF_KEYS: usize = 2;
struct TouchValues {
keys: [bool; NUM_OF_KEYS],
}
fn main() -> Result<(), Box<dyn Error>> {
let mut t_values = Arc::new(Mutex::new(TouchValues {
keys: [false, false],
}));
fn start_ui(mut keyboard: Box<dyn SerialPort>) {
// initialise the UI window
let ui = AppWindow::new().expect("Couldn't create a window!");
// register the callback to toggle the led
ui.on_toggle_led(move |id, state| {
println!("Toggle LED!");
println!("ID: {id}");
// check the button state (toggled or not)
let buf = if state {
[
id.try_into().expect("Could not convert i32 to u8 for id!"),
255,
255,
255,
]
} else {
[
id.try_into().expect("Could not convert i32 to u8 for id!"),
0,
0,
0,
]
};
// then write to the keyboard given as a mutable reference to the start_ui function
match keyboard.write(&buf) {
Ok(l) => println!("Wrote {l} bytes to turn ON {id}!"),
Err(_) => println!("Error while writing to keyboard!"),
};
keyboard.flush();
});
// actually run the application ui
ui.run().expect("Error running UI!");
}
// explained above
let devices = available_ports().expect("Unable to gather Serial Ports");
let keyboard_dev = devices
.iter()
.find(|&device| match device.port_type.clone() {
SerialPortType::UsbPort(usb) => {
println!("Found USB serial device '{}'", device.port_name);
dbg!(usb);
true
}
_ => false,
})
.expect("No USB Serial Device found!");
let mut keyboard = serialport::new(&keyboard_dev.port_name, 115200).open()?;
let mut kb2 = keyboard
.try_clone()
.expect("Couldn't clone SerialPort of keyboard!");
loop {
let mut buf: [u8; 4 * NUM_OF_KEYS] = [0, 0, 0, 0, 0, 0, 0, 0];
match keyboard.read(&mut buf) {
Ok(l) => {
println!("Read {l} bytes:");
dbg!(buf);
let touch_keys: [u32; NUM_OF_KEYS] = [
u32::from_ne_bytes(buf[..4].try_into().expect("Could not convert to u32")),
u32::from_ne_bytes(buf[4..8].try_into().expect("Could not convert to u32")),
];
println!("Actual touch values: {}; {}", touch_keys[0], touch_keys[1]);
for i in 0..NUM_OF_KEYS {
}
}
Err(_) => {}
};
sleep(time::Duration::from_millis(100));
}
}
Unfortunately I didn’t manged to light up the Keys, only output on command line. I had some rust-language-problems X)