networking and communications
linux usb gadgets
For my final project, I want to embed a SoC like a Raspberry Pi into a keyboard. The device as a whole should then be able to function either as a whole computer (with HDMI or Serial out), or as a standalone keyboard (to be plugged into a host computer via USB).
Internally to the device, I can see two ways to accomplish this:
switch-topology
One way is to build or use some kind of USB switching circuit, and connect the keyboard either to the SoC or to the host-facing USB port.
The benefit of this approach is that the keyboard can function as a peripheral even when the SoC unpowered. This is especially important given that e.g. the standard ‘Raspbian’ image for the Raspberry Pi Zero takes more than 10s to boot, which is quite uncomfortable for a plug-and-play device like a keyboard.
man-in-the-middle-topology
Another approach would be to hard-wire the keyboard to the SoC, and then have the SoC itself take on the USB communication:
This would enable some other very interesting uses in addition to using the device soley as a keyboard, such as establishing a wired network connection between the SoC and the host system, data transfer via the mass storage protocol etc.
While there are benefits and drawbacks to both solutions, I want to explore the
second approach. I found out that the Linux kernel contains the
Gadget API, which can be used to implement USB-device features on
supported hardware (In the Raspberry Pi series, only the Zero, Zero W and 4
have the required hardware setup). Using the Gadget API, kernel modules can
implement a USB device. There is also a number of gadgets that ship with the
kernel, such as g_ether, g_hid etc. I had used g_ether once before, but
without understanding it much - what it does is create a virtual ethernet
interface over USB, that allows the host and device to be network-connected.
This is very useful when working with a Raspberry Pi Zero, because it allows
you to SSH into it from the host PC and configure it via USB, rather
than by attaching an extra keyboard and monitor.
However using these modules (g_ether, g_hid, …) only a single module can
control the USB port at one moment. To customize the modules, it is also
necessary to write a kernel module, which is rather complicated. Luckily, there
is another option: The libcomposite module. It allows to create a ‘composite’
USB gadget implementing multiple different ‘functions’ that can be active at
the same time!
The first hurdle to start working like this was to get a stable setup with the Raspberry Pi Zero I had on hand. The Zero has two Micro USB connectors on the board, but one of them can only be used for power, while the other one is used for both host and device mode USB connectivity. This presents a problem: I cannot connect a keyboard (to work on the raspi) and the host computer (to test the configuration) at the same time! I started by powering the raspi via the power port, and then switched the keyboard and device-host cables every time I wanted to test something, but this got tiring very quickly. Therefore I decided to use the raspi serial port to talk to the raspi via UART. I used an Arduino Uno I had as a serial cable by bridging the RESET pin to GND.
I then connected the TX/RX and a ground pin to the Raspberry Pi:
Note that when using the Uno as a Serial adapter, the TX and RX labels may be confusing: they are labelled from the Arduinos perspective, so they do not need to be crossed over when connecting to the raspi - TX to TX, RX to RX.
With the connections made, I could use screen to start a Serial session with
the Pi via the Arduino’s USB serial port:
host $ sudo screen /dev/ttyACM1 115200
I started encountering some problems here. On some boots, the serial connection was very unstable: while my input seemed to always arrive properly, the text printed back over serial often skipped characters and sometimes garbled data. This is probably due to the voltage mismatch, but I just put up with it until I got the USB networking working. I could just write out files on my host PC, and then paste them over serial ‘blindly’ when I had to do something more complex.
The first step to setting up device-mode USB is to enable the device-mode USB
kernel driver. This can be done directly on the Pi or by removing the SD card
and changing the config.txt file on it, I did the former. The only change
necessary is to append these two lines and reboot.
enable_uart=1 # this is optional, but may improve Serial stability
dtoverlay=dwc2
Once the Pi comes back up, the dwc2 module should be loaded and visible
using lsmod. With that sorted out, we can move on to libcomposite. It
can be loaded using sudo modprobe libcomposite and should now also show up in
the module list. If everything worked, we should now see a “UDC” interface:
pi $ lsmod
Module Size Used by
libcomposite 49479 0
dwc2 109939 0
udc_core 12769 2 dwc2,libcomposite
uio_pdrv_genirq 3718 0
uio 10230 1 uio_pdrv_genirq
ipv6 367697 29
pi $ ls /sys/class/udc
20980000.usb
the “UDC” is the device-mode USB port as seen from Linux' side. On the Zero
there is only one, but other systems might have multiple device-mode ports,
so they are identified using directories in /sys/class/udc.
libcomposite is configured by creating and symlinking files and directories
in /sys/kernel/config/gadget_configfs. The whole structure of the config tree
is described here. I started by following a more concrete
guide on setting up ethernet emulation by collabora. I only followed
the first half of the guide. Both the raspbian image and my host Linux do not
use ifconfig anymore, so here is what the network configuration looks like
using iproute2:
pi $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:ae:dc:54:32:ca brd ff:ff:ff:ff:ff:ff
inet 169.254.76.89/16 brd 169.254.255.255 scope global usb0
valid_lft forever preferred_lft forever
inet6 fe80::1562:15d5:7ba:2774/64 scope link
valid_lft forever preferred_lft forever
If you do not see an IPv4 address assigned, try again after a few seconds. On the host a new interface should show up as well:
host $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
...
3: enp0s20u2u3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether e2:0c:50:a8:3c:49 brd ff:ff:ff:ff:ff:ff
However this interface is DOWN by default, and there are no routes set up. The following two commands make this interface functional:
#!/bin/sh
ip addr add 169.254.76.1/16 brd 169.254.255.255 dev enp0s20u2u3
ip link set dev enp0s20u2u3 up
Note that the first IP in the ip addr add command needs to be in the same
subnet that the Pi chose its IP in, but needs to be different from the one the
Pi chose. With the network set up, we can now connect to the Pi:
host $ ssh pi@169.254.76.89
pi $ # ah!
I now started to work on my own script that would establish a HID function next
to the virtual ethernet connection, in order to send keystrokes over the wire.
In the corresponding documentation, the required settings and an
example tool are shown (albeit in C, not using configfs). With this help, I
came up with the following gen-gadget.sh:
#!/bin/sh
set -e
CONFIGFS_ROOT=/sys/kernel/config # adapt to your machine
cd "${CONFIGFS_ROOT}"/usb_gadget
GADGET_NAME=$1
# create gadget directory and enter it
mkdir $GADGET_NAME
cd $GADGET_NAME
# USB ids
# echo 0x1d6b > idVendor
# echo 0x104 > idProduct
# USB strings, optional
mkdir strings/0x409
echo "s-ol" > strings/0x409/manufacturer
echo "Keyboard and Ethernet" > strings/0x409/product
mkdir functions/ecm.usb0
mkdir functions/hid.usb1
echo 0 > functions/hid.usb1/subclass
echo 1 > functions/hid.usb1/protocol
echo 8 > functions/hid.usb1/report_length
sed <<EOF -e 's/\/\*.*\*\///g' -e 's/,//g' -e 's/0x//g' \
| xxd -r -p > functions/hid.usb1/report_desc
0x05, 0x01, /* USAGE_PAGE (Generic Desktop) */
0x09, 0x06, /* USAGE (Keyboard) */
0xa1, 0x01, /* COLLECTION (Application) */
0x05, 0x07, /* USAGE_PAGE (Keyboard) */
0x19, 0xe0, /* USAGE_MINIMUM (Keyboard LeftControl) */
0x29, 0xe7, /* USAGE_MAXIMUM (Keyboard Right GUI) */
0x15, 0x00, /* LOGICAL_MINIMUM (0) */
0x25, 0x01, /* LOGICAL_MAXIMUM (1) */
0x75, 0x01, /* REPORT_SIZE (1) */
0x95, 0x08, /* REPORT_COUNT (8) */
0x81, 0x02, /* INPUT (Data,Var,Abs) */
0x95, 0x01, /* REPORT_COUNT (1) */
0x75, 0x08, /* REPORT_SIZE (8) */
0x81, 0x03, /* INPUT (Cnst,Var,Abs) */
0x95, 0x05, /* REPORT_COUNT (5) */
0x75, 0x01, /* REPORT_SIZE (1) */
0x05, 0x08, /* USAGE_PAGE (LEDs) */
0x19, 0x01, /* USAGE_MINIMUM (Num Lock) */
0x29, 0x05, /* USAGE_MAXIMUM (Kana) */
0x91, 0x02, /* OUTPUT (Data,Var,Abs) */
0x95, 0x01, /* REPORT_COUNT (1) */
0x75, 0x03, /* REPORT_SIZE (3) */
0x91, 0x03, /* OUTPUT (Cnst,Var,Abs) */
0x95, 0x06, /* REPORT_COUNT (6) */
0x75, 0x08, /* REPORT_SIZE (8) */
0x15, 0x00, /* LOGICAL_MINIMUM (0) */
0x25, 0x65, /* LOGICAL_MAXIMUM (101) */
0x05, 0x07, /* USAGE_PAGE (Keyboard) */
0x19, 0x00, /* USAGE_MINIMUM (Reserved) */
0x29, 0x65, /* USAGE_MAXIMUM (Keyboard Application) */
0x81, 0x00, /* INPUT (Data,Ary,Abs) */
0xc0 /* END_COLLECTION */
EOF
mkdir configs/c.1
mkdir configs/c.1/strings/0x409
echo "Default Configuration" > configs/c.1/strings/0x409/configuration
echo 120 > configs/c.1/MaxPower
ln -s functions/*/ configs/c.1/
That intimidating-looking block of C in the middle was taken straight from the
example, and identifies this HID device as a class-compliant Keyboard. This
script can be run as root, passing a name for the gadget. This will create the
gadget in /sys/kernel/config/<name>, but not activate it yet. To do that, we
just have to write the udc’s name into /sys/kernel/config/<name>/UDC, but
that will only work if the UDC is not currently bound to another gadget
configuration. I created a second little script to disable any old and enable
the new gadget, activate.sh:
#!/bin/sh
set -e
CONFIGFS_ROOT=/sys/kernel/config # adapt to your machine
cd "${CONFIGFS_ROOT}"/usb_gadget
GADGET_NAME=$1
UDC=`ls /sys/class/udc`
for gadget in *; do
if [ x"$UDC"x = x`cat $gadget/UDC`x ]; then
echo "disabling $gadget"
echo > $gadget/UDC
fi
done
echo "enabling $GADGET_NAME"
echo $UDC > $GADGET_NAME/UDC
This way I can pivot from the last gadget into the next one with a single command easily, which is necessary since I will lose network connectivity in the process.
Here is what this configuration looks like to the host:
host $ lsusb -v
...
Bus 002 Device 063: ID 0000:0000
Couldn't open device, some information will be missing
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64
idVendor 0x0000
idProduct 0x0000
bcdDevice 4.04
iManufacturer 1
iProduct 2
iSerial 3
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0078
bNumInterfaces 3
bConfigurationValue 1
iConfiguration 4
bmAttributes 0x80
(Bus Powered)
MaxPower 120mA
Interface Association:
bLength 8
bDescriptorType 11
bFirstInterface 0
bInterfaceCount 2
bFunctionClass 2 Communications
bFunctionSubClass 6 Ethernet Networking
bFunctionProtocol 0
iFunction 8
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 2 Communications
bInterfaceSubClass 6 Ethernet Networking
bInterfaceProtocol 0
iInterface 5
CDC Header:
bcdCDC 1.10
CDC Union:
bMasterInterface 0
bSlaveInterface 1
CDC Ethernet:
iMacAddress 6 (??)
bmEthernetStatistics 0x00000000
wMaxSegmentSize 1514
wNumberMCFilters 0x0000
bNumberPowerFilters 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0010 1x 16 bytes
bInterval 9
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 1
bAlternateSetting 0
bNumEndpoints 0
bInterfaceClass 10 CDC Data
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 1
bAlternateSetting 1
bNumEndpoints 2
bInterfaceClass 10 CDC Data
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 7
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 2
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 3 Human Interface Device
bInterfaceSubClass 0
bInterfaceProtocol 1 Keyboard
iInterface 10
HID Device Descriptor:
bLength 9
bDescriptorType 33
bcdHID 1.01
bCountryCode 0 Not supported
bNumDescriptors 1
bDescriptorType 34 Report
wDescriptorLength 63
Report Descriptors:
** UNAVAILABLE **
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x83 EP 3 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0008 1x 8 bytes
bInterval 4
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x02 EP 2 OUT
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0008 1x 8 bytes
bInterval 4
So the device does show up as both Ethernet and HID now! To send HID reports,
we need to write them to the newly-created device file /dev/hidg0. The C tool
included in the gadget_hid documentation can generate
properly-formatted reports for us. We can compile and run it like this:
pi $ gcc -o gadget_test gadget_test.c
pi $ sudo ./gadget_test /dev/hidg0 keyboard
...
a
a
It’s a bit hard to demonstrate this using console logs, but what happens here is that I type an ‘a’ into the console using my physical keyboard and then hit enter. Once I do that, the Pi generates a HID report with that keypress and sends it to my computer. My computer then types another a into the console.
To demonstrate this better, I created a little test script to pipe into
gadget_test:
#!/bin/sh
sleep 1
echo h ; sleep 0.01
echo e ; sleep 0.01
echo l ; sleep 0.01
echo l ; sleep 0.01
echo o ; sleep 0.01
echo --space ; sleep 0.01
echo w ; sleep 0.01
echo o ; sleep 0.01
echo r ; sleep 0.01
echo l ; sleep 0.01
echo d ; sleep 0.01
Here you can see it typing into gedit on my host machine:
UDP on the ESP32
For the IoT light switch, I need to send and receive packets via the WiFi network in my home. Due to the Quarantine however I couldn’t work well on this project, because the soldering setup I have at home doesn’t permit working with the RGB LEDs. Instead, I converted what I had into the control interface for a structural editor.
Since I just needed to send simple commands to represent the different actions the user can perform on the control interface (turning, pressing/releasing and touching/releasing two knobs, touching/releasing three touch pads), I chose to use plain UDP sockets with a simple ad-hoc string protocol. UDP works well in this case because individual packets should arrive quickly and it is not a huge deal if a packet were to get lost (although this is should be unlikely in the small home network). It also means that each datagram already has defined boundaries, so that the messages become very easy to parse.
The message protocol is based on ASCII-strings formatted as follows:
Knob I N: knob move.Iis0or1,Nis-4or4.Knob I DIR: knob press.Iis0or1,DIRisupordown.Knob/Pad I EVT: touch event.Iis0-4,EVTistouchedorreleased.
Knobs move in steps of four because the raw encoder steps dont correspond to detents. I used 4/-4 anyway so that I could add ‘microstepping’ support in a backwards-compatible way later.
TX implementation
On the esp32, most of the standard socket library is implemented. To send UDP
datagrams, all that is necessary is to create a socket (socket(3)), specify
the target sockaddr_in and then to send a byte buffer using sendto(3).
For the moment, the remote is hardcoded using the two #defines UDP_IP and
UDP_PORT.
I added this to a modified version of the FreeRTOS task that I used for logging
events over serial so far. At startup, the Task waits for the
WIFI_CONNECTED_BIT to be set, indicating that the WiFi connection was
established (more on that later).
The task then waits on the queue of detected input events, so it will only be
woken up if there are any. The events are then formatted into a string using
snprintf(3), which uses printf-style formatting but outpus into a fixed size
buffer. 256 bytes are more than enough for my messages, which don’t really vary
in length.
static void reporting_udp_task(void *pvParameters) {
xEventGroupWaitBits(wifi_event_group, WIFI_CONNECTED_BIT, false, true, portMAX_DELAY);
struct sockaddr_in destAddr;
#ifdef MDNS_HOST
destAddr.sin_addr.s_addr = resolve_mdns_host(MDNS_HOST);
#else
destAddr.sin_addr.s_addr = inet_addr(UDP_IP);
#endif
destAddr.sin_family = AF_INET;
destAddr.sin_port = htons(UDP_PORT);
// Create an IPv4 UDP Socket
int sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_IP);
if (sock < 0) {
ESP_LOGE(TAG, "Unable to create socket: errno %d", errno);
vTaskDelete(NULL);
return;
}
ESP_LOGI(TAG, "Socket created");
int32_t count[ENCODER_NUM_CHANNELS] = { 0, 0 };
encoder_event_t event;
for (;;) {
if (xQueueReceive(event_queue, &event, portMAX_DELAY)) {
char msg_buf[256];
size_t msg_len;
const char* channel = event.channel < ENCODER_NUM_CHANNELS ? "Knob" : "Pad";
switch (event.type) {
case ENC_EVT_SWITCH:
msg_len = snprintf(msg_buf, 256, "%s %d %s\n",
channel, event.channel, event.pressed ? "down" : "up");
break;
case ENC_EVT_TOUCH:
msg_len = snprintf(msg_buf, 256, "%s %d %s\n",
channel, event.channel, event.pressed ? "touched" : "released");
break;
case ENC_EVT_TURN:
count[event.channel] += event.delta;
if (count[event.channel] > 3) {
msg_len = snprintf(msg_buf, 256, "%s %d 4\n", channel, event.channel);
count[event.channel] -= 4;
} else if (count[event.channel] < -3) {
msg_len = snprintf(msg_buf, 256, "%s %d -4\n", channel, event.channel);
count[event.channel] += 4;
} else {
continue;
}
break;
default:
return;
}
int err = sendto(sock, msg_buf, msg_len, 0, (struct sockaddr *)&destAddr, sizeof(destAddr));
if (err < 0) {
ESP_LOGE(TAG, "Error occured during sending: errno %d", errno);
break;
}
}
}
vTaskDelete(NULL);
}
Setting up WiFi is a bit more complex, but I could mostly just follow
examples/protocols/sockets/udp_client and
examples/wifi/getting_started/station. The WIFI credentials have to be
#defined as WIFI_SSID and WIFI_PSK.
/*
* WIFI
*/
static EventGroupHandle_t wifi_event_group;
const int WIFI_CONNECTED_BIT = BIT0;
static const char *TAG = "wifi station";
static esp_err_t event_handler(void *ctx, system_event_t *event) {
static int retry_num = 0;
switch (event->event_id) {
case SYSTEM_EVENT_STA_START:
esp_wifi_connect();
break;
case SYSTEM_EVENT_STA_GOT_IP:
ESP_LOGI(TAG, "got ip:%s\n", ip4addr_ntoa(&event->event_info.got_ip.ip_info.ip));
retry_num = 0;
xEventGroupSetBits(wifi_event_group, WIFI_CONNECTED_BIT);
break;
case SYSTEM_EVENT_STA_DISCONNECTED: {
if (retry_num < 3) {
esp_wifi_connect();
xEventGroupClearBits(wifi_event_group, WIFI_CONNECTED_BIT);
retry_num++;
ESP_LOGI(TAG, "retry to connect to the AP\n");
}
ESP_LOGI(TAG, "connect to the AP fail\n");
break;
}
default:
break;
}
return ESP_OK;
}
void wifi_setup() {
#ifdef MDNS_HOST
ESP_ERROR_CHECK(mdns_init());
#endif
// Initialize flash for WiFi options
esp_err_t ret = nvs_flash_init();
if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
ESP_ERROR_CHECK(nvs_flash_erase());
ret = nvs_flash_init();
}
ESP_ERROR_CHECK(ret);
wifi_event_group = xEventGroupCreate();
tcpip_adapter_init();
ESP_ERROR_CHECK(esp_event_loop_init(event_handler, NULL) );
wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
ESP_ERROR_CHECK(esp_wifi_init(&cfg));
wifi_config_t wifi_config = {
.sta = {
.ssid = WIFI_SSID,
.password = WIFI_PSK,
},
};
ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA));
ESP_ERROR_CHECK(esp_wifi_set_config(ESP_IF_WIFI_STA, &wifi_config));
ESP_ERROR_CHECK(esp_wifi_start());
}
For testing I used netcat (nc(1)) on my host computer to simply visualize the
incoming UDP packets:
$ nc -kul 8080 -w 0
RX implementation
On the receiver side, I used luasocket to receive the incoming messages in
MoonScript:
import udp from require 'socket'
sock = udp!
sock\settimeout 0
assert sock\setsockname '0.0.0.0', 8080
…
handlers =
-- nav dial (left)
'1 -4': -> …
'1 4': -> …
'1 touched': -> …
'1 released': -> …
…
love.update = ->
while true
if msg = sock\receive!
evt = msg\match '%w+ (%d+ [%w-]+)'
if handler = handlers[evt]
handler!
else
print "no handler: '#{evt}' / #{msg}"
else
break
First I create a socket at bind it to all IPv4 interfaces, listening on port 8080. I also set the timeout to 0, so that I can poll the socket in my main event loop without locking up the UI.
In that loop, I keep polling the socket until it has no messages left. For each message, I simply chop off the first word for brevity, and index into a table where I stored the handler functions for each type of message. It would be trivial to be smarter about parsing the message, but in this case it’s simply not necessary ;)
service discovery
With the current approach, the IP address and port of the computer that will receive the UDP packets have to be hardcoded in the ESP32 firmware. This is okay in some circumstances, but can be annoying. One solution to this is to use mDNS service discovery. The esp-idf SDK contains a mDNS implementation, but I struggled to get it to work with my PC’s avahi-daemon. This would be nice to figure out in the future.