Week 15 — Interface and application programming

Individual assignment

Per the week brief: embedded UI on a microcontroller, graphical output plus touch input, enough wiring and protocol detail that someone else could repeat the bench setup.

1) Task and starting idea

I wanted one firmware image that showed the words Fab Academy with a tappable button below, exercising both the panel and the touch controller in a single pass. I used an AI assistant inside Cursor to draft that combined sketch. In practice the button graphic did not refresh reliably and the Serial monitor stayed quiet when I tapped, so I stopped guessing and split the work into two programs: display-only bring-up, then touch-focused bring-up. That separation made it obvious whether SPI wiring, reset timing, or the touch stack was the weak link.

2) Learning (pins, controllers, and references)

ILI9341 over 4-wire SPI was the baseline: chip-select, data/command, SPI clock/MOSI (and MISO where the library or readback path needs it), plus backlight and reset. The vendor shipped an STM32-oriented example; I ported the drawing tests to ESP32-Arduino with Adafruit_ILI9341 + Adafruit_GFX, keeping their gamma tables and rotation mapping where it helped this particular IPS panel look correct.

The bigger lesson was strapping pins on the ESP32. I first tied the LCD RST line to GPIO12 because that matched a convenient label on a pinout card. The panel stayed dark even though SPI traffic looked plausible in code. Digging into Espressif’s documentation, GPIO12 is also MTDI, a strapping pin that is sampled at chip reset to configure the flash voltage behavior of the VDD_SDIO domain. At reset, the voltage expected on that strapping state must match the flash fitted on the module (the common WROOM modules use 3.3 V flash). If GPIO12 is driven or pulled to the wrong level during that sampling window, boot or flash access can fail or behave intermittently—which is why many reference designs tell you to treat GPIO12 as “do not use lightly” for outputs such as an active-low reset that might sit in an unsafe state at power-up. After I moved TFT_RST to a normal, non-strapping GPIO (and kept the net short and clean), the ILI9341 initialization sequence finally ran as expected.

Primary references I used while writing this up: ESP32 datasheet (strapping pins / MTDI) and Espressif’s boot mode / strapping overview.

For touch, the panel is capacitive with an FT6336 controller, not a resistive XPT2046 on bit-banged SPI. The first AI-generated sketch assumed resistive SPI touch, which never matched the wiring on my FPC. Switching to an I²C driver path (address commonly 0x38, with I²C SDA/SCL, optional INT, and a controller reset pin) matched the vendor examples and started returning coordinates. In the sketch I ship here, the bring-up messages assume SDA on GPIO22, SCL on GPIO18, and a CTP reset on GPIO5 (see the comments in main_test_touch.cpp and your local board_pins.h).

3) Plan

1. Stabilize 3.3 V, GND, and SPI signals; confirm begin() succeeds.
2. Run the vendor-style color-bar and fill tests to prove the glass is alive.
3. Probe the touch controller on I²C, then map raw coordinates into display space.
4. Capture photos and a short clip for documentation; keep two .cpp entry points so display and touch stay easy to retest.

4) Operation and evidence

The display-only pass re-used the vendor’s visual tests (color bands, rectangles, rotation) after the GPIO12 mistake was corrected. Once that was reliable, the touch pass drew trails and printed normalized points on the serial console whenever the FT6336 reported a stable finger-down edge.

5) AI-assisted LCD UI mockups (sketch → HTML → captures)

To practice interface design at the real pixel budget of the display, I drew a paper wireframe, wrote short prompts describing layout and navigation, and had an AI generate a single HTML/CSS file with five horizontal panels in a swipe container—each panel sized to 320×240 to match the ILI9341 panel’s landscape window. The prototype is in Chinese because that matches how I iterate with the tool and the copy I will eventually ship in the project; the documentation here stays in English.

Five screen exports (one per panel generated from the sketch + prompts):

6) Second UI pass — ASRPRO voice assistant and XIAO ESP32‑S3 over I²C

System integration narrative (v1 bench, display/I²C debug, appendix I²C test): Week 16.

After the HTML-to-TFT port (§5) I wired in the ASRPRO V2.0 module from the Tianwen toolchain: offline wake + command slots, on-board mic and speaker, Mandarin phrase table instead of cloud STT. Wake word for my forest spirit is 灵葭; saying it should start the short dialogue loop that drives ILI9341 UI states on the WROOM‑32D.

6a) Three-board architecture (FINAL bench bundle)

The working stack I debugged on the bench is a three-MCU application, not a single sketch: ASRPRO handles offline voice, the Seeed XIAO ESP32‑S3 is the integration hub (DHT11 + light sensor, WiFi, DeepSeek dialogue), and the ESP32-WROOM-32D from §4 still owns the ILI9341 + FT6336 five-page swipe UI. All three share one I²C bus on the XIAO’s D4/D5 pair (with pull-ups and common GND): the S3 is master, the WROOM‑32D listens at 0x55, and ASRPRO exposes the voice mailbox at 0x56. Getting ASRPRO onto that bus meant adding vendor-side asr_i2c_slave.* inside Tianwen—not the block-only path I started with—then matching phrase IDs in snid_phrase.cpp on the S3 and page routes on the WROOM‑32D.

I archived the full debug tree under code/week15-individual/final/ (PlatformIO projects for XIAO S3 + WROOM‑32D, Tianwen C++ for ASRPRO, plus the v2 screen reference screen-ui.html). Copy s3-hub/src/secrets.h.example → secrets.h before uploading the hub firmware; WiFi + DeepSeek keys stay local and are gitignored.

6b) What the demo does (voice ↔ hub ↔ screen)

The offline phrase table is not open microphone dictation: Tianwen maps each utterance to an integer snid (灵葭 is the wake slot, snid == 0; command phrases are snid ≥ 1). When ASRPRO recognizes a hit, it plays vendor TTS and publishes a three-byte mailbox (valid flag + 16‑bit ID). The S3 hub polls 0x56, logs [ASR] snid=…, routes offline commands to WROOM‑32D UI pages, and—for dialogue intents such as “打开对话” / “问精灵” (snid ≥ 7)—calls DeepSeek when WiFi is up, then pushes assistant text to the ILI9341 chat panel over the 0x55 TLV link. UART at 115200 8N1 on each board still mirrors events for debugging.

Protocol summary: DATA_FLOW.md. Source layout: final/asrpro/, final/s3-hub/, final/wroom-display/.

6c) Hero evidence — talking to 灵葭 on the bench

The recording below is the first bench take where saying 灵葭 woke ASRPRO, got a TTS reply, and kept going through the offline command table. Fab’s brief asks for input and output in one application; here that is mic + speaker on ASRPRO and (when wired) ILI9341 UI pages on the WROOM‑32D, even though the enclosure is still open on the breadboard.

6d) Reproduce the FINAL bundle (upload order)

1. Flash wroom-display/ (PlatformIO env:esp32dev) — ILI9341 + FT6336 swipe UI, I²C slave 0x55.
2. Copy secrets.h.example → secrets.h in s3-hub/src/, fill WiFi + DeepSeek credentials, then flash env:seeed_xiao_esp32s3.
3. Merge final/asrpro/ into your Tianwen ASRPRO project; wire the I²C slave read callback to asr_i2c_slave_next_tx_byte().
4. Bench wiring notes (Chinese): i2c_wiring_notes_zh.txt; overview: final/README.md.

6e) Source code walkthrough — UI process, data flow, and TFT protocols

This section answers the Week 15 assessment line: “Implement a User Interface (UI) using programming and explore protocols to communicate with a microcontroller board that you designed and made.” (Fab Academy Interface and Application Programming learning outcome.) My application is the five-page swipe UI on the ESP32-WROOM-32D + ILI9341 + FT6336 stack; it talks over I²C to boards I designed earlier—the Seeed XIAO sensor hub on my Week 8 carrier (Week 8) and, in the full demo, an ASRPRO voice module—while the WROOM itself also sits on my Week 6 group purple ESP32-WROOM PCB or the breadboard rig documented in §4.

Overall development process

I did not drop a GUI toolkit onto the MCU. The workflow was deliberately staged so each layer could be tested alone:

1. Design at real pixel size — paper wireframe → five-panel 320×240 px HTML/CSS swipe prototype (§5, screen-ui.html).
2. Display bring-up — vendor STM32 examples ported to ESP32-Arduino; SPI + reset/backlight validated in main_test_screen.cpp (§4).
3. Touch bring-up — FT6336 on a separate I²C bus (Wire), coordinate mapping in main_test_touch.cpp.
4. Embedded UI port — each HTML panel became a C++ drawPage*() function in wroom-display/src/main.cpp; swipe direction matches the HTML prototype.
5. Multi-MCU integration — XIAO S3 hub pushes sensor + chat data over I²C (0x55); ASRPRO publishes voice events at 0x56 (§6).

flowchart LR
  subgraph design["Design"]
    SK["Paper sketch"]
    HTML["320×240 HTML prototype"]
  end
  subgraph wroom["ESP32-WROOM — display MCU"]
    UI["main.cpp drawPage*()"]
    TFT["ILI9341 via SPI"]
    CTP["FT6336 via I2C Wire"]
    SLV["env_i2c_slave @ 0x55 Wire1"]
  end
  subgraph hub["XIAO ESP32-S3 — designed Week 8 hub"]
    SENS["DHT11 + light"]
    WIFI["WiFi → DeepSeek"]
    MST["I2C master D4/D5"]
  end
  subgraph voice["ASRPRO V2.0"]
    MIC["Offline wake 灵葭"]
    MB["Mailbox @ 0x56"]
  end
  SK --> HTML --> UI
  UI --> TFT
  CTP --> UI
  MST --> SLV
  SENS --> MST
  WIFI --> MST
  MIC --> MB --> MST
          

How the interface is created

The UI is immediate-mode graphics: every frame is drawn with Adafruit GFX primitives on top of Adafruit_ILI9341—rectangles, rounded rects, circles, and bitmap text—inside five page functions:

• Page 0 — Emoji: temperature mood face (reads DHT telemetry from I²C).
• Page 1 — Plant: nitrogen / plant status placeholders + wall clock line.
• Page 2 — Environment: RH, °C, light bars from telemetry frame 0x01.
• Page 3 — Control: brightness/volume sliders, motion/mood/permission toggles (local touch + status readback to hub).
• Page 4 — Chat: user/assistant bubbles fed by I²C text TLVs when voice dialogue runs.

Navigation is a small state machine: global s_page (0…4), redrawUi() switches on the page index and calls the matching drawer, then paints footer dot indicators. Horizontal swipes are detected in handleSwipeEnd() after the finger lifts; on the Control page, vertical slider drags and button taps are handled first so they do not accidentally change pages. Panel order and colours were copied from screen-ui.html so the bench TFT matches the browser mock-up.

How data is handled and updated

Data lives in three layers—local UI state, I²C snapshots, and version counters that trigger redraws only when something changed:

• Local state (in main.cpp): current page, backlight/volume percentages, control toggles (s_ctrlMotion, s_ctrlMood, s_ctrlPerm), and touch gesture bookkeeping (s_was_down, start/end coordinates).
• Incoming bus data (in env_i2c_slave.cpp): an ISR-safe ring buffer collects bytes from Wire1 (second I²C port, separate from touch). Fixed 8-byte TelemetryPacked frames update EnvFromS3 (RH, temp×10, light×10). Variable-length TLV frames ([magic 0xA5][cmd][len][payload]) update chat lines, WiFi status, offline voice route IDs, wall-clock strings, and streamed assistant chunks—see i2c_env_link.h.
• Outgoing bus data: when the user taps Control buttons, syncHostUiStatus() packs SlaveUiStatusPacked (magic 0x5A) so the XIAO hub can requestFrom(0x55) and adjust DeepSeek tone / safety constraints.
• Redraw policy (loop()): envI2cDataVersion() refreshes Emoji/Environment pages; hostLinkUiDataVersion() refreshes Chat and can jump pages on voice snid routes; hostLinkWallClockDataVersion() refreshes Plant/Environment headers—avoiding full-screen spam on every I²C byte.

How the UI is communicated to the TFT display

The ILI9341 panel is a 4-wire SPI RGB565 display. After setup() configures GPIO and SPI.begin(SCK, MISO, MOSI), Adafruit_ILI9341::begin() sends the controller init sequence at 40 MHz (TFT_SPI_HZ in board_pins.h). Each draw call ultimately becomes SPI transactions: set column/row window, stream 16-bit colour pixels. I ported the vendor STM32 gamma tables (stm32IpsGammaTune()) and rotation mapping (applyStm32DisplayDir()) so this IPS module matches their reference colours. Backlight intensity is PWM on TFT_BL (GPIO25); dragging the Control slider calls backlightSetPct() in real time.

Touch is not on SPI—it uses a second bus: Wire @ 400 kHz to FT6336 address 0x38. Ft6336::read() polls register 0x02; mapSampleToGfx() applies swap/invert/calibration constants so raw coordinates align with the 320×240 landscape framebuffer. Host-link I²C uses Wire1 on GPIO19/21 specifically so touch traffic and hub traffic never share the same driver instance.

Protocols explored (microcontroller ↔ microcontroller)

The hub firmware that masters this bus lives in final/s3-hub/; voice-side C++ for Tianwen is in final/asrpro/.

Libraries, framework, and tools

On-board results (hero evidence)

All captures below were taken on my final integrated bench—the WROOM display stack wired to the XIAO hub I designed (Week 8)—not a desktop simulator. The five stills match the live TFT pages (swipe order left → right).

Final system — five UI pages on the TFT

Bench video and wiring

7) Source code in this repo

Same files as the Downloads block at the top—listed again here for readers who scroll to the conclusion. The §6e walkthrough explains what each module does.

FINAL integrated application (voice + hub + five-page ILI9341 UI — see §6 video):

↓ Browse code/week15-individual/final/ — three-board bundle (asrpro/, s3-hub/, wroom-display/, screen-ui.html)

Display UI originals (wroom-display/src/):

↓ Download main.cpp · ↓ Download env_i2c_slave.cpp · ↓ Download i2c_env_link.h · ↓ Download board_pins.h · ↓ Download ft6336_touch.h · ↓ Download platformio.ini

Earlier Week 15 bring-up sketches (WROOM‑32D + ILI9341 display/touch split tests):

↓ Download main_test_screen.cpp — display tests + optional CTP paint loop

↓ Download main_test_touch.cpp — touch-centric demo with serial traces

↓ Download week15-smart-plant-companion-lcd-ui.html — first 320×240 HTML swipe prototype (§5)

PlatformIO projects pull Adafruit GFX / ILI9341 and DHT libraries via lib_deps; ASRPRO sources drop into the Tianwen toolchain. See final/README.md for pins and upload order.

8) Conclusion

The combined Cursor sketch was a false start: wrong touch chip assumption and TFT reset on GPIO12 left me debugging graphics when boot strapping was the fault. Splitting display vs touch tests and moving reset off MTDI gave a repeatable ILI9341 + FT6336 setup I can carry into the final UI.

The finished application is immediate-mode C++ UI code (§6e): HTML panels ported to drawPage*() functions, SPI pixels to the ILI9341, touch on I²C, and sensor/voice/chat data on a second I²C port to my Week 8 XIAO hub. Original sources are in wroom-display/ with per-file downloads at the top of this page; on-board hero clips are in §6e and the 灵葭 dialogue video in §6c.

The ASRPRO + XIAO + WROOM‑32D bundle in final/ adds voice mailbox (0x56), sensor hub traffic, and the five-page ILI9341 UI over the same I²C plan. Saying 灵葭 on the bench now hits wake, routes snid commands, and can open a DeepSeek chat on the display when WiFi is up; packaging polish is next, not another bus bring-up.