Week 15 — Interface and application programming

Individual assignment

Per the week brief, this is a small embedded UI exercise: a microcontroller application that drives a graphical output and later reads a touch input, with enough detail that someone else could reproduce the wiring and understand the protocols.

1) Task and starting idea

I wanted one firmware image that showed the words Fab Academy with a tappable button below, exercising both the panel and the touch controller in a single pass. I used an AI assistant inside Cursor to draft that combined sketch. In practice the button graphic did not refresh reliably and the Serial monitor stayed quiet when I tapped, so I stopped guessing and split the work into two programs: display-only bring-up, then touch-focused bring-up. That separation made it obvious whether SPI wiring, reset timing, or the touch stack was the weak link.

2) What I learned (pins, controllers, and documents)

ILI9341 over 4-wire SPI was the baseline: chip-select, data/command, SPI clock/MOSI (and MISO where the library or readback path needs it), plus backlight and reset. The vendor shipped an STM32-oriented example; I ported the drawing tests to ESP32-Arduino with Adafruit_ILI9341 + Adafruit_GFX, keeping their gamma tables and rotation mapping where it helped this particular IPS panel look correct.

The bigger lesson was strapping pins on the ESP32. I first tied the LCD RST line to GPIO12 because that matched a convenient label on a pinout card. The panel stayed dark even though SPI traffic looked plausible in code. Digging into Espressif’s documentation, GPIO12 is also MTDI, a strapping pin that is sampled at chip reset to configure the flash voltage behavior of the VDD_SDIO domain. At reset, the voltage expected on that strapping state must match the flash fitted on the module (the common WROOM modules use 3.3 V flash). If GPIO12 is driven or pulled to the wrong level during that sampling window, boot or flash access can fail or behave intermittently—which is why many reference designs tell you to treat GPIO12 as “do not use lightly” for outputs such as an active-low reset that might sit in an unsafe state at power-up. After I moved TFT_RST to a normal, non-strapping GPIO (and kept the net short and clean), the ILI9341 initialization sequence finally ran as expected.

Primary references I used while writing this up: ESP32 datasheet (strapping pins / MTDI) and Espressif’s boot mode / strapping overview.

For touch, the panel is capacitive with an FT6336 controller, not a resistive XPT2046 on bit-banged SPI. The first AI-generated sketch assumed resistive SPI touch, which never matched the wiring on my FPC. Switching to an I²C driver path (address commonly 0x38, with I²C SDA/SCL, optional INT, and a controller reset pin) matched the vendor examples and started returning coordinates. In the sketch I ship here, the bring-up messages assume SDA on GPIO22, SCL on GPIO18, and a CTP reset on GPIO5 (see the comments in main_test_touch.cpp and your local board_pins.h).

3) Plan

1. Stabilize 3.3 V, GND, and SPI signals; confirm begin() succeeds.
2. Run the vendor-style color-bar and fill tests to prove the glass is alive.
3. Probe the touch controller on I²C, then map raw coordinates into display space.
4. Capture photos and a short clip for documentation; keep two .cpp entry points so display and touch stay easy to retest.

4) Operation and evidence

The display-only pass re-used the vendor’s visual tests (color bands, rectangles, rotation) after the GPIO12 mistake was corrected. Once that was reliable, the touch pass drew trails and printed normalized points on the serial console whenever the FT6336 reported a stable finger-down edge.

5) AI-assisted LCD UI mockups (sketch → HTML → captures)

To practice interface design at the real pixel budget of the display, I drew a paper wireframe, wrote short prompts describing layout and navigation, and had an AI generate a single HTML/CSS file with five horizontal panels in a swipe container—each panel sized to 320×240 to match the TFT’s landscape window. The prototype is in Chinese because that matches how I iterate with the tool and the copy I will eventually ship in the project; the documentation here stays in English.

Prototype file: save or open the HTML from this site the same way as other weekly artifacts (browsers often preview instead of saving—use Save As… if you need a local copy).

• Download — smart plant companion 320×240 swipe UI (.html) · Open in new tab

Five screen exports (one per panel generated from the sketch + prompts):

6) Second UI pass — ASRPRO voice assistant and XIAO ESP32‑S3 over I²C

After the first HTML-to-TFT port (§5), I wanted the screen stack to feel like part of the 森之精灵 / Forest Spirit plant companion—not just static panels. The natural next step was to bring in the ASRPRO V2.0 module I had been configuring in the Tianwen (天问) toolchain: offline wake + command slots, on-board microphone and speaker, and a fixed Mandarin phrase table instead of cloud speech-to-text. My forest spirit’s name is 灵葭; saying it should wake the assistant and start a short dialogue loop that eventually maps to UI states on the TFT side.

6a) Three-board architecture (FINAL bench bundle)

The working stack I debugged on the bench is a three-MCU application, not a single sketch: ASRPRO handles offline voice, the Seeed XIAO ESP32‑S3 is the integration hub (DHT11 + light sensor, WiFi, DeepSeek dialogue), and the ESP32‑WROOM from §4 still owns the ILI9341 + FT6336 five-page swipe UI. All three share one I²C bus on the XIAO’s D4/D5 pair (with pull-ups and common GND): the S3 is master, the WROOM listens at 0x55, and ASRPRO exposes the voice mailbox at 0x56. Getting ASRPRO onto that bus meant adding vendor-side asr_i2c_slave.* inside Tianwen—not the block-only path I started with—then matching phrase IDs in snid_phrase.cpp on the S3 and page routes on the WROOM.

I archived the full debug tree under code/week15-individual/final/ (PlatformIO projects for S3 + WROOM, Tianwen C++ for ASRPRO, plus the v2 screen reference screen-ui.html). Copy s3-hub/src/secrets.h.example → secrets.h before uploading the hub firmware; WiFi + DeepSeek keys stay local and are gitignored.

6b) What the demo does (voice ↔ hub ↔ screen)

The offline phrase table is not open microphone dictation: Tianwen maps each utterance to an integer snid (灵葭 is the wake slot, snid == 0; command phrases are snid ≥ 1). When ASRPRO recognizes a hit, it plays vendor TTS and publishes a three-byte mailbox (valid flag + 16‑bit ID). The S3 hub polls 0x56, logs [ASR] snid=…, routes offline commands to WROOM UI pages, and—for dialogue intents such as “打开对话” / “问精灵” (snid ≥ 7)—calls DeepSeek when WiFi is up, then pushes assistant text to the TFT chat panel over the 0x55 TLV link. UART at 115200 8N1 on each board still mirrors events for debugging.

Protocol summary: DATA_FLOW.md. Source layout: final/asrpro/, final/s3-hub/, final/wroom-display/.

6c) Hero evidence — talking to 灵葭 on the bench

The clip below is the moment the stack stopped being “wires plus logs” and started feeling like the character I sketched in the UI mockups: I say 灵葭, the module wakes, answers in the vendor TTS voice, and the exchange continues through the fixed offline command table. That is the application interface Fab Academy asks for—input (microphone / wake + commands) and output (speaker + eventual TFT state)—even though the enclosure and polished UI v2 screens are still in progress.

6d) Reproduce the FINAL bundle (upload order)

1. Flash wroom-display/ (PlatformIO env:esp32dev) — ILI9341 + FT6336 swipe UI, I²C slave 0x55.
2. Copy secrets.h.example → secrets.h in s3-hub/src/, fill WiFi + DeepSeek credentials, then flash env:seeed_xiao_esp32s3.
3. Merge final/asrpro/ into your Tianwen ASRPRO project; wire the I²C slave read callback to asr_i2c_slave_next_tx_byte().
4. Bench wiring notes (Chinese): i2c_wiring_notes_zh.txt; overview: final/README.md.

7) Source code in this repo

FINAL integrated application (voice + hub + five-page TFT — matches the §6 video):

• code/week15-individual/final/ — three-board bundle (asrpro/, s3-hub/, wroom-display/, screen-ui.html).

Earlier Week 15 bring-up sketches (display/touch split tests before the full stack):

• main_test_screen.cpp — display tests + optional CTP paint loop.
• main_test_touch.cpp — touch-centric demo with serial traces.
• week15-smart-plant-companion-lcd-ui.html — first 320×240 HTML swipe prototype (§5).

PlatformIO projects pull Adafruit GFX / ILI9341 and DHT libraries via lib_deps; ASRPRO sources drop into the Tianwen toolchain. See final/README.md for pins and upload order.

8) Conclusion

The combined AI-written UI was a good idea for a demo, but without matching the actual touch controller and respecting ESP32 strapping pins, it was impossible to tell whether I had a graphics bug or a hardware-configuration bug. Splitting the firmware, reading the ESP32 boot pin documentation, and aligning to I²C + FT6336 turned a frustrating blank screen into a repeatable lab setup I can reuse for the final project UI.

The ASRPRO pass turned the UI exercise into a real application boundary: voice events, sensor telemetry, cloud dialogue, and TFT pages are separate firmware images joined by a documented I²C protocol in final/. Saying 灵葭 on the bench now wakes the character, routes commands, and can open a DeepSeek-backed chat on the display—enough for me to iterate on packaging and polish without re-proving the buses.