14. Interface and Application Programming#
hero shot#
When my hand moves away from the sensor, it turns blue, and when my hand gets closer to the sensor, it turns colored.
groupwork#
For details, see the link below:
https://fabacademy.org/2025/labs/kannai/Weekly_Group_Assignment/week14/
my work part#
How to use Colaboratory#
- obtain the “fab_graph.ipynb” file from the download LINK at the bottom of this page and save it to any Google Drive. 2.
- right-click on the “fab_graph.ipynb” file on the Google Drive > open > google Colaboratory.
- the fab_graph Colaboratory will open in a browser-based window.
Coding and Visualization#
Below are the libraries used, the code, and the graphs created for each.
matplotlib#
Matplotlib is one of the most widely used graph drawing libraries in Python. It is suitable for visualizing data and analytical results, and is used in a variety of fields such as research, education, and business.
For more details, please see: https://matplotlib.org/
import matplotlib.pyplot as plt
# Data for Local evaluation
local_labels = [
'40% or more\n(8+ assignments ready)',
'20-35%\n(4–7 assignments ready)',
'5-20%\n(1–3 assignments ready)',
'0%\n(No assignments ready)'
]
local_counts = [53, 57, 64, 10]
# Data for Global evaluation
global_labels = [
'7+ assignments ready',
'1–6 assignments ready',
'Assigned\n but\n no assignments ready',
'No global eval requested'
]
global_counts = [25, 100, 44, 15]
# Create side-by-side subplots
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
# Left: Local evaluation chart
axes[0].bar(local_labels, local_counts, color='skyblue')
axes[0].set_xlabel('Local Evaluation & Assignments')
axes[0].set_ylabel('Number of Students')
axes[0].set_title('Local Evaluation: Student Count vs Assignment Status')
axes[0].tick_params(axis='x', rotation=30)
# Right: Global evaluation chart
axes[1].bar(global_labels, global_counts, color='salmon')
axes[1].set_xlabel('Global Evaluation & Assignments')
axes[1].set_ylabel('Number of Students')
axes[1].set_title('Global Evaluation: Student Count vs Assignment Status')
axes[1].tick_params(axis='x', rotation=30)
plt.tight_layout()
plt.show()
The graph above shares the mid-term results of Fab Academy 2025, expressing the assignment progress status and graduation prospects using a matplotlib bar chart.
seaborn#
Seaborn is a high-level data visualization library based on Matplotlib that allows you to create beautiful statistical charts with less code. Furthermore, Seaborn extends the functionality of Matplotlib, making it easier to integrate with statistical plots and data frames.
For more details, please see: https://seaborn.pydata.org/
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
# Prepare data in long form
data = {
'Type': ['Local'] * 4 + ['Global'] * 4,
'Evaluation': [
'40% or more\n(8+ assignments ready)',
'20–35%\n(4–7 assignments ready)',
'5–20%\n(1–3 assignments ready)',
'0%\n(No assignments ready)',
'7+ assignments ready',
'1–6 assignments ready',
'Assigned but no assignments ready',
'No global eval requested'
],
'Students': [53, 57, 64, 10, 25, 100, 44, 15]
}
df = pd.DataFrame(data)
# Pivot data for heatmap (Evaluation as rows, Type as columns)
pivot = df.pivot(index='Evaluation', columns='Type', values='Students').fillna(0)
# Seaborn theme
sns.set_theme(style="white")
# Plot heatmap
plt.figure(figsize=(8, 6))
sns.heatmap(
pivot,
annot=True,
fmt="g",
cmap="YlOrBr",
cbar_kws={'label': 'Number of Students'}
)
# Labels and title
plt.title("Student Count by Evaluation Type\nand Assignment Status")
plt.xlabel("Evaluation Type")
plt.ylabel("Assignment Status")
plt.yticks(rotation=15, ha='right')
plt.tight_layout()
plt.show()
The graph above shares the mid-term results of Fab Academy 2025, expressing the assignment progress status and graduation prospects using a seaborn heatmap.
plotly#
Plotly is a data visualization library that makes it easier to create interactive and dynamic graphs compared to Matplotlib and Seaborn.
Plotly specializes in creating interactive graphs on a web browser, allowing users to explore data dynamically, such as by zooming and hovering on the graph.
For more details, please see: https://plotly.com/
You need to install plotly in google collaboratory. You can install it by executing the following code.
!pip install plotly
Try running the following code after the installed code.
import plotly.graph_objects as go
import pandas as pd
labels = [
"All", "Local", "Global",
"40% or more (8+)", "20–35% (4–7)", "5–20% (1–3)", "0% (None)",
"7+ ready", "1–6 ready", "Assigned/no ready", "No global req"
]
parents = [
"", "All", "All",
"Local", "Local", "Local", "Local",
"Global", "Global", "Global", "Global"
]
values = [
0, sum([53,57,64,10]), sum([25,100,44,15]),
53,57,64,10,
25,100,44,15
]
base_color = "lightgrey"
highlight_color = "crimson"
colors = [base_color]*len(labels)
line_w = [1]*len(labels)
fig = go.Figure(go.Treemap(
labels=labels, parents=parents, values=values,
marker=dict(colors=colors, line=dict(color="black", width=line_w))
))
local_cats = ["40% or more (8+)", "20–35% (4–7)", "5–20% (1–3)", "0% (None)"]
buttons_local = []
for cat in local_cats:
idx = labels.index(cat)
new_colors = [highlight_color if i==idx else base_color for i in range(len(labels))]
new_line = [3 if i==idx else 1 for i in range(len(labels))]
buttons_local.append(dict(
method="restyle",
label=cat,
args=[{"marker.colors": [new_colors],
"marker.line.width": [new_line]}]
))
global_cats = ["7+ ready", "1–6 ready", "Assigned/no ready", "No global req"]
buttons_global = []
for cat in global_cats:
idx = labels.index(cat)
new_colors = [highlight_color if i==idx else base_color for i in range(len(labels))]
new_line = [3 if i==idx else 1 for i in range(len(labels))]
buttons_global.append(dict(
method="restyle",
label=cat,
args=[{"marker.colors": [new_colors],
"marker.line.width": [new_line]}]
))
fig.update_layout(
updatemenus=[
dict(active=-1, x=0.1, y=1.15, buttons=buttons_local, showactive=True, direction="down", pad=dict(r=10,t=10)),
dict(active=-1, x=0.4, y=1.15, buttons=buttons_global, showactive=True, direction="down", pad=dict(r=10,t=10))
],
title="Interactive Treemap: Highlight Your Category"
)
fig.show()
The video above shares the mid-term results of Fab Academy 2025, expressing the assignment progress status and graduation prospects using a plotly treemap. You can see that the treemap changes interactively when you click on the graph. Also, by selecting the pull tab, the target group is colored. This makes it easier to judge how many members of the whole are targeted.
2.individual work#
Task: Interface and Application Programming
Checklist
- Linked to the group assignment page.
- Documented your process.
- Explained the UI that you made and how you did it.
- Explained how your application communicates with your embedded microcontroller board.
- Explained any problems you encountered and how you fixed them.
- Included original source code (or a screenshot of the app code if that’s not possible).
- Included a ‘hero shot’ of your application running & communicating with your board.
Idea#
I want to create an interface where screen visuals and audio change in real-time based on switches or wireless signals.
Tool Research#
Tool List (Table Format)#
| Tool Name | URL |
|---|---|
| VFX Graph (Unity) | https://unity.com/ja/features/visual-effect-graph |
| openFrameworks | https://openframeworks.cc/ |
| Tidal Cycles | https://tidalcycles.org/ |
| Orca (XXIIVV) | https://wiki.xxiivv.com/site/orca.html |
| VEDA (VJ app for Atom) | https://veda.gl/ |
| Cables | https://cables.gl/ |
| Vue.js | https://vuejs.org/ |
| Pure Data | https://puredata.info/ |
| TouchDesigner | https://derivative.ca/ |
Conclusion#
I decided to use the tool TouchDesigner.
TouchDesigner’s interface is a highly extensible workspace where you can combine operators (functions) to generate various forms of output (audio, visual, data, etc.).
In TouchDesigner, these node arrangements are called networks.
- Mac-like software
- Node-based processing. The nodes themselves are well-organized as a UI and are easy to use.
- Free
- Good for connecting with microcontrollers.
- Can generate sound as well as visuals.
- Official learning documentation is well-organized and easy to learn from.
TouchDesigner Initial Setup#
- Install
- Create an account
- Create a key
Initial screen
Create a project in PlatformIO#
To use the Xiao ESP32-S3, write the following to the ini file:
[env:seeed_xiao_esp32s3]
platform = espressif32
board = seeed_xiao_esp32s3
framework = arduino
upload_speed = 115200
monitor_speed = 115200
Running TouchDesigner#
- Test code for connecting TouchDesigner and the Xiao
#include <Arduino.h> void setup() { pinMode(LED_BUILTIN, OUTPUT); Serial.begin(115200); // Baud rate suitable for TouchDesigner } void loop() { digitalWrite(LED_BUILTIN, HIGH); delay(100); digitalWrite(LED_BUILTIN, LOW); delay(100); Serial.println(analogRead(1)); // Example: Read from A1 delay(50); }
TouchDesigner settings#
* Press Tab > select Serial DAT
When you place the Serial DAT, its parameters will appear on the right.
Configure each parameter as shown in the image. For ‘Port’, select the port you are currently using.
Once configured, turn ‘Active’ ON. You will then be able to receive sensor data as shown in the video below.
Next, let’s try sending the data to a CHOP.
Press Tab > CHOP > select DAT To
Place it and configure as shown in the image. For ‘DAT Name’, enter the name of the Serial DAT created earlier.
It will work if you toggle the DAT’s ‘Active’ parameter OFF and then ON again.
Getting Ultrasonic Distance Sensor Data into a TouchDesigner CHOP#
The sensor used is this one:
Grove-Ultrasonic-Distance-Sensor
Upload the following code to the Seeed Xiao ESP32-S3:
#include <Arduino.h>
#define ULTRASOUND_PIN 6 // ← Correct the pin number
void setup()
{
Serial.begin(115200);
pinMode(ULTRASOUND_PIN, OUTPUT);
}
void loop()
{
// Send trigger pulse
digitalWrite(ULTRASOUND_PIN, LOW);
delayMicroseconds(2);
digitalWrite(ULTRASOUND_PIN, HIGH);
delayMicroseconds(10);
digitalWrite(ULTRASOUND_PIN, LOW);
// Receive echo pulse
pinMode(ULTRASOUND_PIN, INPUT);
long duration = pulseIn(ULTRASOUND_PIN, HIGH, 30000);
pinMode(ULTRASOUND_PIN, OUTPUT);
float distance = duration * 0.034 / 2.0;
Serial.println(distance); // To TouchDesigner
delay(200);
}
It will work if you toggle the DAT’s ‘Active’ parameter OFF and then ON again.
Making Neil Glow Rainbow Colors Based on Sensor Distance#
In TouchDesigner, arrange and connect the nodes as follows:
Select the Math CHOP and configure its Range tab parameters.
- From Range 5 - 25
- To Range 0 - 1
Next, as shown in the video, connect the Math CHOP to the Hue Offset parameter of the HSV Adjust TOP.
HSV Adjust node > Parameters > Hue Offset + > Select the green box.
On the right, enter op('math1')['chan1'].
Check the name of your Math CHOP and ensure you enter the correct node name for your setup.
Also, set the Hue Range to 360.
This allows you to change Neil’s color hue based on the sensor readings.
The TouchDesigner screen looks like this.
Check if it works on your own board#
Changing ULTRASOUND_PIN to 5 will work with kannai board. If you are not sure which pin is connected to the sensor, write the following code and check it on the serial monitor. The following code outputs the information acquired on pins 2 to 8 to the serial monitor.
#include <Arduino.h>
int pins[] = {2, 3, 4, 5, 6, 7, 8}; // test pins
void setup()
{
Serial.begin(115200);
}
void loop()
{
for (int i = 0; i < sizeof(pins) / sizeof(pins[0]); i++)
{
int pin = pins[i];
pinMode(pin, OUTPUT);
digitalWrite(pin, LOW);
delayMicroseconds(2);
digitalWrite(pin, HIGH);
delayMicroseconds(10);
digitalWrite(pin, LOW);
pinMode(pin, INPUT);
long duration = pulseIn(pin, HIGH, 30000);
float distance = duration * 0.034 / 2.0;
Serial.print("Pin ");
Serial.print(pin);
Serial.print(": ");
Serial.println(distance);
}
delay(1000);
}
Below is a video of it in action. The video shows that it is received with 5 pins.
Connect to TouchDesigner again. The following video confirms that it works with the original board.