Week 15
AI prompt:
“now generate image when she started week 15 "Interface and Application Programming" and this week needs to do these points: individual assignment: • Write an application that interfaces a user with an input &/or output device that you made group assignment: • Compare as many tool options as possible For the individual part, she does it in a Unity game scene where directional light is changed by coloure sensore finded color. And for the individual part, she do it her final project part, she generate Unity Game where the player needs to in labirinte finde 10 coins and wine when get all coins in separate time (With arduino IDE and Unity). She designed her Joystick PCB and now need to connect it with Unity Game scene.”
Interface and Application Programming
The idea of creating an interface is already very interesting, because it’s another way to connect the physical and digital worlds. Since I already created web interface examples in Week 11, where I showed how I can control motor speed using a mini web page, and in Week 12 I created 3 different web interfaces, this time creating another web interface would be a bit boring, so I needed to integrate something new.
Hrach and I divided the work, and I needed to create something interesting as an interface using the TCS3472 Color Sensor.
So first I decided to do the classic and simple version, and then try something new 😄
I decided to display the detected color from the sensor directly on a web page using a WiFi IP address.
Since I wanted to create a web interface, besides the sensor I needed an MCU with WiFi capability, so I chose the ESP32-C3, which I have used many times, and the TCS3472 Color Sensor.
After briefly studying how to use the TCS3472, I moved to the connections. I only needed these connections:
- TCS3472 VCC → ESP32C3 3V3
- TCS3472 GND → ESP32C3 GND
- TCS3472 SDA → ESP32C3 GPIO8 (default I2C SDA)
- TCS3472 SCL → ESP32C3 GPIO9 (default I2C SCL)
Since I didn’t have a free ESP32C3 at that moment, I used my RC PCB, and since I couldn’t connect directly to the correct pins, I connected wires from the back and fixed them with tape so they stay stable 😄
Now everything is ready for programming 😄
Here is the full code. In the AI era, it’s easy to get code, you just need to clearly understand what you need and how to connect things correctly. I used Claude for help, but I still modified it and got the final result myself.
Prompt:
“ESP32-C3 colour sensor web interface using the TCS34725 sensor. The system reads RGB colour values, detects the nearest colour name, hosts a live Wi-Fi webpage showing a glowing colour preview with RGB values and HEX code, and continuously sends RGB data to Unity through Serial communication for real-time game interaction.”
/*
ESP32-C3 + TCS34725 Colour Sensor — Wi-Fi Web Interface
=========================================================
Wiring:
TCS34725 VCC → ESP32-C3 3V3
TCS34725 GND → ESP32-C3 GND
TCS34725 SDA → ESP32-C3 GPIO8 (default I2C SDA)
TCS34725 SCL → ESP32-C3 GPIO9 (default I2C SCL)
Libraries needed (install via Arduino Library Manager):
- "Adafruit TCS34725" by Adafruit
- "Adafruit BusIO" by Adafruit (dependency)
- WiFi (built-in with ESP32 core)
Usage:
1. Fill in Wi-Fi SSID and PASSWORD below.
2. Flash the sketch.
3. Open Serial Monitor at 115200 to see the assigned IP.
4. Open that IP in a browser on the same network.
*/
#include <WiFi.h>
#include <WebServer.h>
#include <Wire.h>
#include "Adafruit_TCS34725.h"
struct RGB { uint8_t r, g, b; uint16_t c; bool ok; };
// ── Wi-Fi credentials ──────────────────────────────────────────────
const char* WIFI_SSID = "staff";
const char* WIFI_PASSWORD = "SRGB2020";
// ──────────────────────────────────────────────────────────────────
// TCS34725: 50 ms integration, gain ×4 (tweak if sensor is in bright/dark env)
Adafruit_TCS34725 tcs = Adafruit_TCS34725(TCS34725_INTEGRATIONTIME_50MS,
TCS34725_GAIN_4X);
WebServer server(80);
// ── Colour name lookup (simple nearest-colour in RGB space) ────────
struct NamedColour { const char* name; uint8_t r, g, b; };
const NamedColour PALETTE[] = {
{"Red", 220, 20, 60},
{"Orange", 255, 140, 0},
{"Yellow", 255, 215, 0},
{"Yellow-Green",154, 205, 50},
{"Green", 34, 139, 34},
{"Cyan", 0, 206, 209},
{"Blue", 30, 144, 255},
{"Indigo", 75, 0, 130},
{"Violet", 148, 0, 211},
{"Pink", 255, 105, 180},
{"White", 255, 255, 255},
{"Light Grey", 180, 180, 180},
{"Grey", 128, 128, 128},
{"Dark Grey", 64, 64, 64},
{"Black", 0, 0, 0},
{"Brown", 139, 69, 19},
{"Beige", 245, 245, 220},
};
String nearestColourName(uint8_t r, uint8_t g, uint8_t b) {
long best = LONG_MAX;
const char* name = "Unknown";
for (auto& c : PALETTE) {
long dr = (long)r - c.r, dg = (long)g - c.g, db = (long)b - c.b;
long dist = dr*dr + dg*dg + db*db;
if (dist < best) { best = dist; name = c.name; }
}
return String(name);
}
// ── Read sensor → normalised 0-255 RGB ────────────────────────────
//struct RGB { uint8_t r, g, b; uint16_t c; bool ok; };
RGB readSensor() {
uint16_t raw_r, raw_g, raw_b, raw_c;
tcs.getRawData(&raw_r, &raw_g, &raw_b, &raw_c);
if (raw_c == 0) return {0, 0, 0, 0, false};
// Normalise against clear channel so brightness cancels out
uint8_t r = constrain((uint32_t)raw_r * 255 / raw_c, 0, 255);
uint8_t g = constrain((uint32_t)raw_g * 255 / raw_c, 0, 255);
uint8_t b = constrain((uint32_t)raw_b * 255 / raw_c, 0, 255);
return {r, g, b, raw_c, true};
}
// ── HTML page ─────────────────────────────────────────────────────
const char HTML_HEAD[] PROGMEM = R"rawliteral(
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Colour Sensor</title>
<style>
@import url('https://fonts.googleapis.com/css2?family=Space+Mono:wght@400;700&family=DM+Sans:wght@300;500&display=swap');
:root{--bg:#0e0e12;--panel:#17171f;--border:#2a2a38;--accent:#7c6cfa;--text:#e8e8f0;--sub:#888}
*{box-sizing:border-box;margin:0;padding:0}
body{background:var(--bg);color:var(--text);font-family:'DM Sans',sans-serif;
min-height:100vh;display:flex;flex-direction:column;align-items:center;
justify-content:flex-start;padding:40px 20px}
h1{font-family:'Space Mono',monospace;font-size:1.1rem;letter-spacing:.2em;
text-transform:uppercase;color:var(--sub);margin-bottom:40px}
#swatch{width:220px;height:220px;border-radius:50%;border:3px solid var(--border);
box-shadow:0 0 60px rgba(124,108,250,.15);transition:background .4s,box-shadow .4s;
margin-bottom:32px;background:#222}
#name{font-family:'Space Mono',monospace;font-size:1.5rem;font-weight:700;
letter-spacing:.05em;margin-bottom:8px;transition:color .4s}
#hex{font-family:'Space Mono',monospace;font-size:.9rem;color:var(--sub);margin-bottom:32px}
.stats{display:grid;grid-template-columns:repeat(3,1fr);gap:12px;width:100%;max-width:360px;margin-bottom:32px}
.stat{background:var(--panel);border:1px solid var(--border);border-radius:12px;
padding:16px;text-align:center}
.stat-label{font-size:.65rem;letter-spacing:.15em;text-transform:uppercase;color:var(--sub);margin-bottom:6px}
.stat-value{font-family:'Space Mono',monospace;font-size:1.3rem;font-weight:700}
#status{font-size:.75rem;color:var(--sub);letter-spacing:.1em}
.dot{display:inline-block;width:7px;height:7px;border-radius:50%;
background:var(--accent);margin-right:6px;animation:pulse 1.4s infinite}
@keyframes pulse{0%,100%{opacity:1}50%{opacity:.3}}
</style>
</head>
<body>
<h1>⬡ Colour Sensor · TCS34725</h1>
<div id="swatch"></div>
<div id="name">—</div>
<div id="hex">#------</div>
<div class="stats">
<div class="stat"><div class="stat-label">Red</div><div class="stat-value" id="rv">—</div></div>
<div class="stat"><div class="stat-label">Green</div><div class="stat-value" id="gv">—</div></div>
<div class="stat"><div class="stat-label">Blue</div><div class="stat-value" id="bv">—</div></div>
</div>
<div id="status"><span class="dot"></span>Live · updates every 1 s</div>
<script>
function toHex(n){return n.toString(16).padStart(2,'0').toUpperCase()}
function luma(r,g,b){return 0.299*r+0.587*g+0.114*b}
async function poll(){
try{
const res=await fetch('/data');
const d=await res.json();
const hex='#'+toHex(d.r)+toHex(d.g)+toHex(d.b);
document.getElementById('swatch').style.background=hex;
document.getElementById('swatch').style.boxShadow=`0 0 80px ${hex}55`;
document.getElementById('name').textContent=d.name;
document.getElementById('name').style.color=hex;
document.getElementById('hex').textContent=hex;
document.getElementById('rv').textContent=d.r;
document.getElementById('gv').textContent=d.g;
document.getElementById('bv').textContent=d.b;
}catch(e){document.getElementById('status').textContent='⚠ connection lost'}
}
poll();setInterval(poll,1000);
</script>
</body></html>
)rawliteral";
// ── Route handlers ─────────────────────────────────────────────────
void handleRoot() {
server.send_P(200, "text/html", HTML_HEAD);
}
void handleData() {
RGB px = readSensor();
if (!px.ok) {
server.send(500, "application/json", "{\"error\":\"sensor read failed\"}");
return;
}
String name = nearestColourName(px.r, px.g, px.b);
char buf[128];
snprintf(buf, sizeof(buf),
"{\"r\":%d,\"g\":%d,\"b\":%d,\"clear\":%d,\"name\":\"%s\"}",
px.r, px.g, px.b, px.c, name.c_str());
server.send(200, "application/json", buf);
}
// ── Setup ──────────────────────────────────────────────────────────
void setup() {
Serial.begin(115200);
delay(500);
// I2C on default ESP32-C3 pins (SDA=8, SCL=9) — change if needed
Wire.begin(6, 7);
if (!tcs.begin()) {
Serial.println("[ERROR] TCS34725 not found! Check wiring.");
while (1) delay(1000);
}
Serial.println("[OK] TCS34725 initialised");
// Connect to Wi-Fi
Serial.printf("[WiFi] Connecting to %s", WIFI_SSID);
WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.printf("\n[WiFi] Connected! IP: http://%s\n", WiFi.localIP().toString().c_str());
server.on("/", handleRoot);
server.on("/data", handleData);
server.begin();
Serial.println("[HTTP] Server started");
}
// ── Loop ───────────────────────────────────────────────────────────
void loop() {
server.handleClient(); // Keep your web interface running
RGB px = readSensor();
if (px.ok) {
// Send "R,G,B" to Unity via Serial
Serial.print(px.r); Serial.print(",");
Serial.print(px.g); Serial.print(",");
Serial.println(px.b);
}
delay(50); // Send data frequently for smooth updates
}
Including Libraries
This section imports all required libraries for the project.
#include <WiFi.h>
#include <WebServer.h>
#include <Wire.h>
#include "Adafruit_TCS34725.h"
RGB Structure
This section defines the RGB structure for storing colour values. It helps organize the sensor data into one object. This structure stores:
⦿ Red value (r)
⦿ Green value (g)
⦿ Blue value (b)
⦿ Clear channel brightness (c)
⦿ Sensor read status (ok)
struct RGB { uint8_t r, g, b; uint16_t c; bool ok; };
Wi-Fi Credentials
This section stores the Wi-Fi name and password used by the ESP32 to connect to the internet. After connection, the ESP32 creates a webpage accessible through its local IP address.
const char* WIFI_SSID = "staff";
const char* WIFI_PASSWORD = "SRGB2020";
Initializing the Colour Sensor
This creates the TCS34725 sensor object. Controls how long the sensor collects light. Amplifies the sensor sensitivity.
Adafruit_TCS34725 tcs = Adafruit_TCS34725(
TCS34725_INTEGRATIONTIME_50MS,
TCS34725_GAIN_4X
);
Web Server Creation
Creates a web server running on port 80. This allows users to open the ESP32 webpage from a browser.
WebServer server(80);
Colour Palette Database
This block creates a list of predefined colours. Each colour contains: A name, RGB values.
The program compares sensor readings with these colours to find the nearest matching colour name.
const NamedColour PALETTE[] = {
{"Red", 220,20,60},
...
};
String nearestColourName(uint8_t r, uint8_t g, uint8_t b)
Reading Sensor Data
Normalization removes brightness differences so colours remain consistent under different lighting conditions.
RGB readSensor()
Root Webpage Handler
When the browser opens the ESP32 IP address: http://192.168.x.x this function sends the HTML webpage to the user.
void handleRoot()
Setup Function
This runs once when the ESP32 starts. After successful connection, the ESP32 prints its IP address in the Serial Monitor.
void setup()
Main Loop
This continuously runs the project. The web server stays active. Sensor values are updated. RGB values are sent through Serial communication.
void loop()
And this is the final result:
Now let’s create a game scene interface that reflects in real time the colour detected by the colour sensor on the Cube and Sphere objects placed inside the scene. The Directional Light will also change depending on the detected colour.
At first, together with Claude, I wrote the following code, which creates a real-time colour sensor interface between the ESP32-C3 + TCS34725 colour sensor and the Unity game using serial communication.
AI prompt:
“ESP32-C3 + TCS34725 colour sensor interface for Unity game. Reads RGB colour values from the TCS34725 sensor using I2C communication, converts raw data into normalized RGB values, detects the nearest predefined colour name, and sends real-time JSON data through Serial communication to Unity every 100ms. Unity can use this data to control gameplay elements such as lights, puzzles, materials, doors, or interactions based on detected real-world colours.”
The system reads colours from the TCS34725 colour sensor, then converts the sensor values into RGB colours. It also detects the closest human-readable colour name.
Finally, the data is sent to Unity in JSON format through serial communication.
Arduino Code
#include <Wire.h>
#include "Adafruit_TCS34725.h"
// ════════════════════════════════════════════════════════════════════
// SENSOR READ
// ════════════════════════════════════════════════════════════════════
struct ColourReading { uint8_t r, g, b; uint16_t clear; bool ok; };
#define SERIAL_INTERVAL_MS 100 // how often to send data to Unity
// TCS34725: 50ms integration, gain x4
// Adjust gain if colours look wrong:
// Very bright environment → TCS34725_GAIN_1X
// Normal → TCS34725_GAIN_4X
// Dim environment → TCS34725_GAIN_16X
Adafruit_TCS34725 tcs = Adafruit_TCS34725(
TCS34725_INTEGRATIONTIME_50MS,
TCS34725_GAIN_4X
);
// ════════════════════════════════════════════════════════════════════
// COLOUR NAME LOOKUP
// ════════════════════════════════════════════════════════════════════
struct NamedColour { const char* name; uint8_t r, g, b; };
const NamedColour PALETTE[] = {
{"Red", 220, 20, 60},
{"Orange", 255, 140, 0},
{"Yellow", 255, 215, 0},
{"Yellow-Green", 154, 205, 50},
{"Green", 34, 139, 34},
{"Cyan", 0, 206, 209},
{"Blue", 30, 144, 255},
{"Indigo", 75, 0, 130},
{"Violet", 148, 0, 211},
{"Pink", 255, 105, 180},
{"White", 255, 255, 255},
{"Light Grey", 180, 180, 180},
{"Grey", 128, 128, 128},
{"Dark Grey", 64, 64, 64},
{"Black", 0, 0, 0},
{"Brown", 139, 69, 19},
{"Beige", 245, 245, 220},
};
String nearestColourName(uint8_t r, uint8_t g, uint8_t b) {
long best = LONG_MAX;
const char* name = "Unknown";
for (auto& c : PALETTE) {
long dr = (long)r - c.r;
long dg = (long)g - c.g;
long db = (long)b - c.b;
long dist = dr*dr + dg*dg + db*db;
if (dist < best) { best = dist; name = c.name; }
}
return String(name);
}
ColourReading readSensor() {
uint16_t raw_r, raw_g, raw_b, raw_c;
tcs.getRawData(&raw_r, &raw_g, &raw_b, &raw_c);
if (raw_c == 0) return {0, 0, 0, 0, false};
uint8_t r = (uint8_t)constrain((uint32_t)raw_r * 255 / raw_c, 0, 255);
uint8_t g = (uint8_t)constrain((uint32_t)raw_g * 255 / raw_c, 0, 255);
uint8_t b = (uint8_t)constrain((uint32_t)raw_b * 255 / raw_c, 0, 255);
return {r, g, b, raw_c, true};
}
// ════════════════════════════════════════════════════════════════════
// SETUP
// ════════════════════════════════════════════════════════════════════
void setup() {
Serial.begin(115200);
delay(500);
Serial.println("[BOOT] TCS34725 Colour Sensor starting...");
Wire.begin(6, 7); // SDA=GPIO6, SCL=GPIO7
if (!tcs.begin()) {
Serial.println("[ERROR] TCS34725 not found! Check SDA/SCL wiring.");
while (1) {
delay(1000);
Serial.println("[ERROR] Halted — sensor not detected.");
}
}
tcs.setInterrupt(false); // turn on the sensor LED
Serial.println("[OK] TCS34725 ready — sending JSON to Unity...");
}
// ════════════════════════════════════════════════════════════════════
// LOOP
// ════════════════════════════════════════════════════════════════════
void loop() {
static unsigned long lastSend = 0;
if (millis() - lastSend >= SERIAL_INTERVAL_MS) {
lastSend = millis();
ColourReading px = readSensor();
if (px.ok) {
String name = nearestColourName(px.r, px.g, px.b);
// Newline-terminated JSON — Unity reads line by line
Serial.printf(
"{\"r\":%d,\"g\":%d,\"b\":%d,\"name\":\"%s\"}\n",
px.r, px.g, px.b, name.c_str()
);
}
}
}
What This Code Does
Sensor Configuration
The TCS34725 sensor is initialized with a 50ms integration time and 4× gain. Integration time controls how long the sensor collects light — longer means more accuracy but slower readings. Gain amplifies the signal, which is useful in dim environments.
Adafruit_TCS34725 tcs = Adafruit_TCS34725(
TCS34725_INTEGRATIONTIME_50MS,
TCS34725_GAIN_4X
);
Color Name Lookup
There's a hardcoded table of 17 named colors (Red, Orange, Yellow, Green, Blue, White, Black, Brown, etc.), each with its exact RGB values. When the sensor reads a color, the code calculates the Euclidean distance in 3D RGB space between the measured color and every color in the table, then picks whichever named color is mathematically closest. For example, if the sensor reads (240, 30, 50), it will match to "Red".
Read Sensor Data
This function reads raw light values from the sensor (red, green, blue, and a "clear" channel which is unfiltered light). It then normalizes the raw values relative to the clear channel, scaling them to the standard 0–255 range. This normalization compensates for varying ambient light brightness.
ColourReading readSensor() {
uint16_t raw_r, raw_g, raw_b, raw_c;
tcs.getRawData(&raw_r, &raw_g, &raw_b, &raw_c);
if (raw_c == 0) return {0, 0, 0, 0, false};
uint8_t r = (uint8_t)constrain((uint32_t)raw_r * 255 / raw_c, 0, 255);
uint8_t g = (uint8_t)constrain((uint32_t)raw_g * 255 / raw_c, 0, 255);
uint8_t b = (uint8_t)constrain((uint32_t)raw_b * 255 / raw_c, 0, 255);
return {r, g, b, raw_c, true};
}
Setup part
Runs once on boot. It starts serial communication at 115200 baud, initializes the I²C bus on GPIO pins 6 (SDA) and 7 (SCL), and checks if the sensor is physically connected. If the sensor isn't found, it halts and repeatedly prints an error message.
Loop part
Runs continuously. Every 100 milliseconds it reads the sensor and sends a JSON line over serial that looks like:
{"r":220,"g":18,"b":55,"name":"Red"}
Unity reads these lines one by one and can use the color data in real time — for example, to change an object's color in a game scene based on what the physical sensor sees.
Now as an interface, I will create a Unity game, using the same color sensor.
Since last year I had an interesting experience and learned Unity, now this is the best opportunity to use both my new and old knowledge together.
By the way, you can check my previous game on the About Me page.
Since creating a game in Unity involves using many tools, it will be a long process. I will try to explain the whole creation process in detail and also include some short videos to make it clearer.
Now I will start showing the full game creation process, and after that I will show the Arduino coding part.
After downloading Unity from the Official site, we open New Project and choose 3D Built-in Render Pipeline, because it is easier for beginners and compatible with most older Asset Store packages.
Before starting, go to Edit → Project Settings → Player → Other Settings, and from Api Compatibility Level select .NET Framework (instead of .NET Standard), so the project has access to a larger set of libraries. This is important because we will use external plugins (SDKs). Also, in Active Input Handling, select Both, to avoid errors when one part of the project uses the old system and another uses the new one.
Now about the main components of the game.
When creating a new project, the hierarchy usually includes two default objects:
- Directional Light — provides global lighting
- Main Camera — used to “see” and “hear” the game
Since I want to use the color sensor to build a game interface, I will create a small scene with:
- a main player as a Capsule
- some objects like Cube and Sphere
To do this, right-click in Hierarchy → 3D Object and select the needed objects.
Here is the process shown in a short video.
Programmed Unity
Now I need to program these objects.
First, I give the Capsule player functionality.
From the Inspector, click Add Component and add Character Controller.
Then, in the Assets section, I create a new folder called Scripts to store all code files.
I created a script with Claude called PlayerController.cs for the movement of the Capsule.
To attach it, go to Inspector → Add Component → search “PlayerController” and add it.
Prompt:
“Please write a Unity C# script that creates a first-person player controller using the CharacterController component. It should handle player movement, jumping, gravity, and mouse-based camera rotation. The player should be able to move with WASD, jump using the Space key, and rotate the camera only while holding the right mouse button. The cursor should become hidden and locked during camera rotation and visible again when released.”
Here is the code for player movement.
using UnityEngine;
public class PlayerController : MonoBehaviour {
public CharacterController controller;
public Transform cameraTransform;
[Header("Movement Settings")]
public float speed = 6f;
public float gravity = -9.81f;
public float jumpHeight = 2f;
[Header("Look Settings")]
public float lookSpeed = 2f;
public float lookXLimit = 85f;
private Vector3 velocity;
private float rotationX = 0;
void Start() {
controller = GetComponent();
// We don't lock the cursor at start anymore so you can use the right-click freely
Cursor.lockState = CursorLockMode.None;
Cursor.visible = true;
}
void Update() {
// --- 1. MOVEMENT ---
float x = Input.GetAxis("Horizontal");
float z = Input.GetAxis("Vertical");
Vector3 move = transform.right * x + transform.forward * z;
controller.Move(move * speed * Time.deltaTime);
// --- 2. JUMPING ---
if (Input.GetButtonDown("Jump") && controller.isGrounded) {
// Formula for jump velocity: sqrt(height * -2 * gravity)
velocity.y = Mathf.Sqrt(jumpHeight * -2f * gravity);
}
// --- 3. ROTATION (Only when Right Mouse Button is held) ---
if (Input.GetMouseButton(1)) {
// Hide cursor while rotating
Cursor.visible = false;
Cursor.lockState = CursorLockMode.Locked;
// Rotate player body (Left/Right)
transform.Rotate(0, Input.GetAxis("Mouse X") * lookSpeed, 0);
// Rotate camera (Up/Down)
rotationX += -Input.GetAxis("Mouse Y") * lookSpeed;
rotationX = Mathf.Clamp(rotationX, -lookXLimit, lookXLimit);
cameraTransform.localRotation = Quaternion.Euler(rotationX, 0, 0);
} else {
// Show cursor when button is released
Cursor.visible = true;
Cursor.lockState = CursorLockMode.None;
}
// --- 4. GRAVITY ---
if (controller.isGrounded && velocity.y < 0) {
velocity.y = -2f;
}
velocity.y += gravity * Time.deltaTime;
controller.Move(velocity * Time.deltaTime);
}
}
And here is a short video.
Next, I also need to program the Cube and Sphere so they react to the color detected by the sensor.
For that, in the Hierarchy I created an empty GameObject by selecting Right Click → Create Empty. I named it ColourReader, and attached the SerialColourReader script to it, which I will show below.
Then, inside the Inspector, I assigned all the public parameters of the script.
This image shows the setup in detail.
Here is the ColourReader code, which I again generated together with Claude.
Prompt:
“Unity C# script reads real-time colour data from an ESP32-C3 through Serial communication and applies the detected colour to scene objects and directional light. It uses a background thread to safely read JSON colour packets, parses RGB values, smoothly interpolates colours using Lerp, and updates materials and lighting in the Unity scene for interactive colour-based gameplay.”
using System;
using System.Collections.Concurrent;
using System.IO.Ports;
using System.Threading;
using UnityEngine;
public class SerialColourReader : MonoBehaviour
{
[Header("Serial")]
[SerializeField] string portName = "/dev/ttyACM0"; // change to your port
[SerializeField] int baudRate = 115200;
[Header("Scene objects")]
[SerializeField] Renderer[] colourTargets; // drag 2 cubes + 2 spheres here
[SerializeField] Light directionalLight;
[Header("Smoothing")]
[SerializeField] float lerpSpeed = 8f; // 0 = instant, higher = smoother
// ── internals ──────────────────────────────────────────
SerialPort _port;
Thread _readThread;
volatile bool _running;
readonly ConcurrentQueue _queue = new();
Color _targetColor = Color.white;
Color _currentColor = Color.white;
// ── lifecycle ──────────────────────────────────────────
void Start()
{
_port = new SerialPort(portName, baudRate) { ReadTimeout = 500 };
try
{
_port.Open();
_running = true;
_readThread = new Thread(ReadLoop) { IsBackground = true };
_readThread.Start();
Debug.Log("[ColourReader] Serial port opened.");
}
catch (Exception e) { Debug.LogError($"[ColourReader] {e.Message}"); }
}
void OnDestroy()
{
_running = false;
_readThread?.Join(1000);
if (_port?.IsOpen == true) _port.Close();
}
// ── background thread: just enqueue raw lines ──────────
void ReadLoop()
{
while (_running)
{
try
{
string line = _port.ReadLine(); // blocks up to ReadTimeout
if (!string.IsNullOrWhiteSpace(line))
_queue.Enqueue(line);
}
catch (TimeoutException) { /* normal, keep looping */ }
catch (Exception e)
{
Debug.LogWarning($"[ColourReader] {e.Message}");
_running = false;
}
}
}
// ── main thread: parse + apply ─────────────────────────
void Update()
{
// drain queue (at most a few lines per frame)
while (_queue.TryDequeue(out string line))
TryParseColor(line);
// smooth lerp toward target
_currentColor = Color.Lerp(_currentColor, _targetColor, Time.deltaTime * lerpSpeed);
ApplyColor(_currentColor);
}
void TryParseColor(string json)
{
// Arduino sends: {"r":220,"g":20,"b":60,"name":"Red"}
try
{
var data = JsonUtility.FromJson(json);
_targetColor = new Color(data.r / 255f, data.g / 255f, data.b / 255f);
}
catch { /* malformed packet — ignore */ }
}
void ApplyColor(Color c)
{
foreach (var r in colourTargets)
if (r != null) r.material.color = c;
if (directionalLight != null)
directionalLight.color = c;
}
// ── JSON helper ────────────────────────────────────────
[Serializable]
class ColourPacket { public int r, g, b; public string name; }
}
Serial & Threading Setup (Start)
Opens the serial port (same port and baud rate as the Arduino), then launches a background thread running ReadLoop. This is important — reading serial data is slow and blocking, so doing it on a separate thread prevents the game from stuttering or freezing.
Background Thread (ReadLoop)
Sits in a loop constantly waiting for a new line from the Arduino. When a line arrives it drops it into a ConcurrentQueue — a thread-safe list that both the background thread and main thread can access without conflicts. Timeouts are silently ignored since they're expected behavior.
Main Thread (Update)
Every frame it drains the queue, parses any waiting JSON packets into a Unity Color, then smoothly interpolates _currentColor toward _targetColor using Color.Lerp. The lerpSpeed value controls how fast the transition happens — low values feel sluggish, high values feel instant.
Applying the Color (ApplyColor)
Pushes the current smoothed color to every renderer in the colourTargets array and to the directional light. This means the whole scene — objects and lighting — shifts color together, creating an immersive effect.
Cleanup (OnDestroy)
When the scene stops, it safely shuts down the thread and closes the serial port, preventing crashes or port-locking issues.
Here is the result:
Within the group assignment, you can also review Hrach’s research and contributions on our group assignments page.
For the individual assignment, I decided to create a simulation of my final project in Unity, which will also be a good example of an interface.
The game will be a maze where I control a car using my joystick and collect coins within a certain time. But this time I decided to make it more like an open world, as if the game happens in a forest 😄
I asked Claude:
“Give me an Arduino code that connects to a Unity project, lets me move the Player using a joystick, and at the same time receives data back and prints it on an LCD.”
He gave me a base code, and I made several changes to match my exact needs. Honestly, I’m very thankful to Claude because he saved me a lot of time — without that help, I probably wouldn’t finish this in 4 days ❤️
Here is the full code, and below I’ll explain what I changed and why.
// ============================================================
// JoystickCar.ino
// Unity -> Maze Car Controller
// Hardware: XIAO ESP32C3
// GPIO2 (D0) → KY-023 VRX
// GPIO3 (D1) → KY-023 VRY
// GPIO4 (D2) → Battery voltage divider
// GPIO5 (D3) → KY-023 SW
// GPIO6 (D4) → OLED SDA
// GPIO7 (D5) → OLED SCL
// GPIO21 (D6) → Switch Button
// GPIO10 (D10) → LED
// ============================================================
#include <Wire.h>
#include <WiFi.h>
#include <WiFiUdp.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
// ============================================================
// SECTION 1 — WiFi
// ============================================================
const char* WIFI_SSID = "iPhone";
const char* WIFI_PASSWORD = "Manuela27";
// ============================================================
// SECTION 2 — Network
// ============================================================
const char* UNITY_IP = "172.20.10.3";
const int UNITY_PORT = 7777;
const int LISTEN_PORT = 7778;
// ============================================================
// SECTION 3 — Pins
// ============================================================
#define PIN_VRX 2
#define PIN_VRY 3
#define PIN_BATT 4
#define PIN_JOY_SW 5
#define PIN_SDA 6
#define PIN_SCL 7
#define PIN_SWITCH 21
#define PIN_LED 10
// ============================================================
// SECTION 4 — OLED
// ============================================================
#define OLED_W 128
#define OLED_H 64
#define OLED_ADDR 0x3C
Adafruit_SSD1306 oled(OLED_W, OLED_H, &Wire, -1);
// ============================================================
// SECTION 5 — Joystick calibration
// ============================================================
const int ADC_CENTER = 2048;
const int DEADZONE = 150;
// ============================================================
// SECTION 6 — Timing
// ============================================================
const unsigned long SEND_MS = 20;
const unsigned long DISPLAY_MS = 100;
const unsigned long LED_ON_MS = 100;
// ======================== //
// SECTION 7 — Game state //
// ======================== //
enum State { S_CONNECTING, S_IDLE, S_RUNNING, S_PAUSED, S_WIN, S_GAMEOVER };
State state = S_CONNECTING;
int coins = 0;
unsigned long startTime = 0;
unsigned long pausedTime = 0;
unsigned long lastSend = 0;
unsigned long lastDisplay = 0;
unsigned long ledOff = 0;
bool ledOn = false;
bool swLast = HIGH;
unsigned long swDebounce = 0;
String unityTime = "00:00.0";
WiFiUDP udp;
// ============================================================
// SECTION 8 — Helpers
// ============================================================
float normalize(int raw) {
if (abs(raw - ADC_CENTER) < DEADZONE) return 0.0f;
if (raw < ADC_CENTER)
return map(raw, 0, ADC_CENTER - DEADZONE, -1000, 0) / 1000.0f;
return map(raw, ADC_CENTER + DEADZONE, 4095, 0, 1000) / 1000.0f;
}
int battPct() {
float v = analogRead(PIN_BATT) * (3.3f / 4095.0f) * 2.0f;
return constrain((int)((v - 3.0f) / 1.2f * 100.0f), 0, 100);
}
unsigned long elapsed() {
if (state == S_RUNNING)
return pausedTime + (millis() - startTime);
return pausedTime;
}
void toUnity(const char* msg) {
udp.beginPacket(UNITY_IP, UNITY_PORT);
udp.write((uint8_t*)msg, strlen(msg));
udp.endPacket();
}
// ============================================================
// SECTION 9 — OLED screens
// ============================================================
void screenConnecting() {
static uint8_t dots = 0;
oled.clearDisplay();
oled.setTextColor(SSD1306_WHITE);
oled.setTextSize(1);
oled.setCursor(4, 6); oled.print("Connecting to WiFi");
oled.setCursor(4, 20); oled.print(WIFI_SSID);
oled.setCursor(4, 38);
for (int i = 0; i < dots; i++) oled.print(". ");
dots = (dots + 1) % 7;
oled.display();
}
void screenIdle() {
oled.clearDisplay();
oled.setTextColor(SSD1306_WHITE);
oled.setTextSize(1);
oled.setCursor(0, 0);
oled.print("IP: "); oled.print(WiFi.localIP());
oled.drawLine(0, 11, 127, 11, SSD1306_WHITE);
oled.setTextSize(2);
oled.setCursor(22, 18); oled.print("READY!");
oled.setTextSize(1);
oled.setCursor(2, 54); oled.print("SWITCH = start");
oled.display();
}
void screenRunning() {
oled.clearDisplay();
oled.setTextColor(SSD1306_WHITE);
oled.setTextSize(1);
oled.setCursor(0, 0); oled.print("TIME "); oled.print(unityTime); // ← uses Unity time
oled.setCursor(92, 0); oled.print(battPct()); oled.print("%");
oled.drawLine(0, 11, 127, 11, SSD1306_WHITE);
oled.setTextSize(1);
oled.setCursor(0, 17); oled.print("COINS");
char cbuf[8]; snprintf(cbuf, sizeof(cbuf), "%d/10", coins);
oled.setTextSize(3);
int16_t x1, y1; uint16_t w, h;
oled.getTextBounds(cbuf, 0, 0, &x1, &y1, &w, &h);
oled.setCursor((OLED_W - w) / 2, 26);
oled.print(cbuf);
oled.setTextSize(1);
oled.setCursor(2, 56); oled.print("SWITCH = pause");
oled.display();
}
void screenPaused() {
oled.clearDisplay();
oled.setTextColor(SSD1306_WHITE);
oled.setTextSize(2);
oled.setCursor(10, 2); oled.print("PAUSED");
oled.drawLine(0, 18, 127, 18, SSD1306_WHITE);
unsigned long ms = elapsed();
int s = ms / 1000, m = s / 60;
char tbuf[10]; snprintf(tbuf, sizeof(tbuf), "%02d:%02d", m, s % 60);
oled.setTextSize(1);
oled.setCursor(0, 28); oled.print("Coins : "); oled.print(coins); oled.print(" / 10");
oled.setCursor(0, 40); oled.print("Time : "); oled.print(unityTime);
oled.setCursor(2, 54); oled.print("SWITCH = continue");
oled.display();
}
void screenWin() {
oled.clearDisplay();
oled.setTextColor(SSD1306_WHITE);
oled.setTextSize(2);
oled.setCursor(14, 0); oled.print("YOU WIN!");
oled.drawLine(0, 20, 127, 20, SSD1306_WHITE);
unsigned long ms = elapsed();
int s = ms / 1000, m = s / 60;
char tbuf[14]; snprintf(tbuf, sizeof(tbuf), "%02d:%02d.%d", m, s % 60, (int)((ms % 1000) / 100));
oled.setTextSize(1);
oled.setCursor(0, 26); oled.print("All 10 coins!");
oled.setCursor(0, 38); oled.print("Time: "); oled.print(unityTime);
oled.setCursor(2, 54); oled.print("SWITCH = play again");
oled.display();
}
// NEW: Game Over screen
void screenGameOver() {
oled.clearDisplay();
oled.setTextColor(SSD1306_WHITE);
oled.setTextSize(2);
oled.setCursor(4, 0); oled.print("GAME OVER");
oled.drawLine(0, 20, 127, 20, SSD1306_WHITE);
oled.setTextSize(1);
oled.setCursor(0, 26); oled.print("Coins: "); oled.print(coins); oled.print(" / 10");
unsigned long ms = elapsed();
int s = ms / 1000, m = s / 60;
char tbuf[10]; snprintf(tbuf, sizeof(tbuf), "%02d:%02d", m, s % 60);
oled.setCursor(0, 38); oled.print("Time: "); oled.print(unityTime);
oled.setCursor(2, 54); oled.print("SWITCH = try again");
oled.display();
}
// ============================================================
// SECTION 10 — Game control
// ============================================================
void startGame() {
coins = 0;
pausedTime = 0;
startTime = millis();
state = S_RUNNING;
toUnity("CMD:RESET");
Serial.println("[GAME] Started");
}
void pauseGame() {
pausedTime += millis() - startTime;
state = S_PAUSED;
toUnity("CMD:STOP");
Serial.println("[GAME] Paused");
}
void resumeGame() {
startTime = millis();
state = S_RUNNING;
toUnity("CMD:RESUME");
Serial.println("[GAME] Resumed");
}
// ============================================================
// SECTION 11 — setup()
// ============================================================
void setup() {
Serial.begin(115200);
delay(400);
pinMode(PIN_VRX, INPUT);
pinMode(PIN_VRY, INPUT);
pinMode(PIN_BATT, INPUT);
pinMode(PIN_JOY_SW, INPUT_PULLUP);
pinMode(PIN_SWITCH, INPUT_PULLUP);
pinMode(PIN_LED, OUTPUT);
digitalWrite(PIN_LED, LOW);
Wire.begin(PIN_SDA, PIN_SCL);
delay(100);
if (!oled.begin(SSD1306_SWITCHCAPVCC, OLED_ADDR)) {
Serial.println("OLED not found! Check wiring.");
while (true) { digitalWrite(PIN_LED, !digitalRead(PIN_LED)); delay(150); }
}
oled.clearDisplay(); oled.display();
WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
Serial.print("Connecting WiFi");
while (WiFi.status() != WL_CONNECTED) {
screenConnecting(); delay(300); Serial.print(".");
}
delay(500);
Serial.println(); Serial.print("IP: "); Serial.println(WiFi.localIP());
udp.begin(LISTEN_PORT);
state = S_IDLE;
delay(500);
swLast = HIGH;
Serial.println("[READY] Press switch to start game");
}
// ============================================================
// SECTION 12 — loop()
// ============================================================
void loop() {
unsigned long now = millis();
// Serial.println(WiFi.localIP());
// ── A) Switch button (debounced) ──────────────────────────
bool sw = digitalRead(PIN_SWITCH);//digitalRead(PIN_JOY_SW); //digitalRead(PIN_SWITCH);
//Serial.println(sw);
if (sw == LOW && swLast == HIGH) {// && (now - swDebounce) > 50) {
swDebounce = now;
switch (state) {
case S_IDLE: startGame(); break;
case S_RUNNING: pauseGame(); break;
case S_PAUSED: resumeGame(); break;
case S_WIN: startGame(); break;
case S_GAMEOVER: startGame(); break; // try again
default: break;
}
}
swLast = sw;
// ── B) Incoming UDP from Unity ─────────────────────────────
int pkt = udp.parsePacket();
//Serial.println(pkt);
if (pkt > 0) {
// Serial.print("Packet size: ");
Serial.println(pkt);
char buf[32] = {0};
udp.read(buf, sizeof(buf) - 1);
String msg = String(buf); msg.trim();
Serial.print("[UNITY] "); Serial.println(msg);
// "COIN:N" — parse actual count from Unity
if (msg.startsWith("COIN:") && state == S_RUNNING) {
int n = msg.substring(5).toInt();
coins = n;
Serial.print("Coins updated → ");
Serial.println(coins);
//if (n > coins) coins = n; // only update forward
digitalWrite(PIN_LED, HIGH);
ledOn = true;
ledOff = now + LED_ON_MS;
}
if (msg == "WIN") { // && state == S_RUNNING) {
pausedTime += millis() - startTime;
state = S_WIN;
coins = 10;
digitalWrite(PIN_LED, HIGH);
ledOn = true;
ledOff = now + 600;
}
if (msg.startsWith("GAMEOVER")) { //&& state == S_RUNNING) {
pausedTime += millis() - startTime;
state = S_GAMEOVER;
// Three short LED flashes for game over
for (int i = 0; i < 3; i++) {
digitalWrite(PIN_LED, HIGH); delay(80);
digitalWrite(PIN_LED, LOW); delay(80);
}
}
if (msg.startsWith("TIME:")) {
unityTime = msg.substring(5);
}
}
// ── C) LED off timer ──────────────────────────────────────
if (ledOn && now >= ledOff) {
digitalWrite(PIN_LED, LOW);
ledOn = false;
}
// ── D) Send joystick to Unity ─────────────────────────────
if (state == S_RUNNING && (now - lastSend) >= SEND_MS) {
lastSend = now;
float x = normalize(analogRead(PIN_VRX));
float y = -normalize(analogRead(PIN_VRY));
int b = (digitalRead(PIN_JOY_SW) == LOW) ? 1 : 0;
char pkt2[48];
snprintf(pkt2, sizeof(pkt2), "X:%.2f,Y:%.2f,B:%d", x, y, b);
toUnity(pkt2);
}
// ── E) Refresh OLED ───────────────────────────────────────
if (now - lastDisplay >= DISPLAY_MS) {
lastDisplay = now;
switch (state) {
case S_CONNECTING: screenConnecting(); break;
case S_IDLE: screenIdle(); break;
case S_RUNNING: screenRunning(); break;
case S_PAUSED: screenPaused(); break;
case S_WIN: screenWin(); break;
case S_GAMEOVER: screenGameOver(); break;
}
}
}
Overview
This project demonstrates bidirectional communication, where:
- The ESP32 sends input data (joystick movement and button states) to Unity
- Unity sends game state information (coins, time, win/lose states) back to the ESP32
WiFi and Network Communication
The ESP32 connects to a WiFi network and establishes UDP communication with Unity.
Two ports are used:
7777- One for sending joystick data to Unity7778- One for receiving game data from Unity
I chose UDP because it allows fast, lightweight, real-time communication, which is perfect for game interaction.
const char* WIFI_SSID = "iPhone";
const char* WIFI_PASSWORD = "Manuela27";
const char* UNITY_IP = "172.20.10.3";
const int UNITY_PORT = 7777;
const int LISTEN_PORT = 7778;
Communication with Unity
Sending Data to Unity
The ESP32 continuously sends joystick data in this format:
float x = normalize(analogRead(PIN_VRX));
float y = -normalize(analogRead(PIN_VRY));
int b = (digitalRead(PIN_JOY_SW) == LOW) ? 1 : 0;
char pkt2[48];
snprintf(pkt2, sizeof(pkt2), "X:%.2f,Y:%.2f,B:%d", x, y, b);
toUnity(pkt2);
This allows Unity to control the player movement in real time.
Receiving Data from Unity
The ESP32 listens for messages like:
COIN:N→ updates collected coinsTIME:MM:SS→ updates game timeWIN→ triggers win stateGAMEOVER→ triggers game over state
This creates synchronization between the physical controller and the game.
///// Incoming UDP from Unity
int pkt = udp.parsePacket();
......
String msg = String(buf); msg.trim();
..........
// "COIN:N" — parse actual count from Unity
if (msg.startsWith("COIN:") && state == S_RUNNING) {
int n = msg.substring(5).toInt();
coins = n;
digitalWrite(PIN_LED, HIGH);
ledOn = true;
ledOff = now + LED_ON_MS;
}
if (msg == "WIN") { // && state == S_RUNNING) {
pausedTime += millis() - startTime;
state = S_WIN;
coins = 10;
digitalWrite(PIN_LED, HIGH);
ledOn = true;
ledOff = now + 600;
}
if (msg.startsWith("GAMEOVER")) { //&& state == S_RUNNING) {
pausedTime += millis() - startTime;
state = S_GAMEOVER;
// Three short LED flashes for game over
for (int i = 0; i < 3; i++) {
digitalWrite(PIN_LED, HIGH); delay(80);
digitalWrite(PIN_LED, LOW); delay(80);
}
}
if (msg.startsWith("TIME:")) {
unityTime = msg.substring(5);
}
And print the time from Unity in LCD:
oled.setCursor(0, 0); oled.print("TIME "); oled.print(unityTime);
And print the coin count from Unity in LCD:
oled.setCursor(0, 17); oled.print("COINS");
char cbuf[8]; snprintf(cbuf, sizeof(cbuf), "%d/10", coins);
oled.print(cbuf);
Now I moved to configuring and programming the Unity part.
To create a better and larger environment, I go to 3D Object → Terrain, create a terrain, and in Terrain Settings set the size (for example 150x150).
Before painting the terrain, I will show how to create materials in Unity.
From polyhaven.com I download a texture.
Then I place the downloaded files inside my materials folder in Assets.
Inside it, I see files like .jpg, .png, and .exr.
I create a new material (Right Click → Create → Material), give it a name, and in Inspector drag the files into:
- Albedo
- Normal Map
- Height Map
- Occlusion
Here is a short video of that process.
Now using this method, I can create any material I want.
Then I go to Paint Terrain → Paint Texture, click Edit Terrain Layers → Create Layer, select my texture, and paint the terrain.
To create mountains on the edges, I switch to Raise or Lower Terrain and shape the terrain.
Here is a short video.
From the Unity Asset Store, I search for “Car”, choose one, click Add to My Assets, then go to Window → Package Manager → My Assets, download and import it.
After that, the model appears in Assets, and I can drag it into the scene.
I repeat the same for coins and place them on the terrain.
All objects appear in the Hierarchy, where I can also organize them by creating folders.
I created a Coins folder and placed 10 coin objects there.
I renamed the car to RC and placed the Main Camera inside it, so it moves together with the car.
Before programming, I also create a Canvas for a better interface.
In Hierarchy → Right Click → UI (Canvas)
Inside Canvas:
- Panels for events (
UI (Canvas)→Panel) - Text elements for default UI (
UI (Canvas)→Text – TextMeshPro)
Canvas full view, and tehre can select each Text and change direction.
Again, I asked Claude using my already prepared Arduino code:
AI prompt:
“Please program my car object so I can drive it using my joystick and collect coins within a specific time.”
My favorite Claude gave me a working version of the code, and I adjusted and improved it based on my needs.
Finally, we reached the final version together.
Here is the result, and below I will explain what exactly I changed and why:
// ============================================================
// CarController.cs — Updated Version
// Attach this to your Car GameObject in Unity.
//
// Changes from original:
// - HUD (coins + timer) always visible on Canvas
// - Win Panel shown only on win
// - Game Over Panel shown when time runs out with < 10 coins
// - Timer can count DOWN (set countDown = true in Inspector)
// - Sends "COIN:N" instead of "COIN" so ESP32 can show count on OLED
// - Sends "WIN" back when all 10 collected
// - Handles CMD:RESET / CMD:STOP / CMD:RESUME from ESP32
// ============================================================
using System;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
using TMPro;
public class CarController : MonoBehaviour
{
// ============================================================
// Inspector
// ============================================================
[Header("--- Network ---")]
[Tooltip("Port Unity listens on. Must match UNITY_PORT in Arduino (7777)")]
public int listenPort = 7777;
[Tooltip("Port ESP32 listens on for replies. Must match LISTEN_PORT in Arduino (7778)")]
public int esp32Port = 7778;
[Tooltip("IP address shown on the OLED after ESP32 boots")]
public string esp32IP = "172.20.10.3";
[Header("--- Car Movement ---")]
[Tooltip("How fast the car drives forward/backward (units per second)")]
public float driveSpeed = 8f;
[Tooltip("How fast the car rotates left/right (degrees per second)")]
public float turnSpeed = 90f;
[Header("--- Coins ---")]
[Tooltip("Drag all 10 coin GameObjects here")]
public List<GameObject> coins = new List<GameObject>();
[Header("--- Timer ---")]
[Tooltip("Time limit in seconds. If countDown = true, game over when it hits 0.")]
public float gameOverTimeLimit = 60f;
[Tooltip("If true, timer counts DOWN from gameOverTimeLimit. If false, counts UP.")]
public bool countDown = false;
[Header("--- UI ---")]
[Tooltip("Text that shows Coins: 3 / 10 — always visible on HUD")]
public TextMeshProUGUI coinText;
[Tooltip("Text that shows timer 00:12.4 — always visible on HUD")]
public TextMeshProUGUI timerText;
[Tooltip("The WIN panel (Canvas child). Leave inactive in scene.")]
public GameObject winPanel;
[Tooltip("Text inside win panel showing final time")]
public TextMeshProUGUI winTimeText;
[Tooltip("The GAME OVER panel (Canvas child). Leave inactive in scene.")]
public GameObject gameOverPanel;
[Tooltip("Text inside game over panel showing coins collected")]
public TextMeshProUGUI gameOverCoinsText;
[Header("--- Audio ---")]
public AudioSource gameMusic;
public AudioSource winSound;
public AudioSource gameOverSound;
// ============================================================
// Private
// ============================================================
private UdpClient receiveUDP;
private UdpClient sendUDP;
private Thread receiveThread;
private bool threadRunning = false;
private volatile float joyX = 0f;
private volatile float joyY = 0f;
private volatile bool joyB = false;
private readonly Queue<string> cmdQueue = new Queue<string>();
private readonly object cmdLock = new object();
private enum GState { Idle, Running, Paused, Won, GameOver }
private GState gState = GState.Idle;
private int coinsGot = 0;
private float gameTime = 0f;
private bool timerTick = false;
private Rigidbody rb;
// ============================================================
// Start
// ============================================================
void Start()
{
rb = GetComponent<Rigidbody>();
if (rb == null)
Debug.LogError("[Car] Rigidbody missing on " + gameObject.name +
". Add Component → Physics → Rigidbody.");
sendUDP = new UdpClient();
sendUDP.EnableBroadcast = true;
StartReceiveThread();
// Ensure panels are hidden on start
if (winPanel != null) winPanel.SetActive(false);
if (gameOverPanel != null) gameOverPanel.SetActive(false);
UpdateCoinUI();
UpdateTimerUI();
Debug.Log("[Car] Ready. Waiting for ESP32 to send CMD:RESET");
//if (gameMusic != null) gameMusic.Stop(); // don't play until game starts
}
// ============================================================
// Background UDP receive thread
// ============================================================
void StartReceiveThread()
{
threadRunning = true;
receiveUDP = new UdpClient(listenPort);
receiveThread = new Thread(() =>
{
IPEndPoint ep = new IPEndPoint(IPAddress.Any, 0);
while (threadRunning)
{
try
{
byte[] data = receiveUDP.Receive(ref ep);
string msg = Encoding.UTF8.GetString(data).Trim();
if (msg.StartsWith("CMD:"))
lock (cmdLock) { cmdQueue.Enqueue(msg); }
else
ParseJoystick(msg);
}
catch (SocketException) { break; }
catch (Exception e) { Debug.LogWarning("[Car UDP] " + e.Message); }
}
});
receiveThread.IsBackground = true;
receiveThread.Start();
}
// ============================================================
// Parse "X:0.75,Y:-0.50,B:0"
// ============================================================
void ParseJoystick(string msg)
{
foreach (string part in msg.Split(','))
{
string[] kv = part.Split(':');
if (kv.Length != 2) continue;
switch (kv[0].Trim())
{
case "X": float.TryParse(kv[1], System.Globalization.NumberStyles.Float,
System.Globalization.CultureInfo.InvariantCulture, out joyX); break;
case "Y": float.TryParse(kv[1], System.Globalization.NumberStyles.Float,
System.Globalization.CultureInfo.InvariantCulture, out joyY); break;
case "B": joyB = kv[1].Trim() == "1"; break;
}
}
}
// ============================================================
// Update — main thread
// ============================================================
void Update()
{
// Process commands from ESP32
lock (cmdLock)
{
while (cmdQueue.Count > 0)
{
string cmd = cmdQueue.Dequeue();
Debug.Log("[Car] CMD: " + cmd);
if (cmd == "CMD:RESET") ResetGame();
else if (cmd == "CMD:STOP") { timerTick = false; gState = GState.Paused; }
else if (cmd == "CMD:RESUME") { timerTick = true; gState = GState.Running; }
}
}
// Tick timer
if (timerTick && gState == GState.Running)
{
gameTime += Time.deltaTime;
// Game Over check (only when counting down)
//if (countDown && gameTime >= gameOverTimeLimit && coinsGot < 10)
// {
// TriggerGameOver();
// }
if (countDown && GetDisplayTime() <= 0f && coinsGot < 10)
{
TriggerGameOver();
}
}
// Always update HUD — visible regardless of state
UpdateCoinUI();
UpdateTimerUI();
if (gState == GState.Running && timerTick) {
SendToESP32("TIME:" + FormatTime(GetDisplayTime()));
}
}
// ============================================================
// FixedUpdate — physics
// ============================================================
void FixedUpdate()
{
if (gState != GState.Running || rb == null) return;
Vector3 move = transform.forward * joyY * driveSpeed * Time.fixedDeltaTime;
rb.MovePosition(rb.position + move);
if (Mathf.Abs(joyY) > 0.05f)
{
float turn = joyX * turnSpeed * Time.fixedDeltaTime;
Quaternion rot = Quaternion.Euler(0f, turn, 0f);
rb.MoveRotation(rb.rotation * rot);
}
}
// ============================================================
// Coin collection
// ============================================================
void OnTriggerEnter(Collider other)
{
if (gState != GState.Running) return;
if (!other.CompareTag("Coin")) return;
other.gameObject.SetActive(false);
coinsGot++;
Debug.Log("[Car] Coin " + coinsGot + "/10");
// Send coin count so ESP32 OLED can display it
SendToESP32("COIN:" + coinsGot);
UpdateCoinUI();
if (coinsGot >= 10) WinGame();
}
// ============================================================
// Win
// ============================================================
void WinGame()
{
timerTick = false;
gState = GState.Won;
SendToESP32("WIN");
Debug.Log("[Car] YOU WON! Time: " + FormatTime(GetDisplayTime()));
if (winPanel != null)
{
winPanel.SetActive(false); // make sure it's reset first
winPanel.SetActive(true);
if (winTimeText != null)
winTimeText.text = "You Win!! \n Time: " + FormatTime(GetDisplayTime());
}
if (gameOverPanel != null) gameOverPanel.SetActive(false);
if (gameMusic != null) gameMusic.Stop();
if (winSound != null) winSound.Play();
}
// ============================================================
// Game Over — time ran out
// ============================================================
void TriggerGameOver()
{
timerTick = false;
gState = GState.GameOver;
if (rb != null) rb.linearVelocity = Vector3.zero;
SendToESP32("GAMEOVER" + FormatTime(GetDisplayTime()));
Debug.Log("[Car] GAME OVER! Coins: " + coinsGot + "/10");
if (gameOverPanel != null)
{
gameOverPanel.SetActive(true);
if (gameOverCoinsText != null)
gameOverCoinsText.text = "Coins collected: " + coinsGot + " / 10 \n Time's up!";
}
if (winPanel != null) winPanel.SetActive(false);
if (gameMusic != null) gameMusic.Stop();
if (gameOverSound != null) gameOverSound.Play();
}
// ============================================================
// Reset game — called by CMD:RESET from switch button
// ============================================================
void ResetGame()
{
coinsGot = 0;
gameTime = 0f;
timerTick = true;
gState = GState.Running;
foreach (GameObject c in coins)
if (c != null) c.SetActive(true);
if (winPanel != null) winPanel.SetActive(false);
if (gameOverPanel != null) gameOverPanel.SetActive(false);
// Reset car position and rotation
transform.position = new Vector3(250f, transform.position.y, 100f);
transform.rotation = Quaternion.identity;
if (rb != null)
{
rb.linearVelocity = Vector3.zero;
rb.angularVelocity = Vector3.zero;
}
UpdateCoinUI();
Debug.Log("[Car] Game reset — GO!");
if (gameMusic != null)
{
gameMusic.Play();
}
}
// ============================================================
// Send to ESP32
// ============================================================
void SendToESP32(string msg)
{
try
{
byte[] data = Encoding.UTF8.GetBytes(msg);
sendUDP.Send(data, data.Length, esp32IP, esp32Port);
Debug.Log("SEND → " + msg + " to " + esp32IP + ":" + esp32Port);
}
catch (Exception e)
{
Debug.LogError("[Car] Send error: " + e.Message);
}
}
// ============================================================
// UI helpers
// ============================================================
void UpdateCoinUI()
{
if (coinText != null)
coinText.text = "Coins: " + coinsGot + " / 10";
}
void UpdateTimerUI()
{
if (timerText != null)
timerText.text = "Time:" + FormatTime(GetDisplayTime()) + "";
}
// Returns the time value to DISPLAY
// countDown = true → show (limit - elapsed), clamped to 0
// countDown = false → show elapsed
float GetDisplayTime()
{
if (countDown)
return Mathf.Max(0f, gameOverTimeLimit - gameTime);
return gameTime;
}
string FormatTime(float t)
{
int total = (int)t;
int minutes = total / 60;
int seconds = total % 60;
int tenths = (int)((t % 1f) * 10f);
return string.Format("{0:D2}:{1:D2}.{2}", minutes, seconds, tenths);
}
// ============================================================
// Cleanup
// ============================================================
void OnDestroy()
{
threadRunning = false;
receiveUDP?.Close();
sendUDP?.Close();
receiveThread?.Join(500);
}
}
This Unity script controls the player’s car and manages communication with the ESP32 device. It receives joystick input from the hardware, applies movement to the car, and sends game state updates (coins, time, win/lose) back to the ESP32.
The script also manages the user interface (HUD, win screen, game over screen) and implements the core game logic.
Inspector Variables (User Configuration)
Defines network settings for communication with the ESP32. This enables two-way communication between Unity and hardware.
[Header("--- Network ---")]
[Tooltip("Port Unity listens on. Must match UNITY_PORT in Arduino (7777)")]
public int listenPort = 7777;
[Tooltip("Port ESP32 listens on for replies. Must match LISTEN_PORT in Arduino (7778)")]
public int esp32Port = 7778;
[Tooltip("IP address shown on the OLED after ESP32 boots")]
public string esp32IP = "172.20.10.3";
Controls timer behavior. Can switch between count up and count down and enables game-over condition
[Header("--- Timer ---")]
public float gameOverTimeLimit = 60f;
public bool countDown = false;
References UI elements. Separates logic from UI → clean design.
[Header("--- UI ---")]
public TextMeshProUGUI coinText;
public TextMeshProUGUI timerText;
public GameObject winPanel;
public GameObject gameOverPanel;
Handles game sounds.
[Header("--- Audio ---")]
public AudioSource gameMusic;
public AudioSource winSound;
public AudioSource gameOverSound;
Sending Data to ESP32
This updates OLED display on ESP32
COIN:N → coin count
TIME:MM:SS → timer
WIN / GAMEOVER → game result
UI System
HUD always shows: Coins, Timer
Separate panels for: Win, Game Over
Now I will show the gameplay, when the player manages to collect all 10 coins within the given time limit, before moving to the final design. In this case, I set the maximum game time to 30 seconds to make sure I can collect all the coins.
And here is the Game Over case — here I set the time to 15 seconds.
Now comes my favorite part — bringing my own ideas and scenarios to life inside a small environment.
For the terrain, I created some color combinations for the mountains and also smoothed them a bit, so that later I could place decorations more easily and naturally.
Then, by using simple cubes and turning them into thin walls, I arranged them similar to one of my random maze maps from week 2, just like I did in week 7 when building the real game board. I also used black walls to help align everything in straight lines, and later removed them.
I should mention that I made a small change regarding the coins. In the Assets folder, I created a new folder called Prefab, where I combined the coin and lamp objects into one object, including lighting. This is useful when you plan to reuse the same combination multiple times in the game.
Since I decided that the light should turn off after collecting a coin, I used the initial lighting as a way to guide the player toward where coins might be — which is actually an important detail in game design. I created one example of this combination in the Hierarchy, then dragged it into the Prefab folder, and now I can reuse it as a single complete object in the scene. I removed all the old coins and replaced them with this new prefab.
Now about lighting. You can add lighting to any object. For that, I select the object, then go to Light, and from the options I chose Point Light, because it’s more suitable — it can be seen from a distance and doesn’t only light a small area directly underneath.
Now I will briefly show how I placed images on the UI Game Over and Win panels.
First, I wanted to use Grandma Manuela on these panels respectively for victory and defeat moments.
AI prompt:
"Can you Generate image for Canvas Win panel, to set that image in Win panel background, but please dont set there eny button.”
AI prompt:
“Another image for Game Over Panel, and again not needed there eny buttons, and not set there Time, and coin collected count.”
I imported the images into the Assets folder. Then I clicked on the image, and in the Inspector I changed the Texture Type to Sprite (2D and UI).
After that, I right-clicked on the corresponding panel and selected UI (Canvas) → Image. Then I took the created sprite and placed it inside the Source Image field of the Image component in the Inspector.
Then I also wanted to integrate sound effects into my game.
For that, I downloaded the 3 sounds I needed:
- one for the normal
gameplay, - another for the
Game Over panelactivation, - and another one for the
Win panelactivation.
After that, I created 3 GameObjects:
- GameMusic
- GameOverMusic
- WinMusic
Inside each one, I added an Audio Source component and assigned the corresponding sounds inside the Audio Source → Audio Clip field in the Inspector.
And finally, when everything seemed correctly arranged inside my small and beautiful game world, I was ready to test everything.
Here is the gameplay when the player wins.
And here is the case when the player cannot finish within the given time and loses the game.
After that, we can export the Unity package in the following way: Assets → Export Package.
Since this file is almost 600 MB, I can't provide it, but I can post a drive link for download.
This week was one of the most enjoyable weeks for me because I managed to connect many different worlds together at the same time: electronics, networking, programming, game development, UI design, and even sound design.
I learned how to create interfaces not only as simple web pages, but also inside a real Unity game environment. I used ESP32C3, WiFi communication, UDP networking, Unity, Arduino, sensors, LCDs, audio systems, and UI panels together in one workflow. I also better understood how physical hardware and digital applications can communicate with each other in real time.
Another important thing I learned is that creating a game interface is not only about programming mechanics. Small details like lights, sounds, win/lose screens, timers, and even object placement completely change the feeling of the experience.
The funniest part of this week is probably the fact that my handmade joystick was controlling a Unity car inside a forest world while Grandma Manuela was judging the player from the Win and Game Over panels 😆. Also, every time something finally worked after hours of debugging, I felt like I unlocked a secret level in real life.
Honestly, this week made me realize again how much I enjoy combining game development with electronics, because it feels like giving life to my own ideas.
Individual assignment
- Maze Game - Drive Link
AI prompt:
“And Generate image when she fineshed Week 15”

