Skip to content

12. Mechanical Design, Machine Design#

hero shot#

Group work#

For more details, please see here.

https://fabacademy.org/2025/labs/kannai/Weekly_Group_Assignment/week12/

1.individual work#

Concept Development#

Defining requirements#

We wanted to make the work as good looking as possible using the XY arm.

As we considered the concept, the evaluation items were as follows

  • Small number of parts:

  • 5 or less, 10 or less, 20 or less, 30 or less, 30 or more

  • Installability

  • Interactivity/Controllability
  • Generatability

Concept sketching#

We created the concept in writing and had ChatGPT create a sketch.

Concept 1. Hand of God Othello (Position Shuffle)#

Concept 1. Hand of God Othello (Position Shuffle) prompt|Click for details
Concept 1. Hand of God Othello (Position Shuffle) prompt
  Creating the image 
  We are looking to create a piece of artwork using an arm urban frame that moves in xyz as shown in the attached image.
  Please create a sketch design of the product design with the following details.
  - Overview  
    - An arm equipped with an electromagnet is installed above the Othello board, and at random timings it moves Othello pieces on the board to different positions.
  - Required Elements  
    - A detection method to determine the presence or absence of an Othello piece at each position  
    - Othello pieces with embedded magnets  
    - A concrete mechanism to trigger activation at random timings  
    - For both players, a resignation switch that, when pressed, has a 20% chance of activating the shuffle  
    - When the arm shuffle completes or fails to activate, the opponent’s resignation switch lights up
  - More Detailed Components  
    - Frame  
    - Arm that changes the polarity of the electromagnet  
    - Pieces with built-in magnets

  • Overview

    • An arm equipped with an electromagnet is installed above the Othello board, and at random timings it moves Othello pieces on the board to different positions.

  • Required Elements

    • A detection method to determine the presence or absence of an Othello piece at each position
    • Othello pieces with embedded magnets
    • A concrete mechanism to trigger activation at random timings
    • For both players, a resignation switch that, when pressed, has a 20% chance of activating the shuffle
    • When the arm shuffle completes or fails to activate, the opponent’s resignation switch lights up
  • More Detailed Components
    • Frame
    • Arm that changes the polarity of the electromagnet
    • Pieces with built-in magnets

Concept 2. Hand of God Othello (Obstacle Piece Placement)#

Concept 2. Hand of God Othello (Obstacle Piece Placement) prompt|Click for details
Concept 2. Hand of God Othello (Obstacle Piece Placement) prompt
  Creating the image 
  We are looking to create a piece of artwork using an arm urban frame that moves in xyz as shown in the attached image.
  Please create a sketch design of the product design with the following details.
  - Overview  
    - An arm with built-in pieces is installed above the Othello board, and at random timings it places pieces randomly on the board.  
  - Required Elements  
    - A detection method to determine the presence or absence of an Othello piece at each position  
    - Obstacle pieces may be placed even on top of pieces already placed by the players  
    - A concrete mechanism to trigger activation at random timings  
    - For both players, a resignation switch that, when pressed, has a 20% chance of activating the placement  
    - When the arm placement completes or fails to activate, the opponent’s resignation switch lights up  
  - More Detailed Components  
    - Frame  
    - Arm that properly places the pieces  
    - Pieces  
    - Board
  • Overview
    • An arm with built-in pieces is installed above the Othello board, and at random timings it places pieces randomly on the board.
  • Required Elements
    • A detection method to determine the presence or absence of an Othello piece at each position
    • Obstacle pieces may be placed even on top of pieces already placed by the players
    • A concrete mechanism to trigger activation at random timings
    • For both players, a resignation switch that, when pressed, has a 20% chance of activating the placement
    • When the arm placement completes or fails to activate, the opponent’s resignation switch lights up
  • More Detailed Components
    • Frame
    • Arm that properly places the pieces
    • Pieces
    • Board

Concept 3. Random Pythagorean Switch Generator#

Concept 3. Random Pythagorean Switch Generator prompt|Click for details
Concept 3. Random Pythagorean Switch Generator prompt
  Creating the image 
  We are looking to create a piece of artwork using an arm urban frame that moves in xyz as shown in the attached image.
  Please create a sketch design of the product design with the following details.
  - Overview  
    - When activated, parts are arranged inside the frame, and when a ball is dropped from above, a suitable rolling route over those parts down to the bottom is generated randomly.  
  - Required Elements  
    - Random gimmicks  
      - Stairs  
      - Curves  
      - Straight sections  
      - Tunnels  
      - Pillars  
    - An arm that places these gimmicks, after which the preset marble rolls  
    - A connection check at the ends of gimmicks using magnets and electrical conduction?  
    - Gimmicks support each other with pillars, without using any external supports  
    - A goal is installed at the very bottom, and while connecting gimmicks from the bottom up, a route to the top is constructed  
    - The route is generated by an LLM  
      - It must also be possible to create a blueprint manually  
  - More Detailed Components  
    - Frame  
    - Gimmicks  
    - Electromagnetic placement mechanism  
    - Marble  
    - Device to set and start the marble  
    - Switch to execute the mechanism  
    - Prompt for the LLM to generate the route
  • Overview
    • When activated, parts are arranged inside the frame, and when a ball is dropped from above, a suitable rolling route over those parts down to the bottom is generated randomly.
  • Required Elements
    • Random gimmicks
      • Stairs
      • Curves
      • Straight sections
      • Tunnels
      • Pillars
    • An arm that places these gimmicks, after which the preset marble rolls
    • A connection check at the ends of gimmicks using magnets and electrical conduction?
    • Gimmicks support each other with pillars, without using any external supports
    • A goal is installed at the very bottom, and while connecting gimmicks from the bottom up, a route to the top is constructed
    • The route is generated by an LLM
      • It must also be possible to create a blueprint manually
  • More Detailed Components
    • Frame
    • Gimmicks
    • Electromagnetic placement mechanism
    • Marble
    • Device to set and start the marble
    • Switch to execute the mechanism
    • Prompt for the LLM to generate the route

Concept 4. Karesansui (landscape)#

Concept 4. Karesansui (landscape)Prompt|Click for details
Concept 4. Karesansui (landscape) Prompt
  Creating the image 
  We are looking to create a piece of artwork using an arm urban frame that moves in xyz as shown in the attached image.
  Please create a sketch design of the product design with the following details.
  * Overview
    * When spectators input past memories or emotions, an XYZ arm draws patterns resembling terrain or ripples on the surface of sand (or powder) within a frame.  
    * After a certain time, the surface is reconstructed with another spectator’s memories, and the previous memory is overwritten and disappears—allowing one to experience the poetic message “memories shift like landscapes.”

  * Required Elements
    * An interface (smartphone/tablet) for inputting memories, emotions, keywords, or photos  
    * The XYZ arm unit (motor, drive mechanism, control board)  
    * A tray filled with sand or powder (frame structure)  
    * Pattern generation algorithm and prompts using an LLM  
    * Software to control the pattern drawing  
    * Overhead lighting to emphasize shadows and textures  
    * Ambient sound playback system (wind, water, voices, etc.)  
    * Mechanism for saving and overwriting memory data  
    * Execution and emergency stop switches

  * More Detailed Components
    * LLM prompts  
    * Interface  
    * Safety mechanism

  • Overview

    • When spectators input past memories or emotions, an XYZ arm draws patterns resembling terrain or ripples on the surface of sand (or powder) within a frame.
    • After a certain time, the surface is reconstructed with another spectator’s memories, and the previous memory is overwritten and disappears—allowing one to experience the poetic message “memories shift like landscapes.”

  • Required Elements

    • An interface (smartphone/tablet) for inputting memories, emotions, keywords, or photos
    • The XYZ arm unit (motor, drive mechanism, control board)
    • A tray filled with sand or powder (frame structure)
    • Pattern generation algorithm and prompts using an LLM
    • Software to control the pattern drawing
    • Overhead lighting to emphasize shadows and textures
    • Ambient sound playback system (wind, water, voices, etc.)
    • Mechanism for saving and overwriting memory data
    • Execution and emergency stop switches

  • More Detailed Components

    • LLM prompts
    • Interface
    • Safety mechanism

Conclusion#

After discussion among team members, a modified version of Concept 4. Karesansui (landscape) was adopted.

Project Management Activities#

Background#

With only two weeks to complete the project, efficient execution was critical. Especially since we had set ambitious goals, we wanted to minimize the amount of time someone would have to struggle or interrupt work. So, with my experience in project management, I took the lead. And I had just recently earned my PMP certification!

Actions taken.#

Identifying Constraints#

First, constraints were identified and analyzed. * A limited timeframe of two weeks * The main work was restricted to only two days—Saturday and Sunday—each weekend. * There was no prior experience collaborating on a joint project with these team members. * Furthermore, their practical agile experience and skill levels were unknown. * Evaluation of team members’ skills * Mr. Ito * Strong in project management * Mr. Tokuyama * Strong in all aspects of programming * Mr. Hayashi * Strong in 3D modeling and 3D printing

Framework Creation and Alignment#

  1. Project Charter
    Created a document to clearly define the project’s objectives, scope, and responsibilities, and to obtain approval from stakeholders.
  2. Team Charter
    Created a document summarizing the vision and operating methods on which the team would rely.
  3. Designation of Decision Makers
    Since we needed to move things forward in a short time, we decided to grant each person wide discretion. Therefore, each person’s scope of responsibility was clearly defined along with the requirements, and any changes to specifications within each domain were approved at the person’s discretion.
  4. Meeting Template
    Created a meeting template to facilitate smooth information sharing and task progress management. Meetings were held four times: morning stand-up, before noon, at 4:00 PM, and end-of-day wrap-up. This aimed to add structure to the work and quickly catch up on tasks causing difficulties.
  5. module separation of manufactured items and creation of flowcharts In order to ensure that everyone’s understanding was aligned, we divided the manufactured product into modules and created a flowchart of how they functioned.
  6. Onboarding Preparation
    Held a kickoff meeting on a weekday so that no time would be spent on information sharing on the workday itself.
    To maintain consistency in explanations at the kickoff meeting, an explanatory video for the group work document was created and shared in advance. The video is 20 minutes long, so it is not uploaded here.
  7. Online Kickoff MTG
    Since team members were each working different jobs and had varying availability, online kickoff MTGs were held at each person’s convenience to discuss each person’s responsibilities and task breakdowns to ensure smooth progress on the workday.
  8. timely scrum MTG As mentioned above, since the time frame was only two weeks (four days available for work), we wanted to reduce the amount of time hands were stopped as much as possible. Therefore, I utilized the meeting template to lead and facilitate the MTG so that each member could efficiently share progress and declare the next task.

LLM Prompt Techniques#

This was created following the well-known 26 principles for prompting.

landscape creation prompt#

landscape creation prompt
text: `# ROLE
You are a master designer of karesansui (枯山水) - traditional Japanese Zen rock gardens - with expertise in generating SVG code.

# TASK
Transform the provided image into SVG code that represents the sand patterns (砂紋, samon) of a karesansui garden. 

# TECHNICAL REQUIREMENTS
1. **Simplification:** Extract and simplify the key features and contours from the image while maintaining its essential character.

2. **Continuity:** Create continuous, unbroken lines that could be drawn by a mechanical rake or robotic arm in a single motion where possible.

3. **Minimal Intersections:** Design patterns with minimal crossing lines, as is traditional in karesansui gardens.

4. **Traditional Aesthetics:** Incorporate traditional karesansui design elements:
   - Parallel ripple patterns (波紋, hamon) to represent water
   - Straight lines to create clarity and structure
   - Gentle curves to represent natural flow
   - Asymmetrical balance (非対称の調和)

5. **Technical Specifications:**
   - Optimize for A4 portrait size (210mm × 297mm)
   - Use a clear coordinate system with viewBox="0 0 210 297"
   - Maintain proper scaling and proportions
   - Ensure all lines have appropriate stroke width (typically 0.5-1.0px)
   - Use only stroke paths (no fills)

# OUTPUT FORMAT
You must output ONLY the raw SVG code with no additional text, explanations, comments, markdown formatting, or code block markers. 

The SVG code must:
- Start with the opening `< svg > ` tag including xmlns attribute and viewBox specification
- End with the closing `</svg> ` tag
- Be properly formatted and valid
- Be immediately renderable in a web browser or vector editing software without any modifications

# IMPORTANT
Do not include any explanatory text, introductions, or conclusions.
Do not wrap the SVG code in backticks or markdown code blocks.
Return only the SVG code itself, nothing else.

Example of expected output format:
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 210 297" width="210mm" height="297mm">
  <path d="M10,10 C20,20 30,10 40,10" stroke="black" stroke-width="0.5" fill="none"/>
  <!-- Additional paths as needed -->
</svg>`

Haiku Writing Prompt#

Haiku Writing Prompt
`# Role
You are a master haiku poet with deep knowledge of Japanese aesthetics and Zen philosophy.

# Task
1. Carefully observe the provided image of a Zen rock garden (karesansui).
2. Create a Japanese haiku that captures the essence, visual elements, and atmosphere of the garden.
3. Translate the haiku into English following the translation guidelines below.

# Japanese Haiku Requirements
- Follow the traditional 5-7-5 syllable structure (5-7-5音).
- Include an appropriate seasonal reference (kigo/季語).
- Reflect visual elements of the rock garden (line patterns, sand textures, overall composition).
- Embody Zen spirit and Japanese aesthetics (wabi-sabi/わび・さび, yugen/幽玄).

# English Translation Guidelines
1. **Syllable Structure:** 
   - No need to maintain the strict 5-7-5 syllable pattern in English.
   - Recommended options:
     * 3-5-3 syllables (11 total)
     * 2-3-2 syllables (7 total)
     * Free form but maintain a three-line structure

2. **Seasonal References:**
   - Preserve the seasonal word (kigo) in a culturally appropriate way.
   - Example: "名月" → "harvest moon" rather than literal "famous moon"

3. **Cutting Words:**
   - Transform Japanese cutting words ("や"/"かな", etc.) into appropriate English punctuation (commas, dashes, exclamation marks) or line breaks.

4. **Translation Priorities:**
   - Meaning: Clearly convey the original intent
   - Conciseness: Avoid unnecessary words
   - Imagery: Use vivid, evocative language
   - Rhythm: Maintain natural flow in English

5. **Core Principle:**
   - Translation is a cultural and poetic bridge, not word-for-word conversion.
   - The spirit and beauty of the haiku should resonate with English readers as poetry.

# Output Format
You must strictly adhere to the following format and provide nothing else:

俳句: [JAPANESE HAIKU WITH 5-7-5 STRUCTURE]
英訳: [ENGLISH TRANSLATION FOLLOWING GUIDELINES]

Do not include any additional explanations, comments, or text outside this exact structure. Do not number the entries or add any formatting beyond what is specified.

# Examples
The following are examples of the required format:

俳句: 古池や 蛙飛び込む 水の音
英訳: An old silent pond... A frog jumps into the pond, splash! Silence again.

俳句: 桃の花 散りゆく様は 桜かな
英訳: The peach blossoms As they scatter Look like cherry blossoms.

俳句: 月一輪 星無数空 緑なり
英訳: Around the lone moon countless stars the sky now green.`

Camera Embedded Programming#

I created a program to photograph a karesansui (Zen rock garden) and upload the captured image to the cloud via Wi-Fi.

Environment Used:#

  • PlatformIO
    PlatformIO is an open-source IoT development ecosystem that supports multiple embedded boards and frameworks, providing cross-platform build, library management, debugging, testing, and IDE integration. It specializes in programming in C++.
    For this project, I used PlatformIO to write the code. Therefore, I will also explain how to prepare and install PlatformIO.
  • Seeed Studio XIAO ESP32S3 Sense
    The Seeed Studio XIAO ESP32S3 features a dual-core ESP32-S3 chip with both Wi-Fi and BLE connectivity, and supports battery charging. It includes a built-in camera sensor and digital microphone, 8 MB of PSRAM, 8 MB of flash memory, and an external SD card slot.

1. PlatformIO Setup#

  1. In VS Code (or a VS Code fork), open the Extensions view, search for “PlatformIO,” and install it.

2. Creating the Project Files#

  1. Open PlatformIO Home and select New Project.

  2. Enter the project name, select the board and language, then click Finish. For this project, I chose the XIAO ESP32S3 board and the Arduino framework.

  3. The environment is now ready. In the video below, I create a new sample project so you can see how it behaves.

3. Configuring the platformio.ini File#

The Arduino IDE uses a “one sketch = one project” model, automatically retaining board selection and build settings via GUI and internal settings files (~/.arduinoIDE/settings.json / arduino-cli.yaml), so you don’t need a per-project config file.
However, PlatformIO declaratively manages multiple platforms (MCUs), frameworks, build options, and library dependencies on a per-project basis to reproduce the same source across different environments. Therefore, you need a platformio.ini file:

ini files
[env:seeed_xiao_esp32s3]
platform = espressif32
board = seeed_xiao_esp32s3
framework = arduino
upload_speed = 115200
monitor_speed = 115200
upload_port = /your/devices/ports
board_build.partitions = huge_app.csv
build_flags =
  -DARDUINO_USB_MODE=1
  -DARDUINO_USB_CDC_ON_BOOT=1
lib_deps =
  WiFi
  HTTPClient
  base64
  bblanchon/ArduinoJson @ ^6.21.2

4. Task Breakdown and Sequencing#

To achieve a complex goal, planning is necessary.
First, I defined the goal of this work and then broke down the tasks by working backwards from it.

Next, I considered the sequence of tasks. The important thing is to maintain a working state regardless of when each task is completed.

5. Serial Monitor Test#

It’s advisable to test whether it displays correctly on the serial monitor in PlatformIO as well.

!!!
- In PlatformIO, you must explicitly include <Arduino.h>.
- You can display the serial monitor by selecting the plug icon at the bottom of the screen.

Serial monitor test
#include <Arduino.h>

void setup()
{
  Serial.begin(115200);
  Serial.println("Hello from ESP32");
}

void loop()
{
  Serial.println("Looping...");
  delay(1000);
}

6. Communicating over Wi-Fi#

To connect via Wi-Fi, you need to configure the config file.
Create the following config.h file in the include folder.

!!!
- Wi-Fi credentials are personal information. Please configure your .gitignore in advance so you don’t upload them to GitLab or similar.
- You can check the port information by running pio device list in the terminal.

wifi config setting#

config.h
#ifndef CONFIG_H
#define CONFIG_H

// ====== Wi-Fi setting ======
#define WIFI_SSID     "YourWiFiSSID"
#define WIFI_PASSWORD "YourWiFiPassword"

// ====== Web API setting ======
#define API_URL       "https://machinical-memory-landscapes.cloud/api/upload-image"
#define API_HEADER_CONTENT_TYPE "multipart/form-data"
#define API_IMAGE_FIELD_NAME    "image"
#define API_FILENAME            "image.jpg"

#endif

!!!
- You can build & upload to the microcontroller by running pio run -t upload in the terminal.

Wifi connecting program#

main.cpp
#include <WiFi.h>             // ESP32 Wi-Fi library
#include <HTTPClient.h>       // HTTP client for ESP32
#include <WiFiClientSecure.h> // HTTPS support for ESP32
#include "config.h"

// Wi-Fi connect
void connectWiFi()
{
    WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
    Serial.print("Connecting to WiFi");
    while (WiFi.status() != WL_CONNECTED)
    {
        delay(3000);
        Serial.print(".");
    }
    Serial.println("\nConnected!");
}

// call connecting Wi-Fi
void setup()
{
    Serial.begin(115200);
    delay(3000);
    connectWiFi();
}

Wifi connecting test

7.Camera Shooting#

#include <WiFi.h>
#include <HTTPClient.h>
#include <WiFiClientSecure.h>
#include <ArduinoJson.h>
#include "esp_camera.h"
#include "config.h"

void setupCamera()
{
  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = 15;
  config.pin_d1 = 17;
  config.pin_d2 = 18;
  config.pin_d3 = 16;
  config.pin_d4 = 14;
  config.pin_d5 = 12;
  config.pin_d6 = 11;
  config.pin_d7 = 48;
  config.pin_xclk = 10;
  config.pin_pclk = 13;
  config.pin_vsync = 38;
  config.pin_href = 47;
  config.pin_sccb_sda = 40;
  config.pin_sccb_scl = 39;
  config.pin_pwdn = -1;
  config.pin_reset = -1;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  config.frame_size = FRAMESIZE_QVGA;
  config.jpeg_quality = 10;
  config.fb_count = 1;

  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK)
  {
    Serial.printf("Camera init failed: 0x%x\n", err);
    while (true)
      delay(1000);
  }
}

void setup()
{
  Serial.begin(115200);
  delay(3000);
  Serial.println("Booting...");
  setupCamera();
  Serial.println("Camera ready!");
}

void loop()
{
  digitalWrite(LED_BUILTIN, LOW);
  delay(50);
  camera_fb_t *fb = esp_camera_fb_get();
  if (!fb)
  {
    Serial.println("Capture failed");
  }
  else
  {
    Serial.println("Image captured successfully.");
    Serial.print("Image size: ");
    Serial.print(fb->len);
    Serial.println(" bytes");
  }
  esp_camera_fb_return(fb);
  delay(3000);
}
The following video shows that it is possible to obtain the size of the image taken.

8.Send images captured by camera to webapi via wifi#

Since it is more convenient to test whether the webapi can actually be transmitted by taking pictures with a camera and actually POSTing them over Wi-Fi, we conducted an integrated operation test instead of testing the webapi as a stand-alone unit.

Send images captured by camera to webapi via wifi
#include <WiFi.h>             // ESP32 Wi-Fi library
#include <HTTPClient.h>       // HTTP client for ESP32
#include <WiFiClientSecure.h> // HTTPS support for ESP32
#include <ArduinoJson.h>      // JSON handling library

#include "esp_camera.h" // Camera support for ESP32
#include "config.h"     // Contains WIFI_SSID, WIFI_PASSWORD, API_URL, etc.

void setupCamera();              // Declare function to setup the camera
void postImage(camera_fb_t *fb); // Declare function to send image to server
void connectWiFi();              // Declare function to connect WiFi

void connectWiFi()
{
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  Serial.print("Connecting to WiFi");
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(3000);
    Serial.print(".");
  }
  Serial.println("\nConnected!");
}

void setupCamera()
{
  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = 15;
  config.pin_d1 = 17;
  config.pin_d2 = 18;
  config.pin_d3 = 16;
  config.pin_d4 = 14;
  config.pin_d5 = 12;
  config.pin_d6 = 11;
  config.pin_d7 = 48;
  config.pin_xclk = 10;
  config.pin_pclk = 13;
  config.pin_vsync = 38;
  config.pin_href = 47;
  config.pin_sccb_sda = 40;
  config.pin_sccb_scl = 39;
  config.pin_pwdn = -1;
  config.pin_reset = -1;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  config.frame_size = FRAMESIZE_QVGA;
  config.jpeg_quality = 10;
  config.fb_count = 1;

  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK)
  {
    Serial.printf("Camera init failed: 0x%x\n", err);
    while (true)
      delay(1000);
  }
}

void postImage(camera_fb_t *fb)
{
  digitalWrite(LED_BUILTIN, LOW);
  Serial.println("Starting postImage function");
  WiFiClientSecure client;
  client.setInsecure();

  if (!client.connect("machinical-memory-landscapes.cloud", 443))
  {
    Serial.println("Connection failed!");
    return;
  }

  String boundary = "----WebKitFormBoundary7MA4YWxkTrZu0gW";
  String contentType = "multipart/form-data; boundary=" + boundary;
  String start = "--" + boundary + "\r\n" + "Content-Disposition: form-data; name=\"image\"; filename=\"upload.jpg\"\r\n" + "Content-Type: image/jpeg\r\n\r\n";
  String end = "\r\n--" + boundary + "--\r\n";

  int contentLength = start.length() + fb->len + end.length();

  client.println("POST /api/karesansui-generator HTTP/1.1");
  client.println("Host: machinical-memory-landscapes.cloud");
  client.println("User-Agent: ESP32");
  client.println("Content-Type: " + contentType);
  client.println("Content-Length: " + String(contentLength));
  client.println("Connection: close");
  client.println();

  client.print(start);
  client.write(fb->buf, fb->len);
  client.print(end);

  while (client.connected())
  {
    String line = client.readStringUntil('\n');
    Serial.println(line);
  }

  client.stop();
  Serial.println("Image POST completed");
}

void setup()
{
  Serial.begin(115200);
  delay(3000);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, HIGH);

  connectWiFi();
  setupCamera();
  Serial.println("Setup complete");
}

void loop()
{

  digitalWrite(LED_BUILTIN, LOW);
  for (int i = 0; i < 3; i++)
  {
    digitalWrite(LED_BUILTIN, HIGH);
    delay(100);
    digitalWrite(LED_BUILTIN, LOW);
    delay(200);
  }
  camera_fb_t *fb = esp_camera_fb_get();
  if (!fb)
  {
    Serial.println("Capture failed");
    digitalWrite(LED_BUILTIN, LOW);
    delay(5000);
    return;
  }
  digitalWrite(LED_BUILTIN, LOW);
  Serial.printf("Image captured: %zu bytes\n", fb->len);
  postImage(fb);
  esp_camera_fb_return(fb);
  delay(15000);
}

9. When the trigger is pressed, the internal LED lights up.#

limitswitch_on
#include <Arduino.h>

const int limitSwitchPin = 6; 
const int ledPin = 21; 

void setup()
{
  pinMode(limitSwitchPin, INPUT_PULLUP);
  pinMode(ledPin, OUTPUT);            
  Serial.begin(115200);              
  Serial.println("System startup: monitoring limit switches...");
  digitalWrite(LED_BUILTIN, LOW); 
}

void loop()
{
  int switchState = digitalRead(limitSwitchPin);

  if (switchState == LOW)
  {                            
    digitalWrite(ledPin, HIGH);
  }
  else
  {
    digitalWrite(ledPin, LOW);
  }

  delay(200); 
}

10. When the trigger is pressed, the image captured by the camera is sent to webapi via wifi.#

Integrate all previous content and verify that it works. If it works, then the minimum functionality can be provided.

When the trigger is pressed, the image captured by the camera is sent to webapi via wifi.
#include <WiFi.h>
#include <HTTPClient.h>
#include <WiFiClientSecure.h>
#include <ArduinoJson.h>
#include "esp_camera.h"
#include "config.h"

#define LED_BUILTIN 21
const int limitSwitchPin = 6; // D6 = GPIO6

void setupCamera();
void postImage(camera_fb_t *fb);
void connectWiFi();

void connectWiFi()
{
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  Serial.print("Connecting to WiFi");
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(3000);
    Serial.print(".");
  }
  Serial.println("\nConnected!");
}

void setupCamera()
{
  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = 15;
  config.pin_d1 = 17;
  config.pin_d2 = 18;
  config.pin_d3 = 16;
  config.pin_d4 = 14;
  config.pin_d5 = 12;
  config.pin_d6 = 11;
  config.pin_d7 = 48;
  config.pin_xclk = 10;
  config.pin_pclk = 13;
  config.pin_vsync = 38;
  config.pin_href = 47;
  config.pin_sccb_sda = 40;
  config.pin_sccb_scl = 39;
  config.pin_pwdn = -1;
  config.pin_reset = -1;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  config.frame_size = FRAMESIZE_QVGA;
  config.jpeg_quality = 10;
  config.fb_count = 1;

  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK)
  {
    Serial.printf("Camera init failed: 0x%x\n", err);
    while (true)
      delay(1000);
  }
}

void postImage(camera_fb_t *fb)
{
  digitalWrite(LED_BUILTIN, LOW);
  Serial.println("Starting postImage function");
  WiFiClientSecure client;
  client.setInsecure();

  if (!client.connect("machinical-memory-landscapes.cloud", 443))
  {
    Serial.println("Connection failed!");
    return;
  }

  String boundary = "----WebKitFormBoundary7MA4YWxkTrZu0gW";
  String contentType = "multipart/form-data; boundary=" + boundary;
  String start = "--" + boundary + "\r\n" + "Content-Disposition: form-data; name=\"image\"; filename=\"upload.jpg\"\r\n" + "Content-Type: image/jpeg\r\n\r\n";
  String end = "\r\n--" + boundary + "--\r\n";

  int contentLength = start.length() + fb->len + end.length();

  client.println("POST /api/karesansui-generator HTTP/1.1");
  client.println("Host: machinical-memory-landscapes.cloud");
  client.println("User-Agent: ESP32");
  client.println("Content-Type: " + contentType);
  client.println("Content-Length: " + String(contentLength));
  client.println("Connection: close");
  client.println();

  client.print(start);
  client.write(fb->buf, fb->len);
  client.print(end);

  while (client.connected() || client.available())
  {
    String line = client.readStringUntil('\n');
    Serial.println(line);
  }

  client.stop();
  Serial.println("Image POST completed");
}

void setup()
{
  Serial.begin(115200);
  delay(3000);
  pinMode(limitSwitchPin, INPUT_PULLUP);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, HIGH);

  connectWiFi();
  setupCamera();
  Serial.println("Setup complete");
}

void loop()
{
  int switchState = digitalRead(limitSwitchPin);

  if (switchState == LOW)
  {
    digitalWrite(LED_BUILTIN, LOW);
    for (int i = 0; i < 3; i++)
    {
      digitalWrite(LED_BUILTIN, HIGH);
      delay(100);
      digitalWrite(LED_BUILTIN, LOW);
      delay(200);
    }

    camera_fb_t *fb = esp_camera_fb_get();
    if (!fb)
    {
      Serial.println("Capture failed");
      digitalWrite(LED_BUILTIN, LOW);
      delay(5000);
      return;
    }

    Serial.printf("Image captured: %zu bytes\n", fb->len);
    postImage(fb);
    esp_camera_fb_return(fb);
    delay(15000);
  }

  delay(200);
}

11. Improve camera image quality#

The effort will now be to improve functionality. As the quality of the camera improves, the image size will also increase, which will naturally affect the transmission speed. In addition, if an external storage medium is not used, the microcontroller’s RAM buffer capacity will be limited.

The image quality can be improved by changing the following variables.

config.frame_size

Value Resolution (px) Raw Buffer Size (bytes)
QQVGA 160 × 120 160×120×2=38400
QCIF 176 × 144 176×144×2=50688
HQVGA 240 × 176 240×176×2=84480
QVGA 320 × 240 320×240×2=153600
CIF 400 × 296 400×296×2=236800
HVGA 480 × 320 480×320×2=307200
VGA 640 × 480 640×480×2=614400
SVGA 800 × 600 800×600×2=960000
XGA 1024 × 768 1024×768×2=1572864
HD 1280 × 720 1280×720×2=1843200
SXGA 1280 × 1024 1280×1024×2=2621440
UXGA 1600 × 1200 1600×1200×2=3840000

Initially, we were thinking of verifying whether it would be possible to transmit images without using an external storage medium when image quality was improved, but since saving to an SD card is not that difficult, we decided to do that and then conduct an actual transmission test.

12. Save captured images to SD card#

In order to improve image quality, data needs to be saved to an external medium; code needs to be created to allow saving using an SD card and tested to see if it can actually be saved. This was also integrated and tested since it is used in a series of steps such as “camera shot > photo saved to SD card > POST to Web API”.

Save captured images to SD card
#include <WiFi.h>
#include <HTTPClient.h>
#include <WiFiClientSecure.h>
#include <ArduinoJson.h>
#include "esp_camera.h"
#include "config.h"
#include "FS.h"
#include "SD.h"
#include "SPI.h"
#include <SD.h>

// #define LED_BUILTIN 21  // commented out to avoid redefinition warning
const int limitSwitchPin = 6; // D6 = GPIO6

void setupCamera();
void postImageFromSD(const String &path);
void connectWiFi();

void connectWiFi()
{
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  Serial.print("Connecting to WiFi");
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(1000);
    Serial.print(".");
  }
  Serial.println("\nConnected!");
}

void setupCamera()
{
  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = 15;
  config.pin_d1 = 17;
  config.pin_d2 = 18;
  config.pin_d3 = 16;
  config.pin_d4 = 14;
  config.pin_d5 = 12;
  config.pin_d6 = 11;
  config.pin_d7 = 48;
  config.pin_xclk = 10;
  config.pin_pclk = 13;
  config.pin_vsync = 38;
  config.pin_href = 47;
  config.pin_sccb_sda = 40;
  config.pin_sccb_scl = 39;
  config.pin_pwdn = -1;
  config.pin_reset = -1;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  config.frame_size = FRAMESIZE_QVGA; // High quality (1600x1200)
  config.jpeg_quality = 10;           // High quality JPEG output
  config.fb_count = 1;

  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK)
  {
    Serial.printf("Camera init failed: 0x%x\n", err);
    while (true)
      delay(1000);
  }
}

void postImageFromSD(const String &path)
{
  WiFiClientSecure client;
  client.setInsecure();

  Serial.println("Starting POST...");

  if (!client.connect("machinical-memory-landscapes.cloud", 443))
  {
    Serial.println("Connection failed!");
    return;
  }

  File file = SD.open(path.c_str(), FILE_READ);
  if (!file)
  {
    Serial.println("Failed to open file for reading");
    return;
  }

  String boundary = "----WebKitFormBoundary7MA4YWxkTrZu0gW";
  String contentType = "multipart/form-data; boundary=" + boundary;
  String start = "--" + boundary + "\r\n"
                                   "Content-Disposition: form-data; name=\"image\"; filename=\"upload.jpg\"\r\n"
                                   "Content-Type: image/jpeg\r\n\r\n";
  String end = "\r\n--" + boundary + "--\r\n";

  int contentLength = start.length() + file.size() + end.length();

  client.println("POST /api/upload-image HTTP/1.1");
  client.println("Host: machinical-memory-landscapes.cloud");
  client.println("User-Agent: ESP32");
  client.println("Content-Type: " + contentType);
  client.println("Content-Length: " + String(contentLength));
  client.println("Connection: close");
  client.println();

  Serial.println("Posting image data...");
  client.print(start);
  while (file.available())
  {
    client.write(file.read());
  }
  client.print(end);
  file.close();

  while (client.connected() || client.available())
  {
    String line = client.readStringUntil('\n');
    Serial.println(line);
  }

  client.stop();
  Serial.println("Image POST completed");
}

void setup()
{
  Serial.begin(115200);
  delay(3000);
  Serial.println("Serial monitor initialized. Starting setup...");
  pinMode(limitSwitchPin, INPUT_PULLUP);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, HIGH);

  connectWiFi();
  setupCamera();
  SPI.begin(7, 8, 9, 21); // SCK, MISO, MOSI, CS  // CLK, CMD, D0（仮定ピン）
  if (!SD.begin(21))
  {
    Serial.println("SD_MMC mount failed");
    while (true)
      delay(1000);
  }

  uint8_t cardType = SD.cardType();
  if (cardType == CARD_NONE)
  {
    Serial.println("Error: No SD card attached or card not recognized.");
    while (true)
      delay(1000);
  }
  else
  {
    Serial.println("SD card detected and ready.");
  }
  Serial.print("SD Card Type: ");
  if (cardType == CARD_MMC)
  {
    Serial.println("MMC");
  }
  else if (cardType == CARD_SD)
  {
    Serial.println("SDSC");
  }
  else if (cardType == CARD_SDHC)
  {
    Serial.println("SDHC");
  }
  else
  {
    Serial.println("Unknown");
  }

  uint64_t cardSize = SD.cardSize() / (1024 * 1024);
  Serial.print("SD Card Size: ");
  Serial.print(cardSize);
  Serial.println("MB");

  Serial.println("SD Card initialized.");

  Serial.println("Setup complete");
}

void loop()
{
  // Serial.println("Loop running...");
  int switchState = digitalRead(limitSwitchPin);

  if (switchState == LOW)
  {
    digitalWrite(LED_BUILTIN, LOW);

    camera_fb_t *fb = esp_camera_fb_get();
    if (!fb)
    {
      Serial.println("Capture failed");
      delay(100);
      return;
    }
    Serial.println("Image captured successfully.");

    Serial.println("Saving image to SD card...");
    String filename = "/photo_" + String(millis()) + ".jpg";
    File file = SD.open(filename.c_str(), FILE_WRITE);
    if (!file)
    {
      Serial.println("Failed to open file for writing");
      esp_camera_fb_return(fb);
      return;
    }

    file.write(fb->buf, fb->len);
    file.close();
    Serial.println("Image saved to SD card: " + filename);
    esp_camera_fb_return(fb);

    postImageFromSD(filename);
  }

  delay(200);
}

13. Save camera images to SD card and send those images via Wifi with Webapi#

RAM buffer capacity of the microcontroller, which was one of the constraints for improving image quality, was solved by the introduction of SD cards. However, the POST time increases as the transmission volume increases. It is necessary to test the image quality that can complete transmission in session time.

The results of the experiment are as follows.

Value size transmittable
96×96 18432 OK
QQVGA 38400 OK
QCIF 50688 OK
HQVGA 84480 OK
QVGA 153600 OK
CIF 236800 OK
HVGA 307200 OK
VGA 614400 OK
SVGA 960000 OK
XGA 1572864 OK
SXGA 2621440 NG

The system integration test was conducted in “QVGA” because the transmission was possible, but the transmission time would be long.

14.When the trigger is pressed, the camera takes a picture, saves the camera image to the SD card, and sends the image via Wifi with Webapi.#

Integrate the code so far. 

#include <WiFi.h>
#include <HTTPClient.h>
#include <WiFiClientSecure.h>
#include <ArduinoJson.h>
#include "esp_camera.h"
#include "config.h"
#include "FS.h"
#include "SD.h"
#include "SPI.h"
#include <SD.h>

// #define LED_BUILTIN 21  // commented out to avoid redefinition warning
const int limitSwitchPin = 6; // D6 = GPIO6

void setupCamera();
void postImageFromSD(const String &path);
void connectWiFi();
void connectWiFi()
{
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  Serial.print("Connecting to WiFi");
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(1000);
    Serial.print(".");
  }
  Serial.println("Connected!");

  configTime(0, 0, "pool.ntp.org", "time.nist.gov");
  struct tm timeinfo;
  while (!getLocalTime(&timeinfo))
  {
    Serial.println("Waiting for NTP time...");
    delay(1000);
  }
}

void setupCamera()
{
  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = 15;
  config.pin_d1 = 17;
  config.pin_d2 = 18;
  config.pin_d3 = 16;
  config.pin_d4 = 14;
  config.pin_d5 = 12;
  config.pin_d6 = 11;
  config.pin_d7 = 48;
  config.pin_xclk = 10;
  config.pin_pclk = 13;
  config.pin_vsync = 38;
  config.pin_href = 47;
  config.pin_sccb_sda = 40;
  config.pin_sccb_scl = 39;
  config.pin_pwdn = -1;
  config.pin_reset = -1;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  // config.frame_size = FRAMESIZE_XGA; // High quality (1600x1200)
  // config.jpeg_quality = 8;           // High quality JPEG output
  config.frame_size = FRAMESIZE_QVGA; 
  config.jpeg_quality = 10;   

  config.fb_count = 1;

  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK)
  {
    Serial.printf("Camera init failed: 0x%x\n", err);
    while (true)
      delay(1000);
  }
}

void postImageFromSD(const String &path)
{
  WiFiClientSecure client;
  client.setInsecure();

  Serial.println("Starting POST...");

  if (!client.connect("machinical-memory-landscapes.cloud", 443))
  {
    Serial.println("Connection failed!");
    return;
  }

  File file = SD.open(path.c_str(), FILE_READ);
  if (!file)
  {
    Serial.println("Failed to open file for reading");
    return;
  }

  String boundary = "----WebKitFormBoundary7MA4YWxkTrZu0gW";
  String contentType = "multipart/form-data; boundary=" + boundary;
  String start = "--" + boundary + "\r\n"
                                   "Content-Disposition: form-data; name=\"image\"; filename=\"upload.jpg\"\r\n"
                                   "Content-Type: image/jpeg\r\n\r\n";
  String end = "\r\n--" + boundary + "--\r\n";

  int contentLength = start.length() + file.size() + end.length();

  client.println("POST /api/upload-image HTTP/1.1");
  client.println("Host: machinical-memory-landscapes.cloud");
  client.println("User-Agent: ESP32");
  client.println("Content-Type: " + contentType);
  client.println("Content-Length: " + String(contentLength));
  client.println("Connection: close");
  client.println();

  Serial.println("Posting image data...");
  client.print(start);
  while (file.available())
  {
    client.write(file.read());
  }
  client.print(end);
  file.close();

  while (client.connected() || client.available())
  {
    String line = client.readStringUntil('\n');
    Serial.println(line);
  }

  client.stop();
  Serial.println("Image POST completed");
}

void setup()
{
  Serial.begin(115200);
  delay(3000);
  Serial.println("Serial monitor initialized. Starting setup...");
  pinMode(limitSwitchPin, INPUT_PULLUP);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, HIGH);

  connectWiFi();
  setupCamera();
  SPI.begin(7, 8, 9, 21); 
  if (!SD.begin(21))
  {
    Serial.println("SD_MMC mount failed");
    while (true)
      delay(1000);
  }

  uint8_t cardType = SD.cardType();
  if (cardType == CARD_NONE)
  {
    Serial.println("Error: No SD card attached or card not recognized.");
    while (true)
      delay(1000);
  }
  else
  {
    Serial.println("SD card detected and ready.");
  }
  Serial.print("SD Card Type: ");
  if (cardType == CARD_MMC)
  {
    Serial.println("MMC");
  }
  else if (cardType == CARD_SD)
  {
    Serial.println("SDSC");
  }
  else if (cardType == CARD_SDHC)
  {
    Serial.println("SDHC");
  }
  else
  {
    Serial.println("Unknown");
  }

  uint64_t cardSize = SD.cardSize() / (1024 * 1024);
  Serial.print("SD Card Size: ");
  Serial.print(cardSize);
  Serial.println("MB");

  Serial.println("SD Card initialized.");

  Serial.println("Setup complete");
}

void loop()
{

  int switchState = digitalRead(limitSwitchPin);

  if (switchState == LOW)
  {
    digitalWrite(LED_BUILTIN, LOW);

    camera_fb_t *fb = esp_camera_fb_get();
    if (!fb)
    {
      Serial.println("Capture failed");
      delay(100);
      return;
    }
    Serial.println("Image captured successfully.");
    Serial.print("Image size: ");
    Serial.print(fb->len);
    Serial.println(" bytes");

    Serial.println("Saving image to SD card...");
    struct tm now;
    getLocalTime(&now);
    char filename[32];
    strftime(filename, sizeof(filename), "/%Y%m%d%H%M%S.jpg", &now);
    File file = SD.open(filename, FILE_WRITE);
    if (!file)
    {
      Serial.println("Failed to open file for writing");
      esp_camera_fb_return(fb);
      return;
    }

    file.write(fb->buf, fb->len);
    file.close();
    Serial.print("Image saved to SD card: ");
    Serial.println(filename);
    esp_camera_fb_return(fb);

    postImageFromSD(filename);
  }

  delay(200);
}

Video Editing#

1. creating a storyboard#

A storyboard was created for video creation. I used canva’s whiteboard function to create the storyboard.

2. Create the talk script.#

Next,I created a talk script.

talk script
Each day, we build intricate circuits, write code, and design 3D models.
 At Fab Academy, every moment is a cycle of creation, trial, and error.
And amidst this relentless process, there is one thing that almost inevitably appears.

The bug.

Bugs emerge suddenly.
 They disrupt our thinking.
 They unsettle our hearts.

What we need in those moments—is Zen.
Zen helps us restore stillness within ourselves.
 It teaches us to accept what is in front of us, to observe the root cause, and to focus on a single, deliberate action.
Through our time at Fab Academy, we came to realize something profound:
 The very act of facing bugs, with calm and clarity, is in itself a practice of Zen.
This realization led us to a new challenge—
 To give form to Zen through the tools of technology.
And thus, we created Ryoanji xy2025.
Here's how it works:
 When you upload an image, it is sent to a web server and processed by a Large Language Model.
 This image is transformed into a mindscape.
 Based on that, the XY-arm is activated and constructs a unique Karesansui, or dry landscape garden, just for you.
Once the garden is complete, a switch triggers the camera.
 A photograph of the Karesansui is taken.
 The image is then uploaded via Wi-Fi to a cloud database, and simultaneously interpreted by the LLM.
 The LLM then composes a Haiku—a traditional Japanese poem—based on the garden's design.
And then, in the spirit of Wabi-sabi—the fleeting beauty of impermanence—
 the garden is reset.
Your image, through the synergy of Wi-Fi, LLM, XY-arm, and microcontroller,
 becomes a work of Zen art: a Karesansui, and a Haiku.
This is the form of Zen we have fabricated.
Thank you.

3. create a voice with synthesized speech.#

The following tools were used for voice generation. The reason was that even though it was free, the amount of text read out was as large as 10,000 characters, and it was available in mp3 format.

https://elevenlabs.io

Thus, audio can be generated.

4. collect background music and system effects#

It became necessary to follow the storyboard and collect background music and SEs. The BGM and SE used in this project are as follows.

5. edit our’s video#

We used canva for the main editing and da Vinci Resolve for some of the material processing.

  • canva

    • canva is a free video editing tool with intuitive operation and a wide variety of templates, making it easy for even beginners to create professional-quality videos.
    • Below is a screen shot of an actual video edited by canva. !

  • DaVinci Resolve

    • DaVinci Resolve is an all-in-one video production tool for professionals that integrates video editing, color correction, VFX, motion graphics, and audio editing in one software.
    • Video stabilization was done with the DaVinci Resolve feature.

DaVinci Resolve Transition function#

  1. open DaVinci Resolve and cut the part you want to cut on the timeline by clicking the scissors symbol.
  2. select a transition as shown in the video below, and drag and drop it onto the cut area.

DaVinci Resolve Stabilize function#

  1. cannot be applied to multiple cut videos at once. Therefore, export the previous video and load it again.
  2. stabilize the video according to the following image. The parameters should also follow the image. The following video shows a comparison between before stabilization (left) and after stabilization (right). You can see some differences.

After editing the video, it was determined that the original storyboard would exceed 2 minutes. Therefore, the entire first half was cut.

6. create subtitles#

We used a tool called vrew for the transcription. - vrew - Vrew is a video editing tool that uses AI to automatically recognize audio in videos and generate subtitles. The tool was originally developed to reduce the man-hours required for subtitle editing in TV video production. It allows users to edit subtitles with intuitive operations.

7. reduce capacity to allow upload#

It must be less than 10 MB to upload to gitlab. The following commands were used to compress and reduce the size. 

ffmpeg -i input.mp4 -vf "scale=-2:720,fps=30" -c:v libx264 -preset slow -crf 28 -c:a aac -b:a 128k -movflags +faststart  output.mp4

8. completed#

** Take a look at our passion! **