Week 14: Interface and Application Programming

May 7, 2024

Project Document - ToGo Pathfinding Project

1. Introduction

“The ToGo Pathfinding Project utilizes a Raspberry Pi 4 and Raspberry Pi Camera 2 to detect ArUco markers and calculate the shortest path on a grid while lighting up the path on LED cells. The system controls LEDs via ESP32-C3 modules embedded within cubes that are marked with ArUco markers. The project incorporates a Flask web framework for inter-device communication and control via a user interface.”

Group Assigment Page

group assignment networking link 🚀

2. System Architecture

“ToGO employs a Raspberry Pi 4 as the main control unit. The Raspberry Pi processes real-time video feed from a connected camera module to detect ArUco markers. The detected marker information is processed through a Flask server running on the Raspberry Pi, which sends appropriate commands to ESP32-C3 modules for LED control. Each cube is equipped with ArUco markers on each face, allowing the system to determine the cube’s orientation.”

3. Main Modules and Functions

• Camera Module (Picamera2): Provides real-time video streaming and detects ArUco markers.

• Flask Web Server: Offers a user interface and coordinates command transmission between devices.

• LED Control (ESP32-C3): Activates specific LEDs according to the detected path.

4. User Interface

“Users interact with the system through a web interface provided by Flask. The interface allows users to view camera feeds, take snapshots, adjust LED settings, and review system logs. Additionally, it provides capabilities to check and update the connectivity and settings of the ESP modules.”

Main Page

Camera Settings Function

Aruco Detect

Camera and Grid Settings

5. Testing and Results

“The project underwent extensive testing under various scenarios. Parameters such as ArUco marker detection accuracy, response time of LED control, and system stability were focused on. The tests demonstrated that our system operates with high accuracy and low latency effectively.”

Board Simulation

6. Programming Interface and Application Structure

The app.py file serves as the entry point to our application, orchestrating various functionalities through different modules. We utilize a modular architecture to enhance maintainability and reduce the complexity inherent in handling multiple components. Here’s how the program is structured and the roles of the various imported modules:

• Flask Framework: Used for creating the web server and handling routes. Flask, render_template, jsonify, and request are utilized for web interactions.
• Threading: Allows for parallel processing, particularly useful for managing long-running camera operations without blocking the main application flow.
• Picamera2: Manages camera operations including video capture and processing, crucial for our real-time ArUco marker detection.
• Grid Module: Contains functions like save_image and detect_and_label_grids, responsible for image processing and grid analysis.
• Network Utilities: Includes the check_esp_connections function, used to verify the status of network connections with ESP modules.
• Control ESP: The send_command_to_esp module is crucial for sending commands to ESP32-C3 modules to control LEDs based on detected paths.
• ArUco Handler: Handles the detection and processing of ArUco markers, which are pivotal for navigating the cubes in our grid.

Example Code Snippet

Here is a snippet from app.py illustrating how these components interact:

from flask import Flask, render_template, jsonify, request
import threading
from picamera2 import Picamera2
import grid
from grid import save_image, detect_and_label_grids
from network_utils import check_esp_connections
from control_esp import send_command_to_esp

# Initialize the Flask app
app = Flask(__name__, static_folder='static', static_url_path='/static')

# Define a route to display the main page
@app.route('/')
def index():
    return render_template('index.html')

# Route for initiating camera connection
@app.route('/connect_camera', methods=['GET'])
def connect_camera():
    threading.Thread(target=setup_camera).start()
    return jsonify({'message': 'Camera connection is initiated.'})

# Main function to launch the Flask app
if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

7. Swift Simulation for Pathfinding

The SwiftUI simulation within the ToGo Pathfinding Project leverages SwiftUI to create a dynamic and interactive grid system where users can simulate pathfinding scenarios in real-time. Below are some key features of the Swift simulation:

Togo V4 Main View

Path Calculation

• Interactive Grid: Users can interact directly with the grid to set start and end points, and place obstacles.
• Path Visualization: The application calculates and visually displays the shortest path based on the user-defined conditions.
• Real-Time Updates: Changes to the grid (like adding or removing obstacles) are reflected immediately in the path calculation.

8. Raspberry Pi Pico W and Adafruit IO Dashboard Project

I integrated a Raspberry Pi Pico W with Adafruit IO to create a real-time dashboard. This project demonstrates to work with microcontrollers and cloud-based IoT platforms.

Source Files

Source Folder