System Integration

This week we will focus on the integration of the project systems

The assignment was:

Design and document the system integration for your final project

Integration system result

The "Finger Score Toy" is a Toy for shooting hoops and scoring points with your fingers

System Workflow

This toy combines a sensor, an actuator and a microcontroller integrated into a PCB designed to enhance the system's integration and functionality. This PCB connects via serial communication to a computer to interact with an AI-based hand recognition system and an interactive user interface.

Basketball Hoop Detection Subsystem

Infrared Sensor FC-51:

- Detects when the ball passes through the hoop.

- Sends a signal to the XIAO RP2040 microcontroller.

Test Infrared Sensor FC-51:

- When the ball passes through the hoop, the green LED light on the sensor turns on.

Processing and Control Subsystem

XIAO RP2040 Microcontroller:

- Receives the signal from the infrared sensor FC-51.

- Processes the information and activates the buzzer for auditory feedback.

- Sends data to the computer via serial communication.

Designed PCB:

- Designed in kicad 8.0

- Integrate the system

Test subsystem:

- Xiao RP2040 working.

- IR sensor calibrated and buzzer working.

- Transmits data from the XIAO RP2040 microcontroller to the computer.

Code subsystem:

    int buzzerPin = D5;
    int sensorPin = D8;
    int sensorState = 0;

    void setup() {
    pinMode(buzzerPin, OUTPUT);
    pinMode(sensorPin, INPUT);
    Serial.begin(115200);
    }

    void loop() {
    sensorState = digitalRead(sensorPin);

    if (sensorState == LOW) {  // Detects movement
        digitalWrite(buzzerPin, HIGH);
        delay(1000);  // Buzzer sounds for 1 second
        digitalWrite(buzzerPin, LOW);

        Serial.println('h');  // Send 'h' via serial port
        delay(1000);  // Wait 1 second to avoid multiple signals
    }
    }

                        

Case subsystem:

Hands Detection and User Interface Subsystem

Hands Detection:

The MediaPipe Gesture Recognizer task lets you recognize hand gestures in real time, and provides the recognized hand gesture results along with the landmarks of the detected hands. You can use this task to recognize specific hand gestures from a user, and invoke application features that correspond to those gestures.

More information visit mediapipe website

User Interface:

Tkinter Python interface is used in the "Finger Score Toy" to provide a real-time score display, facilitating user interaction.

- Displays the score in real-time.

- Tracks the points scored.

More information visit Tkinter website

code subsystem:

    import cv2
    import mediapipe as mp
    import tkinter as tk
    from threading import Thread
    import serial
    import time

    class BasketballScoreboard(tk.Tk):
        def __init__(self):
            super().__init__()
            self.title("Finger Score")

            # Score initialization
            self.score = 0

            # Creating labels to display the score
            self.label_team = tk.Label(self, text="Score", font=("Arial", 100))
            self.label_team.grid(row=0, column=0, padx=20, pady=10)
            
            self.label_score = tk.Label(self, text=self.score, font=("Arial", 300))
            self.label_score.grid(row=1, column=0, padx=20, pady=10)

            # Serial port initialization
            self.serial_port = 'COM3'  # Change this to the correct port
            self.baud_rate = 115200
            self.ser = serial.Serial(self.serial_port, self.baud_rate)
            
            # Start the hand detection and serial listening thread
            self.hand_detection_thread = Thread(target=self.detect_hands)
            self.hand_detection_thread.daemon = True
            self.hand_detection_thread.start()

            self.serial_thread = Thread(target=self.listen_serial)
            self.serial_thread.daemon = True
            self.serial_thread.start()

            self.update_score = False

        def increment_score(self, points):
            for _ in range(points):
                self.score += 1
                self.label_score.config(text=self.score)
                time.sleep(0.1)  # Increase slowly

        def listen_serial(self):
            while True:
                if self.ser.in_waiting > 0:
                    signal = self.ser.readline().decode().strip()
                    if signal == 'h':
                        self.update_score = True
                        time.sleep(0.1)  # Short pause to avoid multiple readings

        def detect_hands(self):
            mp_hands = mp.solutions.hands
            mp_drawing = mp.solutions.drawing_utils
            hands = mp_hands.Hands()
            cap = cv2.VideoCapture(0)

            while cap.isOpened():
                ret, frame = cap.read()
                if not ret:
                    break

                frame = cv2.cvtColor(cv2.flip(frame, 1), cv2.COLOR_BGR2RGB)
                results = hands.process(frame)

                if results.multi_hand_landmarks:
                    hand_landmarks = results.multi_hand_landmarks[0]
                    # Draw reference points and connections
                    mp_drawing.draw_landmarks(frame, hand_landmarks, mp_hands.HAND_CONNECTIONS)
                    
                    # Count raised fingers and update score if 'h' signal is received
                    if self.update_score:
                        fingers = self.count_fingers(hand_landmarks)
                        if fingers > 0:
                            self.increment_score(fingers)
                        self.update_score = False

                cv2.imshow('Hand Detection', cv2.cvtColor(frame, cv2.COLOR_RGB2BGR))
                if cv2.waitKey(5) & 0xFF == 27:
                    break

            cap.release()
            cv2.destroyAllWindows()

        def count_fingers(self, hand_landmarks):
            # Count raised fingers
            finger_tips = [4, 8, 12, 16, 20]
            count = 0
            if hand_landmarks.landmark[finger_tips[0]].x < hand_landmarks.landmark[finger_tips[0] - 1].x:
                count += 1  # count thumb
            for tip in finger_tips[1:]:
                if hand_landmarks.landmark[tip].y < hand_landmarks.landmark[tip - 2].y:
                    count += 1
            return min(count, 3)  # Maximum 3 points

    if __name__ == "__main__":
        app = BasketballScoreboard()
        app.mainloop()

                        

Test:

- Finger gestures (1, 2, or 3 fingers raised) are translated into points (1, 2, or 3).

- These points are sent to the user interface (UI) developed with Tkinter.

conclusions:

- Developing this toy allowed me to integrate different technologies into a single system.

- Designing a PCB improved my understanding of electronics.

- Implementing artificial intelligence and a user interface gave me some experience in programming using python, I still find it difficult.

- The toy shows some complexity in real-time data processing and the need for efficient communication between hardware and software.

- The importance of testing to refine functionality is highlighted.

- Creating a product from start to finish gave me immense personal satisfaction.

what I learned?:

- To integrate different technologies in a project.

- To customize a PCB to control the toy.

- To solve some problems and challenges both in the code and in the physical part.

- A Perform multiple tests of each of the subsystems.

- That technology can be used to create fun and educational tools.

Download files

Here we can find and download the design original files