Final Project

About Finger Score Toy

What Finger Score toy does?

Is a toy for shooting hoops and scoring points wit your fingers.

It is an interactive game:

Allowing users to shoot and score points using their fingers, it provides an interactive basketball experience.

Apply artificial intelligence in sports:

It combines traditional gameplay with modern artificial intelligence technology by recognizing hands and fingers through a web camera to score points according to the number of fingers raised.

What did I design?

Structure:

- Backboard: 2D, 3D design process and manufacturing with laser cutting technology

- Hoop: 2D, 3D design process and manufacturing with 3D printing technology

- Column: 2D and 3D design process with CNC Router technology and laser cutting technology

- Column counterweight: 2D and 3D design process with CNC Router technology

Electronics:

- PCB: To control of the input and output devices.

Programming:

- Programming code for the operation of the Xiao RP2040 microcontroller written in Arduino.

- Programming code for the operation of the Hand Landmarks Detection artificial intelligence system written in Python.

- Programming code for the operation of the interface using Tkinter written in Python to keep track of the points.

What materials and components were used?

Structure:

- Backboard: MDF 3.0 mm.

- Hoop: Polylactic acid (PLA) white and sky blue

- Column: leftovers plywood 9.0 mm.

- Column counterweight: leftovers plywood 18.0 mm.

Electronics:

- PCB: Copper-clad bakelite PCB

- 01 infrared sensor FC-51

- 01 buzzer

- 01 Xiao RP2040

- 01 USB-C

- 01 PinHeaderx2

- 01 PinHeaderx3

Where will come from?

- Materials such as plywood and MDF for the structure were recycled and made from leftovers.

- The PLA filament is from FabLab ifurniture.

- The electronic components were purchased in a store in the city of Lima.

- The Xiao RP2040 microcontroller was imported from the USA.

***In short, all materials are recycled.

How much did they cost?

What parts and systems were made?

The Backboard, the support column, the hoop for the board, the ball for testing, and the PCB board were manufactured.

What processes were used?

- Computer aided desing

- Computer controlled cutting

- 3D printing

- Electronics design

- Electronics production

- Embedded programming

- Computer controlled machining

- Output devices

- Input devices

- Networking and comunications

- Interface and application programing

What questions were answered?

What is the possibility of scaling the game to real life in a basketball game?:

- This was a frequently asked question from people who tried the game and I have high hopes that it can be done and I will do everything possible to make it so in the future.

Will players adapt to a system that combines tradition and technology?

- The path of the use of new technologies in sports is outlined and is being carried out, a good example is the Airless Gen1, an innovative 3D printed basketball developed by Wilson.

Will the FC-51 infrared sensor be the right sensor for the Finger Score Toy?

- During the system tests we saw that this infrared sensor has a deficiency, being outdoors it is affected by the sun's rays and at all times it shows that there is an object in front when it is not so, you must work in an environment where the light does not reach direct from the sun.

What worked? what did not?

The Hands Landmarks artificial intelligence works well, although it is still somewhat confusing for me and the user interface to keep the score worked well.

What did not work as expected is the FC-51 Infrared Sensor since its proper functioning depends on the environment in which it is located. It works in low light, it does not work in bright light.

Concept

In basketball, not only do the players use their hands to score baskets in a game, but the referees who are in charge of applying the rules of the game also do so. They use their hands and fingers that meditatively indicate the amount to the official table. of points scored, fouls committed by players, player numbers, etc.

For this reason, The Finger Score Toy introduces artificial intelligence to mark points with your hands using a Mediapipe solution called Hand landmarks detection. The points are displayed on a scoreboard-type digital interface.

For it to work we must integrate an obstacle detection sensor, a buzzer, a microcontroller, an interface and artificial intelligence.

Final project schedule

This table provides an overview of the activities I carried out in each week of the project, some items were adjusted along the way but we achieved the stated objectives.

Previous Design

I love basketball, since I learned to play it at university I haven't stopped doing it.

I would like to play every day but due to the years and my time I can only do it on Saturdays, I am part of a group of people who love basketball, we could say that we are vicious and addicted to this game.

The player who has the ball, is me...

Initially I planned to make an electronic scoreboard that scores points by voice.

To do this, make some sketches and 3D modeling.

Final Design

As we progressed each week with the different units I felt that something else should contain my project and revisiting the bases of the game I realized something that I had not given importance to before and that is the process of how points are scored in basketball.

How are points scored in basketball? The referee uses his hands and fingers making gestures that he directs to the game control table, to score 1 point the referee raises a finger, two points he raises 2 fingers and three points he raises 3 fingers and the scorer on the table transfers it to the electronic scoreboard by pressing a button.

Using hand and finger gestures in basketball is a crucial part of the game and from this information the idea of ​​creating a game that introduces artificial intelligence to the process of marking points on a scoreboard and to record the points was born. A scale board and ball would be needed to complete the entire experience.

Sketches

These are the sketches that would define the final project.

Laser cutting

We manufactured the backboard, we used two pieces and joined it with digital joints, we also marked and engraved the logo on the backboard. We use leftover 3.0 mm MDF.

I also made the covering for the structure using leftover MDF as well.

CNC Router

Manufacture the support commune and the counterweights of the column, use leftover Plywood with a thickness of 9.0 mm for the column and 18.0 mm * 3 units for the counterweight. as always using leftover plywood

3D Printing

I modeled and printed the basketball hoop to scale, modifying the connection with the board so that the infrared sensor enters inside. I also printed the basketballs so I could use them with the game.

The material used was white and light blue PLA filament.

Electronics

Design the PCB electronic board in Kicad and use the Roland Monofab milling machine to mill it, the board will have a Xiao RP2040 microcontroller, an infrared sensor and a buzzer.

PCB electronic board code in Arduino:

int buzzerPin = D5;
int sensorPin = D8;
int sensorState = 0;

void setup() {
pinMode(buzzerPin, OUTPUT);
pinMode(sensorPin, INPUT);
Serial.begin(115200);
}

void loop() {
sensorState = digitalRead(sensorPin);

if (sensorState == LOW) {  // Detects movement
digitalWrite(buzzerPin, HIGH);
delay(1000);  // Buzzer sounds for 1 second
digitalWrite(buzzerPin, LOW);

Serial.println('h');  // Send 'h' via serial port
delay(1000);  // Wait 1 second to avoid multiple signals
}
}

                

Artificial Intelligence & User Interface

We use the Mediapipe Hand Landmark detect solution to detect the signals that referees make with their hands to mark the points, one raised finger is equivalent to 1 point, two raised fingers is equivalent to 2 points and 3 raised fingers are equivalent to 3 points.

The interface shows the score in real time every time the referee raises his fingers after having entered the ball into the hoop.

solution programming code in python:

import cv2
import mediapipe as mp
import tkinter as tk
from threading import Thread
import serial
import time

class BasketballScoreboard(tk.Tk):
def __init__(self):
  super().__init__()
  self.title("Finger Score")

  # Score initialization
  self.score = 0

  # Creating labels to display the score
  self.label_team = tk.Label(self, text="Score", font=("Arial", 100))
  self.label_team.grid(row=0, column=0, padx=20, pady=10)
  
  self.label_score = tk.Label(self, text=self.score, font=("Arial", 300))
  self.label_score.grid(row=1, column=0, padx=20, pady=10)

  # Serial port initialization
  self.serial_port = 'COM3'  # Change this to the correct port
  self.baud_rate = 115200
  self.ser = serial.Serial(self.serial_port, self.baud_rate)
  
  # Start the hand detection and serial listening thread
  self.hand_detection_thread = Thread(target=self.detect_hands)
  self.hand_detection_thread.daemon = True
  self.hand_detection_thread.start()

  self.serial_thread = Thread(target=self.listen_serial)
  self.serial_thread.daemon = True
  self.serial_thread.start()

  self.update_score = False

def increment_score(self, points):
  for _ in range(points):
      self.score += 1
      self.label_score.config(text=self.score)
      time.sleep(0.1)  # Increase slowly

def listen_serial(self):
  while True:
      if self.ser.in_waiting > 0:
          signal = self.ser.readline().decode().strip()
          if signal == 'h':
              self.update_score = True
              time.sleep(0.1)  # Short pause to avoid multiple readings

def detect_hands(self):
  mp_hands = mp.solutions.hands
  mp_drawing = mp.solutions.drawing_utils
  hands = mp_hands.Hands()
  cap = cv2.VideoCapture(0)

  while cap.isOpened():
      ret, frame = cap.read()
      if not ret:
          break

      frame = cv2.cvtColor(cv2.flip(frame, 1), cv2.COLOR_BGR2RGB)
      results = hands.process(frame)

      if results.multi_hand_landmarks:
          hand_landmarks = results.multi_hand_landmarks[0]
          # Draw reference points and connections
          mp_drawing.draw_landmarks(frame, hand_landmarks, mp_hands.HAND_CONNECTIONS)
          
          # Count raised fingers and update score if 'h' signal is received
          if self.update_score:
              fingers = self.count_fingers(hand_landmarks)
              if fingers > 0:
                  self.increment_score(fingers)
              self.update_score = False

      cv2.imshow('Hand Detection', cv2.cvtColor(frame, cv2.COLOR_RGB2BGR))
      if cv2.waitKey(5) & 0xFF == 27:
          break

  cap.release()
  cv2.destroyAllWindows()

def count_fingers(self, hand_landmarks):
  # Count raised fingers
  finger_tips = [4, 8, 12, 16, 20]
  count = 0
  if hand_landmarks.landmark[finger_tips[0]].x < hand_landmarks.landmark[finger_tips[0] - 1].x:
      count += 1  # count thumb
  for tip in finger_tips[1:]:
      if hand_landmarks.landmark[tip].y < hand_landmarks.landmark[tip - 2].y:
          count += 1
  return min(count, 3)  # Maximum 3 points

if __name__ == "__main__":
app = BasketballScoreboard()
app.mainloop()

              

What did I learn?

Learn to integrate different systems in a single project.

I learned to apply different solutions to some problems that arose along the way.

I learned to document each process carried out.

And the most important thing is that I learned how to make an idea a reality by applying all the knowledge acquired in the Fab Academy 2024.