目次

WEEK16 System integration

hito |

WEEK16 System integration

Assignment

Design and document the system integration for your final project

Enclosure

Ideally, the PCB layout should be determined before proceeding with the enclosure design. However, due to the sensor device specifications not being fully finalized and the desire to create training data for the TensorFlow model file as soon as possible, I decided to prioritize the creation of the enclosure. Since there aren't too many parts involved, I believe that even if there are changes in the specifications, only minor modifications will be necessary.
alt text
alt text alt text

The enclosure consists of three parts: the main body, the cover, and the LED window.
The main body and cover were created using a 3D printer, while the LED window was made by cutting a 5mm thick acrylic sheet with a laser cutter.
alt text alt text

alt text alt text

  • Before attaching to the LED window
    alt text alt text
  • After attaching and while charging
    alt text alt text

Fusion 360(.f3d) data

PCB layout

For the final project, I have tested various MPUs such as Xiao RP2040 and ESP32C3, but I decided to proceed with development using the nRF52840 Sense.

I chose the nRF52840 Sense because it has onboard BLE, IMU, and battery charging features, making it ideal for our need to make the device as lightweight as possible.

The nRF52840 Sense indicates battery charging through an on-board LED that lights up. However, this LED is very small and difficult to recognize from outside the enclosure. Therefore, I will add another LED with better visibility that illuminates during charging. Another point is the addition of a button to control the device's sleep and wake functions. Once these functions, currently being tested on a breadboard, are confirmed, I would like to start the PCB layout design.

alt text alt text
alt text

Download: PCB data(.f3z)

Preparation for Machine Learning Using TensorFlow

Obtain acceleration and gyroscopic data from a sensor device attached to the racket. Perform about 10 swings for each shot and save the obtained dataset to a CSV file.

Initially, I received the dataset via BLE, but since it took over 40 seconds to receive data for a single swing, I switched to data acquisition via USB. To obtain swing data, a sufficiently long cable was required, so I connected the sensor to the PC with a 5m USB cable, but the sensor could not be recognized. Maybe the cable is too long.

Ultimately, I resolved the issue by connecting a shorter USB cable (1.5m) to a smartphone, as shown in the photo.

alt text alt text
alt text

I used the following code to obtain the data.
Arduino IDE -> File -> Examples -> Seeed_Arduino_LSM6DS3 ->IMUCapture.ino 

#include <LSM6DS3.h>
#include <Wire.h>

//Create a instance of class LSM6DS3
LSM6DS3 myIMU(I2C_MODE, 0x6A);    //I2C device address 0x6A
float aX, aY, aZ, gX, gY, gZ;
const float accelerationThreshold = 2.5; // threshold of significant in G's
const int numSamples = 119;
int samplesRead = numSamples;

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  while (!Serial);
  //Call .begin() to configure the IMUs
  if (myIMU.begin() != 0) {
    Serial.println("Device error");
  } else {
    Serial.println("aX,aY,aZ,gX,gY,gZ");
  }
}

void loop() {
  // wait for significant motion
  while (samplesRead == numSamples) {
    // read the acceleration data
    aX = myIMU.readFloatAccelX();
    aY = myIMU.readFloatAccelY();
    aZ = myIMU.readFloatAccelZ();

    // sum up the absolutes
    float aSum = fabs(aX) + fabs(aY) + fabs(aZ);

    // check if it's above the threshold
    if (aSum >= accelerationThreshold) {
      // reset the sample read count
      samplesRead = 0;
      break;
    }
  }

  // check if the all the required samples have been read since
  // the last time the significant motion was detected
  while (samplesRead < numSamples) {
    // check if both new acceleration and gyroscope data is
    // available
    // read the acceleration and gyroscope data

    samplesRead++;

    // print the data in CSV format
    Serial.print(myIMU.readFloatAccelX(), 3);
    Serial.print(',');
    Serial.print(myIMU.readFloatAccelY(), 3);
    Serial.print(',');
    Serial.print(myIMU.readFloatAccelZ(), 3);
    Serial.print(',');
    Serial.print(myIMU.readFloatGyroX(), 3);
    Serial.print(',');
    Serial.print(myIMU.readFloatGyroY(), 3);
    Serial.print(',');
    Serial.print(myIMU.readFloatGyroZ(), 3);
    Serial.println();

    if (samplesRead == numSamples) {
      // add an empty line if it's the last sample
      Serial.println();
    }
  }
}

Training data sample

aX,aY,aZ,gX,gY,gZ
-1.441,-0.474,-0.613,161.070,303.590,190.190
-1.567,-0.458,-0.609,164.290,295.750,189.280
-1.706,-0.441,-0.605,165.340,306.040,187.390
-1.778,-0.440,-0.603,164.920,311.150,184.870
-1.909,-0.433,-0.572,167.580,322.420,183.960
-2.099,-0.425,-0.527,150.990,332.430,183.330
-2.223,-0.708,-0.366,129.920,340.270,177.100
-2.368,-0.840,-0.493,135.240,346.360,173.320
-2.355,-0.809,-0.453,137.550,357.000,168.910
-2.379,-0.875,-0.495,147.420,369.740,161.560
-2.495,-0.798,-0.334,149.590,374.500,160.160
-2.675,-0.820,-0.291,142.730,384.580,154.070
-2.873,-0.821,-0.222,138.040,394.310,149.380
-2.881,-0.842,-0.163,133.630,399.070,142.730
-2.989,-0.913,-0.117,128.380,408.170,139.930
-3.074,-0.968,-0.021,124.110,416.640,133.700
-3.114,-1.014,0.064,118.720,420.770,126.700
-3.203,-1.083,0.107,113.820,428.260,123.060
-3.299,-1.170,0.178,112.980,435.330,114.800
-3.343,-1.208,0.262,112.700,438.760,106.400
-3.420,-1.297,0.298,110.530,445.270,100.800
-3.529,-1.335,0.362,112.490,451.220,91.630
-3.575,-1.479,0.603,113.540,453.040,81.410
-3.638,-1.439,0.673,113.330,455.910,75.670
-3.706,-1.519,0.788,117.110,458.500,63.700
-3.737,-1.543,1.002,124.460,458.990,51.800
-3.742,-1.600,1.112,119.980,458.780,45.780
-3.726,-1.718,1.317,116.830,456.890,33.320
-3.709,-1.840,1.561,105.770,455.070,20.720
-3.655,-1.928,1.793,91.350,450.730,13.930
-3.583,-2.082,1.905,86.310,443.590,0.630
-3.542,-2.212,2.074,76.020,441.140,-5.530
-3.483,-1.914,2.522,49.560,431.900,5.460
-3.439,-2.870,2.595,38.850,417.760,-35.840
-3.377,-2.705,2.861,73.710,411.320,-54.460
-3.218,-2.664,3.180,57.400,393.120,-60.830
-3.093,-2.889,3.292,60.830,374.640,-78.400
-3.070,-3.107,3.478,70.280,359.660,-100.800
-2.991,-3.100,3.802,68.950,339.640,-110.530
-2.933,-3.456,3.962,79.940,312.340,-136.360
-2.897,-3.515,4.188,96.880,300.930,-153.230
-2.781,-3.618,4.697,109.480,266.700,-167.860
-2.741,-4.086,4.788,128.660,230.440,-203.630
-2.686,-4.070,5.108,149.940,211.190,-230.580
-2.653,-4.170,5.313,160.510,172.410,-246.400
-2.708,-4.260,5.392,168.000,129.780,-276.570
-2.762,-4.308,5.471,175.700,108.780,-306.950
-2.898,-4.332,5.523,186.550,65.590,-322.350
-3.037,-4.304,5.539,188.160,22.820,-352.730
-3.125,-4.215,5.563,195.160,0.770,-382.550
-3.330,-4.148,5.502,196.980,-42.700,-397.250
-3.588,-3.978,5.450,197.540,-85.540,-425.810
-3.725,-3.754,5.291,196.560,-106.400,-452.760
-4.017,-3.629,5.093,190.960,-146.440,-465.500
-4.341,-3.376,4.976,185.290,-184.100,-490.490
-4.510,-3.101,4.778,182.770,-219.730,-513.940
-4.826,-2.940,4.670,172.760,-236.950,-525.000
-5.161,-2.605,4.612,156.450,-270.690,-545.860
-5.343,-2.258,4.508,147.210,-303.450,-565.110
-5.716,-2.079,4.408,126.070,-319.200,-574.000
-6.085,-1.722,4.359,102.270,-349.930,-591.360
-6.257,-1.339,4.254,89.810,-378.490,-607.180
-6.569,-1.124,4.145,63.350,-392.140,-614.250
-6.876,-0.667,4.104,41.580,-418.040,-626.500
-7.161,-0.128,4.096,32.900,-443.940,-634.970
-7.371,0.153,4.189,18.760,-457.730,-638.820
-7.710,0.688,4.257,4.690,-488.180,-640.220
-8.116,1.164,4.440,1.120,-517.440,-638.120
-8.173,1.411,4.658,-14.280,-533.960,-631.820
-8.456,1.721,4.696,-25.060,-567.700,-629.230
-8.885,1.911,4.822,-24.430,-604.660,-621.390
-9.100,1.971,4.983,-20.860,-624.330,-614.740
-9.655,1.921,5.078,-24.150,-663.740,-612.570
-10.271,1.801,5.355,-21.560,-707.770,-608.650
-10.590,1.689,5.644,-24.080,-731.010,-608.160
-11.347,1.506,5.774,-20.370,-780.220,-608.160
-12.351,1.346,5.944,-12.320,-832.440,-609.630
-12.938,1.249,5.984,2.030,-859.460,-612.920
-14.233,1.133,5.915,15.400,-911.960,-614.810
-15.544,1.055,5.779,20.440,-965.230,-622.230
-15.915,1.263,5.431,28.350,-993.230,-633.710
-15.989,1.610,4.633,72.590,-1042.930,-636.860
-15.989,2.695,4.204,108.570,-1085.630,-637.980
-15.989,4.735,2.969,186.270,-1104.880,-582.680
-15.989,5.933,1.832,217.840,-1130.360,-542.080
-15.989,7.101,1.472,218.750,-1151.220,-473.760
-15.989,7.894,1.023,212.870,-1158.920,-400.680
-15.989,8.039,0.894,198.380,-1174.460,-363.580
-15.989,7.833,0.871,175.980,-1191.400,-297.920
-15.989,6.991,0.661,172.620,-1217.650,-242.690
-15.989,6.516,0.248,181.930,-1210.930,-181.720
-15.989,5.649,0.020,201.880,-1216.740,-170.030
-15.989,4.923,-0.473,213.850,-1217.300,-126.000
-15.989,4.602,-0.989,239.610,-1216.180,-83.440
-15.989,4.027,-1.301,259.210,-1209.880,-62.650
-15.989,3.515,-1.980,256.410,-1195.880,-22.190
-15.989,2.966,-2.788,256.270,-1185.240,10.360
-15.989,2.574,-3.161,259.210,-1158.430,25.690
-15.989,1.991,-3.738,245.910,-1127.840,51.520
-15.989,1.359,-4.376,228.200,-1105.090,72.730
-15.989,1.187,-4.656,206.430,-1061.410,82.880
-15.598,0.613,-4.882,197.330,-1016.750,94.920
-15.019,0.267,-5.111,196.490,-996.590,104.930
-13.922,0.111,-5.326,183.120,-952.910,110.040
-12.875,-0.103,-5.368,177.520,-907.060,117.810
-12.365,-0.382,-5.426,178.990,-886.550,122.430
-11.404,-0.451,-5.403,184.520,-842.800,124.320
-10.427,-0.574,-5.372,183.960,-798.980,128.590
-9.973,-0.626,-5.179,187.810,-761.390,131.250
-9.202,-0.564,-4.949,178.640,-740.040,133.560
-8.527,-0.366,-4.856,168.980,-701.820,139.230
-8.250,-0.147,-4.653,163.100,-666.050,145.460
-7.768,-0.022,-4.513,153.930,-648.480,149.520
-7.325,0.210,-4.450,146.720,-614.810,158.970
-7.117,0.331,-4.360,144.970,-582.260,169.190
-6.749,0.335,-4.327,146.160,-566.650,178.920
-6.442,0.206,-4.340,148.470,-534.100,183.190
-6.029,-0.072,-4.415,148.260,-500.920,189.000
-5.872,-0.214,-4.449,146.160,-484.540,193.130
Use LF (Line Feed) for the line break code when creating the training data files (.csv).

These are graphs showing the acceleration data and gyro data respectively, obtained from 10 shadow swings of a badminton racket for smashing.
alt text alt text

I used the following code to create the graphs. This code is a modified version of the code mentioned in "Getting Started with TensorFlow Lite on Seeed Studio XIAO nRF52840 Sense" to allow selection of any CSV file.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tkinter import Tk
from tkinter.filedialog import askopenfilename

# Display file dialog to select a CSV file
root = Tk()
root.withdraw()  # Hide the Tkinter window
filename = askopenfilename(filetypes=[("CSV files", "*.csv")], title="Select a CSV file")
root.destroy()  # Close the Tkinter window

# Verify if a file was selected
if not filename:
    print("No file selected")
else:
    # Read the selected CSV file
    df = pd.read_csv(filename)

    index = range(1, len(df['aX']) + 1)

    plt.rcParams["figure.figsize"] = (20,10)

    # Plot acceleration data
    plt.plot(index, df['aX'], 'g.', label='x', linestyle='solid', marker=',')
    plt.plot(index, df['aY'], 'b.', label='y', linestyle='solid', marker=',')
    plt.plot(index, df['aZ'], 'r.', label='z', linestyle='solid', marker=',')
    plt.title("Acceleration")
    plt.xlabel("Sample #")
    plt.ylabel("Acceleration (G)")
    plt.legend()
    plt.show()

    # Plot gyroscope data
    plt.plot(index, df['gX'], 'g.', label='x', linestyle='solid', marker=',')
    plt.plot(index, df['gY'], 'b.', label='y', linestyle='solid', marker=',')
    plt.plot(index, df['gZ'], 'r.', label='z', linestyle='solid', marker=',')
    plt.title("Gyroscope")
    plt.xlabel("Sample #")
    plt.ylabel("Gyroscope (deg/sec)")
    plt.legend()
    plt.show()

Generate TensorFlow Lite model file

Generate a TensorFlow Lite model file (model.h), by using csv files.

Train Neural Network

The following are the steps to create an Arduino header file (.h):

  1. Parse the CSV files and transform them into a format that will be used to train the fully connected neural network.
  2. Randomly split the input and output pairs into sets of data: 60% for training, 20% for validation, and 20% for testing.
  3. Build and train a TensorFlow model using the high-level Keras API.
  4. Convert the model to TensorFlow Lite format.
  5. Encode the model in the Arduino header file(.h).

The code is a modified version of the code mentioned in "Getting Started with TensorFlow Lite on Seeed Studio XIAO nRF52840 Sense".

The changes are as follows:
・ Consolidated the previously split code into a single script.
・ Added functionality to select any folder from a displayed file dialog and add all CSV files in the selected folder as parameters to a Python list.
・Made modifications to ensure compatibility with the Windows environment. 

import os
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import tensorflow as tf
from tkinter import Tk
from tkinter.filedialog import askdirectory
import binascii

print(f"TensorFlow version = {tf.__version__}\n")

# Set a fixed random seed value for reproducibility
SEED = 1337
np.random.seed(SEED)
tf.random.set_seed(SEED)

# Display file dialog to select a folder
root = Tk()
root.withdraw()  # Hide the Tkinter window
folder_path = askdirectory(title="Select Folder")
root.destroy()  # Close the Tkinter window

# List of gestures with available data
GESTURES = []

# Get all CSV files in the selected folder and add to GESTURES
for filename in os.listdir(folder_path):
    if filename.endswith(".csv"):
        GESTURES.append(os.path.splitext(filename)[0])

SAMPLES_PER_GESTURE = 119
NUM_GESTURES = len(GESTURES)

# Create a one-hot encoded matrix for the output
ONE_HOT_ENCODED_GESTURES = np.eye(NUM_GESTURES)

inputs = []
outputs = []

# Define expected columns
expected_columns = ['aX', 'aY', 'aZ', 'gX', 'gY', 'gZ']

# Read each CSV file and populate inputs and outputs
for gesture_index, gesture in enumerate(GESTURES):
    print(f"Processing index {gesture_index} for gesture '{gesture}'.")

    output = ONE_HOT_ENCODED_GESTURES[gesture_index]

    df = pd.read_csv(os.path.join(folder_path, gesture + ".csv"))

    # Check if the expected columns are in the dataframe
    if not all(column in df.columns for column in expected_columns):
        print(f"Error: One or more expected columns are missing in {gesture}.csv")
        continue

    # Calculate the number of gesture recordings in the file
    num_recordings = int(df.shape[0] / SAMPLES_PER_GESTURE)
    print(f"\tThere are {num_recordings} recordings of the {gesture} gesture.")

    for i in range(num_recordings):
        tensor = []
        for j in range(SAMPLES_PER_GESTURE):
            index = i * SAMPLES_PER_GESTURE + j
            # Normalize the input data
            tensor.extend([
                (df['aX'][index] + 4) / 8,
                (df['aY'][index] + 4) / 8,
                (df['aZ'][index] + 4) / 8,
                (df['gX'][index] + 2000) / 4000,
                (df['gY'][index] + 2000) / 4000,
                (df['gZ'][index] + 2000) / 4000
            ])
        inputs.append(tensor)
        outputs.append(output)

# Convert lists to numpy arrays
inputs = np.array(inputs)
outputs = np.array(outputs)

print("Data set parsing and preparation complete.")

# Randomize the order of the inputs
num_inputs = len(inputs)
randomize = np.arange(num_inputs)
np.random.shuffle(randomize)
inputs = inputs[randomize]
outputs = outputs[randomize]

# Split the data into training, testing, and validation sets
TRAIN_SPLIT = int(0.6 * num_inputs)
TEST_SPLIT = int(0.2 * num_inputs + TRAIN_SPLIT)

inputs_train, inputs_test, inputs_validate = np.split(inputs, [TRAIN_SPLIT, TEST_SPLIT])
outputs_train, outputs_test, outputs_validate = np.split(outputs, [TRAIN_SPLIT, TEST_SPLIT])

print("Data set randomization and splitting complete.")

# Build and train the model using Input object
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(SAMPLES_PER_GESTURE * 6,)),
    tf.keras.layers.Dense(50, activation='relu'),
    tf.keras.layers.Dense(15, activation='relu'),
    tf.keras.layers.Dense(NUM_GESTURES, activation='softmax')
])
model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])

history = model.fit(inputs_train, outputs_train, epochs=600, batch_size=1, validation_data=(inputs_validate, outputs_validate))

# Wrap the function
@tf.function
def model_predict(inputs):
    return model(inputs)

# Convert to TensorFlow Lite format
converter = tf.lite.TFLiteConverter.from_concrete_functions([model_predict.get_concrete_function(tf.TensorSpec([None, SAMPLES_PER_GESTURE * 6], tf.float32))])
converter.experimental_new_converter = True
converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS,
    tf.lite.OpsSet.SELECT_TF_OPS
]
try:
    tflite_model = converter.convert()
    print("Model successfully converted to TensorFlow Lite format.")
    # Save the model
    with open("gesture_model.tflite", "wb") as f:
        f.write(tflite_model)
    print("Model saved to disk successfully.")
except Exception as e:
    print("Error converting model to TensorFlow Lite format:", e)

# Load the TensorFlow Lite model in binary format
with open("gesture_model.tflite", "rb") as f:
    tflite_model = f.read()

# Convert binary data to C array format
hex_data = binascii.hexlify(tflite_model).decode('utf-8')
hex_array = [hex_data[i:i+2] for i in range(0, len(hex_data), 2)]
formatted_hex_array = ", ".join("0x" + hex_array[i] for i in range(len(hex_array)))

# Generate the header file
header_content = f"const unsigned char model[] = "

with open("model.h", "w") as f:
    f.write(header_content)

# Display file size
model_h_size = os.path.getsize("model.h")
print(f"Header file, model.h, is {model_h_size:,} bytes.")
print("\nOpen the side panel (refresh if needed). Double click model.h to download the file.")

Generating the model file

Here is a part of the message displayed when generating the model file. 

Epoch 102/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.4297e-06 - mae: 6.0069e-04 - val_loss: 1.5543e-05 - val_mae: 0.0015
Epoch 103/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.4089e-06 - mae: 5.9623e-04 - val_loss: 1.5292e-05 - val_mae: 0.0015
Epoch 104/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.3848e-06 - mae: 5.9148e-04 - val_loss: 1.5185e-05 - val_mae: 0.0015
Epoch 105/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.3646e-06 - mae: 5.8718e-04 - val_loss: 1.5039e-05 - val_mae: 0.0015
Epoch 106/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.3453e-06 - mae: 5.8297e-04 - val_loss: 1.4922e-05 - val_mae: 0.0015
Epoch 107/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.3258e-06 - mae: 5.7882e-04 - val_loss: 1.4777e-05 - val_mae: 0.0015
Epoch 108/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.3079e-06 - mae: 5.7482e-04 - val_loss: 1.4559e-05 - val_mae: 0.0015
Epoch 109/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.2872e-06 - mae: 5.7059e-04 - val_loss: 1.4463e-05 - val_mae: 0.0014
Epoch 110/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.2697e-06 - mae: 5.6672e-04 - val_loss: 1.4332e-05 - val_mae: 0.0014
Epoch 111/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.2527e-06 - mae: 5.6288e-04 - val_loss: 1.4231e-05 - val_mae: 0.0014
Epoch 112/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.2360e-06 - mae: 5.5917e-04 - val_loss: 1.4098e-05 - val_mae: 0.0014
Epoch 113/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.2202e-06 - mae: 5.5553e-04 - val_loss: 1.3903e-05 - val_mae: 0.0014
Epoch 114/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 1.2021e-06 - mae: 5.5168e-04 - val_loss: 1.3817e-05 - val_mae: 0.0014
Epoch 115/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.1867e-06 - mae: 5.4817e-04 - val_loss: 1.3699e-05 - val_mae: 0.0014
Epoch 116/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 1.1718e-06 - mae: 5.4468e-04 - val_loss: 1.3606e-05 - val_mae: 0.0014
Epoch 117/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.1571e-06 - mae: 5.4131e-04 - val_loss: 1.3480e-05 - val_mae: 0.0014
Epoch 118/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.1425e-06 - mae: 5.3786e-04 - val_loss: 1.3388e-05 - val_mae: 0.0014
Epoch 119/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.1288e-06 - mae: 5.3467e-04 - val_loss: 1.3192e-05 - val_mae: 0.0014
Epoch 594/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.4864e-07 - mae: 1.9593e-04 - val_loss: 3.5639e-06 - val_mae: 6.2397e-04
Epoch 595/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.4836e-07 - mae: 1.9574e-04 - val_loss: 3.5608e-06 - val_mae: 6.2356e-04
Epoch 596/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 1.4807e-07 - mae: 1.9555e-04 - val_loss: 3.5564e-06 - val_mae: 6.2309e-04
Epoch 597/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.4779e-07 - mae: 1.9536e-04 - val_loss: 3.5532e-06 - val_mae: 6.2269e-04
Epoch 598/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.4752e-07 - mae: 1.9518e-04 - val_loss: 3.5453e-06 - val_mae: 6.2207e-04
Epoch 599/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.4720e-07 - mae: 1.9498e-04 - val_loss: 3.5421e-06 - val_mae: 6.2168e-04
Epoch 600/600
33/33 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 1.4694e-07 - mae: 1.9480e-04 - val_loss: 3.5388e-06 - val_mae: 6.2126e-04
WARNING:absl:Please consider providing the trackable_obj argument in the from_concrete_functions. Providing without the trackable_obj argument is deprecated and it will use the deprecated conversion path.
2024-06-22 15:57:40.945975: I tensorflow/core/grappler/devices.cc:75] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2024-06-22 15:57:40.946332: I tensorflow/core/grappler/clusters/single_machine.cc:361] Starting new session
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
W0000 00:00:1719039460.974008    8296 tf_tfl_flatbuffer_helpers.cc:390] Ignored output_format.
W0000 00:00:1719039460.974389    8296 tf_tfl_flatbuffer_helpers.cc:393] Ignored drop_control_dependency.
Model successfully converted to TensorFlow Lite format.
Model saved to disk successfully.
Header file, model.h, is 889,715 bytes.

Here is a part of the contents of the created "model.h" file. It's hard to understand, isn't it?

const unsigned char model[] = {
0x1c, 0x00, 0x00, 0x00, 0x54, 0x46, 0x4c, 0x33, 0x14, 0x00, 0x20, 0x00, 0x1c, 0x00, 0x18, 0x00, 0x14, 0x00, 0x10, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x08, 0x00, 0x04, 0x00, 0x14, 0x00, 0x00, 0x00, 0x1c, 0x00, 0x00, 0x00, 0x1c, 0x00, 0x00, 0x00, 0x74, 0x00, 0x00, 0x00, 0x9c, 0x3e, 0x02, 0x00, 0xac, 0x3e, 0x02, 0x00, 0x14, 0x44, 0x02, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x34, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0xdc, 0xff, 0xff, 0xff, 0x0d, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x13, 0x00, 0x00, 0x00, 0x43, 0x4f, 0x4e, 0x56, 0x45, 0x52, 0x53, 0x49, 0x4f, 0x4e, 0x5f, 0x4d, 0x45, 0x54, 0x41, 0x44, 0x41, 0x54, 0x41, 0x00, 0x08, 0x00, 0x0c, 0x00, 0x08, 0x00, 0x04, 0x00, 0x08, 0x00, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x13, 0x00, 0x00, 0x00, 0x6d, 0x69, 0x6e, 0x5f, 0x72, 0x75, 0x6e, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x00, 0x0e, 0x00, 0x00, 0x00, 0x24, 0x3e, 0x02, 0x00, 0x1c, 0x3e, 0x02, 0x00, 0xb4, 0x3b, 0x02, 0x00, 0xec, 0x2f, 0x02, 0x00, 0xb4, 0x2f, 0x02, 0x00, 0x68, 0x2f, 0x02, 0x00, 0x90, 0x2e, 0x02, 0x00, 0xb0, 0x00, 0x00, 0x00, 0xa8, 0x00, 0x00, 0x00, 0xa0, 0x00, 0x00, 0x00, 0x98, 0x00, 0x00, 0x00, 0x90, 0x00, 0x00, 0x00, 0x70, 0x00,

Inference

Incorporate the generated header file into the following code to perform swing type determination.

Arduino IDE -> File -> Examples -> Seeed_Arduino_LSM6DS3 ->IMU_Classifier 

#include <LSM6DS3.h>
#include <Wire.h>

#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_error_reporter.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>
//#include <tensorflow/lite/version.h>

#include "model.h"

// const float accelerationThreshold = 2.5; // threshold of significant in G's
const float accelerationThreshold = 5.5; // threshold of significant in G's
const int numSamples = 119;

int samplesRead = numSamples;

LSM6DS3 myIMU(I2C_MODE, 0x6A);  

// global variables used for TensorFlow Lite (Micro)
tflite::MicroErrorReporter tflErrorReporter;

// pull in all the TFLM ops, you can remove this line and
// only pull in the TFLM ops you need, if would like to reduce
// the compiled size of the sketch.
tflite::AllOpsResolver tflOpsResolver;

const tflite::Model* tflModel = nullptr;
tflite::MicroInterpreter* tflInterpreter = nullptr;
TfLiteTensor* tflInputTensor = nullptr;
TfLiteTensor* tflOutputTensor = nullptr;

// Create a static memory buffer for TFLM, the size may need to
// be adjusted based on the model you are using
constexpr int tensorArenaSize = 8 * 1024;
byte tensorArena[tensorArenaSize] __attribute__((aligned(16)));

// array to map gesture index to a name
//const char* GESTURES[] = {
//  "punch",
//  "flex"
//};

const char* GESTURES[] = {
  "Smash L",
  "Forehand net L",
  "Forehand drive L",
  "Forehand lift L"
};

#define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))

void setup() {
  Serial.begin(9600);
  while (!Serial);

  if (myIMU.begin() != 0) {
    Serial.println("Device error");
  } else {
    Serial.println("Device OK!");
  }

  Serial.println();

  // get the TFL representation of the model byte array
  tflModel = tflite::GetModel(model);
  if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("Model schema mismatch!");
    while (1);
  }

  // Create an interpreter to run the model
  tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize, &tflErrorReporter);

  // Allocate memory for the model's input and output tensors
  tflInterpreter->AllocateTensors();

  // Get pointers for the model's input and output tensors
  tflInputTensor = tflInterpreter->input(0);
  tflOutputTensor = tflInterpreter->output(0);
}

void loop() {
  float aX, aY, aZ, gX, gY, gZ;

  // wait for significant motion
  while (samplesRead == numSamples) {
      // read the acceleration data
      aX = myIMU.readFloatAccelX();
      aY = myIMU.readFloatAccelY();
      aZ = myIMU.readFloatAccelZ();

      // sum up the absolutes
      float aSum = fabs(aX) + fabs(aY) + fabs(aZ);

      // check if it's above the threshold
      if (aSum >= accelerationThreshold) {
        // reset the sample read count
        samplesRead = 0;
        break;
      }
  }

  // check if the all the required samples have been read since
  // the last time the significant motion was detected
  while (samplesRead < numSamples) {
    // check if new acceleration AND gyroscope data is available
      // read the acceleration and gyroscope data
      aX = myIMU.readFloatAccelX();
      aY = myIMU.readFloatAccelY();
      aZ = myIMU.readFloatAccelZ();

      gX = myIMU.readFloatGyroX();
      gY = myIMU.readFloatGyroY();
      gZ = myIMU.readFloatGyroZ();

      // normalize the IMU data between 0 to 1 and store in the model's
      // input tensor
      tflInputTensor->data.f[samplesRead * 6 + 0] = (aX + 4.0) / 8.0;
      tflInputTensor->data.f[samplesRead * 6 + 1] = (aY + 4.0) / 8.0;
      tflInputTensor->data.f[samplesRead * 6 + 2] = (aZ + 4.0) / 8.0;
      tflInputTensor->data.f[samplesRead * 6 + 3] = (gX + 2000.0) / 4000.0;
      tflInputTensor->data.f[samplesRead * 6 + 4] = (gY + 2000.0) / 4000.0;
      tflInputTensor->data.f[samplesRead * 6 + 5] = (gZ + 2000.0) / 4000.0;

      samplesRead++;

      if (samplesRead == numSamples) {
        // Run inferencing
        TfLiteStatus invokeStatus = tflInterpreter->Invoke();
        if (invokeStatus != kTfLiteOk) {
          Serial.println("Invoke failed!");
          while (1);
          return;
        }

        // Loop through the output tensor values from the model
        for (int i = 0; i < NUM_GESTURES; i++) {
          Serial.print(GESTURES[i]);
          Serial.print(": ");
          Serial.println(tflOutputTensor->data.f[i], 6);
        }
        Serial.println();
      }
  }
}

The initial determination results using the first header file created were very low. Upon checking the swing data obtained during the header file creation, it was found that the cause was the sensor being too sensitive due to inappropriate threshold settings. As a result, the start and end of the swing could not be accurately determined, and it seemed that multiple sets of swing data were being captured for a single swing. To find the optimal threshold, the threshold value in IMUCapture.ino was gradually increased and tests were repeated, revealing that 5.5 was an appropriate value. In hopes of even a slight improvement in determination accuracy, a header file was recreated by consciously performing swings with a focus on the starting and ending points for each practice swing.

alt text
It's not perfect, but it has become almost accurate in determining the types of swings.

Velocity calculation

Next, calculate the speed for each swing.
Below is the code for adding a speed calculation feature to the IMU_Classifier. While I also performed speed calculations with the sensor in Assignment 11, this time, the calculation results varied significantly, with extremely low and high values, making it very time-consuming to obtain stable results.

Upon investigation, I found that noise removal and sensor calibration are effective for obtaining stable results, so I used ChatGPT to add the following features:

  • Static sensor calibration
    Sample:
    void calibrate() {
  float offsetX = 0, offsetY = 0, offsetZ = 0;
  const int samples = 100;

  for (int i = 0; i < samples; i++) {
    sensors_event_t event;
    lis.getEvent(&event);
    offsetX += event.acceleration.x;
    offsetY += event.acceleration.y;
    offsetZ += event.acceleration.z;
    delay(10);
  }

  offsetX /= samples;
  offsetY /= samples;
  offsetZ = (offsetZ / samples) - 9.81; // Z軸のオフセットに重力を考慮

  Serial.print("Offset X: "); Serial.println(offsetX);
  Serial.print("Offset Y: "); Serial.println(offsetY);
  Serial.print("Offset Z: "); Serial.println(offsetZ);
}

void setup() {
  Serial.begin(115200);
  if (!lis.begin(0x18)) {
    Serial.println("Could not start LIS3DH");
    while (1);
  }
  lis.setRange(LIS3DH_RANGE_2_G);
  calibrate();
}

  • Low-pass filter
    Sample:

    const float cutoff_freq = 5.0; // Cutoff frequency (Hz)
float rc = 1.0 / (2 * PI * cutoff_freq);
float dt_filter = 0.0;
float aX_filtered = 0.0, aY_filtered = 0.0, aZ_filtered = 0.0;

void applyLowPassFilter(float& input, float& output) {
  output = (dt_filter / (rc + dt_filter)) * output + (rc / (rc + dt_filter)) * input;
}

// Usage in the loop function
applyLowPassFilter(aX, aX_filtered);
applyLowPassFilter(aY, aY_filtered);
applyLowPassFilter(aZ, aZ_filtered);

  • Moving average filter
    Sample:

    float movingAverage(float* buffer, float newValue) {
  buffer[filterIndex] = newValue; // Add new sample to the buffer
  float sum = 0.0;
  for (int i = 0; i < filterSize; i++) {
    sum += buffer[i]; // Sum all samples in the buffer
  }
  filterIndex = (filterIndex + 1) % filterSize; // Move to the next index
  return sum / filterSize; // Return the average value
}

Modified IMU_Classifier

#include <LSM6DS3.h>
#include <Wire.h>
#include <math.h>

#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_error_reporter.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>
#include "model.h"

// Threshold for significant acceleration in G's
const float accelerationThreshold = 5.5;
const int numSamples = 119;

int samplesRead = 0; // Number of samples read
bool inSwing = false; // Flag to indicate if a swing is in progress

LSM6DS3 myIMU(I2C_MODE, 0x6A); // Initialize the IMU sensor

// Global variables used for TensorFlow Lite (Micro)
tflite::MicroErrorReporter tflErrorReporter;
tflite::AllOpsResolver tflOpsResolver;

const tflite::Model* tflModel = nullptr; // Pointer to the model
tflite::MicroInterpreter* tflInterpreter = nullptr; // Pointer to the interpreter
TfLiteTensor* tflInputTensor = nullptr; // Pointer to the input tensor
TfLiteTensor* tflOutputTensor = nullptr; // Pointer to the output tensor

// Define the size of the tensor arena
constexpr int tensorArenaSize = 8 * 1024;
// Define the tensor arena (aligned to 16 bytes)
byte tensorArena[tensorArenaSize] __attribute__((aligned(16)));

// Array to map gesture index to a name
const char* GESTURES[] = {
  "Smash L",
  "Forehand net L",
  "Forehand drive L",
  "Forehand lift L"
};

#define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))

// Parameters for moving average filter
const int filterSize = 10;
float aX_buffer[filterSize] = {0};
float aY_buffer[filterSize] = {0};
float aZ_buffer[filterSize] = {0};
int filterIndex = 0;

float velocity[3] = {0.0, 0.0, 0.0}; // Initial velocity
float accel_scale = 16384.0; // Accelerometer scale factor
float g = 9.80665; // Acceleration due to gravity in m/s^2
unsigned long lastTime = 0; // Last recorded time
float aX_offset = 0.0, aY_offset = 0.0, aZ_offset = 0.0; // Offset values
const int calibrationSamples = 100; // Number of samples for calibration

// Complementary filter parameters
const float alpha = 0.98; // Adjust this value based on sensor noise

// Variables for angle calculation
float pitch = 0.0;
float roll = 0.0;

// Low-pass filter parameters
const float cutoff_freq = 5.0; // Cutoff frequency (Hz)
float rc = 1.0 / (2 * PI * cutoff_freq);
float dt_filter = 0.0;
float aX_filtered = 0.0, aY_filtered = 0.0, aZ_filtered = 0.0;
float gX_filtered = 0.0, gY_filtered = 0.0, gZ_filtered = 0.0;

// Array to store swing data
float swingData[1000][6];
int swingDataIndex = 0;

// Function to calibrate the sensor
void calibrateSensor() {
  float aX, aY, aZ;
  for (int i = 0; i < calibrationSamples; i++) {
    aX = myIMU.readFloatAccelX();
    aY = myIMU.readFloatAccelY();
    aZ = myIMU.readFloatAccelZ();
    aX_offset += aX;
    aY_offset += aY;
    aZ_offset += aZ;
    delay(10);
  }
  aX_offset /= calibrationSamples;
  aY_offset /= calibrationSamples;
  aZ_offset /= calibrationSamples;
}

// Setup function
void setup() {
  Serial.begin(9600);
  while (!Serial);

  if (myIMU.begin() != 0) {
    Serial.println("Device error");
  } else {
    Serial.println("Device OK!");
  }

  Serial.println();

  calibrateSensor(); // Calibrate the sensor

  // Get the TFL representation of the model byte array
  tflModel = tflite::GetModel(model);
  if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("Model schema mismatch!");
    while (1);
  }

  // Create an interpreter to run the model
  tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize, &tflErrorReporter);

  // Allocate memory for the model's input and output tensors
  tflInterpreter->AllocateTensors();

  // Get pointers for the model's input and output tensors
  tflInputTensor = tflInterpreter->input(0);
  tflOutputTensor = tflInterpreter->output(0);

  lastTime = millis(); // Set the initial time in milliseconds
}

// Moving average filter function
float movingAverage(float* buffer, float newValue) {
  buffer[filterIndex] = newValue;
  float sum = 0.0;
  for (int i = 0; i < filterSize; i++) {
    sum += buffer[i];
  }
  filterIndex = (filterIndex + 1) % filterSize;
  return sum / filterSize;
}

// Function to calculate velocity from saved swing data
void calculateVelocity() {
  float dt = 0.0084; // Sampling rate (119 Hz)
  for (int i = 1; i < swingDataIndex; i++) {
    float aX_world = swingData[i][0] * cos(pitch * PI / 180.0) + swingData[i][1] * sin(pitch * PI / 180.0) * sin(roll * PI / 180.0) + swingData[i][2] * cos(roll * PI / 180.0) * sin(pitch * PI / 180.0);
    float aY_world = swingData[i][1] * cos(roll * PI / 180.0) - swingData[i][2] * sin(roll * PI / 180.0);
    float aZ_world = -swingData[i][0] * sin(pitch * PI / 180.0) + swingData[i][1] * cos(pitch * PI / 180.0) * sin(roll * PI / 180.0) + swingData[i][2] * cos(pitch * PI / 180.0) * cos(roll * PI / 180.0) - g;

    // Update velocity using trapezoidal integration
    velocity[0] += aX_world * dt;
    velocity[1] += aY_world * dt;
    velocity[2] += aZ_world * dt;
  }

  // Apply a scaling factor to adjust the velocity
  float scalingFactor = 2.0; // This factor can be adjusted based on observed results
  velocity[0] *= scalingFactor;
  velocity[1] *= scalingFactor;
  velocity[2] *= scalingFactor;

  // Calculate speed in km/h
  float speed_ms = sqrt(velocity[0] * velocity[0] + velocity[1] * velocity[1] + velocity[2] * velocity[2]);
  float speed_kmh = speed_ms * 3.6;
  Serial.print("Estimated speed: ");
  Serial.print(speed_kmh, 2);
  Serial.println(" km/h");

  // Reset velocity and swing data index
  velocity[0] = 0.0;
  velocity[1] = 0.0;
  velocity[2] = 0.0;
  swingDataIndex = 0;
}

// Main loop function
void loop() {
  float aX, aY, aZ, gX, gY, gZ;
  unsigned long currentTime = millis();
  float dt = (currentTime - lastTime) / 1000.0; // Convert to seconds
  lastTime = currentTime;

  dt_filter = dt; // Set dt for low-pass filter

  // Read the acceleration and gyroscope data
  aX = myIMU.readFloatAccelX() - aX_offset;
  aY = myIMU.readFloatAccelY() - aY_offset;
  aZ = myIMU.readFloatAccelZ() - aZ_offset;

  gX = myIMU.readFloatGyroX();
  gY = myIMU.readFloatGyroY();
  gZ = myIMU.readFloatGyroZ();

  // Apply moving average filter
  aX = movingAverage(aX_buffer, aX);
  aY = movingAverage(aY_buffer, aY);
  aZ = movingAverage(aZ_buffer, aZ);

  // Apply low-pass filter
  aX_filtered = (dt_filter / (rc + dt_filter)) * aX_filtered + (rc / (rc + dt_filter)) * aX;
  aY_filtered = (dt_filter / (rc + dt_filter)) * aY_filtered + (rc / (rc + dt_filter)) * aY;
  aZ_filtered = (dt_filter / (rc + dt_filter)) * aZ_filtered + (rc / (rc + dt_filter)) * aZ;
  gX_filtered = (dt_filter / (rc + dt_filter)) * gX_filtered + (rc / (rc + dt_filter)) * gX;
  gY_filtered = (dt_filter / (rc + dt_filter)) * gY_filtered + (rc / (rc + dt_filter)) * gY;
  gZ_filtered = (dt_filter / (rc + dt_filter)) * gZ_filtered + (rc / (rc + dt_filter)) * gZ;

  // Calculate pitch and roll from accelerometer data
  float pitchAcc = atan2(aY_filtered, sqrt(aX_filtered * aX_filtered + aZ_filtered * aZ_filtered)) * 180.0 / PI;
  float rollAcc = atan2(-aX_filtered, aZ_filtered) * 180.0 / PI;

  // Integrate gyro data to get angle
  pitch = pitch + gY_filtered * dt;
  roll = roll + gX_filtered * dt;

  // Apply complementary filter
  pitch = alpha * pitch + (1 - alpha) * pitchAcc;
  roll = alpha * roll + (1 - alpha) * rollAcc;

  // Detect swing start
  if (!inSwing && fabs(aX) + fabs(aY) + fabs(aZ) >= accelerationThreshold) {
    inSwing = true;
    samplesRead = 0;
    velocity[0] = 0.0;
    velocity[1] = 0.0;
    velocity[2] = 0.0;
    swingDataIndex = 0;
  }

  if (inSwing && samplesRead < numSamples) {
    // Save swing data
    if (swingDataIndex < 1000) {
      swingData[swingDataIndex][0] = aX_filtered;
      swingData[swingDataIndex][1] = aY_filtered;
      swingData[swingDataIndex][2] = aZ_filtered;
      swingData[swingDataIndex][3] = gX_filtered;
      swingData[swingDataIndex][4] = gY_filtered;
      swingData[swingDataIndex][5] = gZ_filtered;
      swingDataIndex++;
    }

    // Normalize the IMU data between 0 to 1 and store in the model's input tensor
    tflInputTensor->data.f[samplesRead * 6 + 0] = (aX + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 1] = (aY + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 2] = (aZ + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 3] = (gX + 2000.0) / 4000.0;
    tflInputTensor->data.f[samplesRead * 6 + 4] = (gY + 2000.0) / 4000.0;
    tflInputTensor->data.f[samplesRead * 6 + 5] = (gZ + 2000.0) / 4000.0;

    samplesRead++;
  }

  // Detect swing end and run inference
  if (samplesRead == numSamples) {
    inSwing = false;

    // Run inferencing
    TfLiteStatus invokeStatus = tflInterpreter->Invoke();
    if (invokeStatus != kTfLiteOk) {
      Serial.println("Invoke failed!");
      while (1);
    }

    // Loop through the output tensor values from the model
    for (int i = 0; i < NUM_GESTURES; i++) {
      Serial.print(GESTURES[i]);
      Serial.print(": ");
      Serial.println(tflOutputTensor->data.f[i], 6);
    }

    // Calculate and print the velocity
    calculateVelocity();

    // Reset samplesRead for the next swing
    samplesRead = 0;
  }
}

It still needs adjustments, but I am now able to obtain stable results.

alt text

BLE

I will combine the BLE communication code created in Assignment 14 with the Modified IMU_Classifier code.

Modified IMU_Classifier w/BLE

#include <LSM6DS3.h>
#include <Wire.h>
#include <math.h>
#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_error_reporter.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>
#include "model.h"
#include <bluefruit.h> // Include the Bluefruit library

const bool DEBUG_MODE = true; // Set this to false to disable serial output

const float accelerationThreshold = 5.5; // threshold of significant in G's
const int numSamples = 119;

int samplesRead = 0;
bool inSwing = false;

LSM6DS3 myIMU(I2C_MODE, 0x6A);

// global variables used for TensorFlow Lite (Micro)
tflite::MicroErrorReporter tflErrorReporter;
tflite::AllOpsResolver tflOpsResolver;

const tflite::Model* tflModel = nullptr;
tflite::MicroInterpreter* tflInterpreter = nullptr;
TfLiteTensor* tflInputTensor = nullptr;
TfLiteTensor* tflOutputTensor = nullptr;

constexpr int tensorArenaSize = 8 * 1024;
byte tensorArena[tensorArenaSize] __attribute__((aligned(16)));

// array to map gesture index to a name
const char* GESTURES[] = {
  "Smash L",
  "Forehand net L",
  "Forehand drive L",
  "Forehand lift L"
};

#define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))

// Parameters for moving average filter
const int filterSize = 10;
float aX_buffer[filterSize] = {0};
float aY_buffer[filterSize] = {0};
float aZ_buffer[filterSize] = {0};
int filterIndex = 0;

float velocity[3] = {0.0, 0.0, 0.0};  // Initial velocity
float accel_scale = 16384.0;  // Accelerometer scale factor
float g = 9.80665;  // Acceleration due to gravity in m/s^2
unsigned long lastTime = 0;
float aX_offset = 0.0, aY_offset = 0.0, aZ_offset = 0.0;
const int calibrationSamples = 100;

// Complementary filter parameters
const float alpha = 0.98; // Adjust this value based on sensor noise

// Variables for angle calculation
float pitch = 0.0;
float roll = 0.0;

// Low-pass filter parameters
const float cutoff_freq = 5.0; // Cutoff frequency (Hz)
float rc = 1.0 / (2 * PI * cutoff_freq);
float dt_filter = 0.0;
float aX_filtered = 0.0, aY_filtered = 0.0, aZ_filtered = 0.0;
float gX_filtered = 0.0, gY_filtered = 0.0, gZ_filtered = 0.0;

// List to store swing data
float swingData[1000][6];
int swingDataIndex = 0;

// BLE UART service
BLEUart bleuart;

// Define the DebugLog function
extern "C" void DebugLog(const char* s) {
  if (DEBUG_MODE) {
    Serial.println(s);
  }
}

// Define the test_over_serial namespace functions
namespace test_over_serial {
  void SerialWrite(const char* s) {
    if (DEBUG_MODE) {
      Serial.print(s);
    }
  }

  int SerialReadLine(int timeout_ms) {
    // Add implementation for reading a line from Serial
    return 0;
  }
}

void calibrateSensor() {
  float aX, aY, aZ;
  for (int i = 0; i < calibrationSamples; i++) {
    aX = myIMU.readFloatAccelX();
    aY = myIMU.readFloatAccelY();
    aZ = myIMU.readFloatAccelZ();
    aX_offset += aX;
    aY_offset += aY;
    aZ_offset += aZ;
    delay(10);
  }
  aX_offset /= calibrationSamples;
  aY_offset /= calibrationSamples;
  aZ_offset /= calibrationSamples;
}

void setup() {
  if (DEBUG_MODE) {
    Serial.begin(9600);
    while (!Serial);
  }

  if (myIMU.begin() != 0) {
    if (DEBUG_MODE) {
      Serial.println("Device error");
    }
  } else {
    if (DEBUG_MODE) {
      Serial.println("Device OK!");
    }
  }

  if (DEBUG_MODE) {
    Serial.println();
  }

  calibrateSensor();

  // get the TFL representation of the model byte array
  tflModel = tflite::GetModel(model);
  if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
    if (DEBUG_MODE) {
      Serial.println("Model schema mismatch!");
    }
    while (1);
  }

  // Create an interpreter to run the model
  tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize, &tflErrorReporter);

  // Allocate memory for the model's input and output tensors
  tflInterpreter->AllocateTensors();

  // Get pointers for the model's input and output tensors
  tflInputTensor = tflInterpreter->input(0);
  tflOutputTensor = tflInterpreter->output(0);

  lastTime = millis();  // Set the initial time in milliseconds

  // Initialize BLE
  Bluefruit.begin();
  Bluefruit.setTxPower(4); // Set maximum transmission power
  Bluefruit.setName("IMU_Classifier");

  // Start BLE UART service
  bleuart.begin();

  // Advertising setup and start
  Bluefruit.Advertising.addFlags(BLE_GAP_ADV_FLAGS_LE_ONLY_GENERAL_DISC_MODE);
  Bluefruit.Advertising.addTxPower();
  Bluefruit.Advertising.addService(bleuart); // Advertising UART service
  Bluefruit.Advertising.addName();
  Bluefruit.Advertising.restartOnDisconnect(true);
  Bluefruit.Advertising.start(0); // 0 = Continue advertising without timeout
}

float movingAverage(float* buffer, float newValue) {
  buffer[filterIndex] = newValue;
  float sum = 0.0;
  for (int i = 0; i < filterSize; i++) {
    sum += buffer[i];
  }
  filterIndex = (filterIndex + 1) % filterSize;
  return sum / filterSize;
}

void calculateVelocity() {
  // Calculate velocity from saved swing data
  float dt = 0.0084; // Sampling rate (119 Hz)
  for (int i = 1; i < swingDataIndex; i++) {
    float aX_world = swingData[i][0] * cos(pitch * PI / 180.0) + swingData[i][1] * sin(pitch * PI / 180.0) * sin(roll * PI / 180.0) + swingData[i][2] * cos(roll * PI / 180.0) * sin(pitch * PI / 180.0);
    float aY_world = swingData[i][1] * cos(roll * PI / 180.0) - swingData[i][2] * sin(roll * PI / 180.0);
    float aZ_world = -swingData[i][0] * sin(pitch * PI / 180.0) + swingData[i][1] * cos(pitch * PI / 180.0) * sin(roll * PI / 180.0) + swingData[i][2] * cos(pitch * PI / 180.0) * cos(roll * PI / 180.0) - g;

    // Update velocity using trapezoidal integration
    velocity[0] += aX_world * dt;
    velocity[1] += aY_world * dt;
    velocity[2] += aZ_world * dt;
  }

  // Apply a scaling factor to adjust the velocity
  float scalingFactor = 2.0; // This factor can be adjusted based on observed results
  velocity[0] *= scalingFactor;
  velocity[1] *= scalingFactor;
  velocity[2] *= scalingFactor;

  // Calculate speed in km/h
  float speed_ms = sqrt(velocity[0] * velocity[0] + velocity[1] * velocity[1] + velocity[2] * velocity[2]);
  float speed_kmh = speed_ms * 3.6;
  if (DEBUG_MODE) {
    Serial.print("Estimated speed: ");
    Serial.print(speed_kmh, 2);
    Serial.println(" km/h");
  }

  // Send speed via BLE
  char buffer[32];
  snprintf(buffer, sizeof(buffer), "Speed: %.2f km/h", speed_kmh);
  bleuart.write(buffer);

  // Reset velocity and swing data index
  velocity[0] = 0.0;
  velocity[1] = 0.0;
  velocity[2] = 0.0;
  swingDataIndex = 0;
}

void loop() {
  float aX, aY, aZ, gX, gY, gZ;
  unsigned long currentTime = millis();
  float dt = (currentTime - lastTime) / 1000.0;  // Convert to seconds
  lastTime = currentTime;

  dt_filter = dt; // Set dt for low-pass filter

  // Read the acceleration and gyroscope data
  aX = myIMU.readFloatAccelX() - aX_offset;
  aY = myIMU.readFloatAccelY() - aY_offset;
  aZ = myIMU.readFloatAccelZ() - aZ_offset;

  gX = myIMU.readFloatGyroX();
  gY = myIMU.readFloatGyroY();
  gZ = myIMU.readFloatGyroZ();

  // Apply moving average filter
  aX = movingAverage(aX_buffer, aX);
  aY = movingAverage(aY_buffer, aY);
  aZ = movingAverage(aZ_buffer, aZ);

  // Apply low-pass filter
  aX_filtered = (dt_filter / (rc + dt_filter)) * aX_filtered + (rc / (rc + dt_filter)) * aX;
  aY_filtered = (dt_filter / (rc + dt_filter)) * aY_filtered + (rc / (rc + dt_filter)) * aY;
  aZ_filtered = (dt_filter / (rc + dt_filter)) * aZ_filtered + (rc / (rc + dt_filter)) * aZ;
  gX_filtered = (dt_filter / (rc + dt_filter)) * gX_filtered + (rc / (rc + dt_filter)) * gX;
  gY_filtered = (dt_filter / (rc + dt_filter)) * gY_filtered + (rc / (rc + dt_filter)) * gY;
  gZ_filtered = (dt_filter / (rc + dt_filter)) * gZ_filtered + (rc / (rc + dt_filter)) * gZ;

  // Calculate pitch and roll from accelerometer data
  float pitchAcc = atan2(aY_filtered, sqrt(aX_filtered * aX_filtered + aZ_filtered * aZ_filtered)) * 180.0 / PI;
  float rollAcc = atan2(-aX_filtered, aZ_filtered) * 180.0 / PI;

  // Integrate gyro data to get angle
  pitch = pitch + gY_filtered * dt;
  roll = roll + gX_filtered * dt;

  // Apply complementary filter
  pitch = alpha * pitch + (1 - alpha) * pitchAcc;
  roll = alpha * roll + (1 - alpha) * rollAcc;

  // Detect swing start
  if (!inSwing && fabs(aX) + fabs(aY) + fabs(aZ) >= accelerationThreshold) {
    inSwing = true;
    samplesRead = 0;
    velocity[0] = 0.0;
    velocity[1] = 0.0;
    velocity[2] = 0.0;
    swingDataIndex = 0;
  }

  if (inSwing && samplesRead < numSamples) {
    // Save swing data
    if (swingDataIndex < 1000) {
      swingData[swingDataIndex][0] = aX_filtered;
      swingData[swingDataIndex][1] = aY_filtered;
      swingData[swingDataIndex][2] = aZ_filtered;
      swingData[swingDataIndex][3] = gX_filtered;
      swingData[swingDataIndex][4] = gY_filtered;
      swingData[swingDataIndex][5] = gZ_filtered;
      swingDataIndex++;
    }

    // Normalize the IMU data between 0 to 1 and store in the model's input tensor
    tflInputTensor->data.f[samplesRead * 6 + 0] = (aX + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 1] = (aY + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 2] = (aZ + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 3] = (gX + 2000.0) / 4000.0;
    tflInputTensor->data.f[samplesRead * 6 + 4] = (gY + 2000.0) / 4000.0;
    tflInputTensor->data.f[samplesRead * 6 + 5] = (gZ + 2000.0) / 4000.0;

    samplesRead++;
  }

  // Detect swing end and run inference
  if (samplesRead == numSamples) {
    inSwing = false;

    // Run inferencing
    TfLiteStatus invokeStatus = tflInterpreter->Invoke();
    if (invokeStatus != kTfLiteOk) {
      if (DEBUG_MODE) {
        Serial.println("Invoke failed!");
      }
      while (1);
    }

    // Find the gesture with the highest confidence
    int maxIndex = 0;
    float maxConfidence = tflOutputTensor->data.f[0];
    for (int i = 1; i < NUM_GESTURES; i++) {
      if (tflOutputTensor->data.f[i] > maxConfidence) {
        maxConfidence = tflOutputTensor->data.f[i];
        maxIndex = i;
      }
    }

    // Print the detected gesture
    if (DEBUG_MODE) {
      Serial.print("Detected gesture: ");
      Serial.println(GESTURES[maxIndex]);
    }

    // Send gesture via BLE
    bleuart.write("Gesture: ");
    bleuart.write(GESTURES[maxIndex]);
    bleuart.write("\n");

    // Calculate and print the velocity
    calculateVelocity();

    // Reset samplesRead for the next swing
    samplesRead = 0;
  }
}
It worked well!

alt text

Here are the last two features to be added to Modified IMU_Classifier:

Sleep & Resume Function:
Pressing the button on the device will alternately activate the sleep and resume functions. This allows for reduced battery consumption.

Charging LED Controller:
When the XIAO nRF52840 is powered via the USB-C port, it automatically starts charging the attached battery. An additional LED will be lit to indicate that charging is in progress, addressing the issue of the onboard charging LED being hard to see.

alt text alt text 

#include <LSM6DS3.h>
#include <Wire.h>
#include <math.h>
#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_error_reporter.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>
#include "model.h"
#include <bluefruit.h> // Include the Bluefruit library

#define BUTTON_PIN D2       // Define the button pin (D2)
#define LED_PIN LED_BUILTIN // Define the LED pin (built-in LED)

const bool DEBUG_MODE = true; // Set this to false to disable serial output
const float accelerationThreshold = 5.5; // Threshold of significant acceleration in G's
const int numSamples = 119; // Number of samples to read

int samplesRead = 0;
bool inSwing = false;
volatile bool sleepMode = false; // Flag to manage sleep mode
volatile bool buttonPressed = false; // Flag to indicate if
volatile bool buttonPressed = false; // Flag to indicate if the button was pressed

LSM6DS3 myIMU(I2C_MODE, 0x6A); // Initialize the IMU sensor

// Global variables used for TensorFlow Lite (Micro)
tflite::MicroErrorReporter tflErrorReporter;
tflite::AllOpsResolver tflOpsResolver;

const tflite::Model* tflModel = nullptr;
tflite::MicroInterpreter* tflInterpreter = nullptr;
TfLiteTensor* tflInputTensor = nullptr;
TfLiteTensor* tflOutputTensor = nullptr;

constexpr int tensorArenaSize = 8 * 1024;
byte tensorArena[tensorArenaSize] __attribute__((aligned(16)));

// Array to map gesture index to a name
const char* GESTURES[] = {
  "Smash L",
  "Forehand net L",
  "Forehand drive L",
  "Forehand lift L"
};

#define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))

// Parameters for moving average filter
const int filterSize = 10;
float aX_buffer[filterSize] = {0};
float aY_buffer[filterSize] = {0};
float aZ_buffer[filterSize] = {0};
int filterIndex = 0;

float velocity[3] = {0.0, 0.0, 0.0};  // Initial velocity
float accel_scale = 16384.0;  // Accelerometer scale factor
float g = 9.80665;  // Acceleration due to gravity in m/s^2
unsigned long lastTime = 0;
float aX_offset = 0.0, aY_offset = 0.0, aZ_offset = 0.0;
const int calibrationSamples = 100; // Number of samples for calibration

// Complementary filter parameters
const float alpha = 0.98; // Adjust this value based on sensor noise

// Variables for angle calculation
float pitch = 0.0;
float roll = 0.0;

// Low-pass filter parameters
const float cutoff_freq = 5.0; // Cutoff frequency (Hz)
float rc = 1.0 / (2 * PI * cutoff_freq);
float dt_filter = 0.0;
float aX_filtered = 0.0, aY_filtered = 0.0, aZ_filtered = 0.0;
float gX_filtered = 0.0, gY_filtered = 0.0, gZ_filtered = 0.0;

// List to store swing data
float swingData[1000][6];
int swingDataIndex = 0;

// BLE UART service
BLEUart bleuart;

// Define the DebugLog function
extern "C" void DebugLog(const char* s) {
  if (DEBUG_MODE) {
    Serial.println(s);
  }
}

// Define the test_over_serial namespace functions
namespace test_over_serial {
  void SerialWrite(const char* s) {
    if (DEBUG_MODE) {
      Serial.print(s);
    }
  }

  int SerialReadLine(int timeout_ms) {
    // Add implementation for reading a line from Serial
    return 0;
  }
}

// Charging state pin and LED
const int chargePin = 23; // P0.17 pin
const int chargeLED = 3;  // Pin connected to additional LED

void calibrateSensor() {
  // Calibrate the sensor by averaging the readings
  float aX, aY, aZ;
  for (int i = 0; i < calibrationSamples; i++) {
    aX = myIMU.readFloatAccelX();
    aY = myIMU.readFloatAccelY();
    aZ = myIMU.readFloatAccelZ();
    aX_offset += aX;
    aY_offset += aY;
    aZ_offset += aZ;
    delay(10);
  }
  aX_offset /= calibrationSamples;
  aY_offset /= calibrationSamples;
  aZ_offset /= calibrationSamples;
}

void buttonISR() {
  buttonPressed = true; // Set flag when button is pressed
}

void setup() {
  // Initialize serial communication
  if (DEBUG_MODE) {
    Serial.begin(9600);
    while (!Serial);
  }

  // Initialize LED and button pin
  pinMode(LED_PIN, OUTPUT);
  pinMode(BUTTON_PIN, INPUT_PULLUP);
  attachInterrupt(digitalPinToInterrupt(BUTTON_PIN), buttonISR, FALLING); // Attach interrupt to button

  // Flash LED to indicate setup
  digitalWrite(LED_PIN, HIGH);
  delay(1000);
  digitalWrite(LED_PIN, LOW);
  delay(1000);
  digitalWrite(LED_PIN, HIGH);
  delay(1000);
  digitalWrite(LED_PIN, LOW);

  // Initialize IMU sensor
  if (myIMU.begin() != 0) {
    if (DEBUG_MODE) {
      Serial.println("Device error");
    }
  } else {
    if (DEBUG_MODE) {
      Serial.println("Device OK!");
    }
  }

  calibrateSensor();

  // Get the TFL representation of the model byte array
  tflModel = tflite::GetModel(model);
  if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
    if (DEBUG_MODE) {
      Serial.println("Model schema mismatch!");
    }
    while (1);
  }

  // Create an interpreter to run the model
  tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize, &tflErrorReporter);

  // Allocate memory for the model's input and output tensors
  tflInterpreter->AllocateTensors();

  // Get pointers for the model's input and output tensors
  tflInputTensor = tflInterpreter->input(0);
  tflOutputTensor = tflInterpreter->output(0);

  lastTime = millis();  // Set the initial time in milliseconds

  // Initialize BLE
  Bluefruit.begin();
  Bluefruit.setTxPower(4); // Set maximum transmission power
  Bluefruit.setName("IMU_Classifier");

  // Start BLE UART service
  bleuart.begin();

  // Advertising setup and start
  Bluefruit.Advertising.addFlags(BLE_GAP_ADV_FLAGS_LE_ONLY_GENERAL_DISC_MODE);
  Bluefruit.Advertising.addTxPower();
  Bluefruit.Advertising.addService(bleuart); // Advertising UART service
  Bluefruit.Advertising.addName();
  Bluefruit.Advertising.restartOnDisconnect(true);
  Bluefruit.Advertising.start(0); // Continue advertising without timeout

  // Setup charge detection and LED
  pinMode(chargePin, INPUT_PULLUP);
  pinMode(chargeLED, OUTPUT);
  digitalWrite(chargeLED, LOW);  // Turn off LED initially

  if (DEBUG_MODE) {
    Serial.println("System initialized and running.");
  }
}

float movingAverage(float* buffer, float newValue) {
  // Compute the moving average
  buffer[filterIndex] = newValue;
  float sum = 0.0;
  for (int i = 0; i < filterSize; i++) {
    sum += buffer[i];
  }
  filterIndex = (filterIndex + 1) % filterSize;
  return sum / filterSize;
}

void calculateVelocity() {
  // Calculate velocity from saved swing data
  float dt = 0.0084; // Sampling rate (119 Hz)
  for (int i = 1; i < swingDataIndex; i++) {
    float aX_world = swingData[i][0] * cos(pitch * PI / 180.0) + swingData[i][1] * sin(pitch * PI / 180.0) * sin(roll * PI / 180.0) + swingData[i][2] * cos(roll * PI / 180.0) * sin(pitch * PI / 180.0);
    float aY_world = swingData[i][1] * cos(roll * PI / 180.0) - swingData[i][2] * sin(roll * PI / 180.0);
    float aZ_world = -swingData[i][0] * sin(pitch * PI / 180.0) + swingData[i][1] * cos(pitch * PI / 180.0) * sin(roll * PI / 180.0) + swingData[i][2] * cos(pitch * PI / 180.0) * cos(roll * PI / 180.0) - g;

    // Update velocity using trapezoidal integration
    velocity[0] += aX_world * dt;
    velocity[1] += aY_world * dt;
    velocity[2] += aZ_world * dt;
  }

  // Apply a scaling factor to adjust the velocity
  float scalingFactor = 2.0; // This factor can be adjusted based on observed results
  velocity[0] *= scalingFactor;
  velocity[1] *= scalingFactor;
  velocity[2] *= scalingFactor;

  // Calculate speed in km/h
  float speed_ms = sqrt(velocity[0] * velocity[0] + velocity[1] * velocity[1] + velocity[2] * velocity[2]);
  float speed_kmh = speed_ms * 3.6;
  if (DEBUG_MODE) {
    Serial.print("Estimated speed: ");
    Serial.print(speed_kmh, 2);
    Serial.println(" km/h");
  }

  // Send speed via BLE
  char buffer[32];
  snprintf(buffer, sizeof(buffer), "Speed: %.2f km/h", speed_kmh);
  bleuart.write(buffer);

  // Reset velocity and swing data index
  velocity[0] = 0.0;
  velocity[1] = 0.0;
  velocity[2] = 0.0;
  swingDataIndex = 0;
}

// Function to enter sleep mode
void enterSleepMode() {
  // Indicate entering sleep mode with LED blink
  digitalWrite(LED_PIN, LOW);
  delay(500);
  digitalWrite(LED_PIN, HIGH);
  delay(500);
  digitalWrite(LED_PIN, LOW);

  // Print message to serial monitor
  if (DEBUG_MODE) {
    Serial.println("Entering sleep mode...");
  }

  // Stop serial communication and BLE
  if (DEBUG_MODE) {
    Serial.end();
  }
  Bluefruit.Advertising.stop();
  Bluefruit.disconnect(0); // Disconnect all BLE connections

  // Configure the button to wake up the system
  nrf_gpio_cfg_sense_input(digitalPinToInterrupt(BUTTON_PIN), NRF_GPIO_PIN_PULLUP, NRF_GPIO_PIN_SENSE_LOW);

  // Enter System ON sleep mode
  __WFI(); // Wait for interrupt to wake up
}

// Function to wake up from sleep mode
void wakeup() {
  // Re-initialize serial communication
  if (DEBUG_MODE) {
    Serial.begin(9600);
    while (!Serial);
    Serial.println("Waking up...");
  }

  // Indicate waking up with LED blink
  digitalWrite(LED_PIN, HIGH);
  delay(1000);
  digitalWrite(LED_PIN, LOW);

  // Re-initialize BLE
  Bluefruit.begin();
  Bluefruit.setTxPower(4); // Set maximum transmission power
  Bluefruit.setName("BLE_Sample"); // Set device name

  // Restart BLE UART service
  bleuart.begin();

  // Restart BLE advertising
  Bluefruit.Advertising.addFlags(BLE_GAP_ADV_FLAGS_LE_ONLY_GENERAL_DISC_MODE);
  Bluefruit.Advertising.addTxPower();
  Bluefruit.Advertising.addService(bleuart); // Add BLE UART service to advertising
  Bluefruit.Advertising.addName();
  Bluefruit.Advertising.restartOnDisconnect(true);
  Bluefruit.Advertising.start(0); // Continue advertising without timeout
}

void loop() {
  // Check if the button was pressed
  if (buttonPressed) {
    buttonPressed = false; // Reset the flag
    sleepMode = !sleepMode; // Toggle sleep mode state

    if (sleepMode) {
      if (DEBUG_MODE) {
        Serial.println("Switching to sleep mode...");
      }
      enterSleepMode(); // Enter sleep mode
    } else {
      if (DEBUG_MODE) {
        Serial.println("Switching to wakeup mode...");
      }
      wakeup(); // Wake up from sleep mode
    }
  }

  if (!sleepMode) {
    // Check charging state
    int chargeState = digitalRead(chargePin);
    if (chargeState == LOW) { // LOW means charging (pull-up resistor in place)
      digitalWrite(chargeLED, HIGH); // Turn on LED to indicate charging
    } else {
      digitalWrite(chargeLED, LOW); // Turn off LED when not charging
    }

    float aX, aY, aZ, gX, gY, gZ;
    unsigned long currentTime = millis();
    float dt = (currentTime - lastTime) / 1000.0;  // Convert to seconds
    lastTime = currentTime;

    dt_filter = dt; // Set dt for low-pass filter

    // Read the acceleration and gyroscope data
    aX = myIMU.readFloatAccelX() - aX_offset;
    aY = myIMU.readFloatAccelY() - aY_offset;
    aZ = myIMU.readFloatAccelZ() - aZ_offset;

    gX = myIMU.readFloatGyroX();
    gY = myIMU.readFloatGyroY();
    gZ = myIMU.readFloatGyroZ();

    // Apply moving average filter
    aX = movingAverage(aX_buffer, aX);
    aY = movingAverage(aY_buffer, aY);
    aZ = movingAverage(aZ_buffer, aZ);

    // Apply low-pass filter
    aX_filtered = (dt_filter / (rc + dt_filter)) * aX_filtered + (rc / (rc + dt_filter)) * aX;
    aY_filtered = (dt_filter / (rc + dt_filter)) * aY_filtered + (rc / (rc + dt_filter)) * aY;
    aZ_filtered = (dt_filter / (rc + dt_filter)) * aZ_filtered + (rc / (rc + dt_filter)) * aZ;
    gX_filtered = (dt_filter / (rc + dt_filter)) * gX_filtered + (rc / (rc + dt_filter)) * gX;
    gY_filtered = (dt_filter / (rc + dt_filter)) * gY_filtered + (rc / (rc + dt_filter)) * gY;
    gZ_filtered = (dt_filter / (rc + dt_filter)) * gZ_filtered + (rc / (rc + dt_filter)) * gZ;

    // Calculate pitch and roll from accelerometer data
    float pitchAcc = atan2(aY_filtered, sqrt(aX_filtered * aX_filtered + aZ_filtered * aZ_filtered)) * 180.0 / PI;
    float rollAcc = atan2(-aX_filtered, aZ_filtered) * 180.0 / PI;

    // Integrate gyro data to get angle
    pitch = pitch + gY_filtered * dt;
    roll = roll + gX_filtered * dt;

    // Apply complementary filter
    pitch = alpha * pitch + (1 - alpha) * pitchAcc;
    roll = alpha * roll + (1 - alpha) * rollAcc;

    // Detect swing start
    if (!inSwing && fabs(aX) + fabs(aY) + fabs(aZ) >= accelerationThreshold) {
      inSwing = true;
      samplesRead = 0;
      velocity[0] = 0.0;
      velocity[1] = 0.0;
      velocity[2] = 0.0;
      swingDataIndex = 0;
    }

    if (inSwing && samplesRead < numSamples) {
      // Save swing data
      if (swingDataIndex < 1000) {
        swingData[swingDataIndex][0] = aX_filtered;
        swingData[swingDataIndex][1] = aY_filtered;
        swingData[swingDataIndex][2] = aZ_filtered;
        swingData[swingDataIndex][3] = gX_filtered;
        swingData[swingDataIndex][4] = gY_filtered;
        swingData[swingDataIndex][5] = gZ_filtered;
        swingDataIndex++;
      }

      // Normalize the IMU data between 0 to 1 and store in the model's input tensor
      tflInputTensor->data.f[samplesRead * 6 + 0] = (aX + 4.0) / 8.0;
      tflInputTensor->data.f[samplesRead * 6 + 1] = (aY + 4.0) / 8.0;
      tflInputTensor->data.f[samplesRead * 6 + 2] = (aZ + 4.0) / 8.0;
      tflInputTensor->data.f[samplesRead * 6 + 3] = (gX + 2000.0) / 4000.0;
      tflInputTensor->data.f[samplesRead * 6 + 4] = (gY + 2000.0) / 4000.0;
      tflInputTensor->data.f[samplesRead * 6 + 5] = (gZ + 2000.0) / 4000.0;

      samplesRead++;
    }

    // Detect swing end and run inference
    if (samplesRead == numSamples) {
      inSwing = false;

      // Run inferencing
      TfLiteStatus invokeStatus = tflInterpreter->Invoke();
      if (invokeStatus != kTfLiteOk) {
        if (DEBUG_MODE) {
          Serial.println("Invoke failed!");
        }
        while (1);
      }

      // Find the gesture with the highest confidence
      int maxIndex = 0;
      float maxConfidence = tflOutputTensor->data.f[0];
      for (int i = 1; i < NUM_GESTURES; i++) {
        if (tflOutputTensor->data.f[i] > maxConfidence) {
          maxConfidence = tflOutputTensor->data.f[i];
          maxIndex = i;
        }
      }

      // Print the detected gesture
      if (DEBUG_MODE) {
        Serial.print("Detected gesture: ");
        Serial.println(GESTURES[maxIndex]);
      }

      // Send gesture via BLE
      bleuart.write("Gesture: ");
      bleuart.write(GESTURES[maxIndex]);
      bleuart.write("\n");

      // Calculate and print the velocity
      calculateVelocity();

      // Reset samplesRead for the next swing
      samplesRead = 0;
    }

    // Print running status to serial monitor
    if (DEBUG_MODE) {
      Serial.println("System running...");
    }
  }
}

PC app development

Now, I create an application that receives and displays data sent via BLE.

In this final project, I develop for the PC environment using Python with Kivy and Bleak installed.

I base our work on the Python code created in Assignment 11.

This code will display sensor data sent via BLE on the PC, and I add the following features:

  • Control of the additional charging LED
  • Control of Sleep and Wakeup buttons
  • Display of images according to the swing type
  • Counting the number of shadow swings
  • In recent times, with the frequent use of ChatGPT, I have learned that when creating code, starting by sending prompts to create small sample programs according to the purpose can make the process smoother. 

from kivy.app import App
from kivy.uix.dropdown import DropDown
from kivy.uix.button import Button
from kivy.uix.floatlayout import FloatLayout
from kivy.uix.label import Label
from kivy.uix.image import Image
from kivy.core.window import Window
from kivy.clock import Clock, mainthread
import asyncio
from bleak import BleakClient, BleakError
import threading

Window.size = (720, 600)

class CustomDropDown(DropDown):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.auto_width = False
        self.width = Window.width
        self.max_height = 200

    def add_widget(self, widget):
        if isinstance(widget, Button):
            widget.halign = 'center'
            widget.valign = 'middle'
            widget.text_size = (None, None)
            widget.bind(size=self.update_button_padding, texture_size=self.update_button_padding)
            widget.bind(text=self.update_button_padding)
        super().add_widget(widget)

    def update_button_padding(self, button, *args):
        button.text_size = (button.width, None)
        button.texture_update()
        button.padding_x = (button.width - button.texture_size[0]) / 2

class GetAccGyrodata_app(App):
    def build(self):
        self.counts = {1: 0, 2: 0, 3: 0, 4: 0}
        self.data_buffer = {"S": None, "V": None}
        self.is_sleeping = False
        self.client = None
        self.image_paths = {
            1: "/mnt/data/1.jpg",
            2: "/mnt/data/2.jpg",
            3: "/mnt/data/3.jpg",
            4: "/mnt/data/4.jpg"
        }

        main_layout = FloatLayout()

        self.mainbutton = Button(
            text='MENU', size_hint=(1, None),
            height=75, pos_hint={'top': 1}
        )
        main_layout.add_widget(self.mainbutton)

        self.dropdown = CustomDropDown()

        sleep_btn = Button(text='SLEEP', size_hint_y=None, height=40)
        sleep_btn.bind(on_release=self.sleep)
        self.dropdown.add_widget(sleep_btn)

        quit_btn = Button(text='QUIT', size_hint_y=None, height=40)
        quit_btn.bind(on_release=self.stop)
        self.dropdown.add_widget(quit_btn)

        self.mainbutton.bind(on_release=self.open_dropdown)
        self.dropdown.bind(on_select=lambda instance, x: setattr(self.mainbutton, 'text', x))

        self.status_label = Label(text='Connecting to device...', size_hint=(1, None), font_size=48, pos_hint={'top': 0.95})
        self.smash_label = Label(text='Waiting for data...', size_hint=(1, None), font_size=48, pos_hint={'top': 0.8})
        self.speed_label = Label(text='', size_hint=(1, None), font_size=48, pos_hint={'top': 0.65})
        self.reps_label = Label(text='', size_hint=(1, None), font_size=48, pos_hint={'top': 0.5})
        self.image = Image(size_hint=(1, None), height=200, pos_hint={'top': 0.35})

        main_layout.add_widget(self.status_label)
        main_layout.add_widget(self.smash_label)
        main_layout.add_widget(self.speed_label)
        main_layout.add_widget(self.reps_label)
        main_layout.add_widget(self.image)

        return main_layout

    def open_dropdown(self, widget):
        self.dropdown.open(widget)

    def sleep(self, instance):
        print("Entering sleep mode...")
        self.is_sleeping = True
        self.update_status_label("Sleeping...")
        # Disconnect the BLE client if connected
        if self.client and self.client.is_connected:
            asyncio.run_coroutine_threadsafe(self.client.disconnect(), asyncio.get_event_loop())

    def wakeup(self):
        print("Waking up from sleep mode...")
        self.is_sleeping = False
        self.update_status_label("Reconnecting...")
        asyncio.run_coroutine_threadsafe(self.reconnect(), asyncio.get_event_loop())

    def stop(self, *args):
        self.running = False
        if self.client:
            asyncio.run_coroutine_threadsafe(self.client.disconnect(), asyncio.get_event_loop())
        App.get_running_app().stop()

    def on_start(self):
        self.running = True
        threading.Thread(target=self.bleak_thread, daemon=True).start()

    def bleak_thread(self):
        asyncio.run(self.connect_and_receive_data())

    async def connect_and_receive_data(self):
        DEVICE_ADDRESS = "E6:B0:FA:69:60:0B"
        UART_RX_CHARACTERISTIC_UUID = "6E400003-B5A3-F393-E0A9-E50E24DCCA9E"

        while self.running:
            if self.is_sleeping:
                await asyncio.sleep(1)
                continue

            try:
                self.client = BleakClient(DEVICE_ADDRESS, disconnected_callback=self.on_disconnected)
                await self.client.connect()
                if not self.client.is_connected:
                    print("Unable to connect to the device!")
                    self.update_status_label("Disconnected from device")
                    await asyncio.sleep(5)
                    continue

                print(f"Connected to device with address {DEVICE_ADDRESS}!")
                self.update_status_label("Connected to device")

                def handle_notification(char_uuid, data):
                    if self.running and not self.is_sleeping:
                        decoded_data = data.decode()
                        print(f"Received data: {decoded_data}")  # Print received data to terminal
                        self.process_received_data(decoded_data)

                await self.client.start_notify(UART_RX_CHARACTERISTIC_UUID, handle_notification)

                print("Receiving data...")
                while self.running and not self.is_sleeping:
                    await asyncio.sleep(1)

                await self.client.stop_notify(UART_RX_CHARACTERISTIC_UUID)
                print("Disconnected from device.")
                self.update_status_label("Disconnected from device")

            except BleakError as e:
                print(f"Error: {str(e)}")
                self.update_status_label("Disconnected from device")
                await asyncio.sleep(5)

    def on_disconnected(self, client):
        if self.is_sleeping:
            return
        print("Device was disconnected, callback activated.")
        self.update_status_label("Disconnected from device")
        if self.running and not self.is_sleeping:
            asyncio.run_coroutine_threadsafe(self.reconnect(), asyncio.get_event_loop())

    async def reconnect(self):
        while self.running and not self.is_sleeping:
            try:
                print("Attempting to reconnect...")
                self.client = BleakClient("E6:B0:FA:69:60:0B", disconnected_callback=self.on_disconnected)
                await self.client.connect()
                if self.client.is_connected:
                    print("Reconnected successfully!")
                    self.update_status_label("Reconnected to device")

                    def handle_notification(char_uuid, data):
                        if self.running and not self.is_sleeping:
                            decoded_data = data.decode()
                            print(f"Received data: {decoded_data}")  # Print received data to terminal
                            self.process_received_data(decoded_data)

                    await self.client.start_notify("6E400003-B5A3-F393-E0A9-E50E24DCCA9E", handle_notification)
                    return
            except BleakError as e:
                print(f"Reconnect error: {str(e)}")
            await asyncio.sleep(5)

    @mainthread
    def update_status_label(self, status):
        self.status_label.text = status

    def process_received_data(self, data):
        print(f"Processing data: {data}")  # Debugging statement to print data to terminal
        if "S:" in data:
            number = int(data.split("S:")[1].split("-")[0])
            self.data_buffer["S"] = number
            self.update_image(number)  # Update the image whenever "S:" data is received
        elif "V:" in data:
            speed = data.split("V:")[1].split("-")[0]
            self.data_buffer["V"] = speed

        if self.data_buffer["S"] is not None and self.data_buffer["V"] is not None:
            self.update_data_label()

    @mainthread
    def update_image(self, number):
        if number in self.image_paths:
            self.image.source = self.image_paths[number]
            self.image.reload()

    @mainthread
    def update_data_label(self):
        number = self.data_buffer["S"]
        speed = self.data_buffer["V"]

        if number in self.counts:
            self.counts[number] += 1

        smash_type = ['Smash', 'Net', 'Drive', 'Lift'][number - 1]
        smash_text = smash_type if smash_type else "unmeasurable"
        speed_text = f"Speed: {speed}km/h" if speed else "unmeasurable"
        reps_text = f"Reps: {self.counts[number]}"

        self.smash_label.text = smash_text
        self.speed_label.text = speed_text
        self.reps_label.text = reps_text

        # Clear the status label once data is received
        self.status_label.text = ""

        # Clear the buffer
        self.data_buffer = {"S": None, "V": None}

if __name__ == "__main__":
    GetAccGyrodata_app().run()
Now, let's begin the comprehensive test.

alt text alt text

Application Execution Screen:

alt text alt text

Each time the racket is swung, the swing type, speed, and the number of shadow swings for each swing are displayed.

*Current Issues:

  • Data reception via BLE can become unstable.
  • Speed calculation needs adjustment.
  • Additionally, the swing detection is not always accurate, indicating that there are areas for improvement starting from the TRAIN DATA of Tensorflow section.