Final Project Requirements

In this week an integrated design of the final project will be made, using both additive and subtractive manufacturing techniques to fabricate 3D designed components based on 2D sketched features. In addition, a microcontroller will also be manufactured, and programmed depending on input and output devices. The final project also combines my proposed Master's Thesis on classifying facially expressed emotions using Fast R-CNN object detector in real-time. The process of creating a convolution neural network based object detector for such purpose will also be documented.

The main structure of the project consists of a subtractively manufactured wood components which are joined using press fitting technique, forming mainly a box, a platform which would hold sensor components, a supporting structure, and a top component. Aluminium rods will be placed on the box, supported a back and top wood structures. The aluminium rods would help the sensor/webcam platform to move smoothly on a vertical axis with the help of bearings. The structure movement will be powered using a stepper motor belt mechanism to move the sensor/webcam platform vertically. Components which house the bearings, join the sensor and webcam with wood structures will be designed and additively manufactured using a 3D printer.

The box's 2D sketches and 3D design is documented in Week 8.

The sensor/webcam wooden platform which will move along the belt vertically is sketched to be rectangular with various M4 screws to allow mounting of 3D printed components for both sensors and bearings.

The 2D sketch is extruded to a 3D designed components with thickness that matches the wood sheet intended to be used for this project, 18 mm. It is easier to design components that would mount the bearings, sensor, and webcam by creating a sketch on platform's 3D surface. Such approach maintains existing features such as screw locations and would define upcoming component dimensions.

The front panel with 3D components designed on its surface. The sensor and webcam brackets are fixed on the front side, and on the back, bearing brackets are fixed.

The bearing socket is designed to match LM12UU bearing, which is chosen based on the 12 mm rod intended to be used for the project.

The webcam socket, to allow fixing the webcam on the wooden front panel of the platform professionally using screws is designed with a base sketched on the front panel, extruded, and then another sketch was drawn on the extruded surface, this sketch included the screw feature of the webcam, to allow fixing the webcam on the 3D printed component using a typical tripod screw.

The front panel, and 3D components has been considered for the vertical motion of the system. Now it is required to consider the position of the stepper motor, the belt, top wooden panel, and back wooden support sheet. Assembling the front panel, with bearing sockets, along with rods and belt is an important step. Afterwards, the over all platform's height, such as the back panel, and rod height can be adjusted according to component availability. In my case, a 2000m rod was not available, the design was then adjusted for 1,500m. Then, placing a stepper motor at the bottom of the front panel, on top of the bottom's box is an important step. The bracket which would hold the stepper motor can be designed to be printed using additive manufacturing. Also this will ensure the correct bracket dimension and that it will fit in the correct place without obstructing the panel movement, or the rods. Steps documenting how a stepper motor bracket is designed has been documented during the mechanical design week of the group project mechanical design week of the group project

The stepper motor is being positioned as close as possible to the belt which when moved can vertically move the front panel of the platform.

A custom designed stepper motor bracket is shown in the screen shot below. The belt is seen to be positioned in the middle of the bracket. A pulley attached to the stepper motor will move the belt. It is also required to design a similar bracket to be attacked on the top panel, were a free running pulley can be placed.

With the above approach and technique, the designer is sure that every component is placed in its suitable position. Following this step, we will start first with subtractive manufacturing, and additive manufacturing of all designed components, during the cutting/printing time, the time will be used for programming both a control algorithm, and a MATLAB code to establish a serial communication between MATLAB IDE and ATMEGA328P.

The above 9 components will all be manufactured using the shop bot machine. Steps on how to prepare and operate the machine are described in the following guide of Week 8. First the shop bot's bed is vacuumed, placing the vacuum machine on top of the shop bot is completely safe would make the vacuum work more easy. Then a new 18 mm wood sheet is added.

The tool used to cut the above components was 6 mm. Spindle speed was set at 13,000 RPM, Feed rate at 2000 mm/min and plunge rate at 400 mm/min. All tool paths were created, while considering cutting inner paths first and then outside ones. Three tabs are added for each component to ensure it does not move after being milled completely.

A neat result as shown below, all components were removed from the shop bot, a sand tool was used to smoothen corners and sharp wooden materials sticking out from the components. Then using a rubber hammer, components were joined together using press fitting. It was best to join all sides together, and then top/bottom parts.

Additive Manufacturing - 3D Printing

Using additive manufacturing, 3D printing, sockets for bearing, bracket for stepper motor, and bracket for free running pulley are produced. The bearing must be inserted inside the socket, the same way press fitting is done. If the bearing was inserted using only hand force, then the socket must be reprinted using a smaller hole diameter to ensure press fitting using a rubber hammer.

As we learned previously, it is important to laser cut components before manufacturing them subtractively, this information is also important for additive manufacturing, this step can be used to confirm if the diameter of screw holes of 3D printed components are not loose enough to prevent any vibration caused from any mechanical motion.

The following component is made as a bracket for the stepper motor.

The following component is made for the free running pulley.

Assembling Components - Test

All subtractively and additively manufactured components have been assembled together. The most important test at this point was to verify whether the belt is actually moving the front panel, to verify if anything will be obstructing the motion, to verify if all components have indeed been manufactured correctly.

Interfacing Electronics

The electronics consist of an ATMEGA328P, Stepper motor, stepper motor driver, power supply, time-of-flight sensor and a webcam. The ATMEGA328P and the webcam will be directly connected to the laptop. Rest of the components will be interfaced with the ATMEGA328P. Stepper motor was the first component to be connected to the ATMEGA328P through stepper motor driver board. Initial test were carried to validate if the wiring is correct, the power from the power supply is enough to power the stepper motor. Then, the next test was on programming the stepper motor correctly, this required referring to TB6560 Stepper Motor Datasheet in order to interface the component correctly with the ATMEGA328P and the power source.

Board Microcontroller - ATMEGA328P

In Week 11, an ATMEGA328P based microcontroller has been designed and fabricated. The same board will be used for this project

Stepper Motor Driver - TB6560

For belt driven systems which use stepper motors, it is important to calculate step per millimeter. In this website, a steps per millimeter calculator can be found. A full rotation means 360 degrees, with a typical motor a single step results in a 1.8 degree rotation, thus there are 200 steps per revolution. Microstepping is 1/8, with lower microstepping the motor's resolution is increased but at the stake of lower torque. The GT2 belt has a pitch of 2 mm, and the pulley mounted on the stepper motor has 20 teeth. With the above configuration, there are 40 steps per millimeter.

Stepper Motor Driver - Program

A simple code was written to program the ATMEGA328P to control the stepper motor driver, the code simply writes sends output to the driver from pin 8 and pin 9. Pin 8 is the direction pin, and pin 9 is the signal pin used to send a sequence of HIGH and LOW signal separated by time delay equivalent to a single step. The code have been obtained from this link.

    int Distance = 0; // track steps

    void setup() {

    pinMode(8, OUTPUT); // direction pin
    pinMode(9, OUTPUT); // signal pin
    digitalWrite(8, LOW); // digital low
    digitalWrite(9, LOW); // digital low

    }

    void loop() {

    digitalWrite(9, HIGH); // digital high
    delayMicroseconds(1); // step delay
    digitalWrite(9, LOW); // digital low
    delayMicroseconds(1); // step delay
    
    // the above sequence = single step

    Distance = Distance + 1; // record this step // Check to see if we are at the end of our move
    // two rotation for 1/8 bridge and 1 rotation for 1/6 bridge (for this code)

    if (Distance == 3200) { // We are! Reverse direction (invert DIR signal)
    if (digitalRead(8) == LOW) {
    digitalWrite(8, HIGH); }

    else {
    digitalWrite(8, LOW);
    } // Reset our distance back to zero since we're // starting a new move

    Distance = 0; // Now pause for half a second delay(500);

    }

    }
        

Time-of-Flight Sensor

In week 11, the use of time-of-flight sensor as an input to the ATMEGA328P is documented.

Before moving any further with the project, the interfacing between the ATMEGA328P, ToF sensor, stepper driver and stepper motor is tested. Combing the above documented codes, if the ToF sensor detects an object within a specified distance, then the the ATMEGA328P would output a signal to the stepper motor to move upwards, and then downwards.

MATLAB

The MATLAB code is based on the following code obtained from this source, with the following license. The source contains a code used to detect and track multiple faces in real-time. The code has been modified to load the trained Fast R-CNN object detector trained to classify facially expressed emotions in real-time. The code loads the webcam, cascade object detector and multi object tracker to its workspace as objects. The code also loads the trained facial emotion expression object detector.

The code then constantly takes a snapshot, an image from the video object detector and create sort of a playback video. Then using the above loaded object detector to detect faces, the code iterates the detector until it detects a face, this code is part of the original code used to detect faces in real-time.

The following part of the code which is based on the original, takes a snapshot of the video object, convert it to gray scale, which is then used by the face detector algorithm in the following line of code. The same captured image, however not converted to gray scale, processed using the trained emotion detector. The line of code then returns the region of interest, which points to the expression location, score, and type of classified emotion.

The following line of code, keeps on detecting the emotions, and if no face is detected, then the program will stop running the object detectors until a face is again detected.

The above code still does not serially communicate with the ATMEGA328P. The documentation on Serial Port Devices guide on MATLAB describes in details the commands required to establish a serial communication with any device, in our case the ATMEGA328P, how to read information from the serial port and how to close it.

The below command creates an object in MATLAB's workspace as a serial object device on the specified come and baud rate:

serial('COM7','BaudRate',9600);

The following command initializes the serial communication between the serial object created and the ATMEGA328P controller:

fopen(arduino);

The next code is used to verify whether the serial communication has been established with the ATMEGA328P:

if isvalid (arduino) == 1..

In short, the above commands has been added as seen in the below image at the start of the code. If the serial communication is established, then there will be an if statement, if the signal received from the controller is equal to 1, then the classification process would start, else remain on standby.

This command checks if there is any byte available and if the statement is true, the next command reads the received byte from the serial port, and the one after it is an if statement to check if the value is 0 or 1, with 1 the classification process starts, with 0 the system goes into standby mode.

if arduino.BytesAvailable

parameter = fgetl(arduino);

if parameter == '1'

Embedded Programming - ATMEGA328P

Previously in the guide, the code to test a stepper motor have been documented, along with a MATLAB code which starts the facial emotion recognition object detector based on the command received from the controller via the established serial communication. The following code defines several system status such as SLEEPING, CHECKING, MATLAB, and HOMING according to the suitable devised algorithm for the system. Once the system is turned ON, the first status is SLEEPING status, the ATMEGA328P enters a SLEEPING status awaiting a person to stand within a previously specified distance detected by the time-of-flight sensor. Once the user is detected, the ATMEGA328P enters a CHECKING status, a status by which the system moves its front panel (webcam+sensor) vertically until the panel is positioned on a level parallel to user's face. The system then enters a MATLAB status, a status where the ATMEGA328P sends a serial signal to MATLAB, and MATLAB initiates the facial emotion expression classification process, displaying results on the laptop screen. Other parameters such as ACTIVATIONDISTANCE and MAXIMUMDISTANCE has been defined. Once a user stands within a distance less than 500 mm in front of the platform, specified by the activation distance as defined in the program, the system is activated. The maximum distance of 1400 mm refer's to the maximum height the stepper is allowed to move the front panel before hitting the top layer of the platform.

As previously documented, direction pin and stepper pin are also defined along with stepsPerRevolution and stepsPerMM, which have been estimated using the stepper per millimeter calculations documented earlier.

In line 14, DEBUG is declared as boolean, if DEBUG is equal to true, then an if statement will be true in several sections of the code, by which the serial monitor prints the system status to the serial monitor. In a code of more than 100 lines, it is difficult to point out any issues, or to verify the system indeed have entered a certain status, that is why with the help of the debug if statement, the user can troubleshoot the code easier as he knows from the serial monitor in which status the system faced an issue.

As an example, the system constantly reads the distance from the ToF sensor, and at the same time writes to the serial monitor that the system is in "SLEEPING" status.

The system continuously checks if the distance is less than the activation distance, checks if a person is indeed standing infront of the ToF sensor within the activation distance. If the condition is true, and the system was in "SLEEPING" status, the statement is true, and the system status is changed to "CHECKING" status. Now the next else if statement is also true, the system will continuously check if the person is still standing infront of the system, and move the front panel in an upward position, while reading the distance from the ToF sensor.

The front panel would be moving upward until the distance sensor no longer detecting a person standing infront of the system, as in the front panel is now in a higher height than that of the person. Then the front panel is moved in 20 cm downwards, and this way the front panel is approximately positioned infront of user's face. This method is suitable for users of various heights. The system status is then changed to "MATLAB".

While the user is still standing infront of the system, the ATMEGA328P will serially send "1" to MATLAB, thus initiating the facial expression recognition object detector to start classification the emotions expressed through user's face. Once the user moves out of the ToF sensor scope, and the distance is higher than that of the activation distance, the system enters into a "HOMING" status.

During the "HOMING" status, the system sends "0" serially to the MATLAB, stopping any classification process and to restart MATLAB's workspace until another user is presented infront of the system. In addition the front panel which houses the ToF sensor and webcam travels back to its original position, as its distance is tracked using currentDistance as declared in the beginning of the code. The system status is now changed to "SLEEPING"

Two void declared functions were used in the code, MovePlate() and readDistance(), positioned at the bottom of the code structure. MovePlate declares two letters, j and d, equivalent to MovePlate(10,LOW) for example. The value refers to number of steps, and the HIGH/LOW boolean refers to direction pin output to the stepper motor driver, to driver the stepper motor either upwards or downwards, for LOW, it is upwards in this example. The void function first checks if the boolean to move the front panel upwards or downwards is HIGH or LOW, in case it was LOW (moving upward), the current distance will continuously be incremented. Else the value would be decremented for the number of steps specified in the code.

As a safety measure, as defined earlier, the maximum distance is used as an if statement for the function to check if the incremented currentDistance does not reach a higher value of that of the maximum distance. The function only moves the stepper motor to a level less that is less than the maximum distance level. The following line of codes have been documented earlier in this guide, pertaining to incrementing each step taken by the stepper motor using a HIGH-delay-LOW signal equivalent to a single step.

The following line of codes pertaining to reading the distance from the PWM signal of the ToF sensor have been documented in week 11.

Project Presentation