Final Project Proposal
Overview
This proposal outlines a comprehensive final project that integrates various units of study. The project will be designed to demonstrate mastery of multiple skills, including 2D and 3D design, fabrication processes, electronics design and production, and system integration.
Project Objectives
What will it do?
- For now, a personal secretary, assisting personal daily work and organize the schedule, interactable, memorizeable.
- In the future, record all daily data(like eating, sleeping, daily work schedule, etc) of a person, and organize daily work, all data local.
Who's done what beforehand?
If just one image:
- Not a technology enthusiasts would not find the appeal of creating something like Iron Man's AI assistant,
Jarvis, orFriday, or others irresistible. - And the method of designing the assistant is now simpler with the blessing of the LLM.
- A project named "RAGflow," hosted on GitHub. RAGflow is referred to as a Retrieval-Augmented Generation Engine, designed to facilitate a streamlined workflow for both businesses and individuals interested in technology development.
Thus I am planning to build a simple one.
What will you design?
- A mobile recording module, recording and recognising my daily words, which supports Wi-Fi wireless data transmission
- An interactable website, running local LLM, offering original dialogue.
- A simple system enabling recorded words becoming standard calendar, and file storage function(RAG-supported).
- A mobile displaying module, displaying the calendar information from my reComputer, and being able to do something.
Implementation Details
Materials and Components:
Updated on 7.8th, I have updated my project, with both input and output. Thus, my materials have some changes.
Embedded Microcontroller Interfacing and Programming:
For the first is my software testing part:
(Input) Mobile Module - Raw Audio Data to WAV file
The function: Powering it the board up and it will record and generate a WAV file and transmit to a server, where the IP is given by the nework.
The reComputer can read the IP and download the file from the temporary server.
More details on the (input) Mobile Module - Raw Audio Data to Words
WAV file to text on reComputer
Convert WAV file to text using canary-1b and input the text into the LLM(Ollama API):
More details on the (input) Mobile Module - Raw Audio Data to Words
To achieve this part, I need to set up a LLM-powered bot on my reComputer:
More details on the Chat Bot(Local Server)
For making this bot looking better, I use GPT itself to generate code to looking better:
More details on the Operating UI Setting
Audio Text to Designed calendar standards(JSX)
I need to be able to translate my words(audio text file) into a calendar JSX standard file that docusaurus platform can render:
More details on the Local LLM and Auto-generation tsx file
For using LLM correctly, I need to learn and apply well using Prompt Enginner:
More details on the Prompt Setup
(Output) Mobile Module - Receive and Display Calendar Information
After the reComputer generating the standard JSX calendar files, I shall transmit these message to a mobile module, displaying it, and maybe controlling something as well.
More details on the Mobile Module - Receive and Display Calendar Information
For applying this, it requires MQTT function and I have apply mosquitto in reComputer, let it functioning as the broker.
More details on the week 15 - MQTT connect with XIAO boards and Docusaurus-website
And using Docusaurus page to construct WebSocket and implement MQTT function under 192.168.6.1 network(reRouter)
More details on the week 15 - MQTT connect with XIAO boards and Docusaurus-website
Interfacing
I design the bot using my assignment website page and put all wireless function together. These can be done as automated.
This is great and convenient for the users, but they can always manually control, under the FAB Academy MQTT. For example:
Electronics Design and Production:
Updated on 7.8th, I have updated my project, with both input and output. Thus, my function is upgraded, using my own PCB.
(Input) Mobile Module - Raw Audio Data to Words
I want this module whiling receiving my voice it can display some RGB LED lights showing it is working, or other things. I then connect two 3 Pin Header SMD in it.
More details on the (input) Mobile Module - PCB design
(Output) Mobile Module - Receive and Display Calendar Information
I want this module whiling receiving the calendar information it can control something, like a relay. I then add a Grove port and a 8 PinHeader P2.54mm on my PCB:
More details on the (Output) Mobile Module - PCB design
Fabrication Processes:
Updated on 7.8th, I have updated my project, with both input and output. Thus, my design are changing here:
The case for input module:
Top cover(3D printing):
Middle part(3D printing):
And the RGB light cover(2D design):
Eventually:
Extend the rgb lights out, along with XIAO antenna:
Finally put the acrylic plate init can cover it up:
The case for output module:
Display holder and Grove Relay holder(3D printing):
Eventually, it looks fine:
The system case design - 2D laser cutting
This part is to make the work look good, tidy and logical.
- The RGB display is best with a filter layer. So cut some acrylic sheets against the blueprints:
- Since I used some big equipments/devices (reComputer and reRouter), I'm going to have two large acrylic sheets to hold everything together:
Thus the laser cutting is necessary:
The system case design - 3D printing
This part is to make the work look good, tidy and logical.
- My Devices need to be supported, fixed and not moving everywhere. Thus I need to design some 3D parts holding them:
For other consideration, I remove the original.
System Integration and Packaging:
The whole system basic idea:
- Main computing device: reComputer, offering MQTT broker, AI computing, Running LLM and Browser Website.
- Networking device(I want to ensure all thing is local): reRouter, offering Wi-Fi wireless connection and wire connection.
- One mobile module with INMP441 is inputting voice
- One mobile module with ILI9341 is display the processed information.
The whole system carraging:
The packaging is big, even though I remove the original cases of computing devices. It is still big, no mention the power adapter. But this can be carried:
Key Questions and Evaluation for Localized LLMs with Mobile Modules
Key Questions
-
System Stability: How can we ensure system stability under continuous use?
- Implement robust testing protocols including stress testing to ensure durability.
- Use high-quality, durable components that can withstand continuous operation.
- Regular software updates to address any vulnerabilities or bugs that could affect stability.
- Employ advanced cooling systems to prevent overheating, which can degrade performance over time.
-
Data Security: What are the best practices for securing data locally on the device?
- Encrypt all sensitive data stored on the device using strong encryption standards.
- Use secure hardware elements like TPMs (Trusted Platform Modules) for storing cryptographic keys.
- Implement strict access controls and authentication mechanisms, including biometric security features if applicable.
-
User Interface Optimization: How can the user interface be optimized for ease of use and accessibility?
- Design the interface with simplicity and intuitiveness in mind, ensuring that all functionalities are easily accessible.
- Include accessibility features such as voice commands, large text, high contrast modes, and screen readers to accommodate all users.
- Conduct usability testing with a diverse group of users to gather feedback and make necessary adjustments.
-
Power Consumption: What are the power consumption metrics, and how can battery life be maximized in mobile modules?
- Optimize software to ensure efficient use of power, avoiding unnecessary processing.
- Use energy-efficient components that consume less power without compromising performance.
- Implement power-saving modes that reduce energy consumption when the device is idle or not performing intensive tasks.
-
Module Communication: How will different modules communicate efficiently and securely via MQTT?
- Use MQTT protocol with TLS/SSL encryption for secure data transmission between modules.
- Implement quality of service (QoS) levels within MQTT to prioritize critical data transmission.
- Regularly update and patch the MQTT brokers and clients to protect against vulnerabilities.
Evaluation
-
Functionality:
- Test whether the system performs all its intended functions such as voice recognition, data storage, and schedule organization accurately and reliably.
-
User Experience:
- Conduct extensive user testing to evaluate ease of use, collecting feedback on the UI's intuitiveness and interaction smoothness.
- Implement adjustments based on user feedback to improve the overall user experience.
-
Performance:
- Measure system responsiveness and reliability during various operations, focusing on the speed and accuracy of voice recognition and command execution.
-
Scalability and Modularity:
- Assess whether the system can be easily expanded with additional modules or features without requiring significant redesign or system overhaul.
-
Security:
- Evaluate the effectiveness of the implemented data privacy and security measures through penetration testing and security audits.
- Ensure that all security features are up to date and compliant with current data protection regulations.
Additional Notes
-
DIY Approach:
- Components will be fabricated in-house wherever possible to demonstrate hands-on skills and innovation. This includes 3D printing cases, designing custom PCBs, and manually assembling all electronic components.
-
Project Format:
- Projects can be either individual or collaborative efforts, provided they demonstrate individual mastery and can operate independently. This flexibility allows for varying project scales and scopes, accommodating different levels of expertise and resource availability.
Summary
This project aims to develop a modular, scalable, and secure system capable of performing a variety of functions such as voice recognition, data storage, and schedule organization. Key questions focus on ensuring system stability, security, and efficient power management. The system's design emphasizes a DIY approach, encouraging the use of in-house fabricated components to foster innovation and hands-on skills development. Evaluation will involve assessing functionality, user experience, performance, scalability, and security, ensuring a comprehensive understanding of the system’s capabilities and limitations. The flexible project format allows for both individual and collaborative efforts, accommodating diverse expertise and resource availability to maximize learning and development outcomes.