Skip to main content

Final Project and Ongoing Things 🔁

info

Written on 2024.5.13th, some memo.

I have updated my week 2 assignment today because I realized handwriting never looks good, and I need professional tools to help me sort it out.

I found the size parameters on the official website of each product listed below, which can help me build the 3D model in the future.

image0

info

Written on 2024.5.10th, some memo.

I have updated my pages today, and wrote some thinkings, as well as finished my personal details. It is a fun journey indeed, esspecially since I have to deal with a lot of stress from my new responsibilities at work. I have a new me.

info

Written on 2024.4.28th, a little update about my work.

I was thrilled by receiving my reComputer J4012 several days ago:

And I have set it up right on the top of my working position. This will be the prototype of my final project: a local, high AI-enabled computing device, running a smart personal assistant mode, ready to help me with everywhere I go(the calendar is just a simple function of it).

But I faced a big trouble of the device - I can't run the docusaurus page on the Jetson, and here are some of my progress:

  1. One requirement for running docusaurus pages is nodejs and its minimum version is 18 but the Jetson only supports 10
  2. I learnt that I can use apt-get tool to install the nodejs-10 but that's it. So I have to find the other ways from the nodejs official website
  3. I try to download the binaries files for the lastest one, but I accidently download the wrong one, that the reComputer is ARM-architecture Linux computer and I downloaded the X86-architecture Liunx version.
  4. After facing some problems about environment setting(command echo, ln, etc, this costs me so much time) I have finally searched for the right one.
  5. So it appears that I only need to locate the downloaded binaries file in my terminal, then excute sudo cp -R * /usr/local/ will do the work
    • cp is for copy -R is the option for recursively copy directories and their contents.
    • * represents the wildcard character that represents all files and folders in the current directory.
    • /usr/local/ this is the PATH and the environment setting I am looking for. Copying to this location means that Node.js will be installed system-wide and accessible to all users.
  6. Finally I can run npm init docusaurus to install the docusaurus initial pages and replace the files by my fablab folder. Then I run npm run start and finally have my website, there goes my calendar(so not easy...):

But I did it! So interesting! The next steps(running ollama and do the auto-generation jsx files) will be easy.

Install Ollama and run Llama3:

There requires curl tool to pull the latest Ollama and here is the way to install it on the Jetson(ARM-architecture):

wget https://curl.se/download/curl-7.51.0.tar.bz2
tar xvf url-7.51.0.tar.bz2
cd curl-7.51.0
sudo ./configure
sudo make
sudo make install

And then I can use this official command curl -fsSL https://ollama.com/install.sh | sh to download the Ollama. I then use ollama run llama3 to pull the library on my device.

info

Written on 2024.4.27th, a little update about my work.

Continued on the AI recitation... Still have half vedio to process...

Combining technique and do something personal:

Retrieval Augmented Generation(RAG): Provide the domain knowledge documents, where the document is chunked(differnet blocks) and context semantically vectorized(like the number vectorized). But it requires curation. This skill does not required amount of documents.

Fine-tuning: require amount of documents, the size is big

AI tools:

  • Whisper Model: speecch-to-text, can transcribe audio files in 100 languages
  • WhisperWriter is a small speech-to-text app that uses OpenAI's Whisper model to auto-transcribe recordings from a user's microphone.
  • Local LLM Apps: there are different ones, and I have discovered a little bit and decided to use local LLM(Ollama) and use docusaurus framework to build my website(UI).
info

Written on 2024.4.26th, a little update about my work.

When I am busy with my reComputer(arrived a couple days ago) developing and sad about the progress and worry about my other assignments... I just found the recitation about AI(LLM) toolsXD Oh My God - that is what I wanted! So much! I have the idea, I have the knowledge about LLM, but at the same time I have the job.... - which means I don't have much time to start at the beginning and this is the reason I couldn't sleep well these days...

I will watch this video and finish it before I went to bed today... And... That some notes here:

LLM-powered CAD:

  • Method 1: Using DSL(Domain Specific Language)
    • prompt: give a function box(x, y, w, h, d) and ask GPT to generate
  • Method 2: OpenJSCAD: multipe one, not sure which one (this is GitHub repo)
    • The idea is using JS to describe the CAD design, and the JS code can be generated by LLM
  • Method 3: Python - PyVista
    • The idea is using Python to describe the CAD design, and the Python code can be generated by LLM
  • Method 4: text-to-cad.zoo.dev
    • Directly using text to generate CAD files

LLM-powered Laser Cutting:

  • Method 1: not sure right now

LLM-powered CNC:

analyze the models using ChatGPT - which I might not be able to use.. analyze the performance with the models

CAD - Strengths and Limitations

Strengths:

  • Respects high-level spatial constraints.such as a desian element's absolute size orits position relative to another element of thedesign.
  • lteration Support
  • Parametric design: ability to createparameters, bounds and constraints fortext-based designs and already existing, andinterpolate and change designs
  • Modularity and Hierarchy: works better when submodules are formed first.

Limitations:

  • Lack of spatial awareness created difficulties with constraint handling
  • Problem with Scalability

LLM-powered 3d Model:

LLM-powered Electronics Design:

  • Method 1: chat with any PDF: ask the datasheet about the components, microcontrollers, other modules. Which pin can connect, what code might be, etc.
  • Method 2: BoardDesignerGPT: need gpt API

Example by student OpenAir

info

Written on 2024.4.17th, a little update about my work.

Achieve the function of generating the local content from LLM and convert it into a formal standard jsx file: here and I have my own calendar UI....

info

Written on 2024.4.15th, a little update about my work.

I think I need to make a UI for presenting both calendar and the communication with the LLM, for knowing if it is workable. I have searched for multiple opensource calendar software and they all seem meet my requirments, with customizable parameters. But they all seem complicated - They do have many functions and support various extensions, but it takes too much time to understand and learn one by one. I don’t have enough time, so I plan to study it later.

  • But I still need an automated calendar UI to help me convert my recording content into what I need to do and display it so that I can see it at any time, and it has to be local deployment.
  • I suddenly realized that for locally deployed web pages, maybe "Docusaurus" has this scalability, so I started looking for. Then I did find it - There is a date picker component for React called "DayPicker" supporting Docusaurus.

I then have the idea of doing it:

  1. Ensure the DayPicker is functional on my FabLab pages
  2. For the output content from LLM, making it as format as a JS file, including some important information, and stored as a text file
    • I also need to make a continuous monitoring capability, to watch the folder of the files containing converted files. The "chokidar" might be a good idea.
  3. Then use "Handlebars" and node.js to generate React components with basically the same structure - the different ones will be the information of me recording contents
    • These components will be stored in the path of "/src/components/calendar", named one by one
  1. Finally, creating a TSX page, to input all the components one by one - Then I will have my contents of me working and the timeline

This will be the function of "Ensure the standard/formal content output by large language models to be correctly convert to task chasing, presented as the UI of calendar".

By the way, the Google Calendar sounds also good, and might be easier:

  1. I can first deploy the Google Workspace in my computer with python.
  2. Then use python as well to create the JS file with standards to upload the event/my working to the cloud.
    • The JS file can be indeed generated automatically
  3. For the UI itself - I go to the Google Calendar and all will be fine.

But it requires the cloud services, and the UI is not very well. I like the local services. And the method of Docusaurus might be not good looking at the first place, but I can customize it, which means I can make the UI with my ideas. I choose the DayPicker.:D

info

Written on 2024.4.6th, my work is finally on track and it is time to do this lab right.

As a content and web manager for my company, my role involves extensive coordination with internal team members, as well as overseeing an additional project that requires engaging with numerous external stakeholders. Amid these responsibilities, I am also dedicating time to learning and developing within the fablab space. Given the multifaceted nature of my work, effective time and project management are not just beneficial—they are essential.

Recognizing the need for assistance in managing this complex array of duties, I am exploring the potential of integrating a personal AI assistant into my workflow. The recent advancements in large language models (LLMs) have caught my attention, presenting a promising solution that could offer the precision and efficiency that physical assistance may not always be able to provide. I am interested in leveraging the capabilities of an AI to enhance my productivity, especially in the context of the fablab project, where innovation and meticulous coordination are key.

My Final Project Idea - Task Management Calendar​

I plan for an AI assistant with a physical calendar interface, powered by an NVIDIA Jetson unit running a large language model, with an external display and a portable, voice-activated module.

Conceptualization Phase - ✅ on 4.6th​

Define Objectives​

  • It is a task management AI assistant with scheduling capability.
  • In the future, it might have reminder, remote support, wireless function as well.

Current User - Matthew​

  • Manage different types of products, ensure the description and other content meet the standards.
  • Output the priority of the tasks, ensure the meet the timeline, for day/week/month.

Requirements Gathering​

  • NVIDIA Jetson Orin NX module-powered devices, LLM integrated.
  • Displays supporting the devices above, with the suitable size.
  • Microphone and voice stored module, voice recognition.

Design Phase - 🔁 keep going​

Hardware Design​

  • NVIDIA Jetson devices - reComputer J4012, with mechancial parameters: 130mm x 120mm x 58.5mm.

  • The size of display is 7-inch, with HDMI connector.

  • The voice recoginiction module is DIY, with Seeed Studio XIAO board, recorder chip, button, SD card and USB-connector.

  • The assembled and diy cutting things for integrate all parts.

Software Design​

  1. The LLM model supported on reComputer.
  2. UI design on the display.
  3. The code about pressing button and record voice, with XIAO.
  4. The program that automatically copy the voice files into the Jetson platform.
  5. The program that convert voice into words(LLM) then convert to the standard command and store the csv files.
  6. The program that read the csv files and output on the display.

The Development Phase - 🔁 keep going​

Since I am pretty busy before 2024.4.6th, I should catch up from week11, in three portions:

  • The main courses
  • The courses to catch up with
  • Development of my final project own

The previous​

June 5th will be the final and I still have 8 weeks to catch up. Here are something I currently finishing and assignments that I need to catch up with:

The plan from 2024.4.6th​

As a project manager in the company, there should be a plan for me doing and catching:

Going​