Lei Feng Fab Academy 2025

Appearance

Week 15: Interface and Application Programming ​

The content of this document was refined and generated by AI after I provided Claude 3.7 Sonnet with the course outline and the subtitles from the course video conference.

Course Overview ​

This course primarily introduces how to build software interfaces, write programs, create graphics, sound, video, perform mathematical operations, improve performance, and introduce machine learning and application deployment. The core objective of the course is to build interfaces between users and input/output devices, achieve data visualization, and send messages from input devices to output devices. This course will explore various programming languages, user interface frameworks, graphics rendering techniques, sound processing, data visualization, performance optimization, and basic applications of machine learning.

Detailed Course Content Introduction ​

1. Choosing a Programming Language ​

Programming languages are the foundation for building interfaces, with different languages suitable for different application scenarios:

1.1 Traditional Programming Languages ​

  • C language: A classic language between low-level and high-level
  • C++: An extension of C that supports object-oriented programming; the Arduino IDE uses C++
  • .NET and C#: More modern frameworks from Microsoft based on C
  • Go and Rust: More modern languages that address C's security and stability issues
    • Rust is becoming increasingly popular in embedded applications, high-performance and more reliable

1.2 Functional and Scripting Languages ​

  • Functional languages: Haskell, Lisp, etc., programming by evaluating functions rather than describing steps
  • Scripting languages: Bash, Perl, etc., suitable for writing simple scripts
  • Python: Beautifully designed, well-documented, cross-platform, and free to use
    • Combined with Conda, different toolkits can be set up for different environments (embedded programming, web development, machine learning)
    • Almost all machine learning is now implemented in Python

1.3 Web and Interactive Programming Languages ​

  • Processing: The inspiration for Arduino, suitable for graphics and sound processing in desktop environments
    • Also has a JavaScript version: P5.js
  • JavaScript: Unrelated to Java, originally designed as a scripting language for early browsers
    • Has received significant investment due to the commercial importance of the web
    • Modern JavaScript engines implement just-in-time compilation, with performance comparable to C
    • Node.js is a version of JavaScript that can run outside the browser

1.4 Low-code/No-code Environments ​

If you don't like writing code, you can use visual data flow environments:

  • LabView: For laboratory automation
  • Simulink: For simulation
  • Max: For music
  • Scratch: Block-based programming for children
  • App Inventor: Making applications with a block-based framework
  • Node-RED: Data flow framework for event interfaces
  • Mods: Another data flow framework

2. Device Interface Methods ​

To communicate with devices, you need to choose an appropriate interface method:

2.1 Serial Communication (RS-232) ​

  • Python: Using the PySerial library
  • Node.js: Built-in serial port support
  • Web browsers: Chrome-series browsers support the WebSerial API
    • Note: Firefox and Apple browsers do not support this for security reasons

2.2 Other Communication Protocols ​

  • I2C: Python and Node have interfaces
  • USB: Can use more advanced protocols rather than just serial communication
  • MQTT: Used to manage multi-device ecosystems
    • Based on a publish/subscribe model with a centralized "broker"
    • Devices can publish results to the broker, and other devices can subscribe to these results

3. Data Organization ​

Data organization is crucial for application development:

  • JSON: JavaScript Object Notation, a lightweight data interchange format
  • XML: Extensible Markup Language, more complex than JSON
  • CSV: Comma-Separated Values, simple text format for tabular data
  • Database: For more complex data storage and retrieval
    • SQLite: Lightweight database embedded in the application
    • MongoDB: Document-oriented NoSQL database

4. Building User Interfaces ​

User interfaces are the bridge between users and applications:

4.1 Desktop Applications ​

  • Tkinter: Python's standard GUI package
  • PyQt/PySide: Python bindings for the Qt framework
  • Electron: Build desktop applications with web technologies

4.2 Mobile Applications ​

  • React Native: Build native mobile apps using React
  • Flutter: Google's UI toolkit for building natively compiled applications
  • Swift UI: For iOS applications

4.3 Web Interfaces ​

  • HTML forms: An early part of web standards, can be used to build user interfaces
  • jQuery: Originally written to solve JavaScript limitations, now with more features
  • Other JavaScript frameworks: Such as React, Vue, Angular, etc.

4.4 Cross-platform Frameworks ​

  • Design applications for different device sizes and operating systems
  • Write once, export to any type of device and language

5. Graphics Programming ​

Graphics are important for data visualization and user interaction:

5.1 JavaScript Graphics ​

  • Canvas: 2D drawing API for HTML5
  • SVG: Scalable Vector Graphics, XML-based vector image format
  • WebGL: JavaScript API for rendering 3D graphics in browsers

5.2 Python Graphics ​

  • Matplotlib: Comprehensive library for creating static, animated, and interactive visualizations
  • Pygame: Set of Python modules designed for writing video games
  • OpenCV: Computer vision library with Python bindings

6. Sound and Video Processing ​

Sound and video add richness to applications:

6.1 Sound Processing ​

  • Web Audio API: High-level JavaScript API for processing and synthesizing audio
  • PyAudio: Python bindings for PortAudio, a cross-platform audio I/O library
  • Librosa: Python package for music and audio analysis

6.2 Video Processing ​

  • OpenCV: For real-time computer vision
  • FFmpeg: Complete solution to record, convert, and stream audio and video
  • MediaPipe: Cross-platform framework for building multimodal ML pipelines

7. Virtual Reality and Augmented Reality ​

  • VR: Virtual Reality, inserting into sensory perception through goggles
  • AR: Augmented Reality, adding virtual elements to the environment
  • Three.js: Can convert 3D environments into VR or AR experiences

8. Mathematics and Data Visualization ​

Data analysis and visualization are crucial for understanding data:

  • NumPy: Extends Python to handle mathematical data objects
  • Matplotlib: Framework for making charts
    • Can create static or dynamically updated charts
  • Jupyter Notebook: Create interactive notebooks containing code, results, and descriptions
  • Plotly: Open-source plotting framework for JavaScript and Python
  • D3.js: For creating complex and interactive data visualizations

9. Performance Optimization ​

When dealing with large amounts of data, performance becomes key:

  • NumPy: 10 times faster than pure Python
  • Numba: Compiles Python code to improve speed
  • Jax: Accelerates Python and supports computing derivatives
  • JavaScript Workers: Run code in parallel on different cores
  • Rust parallel libraries: Like Rayon, automatically parallelizes code
  • MPI, CUDA: For large computers and graphics hardware acceleration

10. Introduction to Machine Learning ​

Machine learning can learn patterns from data:

  • Basic concepts: Network architecture, layers, nodes, functions
  • Training process: Defining error metrics, training strategies, backpropagation
  • Frameworks: TensorFlow, PyTorch
  • Deployment: Hugging Face, cloud service provider APIs

11. Deployment and Security ​

Deploying applications to production environments and ensuring security:

  • Cloud services: AWS, Google Cloud, Azure
  • Security considerations: Dependencies are a historical source of hacker attacks
    • Keeping dependencies updated is crucial

Assignment Requirements ​

The goal of this week's assignment is to build a software interface to interact with an input or output device you've made. Specific requirements:

  1. Choose a programming language or framework to build the interface
  2. Implement communication with a device you've previously made
  3. Build a user interface to visualize data from input devices or send messages to output devices
  4. Document the entire process, including:
    • The chosen programming language/framework and rationale
    • Detailed description of the interface implementation
    • Code examples and explanations
    • Functional demonstrations
    • Problems encountered and solutions

Note:

  • If you're behind on weekly assignments, you can use your final project to complete weekly assignments
  • But you still need to demonstrate in your documentation how you use the relevant skills in your final project
  • Time management is crucial; only four weeks remain in the course

Learning Resources ​

Programming Languages ​

Device Communication ​

Graphics and User Interfaces ​

Data Visualization ​

Machine Learning ​

Video Tutorials ​

Low-code/No-code Options ​