14 - interface and application programming

introduction

The group assignment for this week is to compare as many tool options as possible, document your work on the group work page and reflect on your individual page what you learned. As individual assignment the goal is to write an application that interfaces a user with an input and/or output device(s) on a board that you made.

group assignment

We start with instruction by Bas and Erwin. The first tool discussed is processing.org. This is a good tool for connection with local hardware, many libraries. Nice link between default settings and freedom to code (so all the JAVA complexity dissapears under the hood) Bas’ tutorial can be found here. Below the notes taken during the session:

• in Processing.org the order of function calls is important (i.e. background() after a draw() instruction)
• exporting applications for deployment can be an issue (drivers have to be installed, hardware compatibility and Java dependencies)
• better to use P5.js -editor.p5js.org (oracle free :) apparently this might be a future issue since Oracle’s policies have changed post on oracle java licensing - check
• Other good news is that p5.js is supported in Vscode, and that for use in your gitlab pages markdown documentation integration is also possible, so you can make interaction work in your documentation (see Bas’ doc)
• best P5 references is the coding train -> shiffman (youtube)
• For work in VScode use (install) the following extensions: live-p5, p5.vscode
• best trick in vscode: use a ‘>’ (F1) in the searchbar. Start a new p5 project or open the live-viewer
• Also in p5 the order of operations (fill, stroke, colour) is important.. you ‘dip’ your brush..
• (pro tip: when teaching object oriented programming using the bouncing balls example, call your object a blob, otherwise you are referring to ‘ball myBall(init pos);’)
• When you would like to restart the p5 ‘sketch’ in vscode: save… the browser will automagically re-load.
• select multicursor (vscode trick) cmd d (!) overview of tools.
• can I use .com -> webserial can be used in firefox (future?)
• web p5 serial TODO: rewrite my Teaching programming examples!! (see individual assignment #2)
• p5 reference (on its site) -> DOM section = interaction with webpage
• serialisation is a serious problem.. solving solutions:
• JSON -> good for getting values from one side to the other
• webmidi… = connectivity for P5.js
• websocket -> socket server .. not peer to peer
• MQTT ? server in the fabcloud (message broker) (slow)
• you can setup a websocket server -> node.js
• websocket client test plugin (fast)
• ngrok.com -> commercial service om een TCP tunnelling service.
• [NOTE to SELF] - check port forwarding of WIFI lightbulbs…
• data sources remote: https://gadgets.buienradar.nl/data/raintext?lat=52.09&lon=5.11

As group assignment we engage in a bit of ‘parallel play’. Vera is debugging her BLE connection with help of Bas, Leo works on a websocket interface and webserial in P5.js. Joany and Joe give processing a go. After the first steps in VScode and P5.js I try to get more out of Python and OpenSCAD’s command line interface, described in the next section:

individual assignment

The first idea is to see if it would be possible to use OpenSCAD as for a live animation, i.e. using (physical) sliders or other input to do live adjustment of a design. Next up I will try to break away from processing and use P5.js to communicate with my hardware

controlling OpenSCAD

There are a great many of *.js implementations of OpenSCAD - like visualisers. However, like the OpenSCAD environment itself, it is not meant as a live visualisation, but with a specific edit file -> compile -> visualise loop. Examples are

• openjscad uses an openSCAD like modelling in Javascript
• scriptcad - example at https://scriptcad.org/
• scad-js - a transpiler to gerate openscad files from javascript
• https://ochafik.com/openscad2/ uses actual openSCAD file structure
• solidpython also generates OpenSCAD code from python language

I am looking at some way to change an OpenSCAD model on the fly (or render so quickly that it ’looks’ like animating)

There is a command line interface for OpenSCAD. I give it a go.

• add the path to OpenSCAD to my .zshrc file’s PATH environment:

# added on 28-04-2024 by Edwin for the pico SDK and OpenSCAD command line access
export PATH="/Applications/OpenSCAD.app/Contents/MacOS/:/Users/Dertien/git/pico-sdk:$PATH" 

• test by calling OpenSCAD sphere.scad -D diameter=100. A window is opened and the diameter variable is passed
• when I call the same instruction again from a new terminal, it ’looks’ like the same window stays open with a new drawing. (I might have had to verify this)
• so, next up I try a python program. Python is not I program regularly, so for every instruction (how to make a loop, how to invoke command line commands, how to concatenate strings and converting integer to string and how to wait I have to google/stackexchange). The program I end up with (to animate a sphere) is the following:

import os
import time
os.system('date')
for diam in range(1,100,10):
     output = os.popen('OpenSCAD sphere.scad -D diameter='+str(diam) + ' &')
     print(output.close())
     time.sleep(0.2)
     os.system('exit OpenSCAD')

• shell_test.py
• sphere.scad (the file that is being opened, has to reside on desktop (or correct location mentioned in script))

The problem is that apparently new windows are being opened, and a new window is only opened (with a new sphere drawing) when the previous one is closed. Closing it using popen and close does not help. With the & added to the commandline invocation at least the script does not wait for me to close a window, but opens a new window for every instance. So, it would have been nice if OpenSCAD would have a more interactive API - the trick described above might work well for creating a batch of objects. Think of a garden gnome to be printed in different poses for stop motion film (and a lot of printed material)

There are integrations of python and OpenSCAD such as the pythonscad project, but still do not allow for what I would like (interact with my OpenSCAD models or generate multiple). A typical approach would be blender or unity, but that would require me to compile solid objects (STL) and render (and rig) them. Also graph-editor is an interactive editor using openSCAD which might be used for something ’live’. The following is also fun: My final project 3D CAD in one hyperlink for a visualiser

Note that this link might take a long time to load (as you are parsing 10kB of information). Apparently it is no problem to pass an entire 10k project file as a hyperlink. Long live OpenSCAD :)

processing to P5.js

In my course on Sensors and Physical Computing (if you want to check, log in with ‘student’ and as pwd ‘create’) I have been using a nice ‘graphwriter’ tool (similar to Arduino’s plotter) for quite a while. It is not autoscaling (such as Arduino’s version) but uses key-presses for scaling, can dump output in a *.csv file, colours, time stamp, grid etc. is customiseable, so all in all a very useful tool. After scaring us with news about Oracle’s Java Runtime environment I thought it was a good idea to try to rewrite this sketch for P5.js.

You can find the original graphwriter source here.

A very helpful resource for P5.js is the webreference. In order to debug the code (your processing editor is no more) VScode provide you with compiler error/warnings in the PROBLEMS tab. Also, by hitting the ‘go live’ button in the bottom right corner, a server on localhost port 5500 is started which allows you to browse and view the running *.js example. Note that my standard browser (Mozilla Firefox) does not support webserial (yet) so I have to copy the link to chromium instead. (chromium can only be started from command line from now, see the install necessary for using mods and UGS).

In chromium I use the ‘developers tool’ which can be found in the ‘view’ menu. console.log("hello world" will be visible in a nice debug pane.

I set up the sources and start writing from A to B (note, my experience with Javascript is VERY limited.)

• in vscode support for P5.js is added
• new P5.js project started (also live view), start with ‘>’ in the search bar and go for ’new p5 project’

The new project contains a style.css, a index.hml, jsconfig.json (does not need change), a sketch.js (important file) and a libraries folder.

• download p5.webserial from https://github.com/gohai/p5.webserial and put in libraries folder
• add p5.webserial to the index.html

I used the example as explained by Bas Pijls together with the ‘basic’ example from github webserial. Next up I need to find out a lot about javascript, useful resources are mdn and the ever so necessary stackoverflow.

Some notes on getting from Processing (java) to P5.js (javascript). (I don’t know whether this is actually correct. It is what I ’think’ is working now.. at least a bit.)

• no variable types, so use const for parameters and let for variables. The way you initialise a variable, i.e. with an array or a floating point number probably defines its type
• try and catch do work (but for now I don’t need them anymore)
• console.log(" "); is your alternative to a println("");
• color spaces work slightly different. Might need to fix this (or kill colour… )
• functions start with function (and not with the return type)
• easy access to a text file (storing *.csv data) is probably not possible as easily.. I might need to use json for that.

First I setup the Raspberry pi Pico (W) on my board to send analogRead values to the standard serial port (not using the debug probe). The code is very straightforward:

void setup(){
    Serial.begin(115200);
}
void loop(){
    static unsigned long looptime;
    if(millis()>looptime+19){
        looptime = millis();
        Serial.print(analogRead(A0));
        Serial.print(',');
        Serial.println(analogRead(A1));
    }
}

First I try a simple example to make sure the port can be opened and data can be read:

let port;
const BAUDRATE = 115200;

function setup() {
  createCanvas(400, 400);
  port = createSerial();

  // fetch the list of serial ports that were previously approved
  let usedPorts = usedSerialPorts();
  if (usedPorts.length > 0) {
    port.open(usedPorts[0], BAUDRATE);
  }
  else { // if no ports have been approved yet
    // add a button to open the serial port
    let button = createButton('open serial port');
    button.mousePressed(() => {
      // open the port
      port.open(BAUDRATE); // this will show a popup window asking the user to select a serial port
    });
  }
}

function draw() {
  background(220);
  if (port.available() > 0) {
    let data = port.readUntil('\n');
    if (data !== null) {
      let sensorValue = (data);
      text(sensorValue, 10, 10);
    }
  }
}

In the top-left corner I now see data coming in, however it is -just- a string, not interpreted as number yet. The full sketch can be seen here with the following source (note: I don’t add sensors yet, the inputs are just floating. Later on in the video I touch the inputs with my finger)

Anyway, I can now move on to the bit of the graphwriter sketch that splits a string into separate values and uses them as numbervalue. Instead of a separate callback function (as used in the Processing sketch) I simply use the following in the javascript loop() function:

  if (port.available () > 0) {
    let data = port.readUntil('\n');
    if(data!== null){
      let values = split(data, ',');
      for (z=0; z<2; z++) { // change this later to show all desired values, not just 2
        value[z] = values[z];
      }
    }
  }

For connecting to the serial port I used the example button code from github:

port = createSerial();

  // fetch the list of serial ports that were previously approved
  let usedPorts = usedSerialPorts();
  if (usedPorts.length > 0) {
    port.open(usedPorts[0], BAUDRATE);
  }
  else { // if no ports have been approved yet
    // add a button to open the serial port
    let button = createButton('open serial port');
    connectBtn.position(80, 200);
    button.mousePressed(() => {
      // open the port
      port.open(BAUDRATE); // this will show a popup window asking the user to select a serial port
    });
  }

This code seems to be working fine for now. The full sketch is available here and the main source here.

webmidi

Using a sketch example from the P5.js documentation site I build a quick webmidi test. I need to make sure midi control is supported from the chromium browser, and have to check the right webmidi.js sources. I don’t manage to get version 3.1.9 to work, but an (older) 2.0.0 version does work. Another potential ‘snag’ is to make sure the WebMidi script is included in the html file before the sketch.js script is being run. In the process I check the documentation mentioned above and this example project.

Again, the debug console in Chromium is very helpful for printing status / debug messages. The following script paces a button and sends a midi command. I use last week’s simple USB-MIDI implementation, so the sketch in the Pico (on the MakerIO board this time)will play a furby sample from memory upon receiving a MIDI note-on command.

On the pico board I use:

• furby.h
• midiFurb.ino

THe following javascript file is used, sources can be found here:

• index.html
• sketch.js
• style.css

let midiOutput;
let button;
function setup() {
  //noCanvas(); 
  button = createButton("Click Me", width/2, height/2);
  button.mousePressed(buttonPressed);
}
function buttonPressed(){
	if(midiOutput){
		// Play note "C2" on channel 10 for half a second and with 0.5 velocity
  	// (a noteOff message will be sent in half a second)
		midiOutput.playNote("C2", 10, {duration: 100, velocity: 0.5});
	}
	else{
		console.log("Looks like there is no MIDI output device. Check if your Arduino is connected.");  
	}
}
WebMidi.enable(function (err) {
  if (err) {
    console.log("WebMidi could not be enabled.", err);
  } 
  // Print available MIDI outputs
  for(let i = 0; i < WebMidi.outputs.length; i++){
  	console.log(WebMidi.outputs[i].name);
  }
  // From the list on the console, pick an output name:
  // midiOutput = WebMidi.getOutputByName("IAC Driver IAC Bus 1");
  midiOutput = WebMidi.getOutputByName("Pico");
});

I also try to make the code run on my week06 board with the I2S DAC. Although the ‘simpletone’ example still works (note: when using Adafruit’s tinyUSB instead of PICO SDK for USB the compiler throws an error) the combination of tinyUSB (for the MIDI support) and I2S sound output do not work(yet). In the video you see the WebSerial midi output triggering the sample playback, however, on I2S there is no sound output.

The source is the following

In the final project the main source of input will be the analog (CV, gate, modulation) triggers, but USB MIDI is definitely next, if only for configuration purpose. Probably have to look into the PIO assignment or move the sample playback to a different core anyway.

learning outcomes

Implement a User Interface (UI) using programming and explore protocols to communicate with an embedded board.

evaluation checklist

• Linked to the group assignment page.
• Documented your process.
• Explained the UI that you made and how you did it.
• Explained how your application communicates with your embedded board.
• Explained any problems you encountered and how you fixed them.
• Included original source code (or a screenshot of the app code if that’s not possible).
• Included a ‘hero shot’ of your application running & communicating with your board.

lessons learned, tips and tricks

(or, the most insightful mistakes I made)

• Might be a tricky thing to debug: why does USB-MIDI conflict with I2S playback
• OpenSCAD is not for live rendering. (sigh…)

left for todo

• finish the search for other nice ways of scripting OpenSCAD
• continue with P5.js: add also webmidi and try websocket/mqtt
• integration of javascript with Hugo / web so sketches can be visualised and run directly in a div, rather than using a separate link.
• now start the big overhaul of all my nice educational processing sketches and make them work in-browser..
• try anaconda (again) for python installation, distribution and management

reflection

A whole area of new things to (re)discover such as python, and python/BASH scripting, the OpenSCAD command line interface. Very nice and powerful stuff, so much more to explore. Trying to debug P5.js and convert applications from Processing to P5 was a bit of a struggle (javascript is also pretty much new territory). Still, three new languages and a whole new road ahead, not a bad score.

copyrights and references