alv structural editor

(fullscreen recommended)

This week I built a structural editing interface for the alv livecoding language I am working on. It is a programming language optimised to be edited while it is running, and allows such changes to be made to local regions of the code without affecting the rest of the program.

With alv I am trying to push for new and better coding user experiences. Since the biggest user-facing part of programming is editing the program text in the editor, the editing flow is what alv focuses on. Unlike other livecoding languages (e.g. SuperCollider, TidalCycles, etc), the text editor is not just a scratchboard to type up commands in before executing them; the editor contents are directly and permanently tied to the actual running program.

Reifying the source code in this way opens the door for robustly implementing advanced editor and debugging features similar to e.g. Processing’s tweak mode (but within the same mode as regular editing!). While alv has been explicitly designed with this in mind, so far I had limited myself to work only on the langauge itself to constrain the scope of the project.

Given the topic of this week, I decided to build a hardware-enabled structural editor for alv to start to experiment with such editor features. Structural editing refers to editing the code not strictly as text, but rather in terms of its structural properties. Since alv uses an S-Expression (Lisp) syntax, the code follows a very simple yet expressive structure - it consists only of five types of things:

• whitespace and comments (meaningless to the computer)
• numbers
• strings
• symbols (“variable names”)
• cells - lists of numbers, strings or symbols encapsulated in parenthesis.

Given this very regular structure that is naturally representable as a tree, structural editing is a natural choice.

hardware

I reused the ESP32-based prototype from the input devices week, that was originally meant to become a smart light switch. Due to COVID-related compromises that never happened, so I instead added a second knob and three touch pads. Since this is off-topic here, I will document that in the log mentioned above.

design

I dedicated the smaller knob to the traversal of the tree. Moving around the tree is probably the biggest part of using a structural editor, but it does not require as much precision as entering an exact number or choosing the correct one from a long list of symbols, since there are far less than ten children at every level of the tree usually. This let me use the bigger knob, which gives finer control with its larger diameter, for the ‘tweaking’ of constants. Both the cursor and the tweak knobs are capacitive-touch-enabled. The cursor is highlighted when touched, and the tweak UI is only visible while the corresponding knob is held.

In structured editing, navigation consists not only of moving left and right in through the program text, but also moving in and out of structural layers (in this case Cells). Originally I used the naive approach of mapping all four movements to individual actions (forward/backward on knob turn, inward/outward on touch pads), but I quickly noticed that this navigation style was very cumbersome: given the often very short expressions used in alive, a lot of ‘level switching’ is often required to make a specific movement. This requires both hands and takes time and effort, while navigation really should be able to happen almost subconsciously.

The first change I made to adress this was to allow ‘bumping’ into the ends of Cells: when the selection is at a certain depth in the tree, navigating past the end of the cell normally does nothing. Instead, the cursor can automatically navigate outwards as many levels as required to reach the closest right neighbour in the tree, and focus that. However accidentally going past the end of the cell could be quite frustrating, since this may involve moving outwards by multiple levels in a single movement, which would have to be undone using a complex series of multiple inputs. Therefore I decided to implement it this way: if an input causes no change because it moves past the end of the cell, the cursor becomes red to signify that it ‘bumped’ into the boundary. Inputting the same motion again then triggers the outwards-and-sideways motion mentioned above, while any other input clears the bump-mode. This gives some extra leeway when quickly scrolling to the start or end of a cell.

This really solved the navigation out of nested structure, but navigating deeper into nested expressions was still pretty slow. To fix this I added the ‘linear navigation’ mode: while pressing and holding down the navigation dial, the cursor always descends onto the most nested node of the tree, completely skipping Cells. This allows to traverse the code linearly atom-by-atom:

This mode is very useful to select a specific atom in a deeply nested structure, but since it does not skip any content using this mode alone would require turning the dial very far. In combination with the other navigation features though, it becomes very useful: the other modes can be used to move to the surrounding exression, at which point the linear mode can be used to quickly dig deeper into the tree.

implementation

I considered a number of frameworks for implementing this project with, here is a list of them and my thoughts on using each for this project:

• OpenFrameworks (C++)
  • pro: native performance
  • con: slow development cycle; overkill
• Web (HTML, CSS, JS)
  • pro: easy to make beautiful; omnipresent
  • con: cannot access FS, network; not native
• Atom (HTML, CSS, JS)
  • pro: existing text editing engine, [alv parser][langauge-alv]
  • con: doesnt fulfil this assignments criteria
• wxWidgets (C++ or Lua)
  • pro: proven UI system
  • con: i’m trying to innovate on standard UI
• LÖVE (Lua)
  • pro: can reuse existing alv parser
  • con: hard to package

I ended up going with LÖVE, since this is really more of a testbed than a final application, and programming in Lua/MoonScript allows me to prototype incredibly quickly. Using an existing UI framework like wxWidgets, GTK, Qt or even the web wouldn’t really have added anything since my interface should not use buttons, sliders and input boxes anyway.

I reused the existing parser I wrote for alv, which uses the LPeg Parsing Expression Grammars library. The output of the parser is the language’s AST, which is the ideal representation to build a structural editor on top of.

The AST is drawn using a recursive function, which branches into different behaviour based on the type of AST node - Cell, Constant value or ‘Tag’. In both cases, the draw function simply draws the text on the screen using a monospace font, recursively calling itself for any children in the current node. When text is drawn on the screen, a virtual cursor is moved accordingly: It moves one column towards the right for each printed character, and is reset to x=0, y=y+1 for each newline character.

Apart from navigating the AST using enter/exit/forward/backwards commands, the structural editor also allows ‘tweaking’ the currently selected node. When the tweak dial is touched, a Tweak instance is created depending on the type of node. When the node is drawn, the Tweak instance takes over and handles the node instead of the default drawing behaviour. This allows it to render extra content, such as a background for highlighting or the bar of suggestions on another layer before continuing with the rest of the program.

links:

• project source code (fabcloud)
• alv documentation