Final project

In general, I'm interested in experimenting with non-screen-based human-computer interaction and interfaces.
I want to explore rebalancing the interaction more from the user's creative sources towards technology and less from technology to the user. I hope to empower the user to become more of a producer of new ideas and perspectives, rather than consumer of computer mediated ideas, and augmenting that with technology.

My hypothesis is that leaving the screen out of the interaction frees up the user's vision to source from their inner resources instead of being fully involved in the interaction. There are many non-screen solutions with audio-based interface, but how about physical interaction?

Starting point: a portable low-treshold notetaking tool

I want notetaking to be as easy as possible. I think active notetaking is an excellent way to reduce information overwhelm as well as enhance thinking and creativity as thoughts can get externalized as they appear instead of trying to remember them. I want to make notetaking as easy as possible.

Requirements:
- Small enough to be carried anywhere with ease
- Connection to phone or computer (and cloud) for storage and organizing the notes
- There should be multiple documents to write on which can be toggled on the device
- Cheap(ish) to produce (there are a lot expensive devices for this purpose)
- Automated notes-to-text integration

The ideation process for the final project idea ▾

The early chain of thought

The earliest idea came from the first week:
A portable sheet that has sensors on the sides so that it knows a location of pen that is "drawing" on the sheet. The pen has a sensor which senses the pressure. There is also a projector on top of the sheet, projecting on it what has been drawn. What is being drawn is also visible on the computer screen (on the right). I'm looking into a possibility to integrate it into a open source design software, such as GIMP, Inkscape or even FreeCad.

I draw a sketch of this idea when I was practicing InkScape:

After discussing with Kris of this idea, he suggested that I'd make my own "rand tablet", an early version of a tablet and a stylus from 1964. Here's link to original paper and youtube video that explain how it works: it sends a pulse in speficic times to each wire under surface. The pen senses pulses, sends them to computer through wire, and based on which time-coded pulses it receives, an accurate position of it is calculated and its input controls the software.

Followed by this chain of thought I made a cardboard prototype of it (or similar at least) in the laser cutting week. It has an integrated swipe for cleaning up the surface.

Idea: Pocket-sized note-taking device

A small tablet type of device that can fit into pocket and has a pen that leaves an impermanent mark on the surface (or doesn't).

Claude gave a simulation of this as we were discussing with it tp make sense of the Rand Tablet (see below).

I continued exploring this idea in the Week 8: Input devices and made a capacitance sensing platform out of copper stripes and epoxy tape as well as a 3D printed "pen", but I couldn't make this idea work and left it to be able to continue to other things.

Idea: Universal pen

A pen that can write on (almost) any surface and it saves the notes based on where the pen tip has moved.

Here's a paper on FlashPen that does something similar with an high-accuracy optic sensor as in a gaming mouse.

After being selected by random selector in the review part of the global lecture, Neil mentioned that an optical sensor is not accurate enough and I should use an IMU sensor to spot patterns in the pen movements.

This idea led to the idea of my Final Project.

The Idea: Omni Pen

A pen is a device with long and tested history of making human creativity into reality and even today almost everyone rely on it at some point. How could we enhance pen's simple yet powerful design with technology, without reducing its original purpose?

A pen with which I can write on any surface, have the text output saved in digital format and eventually be sent wirelessly to another device to view.

With this pen I can add popping ideas to my notes anywhere and anytime without having to carry a notebook etc. with me: all I need is a flat surface.

Bill of Materials:

Timetable:

First iteration: 2 weeks
- Simple 3D printed pen structure
- PCB/electronics for XIAO nRF52840 and IMU
- Developing the ML model to recognize letters (patterns on my handwriting) ​

Second iteration: 2 weeks
- Wireless networking
- Add battery and memory
- Further embedding the electronics in the 3D print
- Interface on a computer ​

Third iteration: 2 weeks
- Make video and slides of the project
- Add toggle to the pen which changes the document to write on (eg. 10 options)
- Update 3D print, electronics and code for the toggle (and others)
- Other necessary updates ​

Future possibilities

- More sensor precision to qualify for catching free handwriting and drawing.

- A widget to be attached on any pen that does the same + leaves the markings of the original pen.

- An augmented reality application that shows with phone camera what was written on a surface (allows for example secret writings).

Log: Development process

TUESDAY 28.4.: Pondering the design

I began to design the first prototype with KiCad as I knew which components were needed. Having the components on a board helps in designing the first 3D print. However, I don’t yet know how is IMU, battery, or EEPROM is used in a PCB. Also how do the components in the Fablab Kicad library correspond to the components I’m using.

So I began a conversation with Gemini about the components, PCB schematics and what needs to be taken into account.

The main takeaways from Geminis answer were:
- IMU and EEPROM communicate through I2C so there’s a need for different addresses. Also pull-up resistors (4.7kohm) for both of these are needed to have a signal working with their pins.
- For the battery, a capacitor (0.1 uF) is needed and placed near the VCC pin of IMU and EEPROM to reduce noise (for pattern recognition). Also having battery protection IC is important in preventing electric shock as it is near the users hand.
- Double sided PCB is needed because it’s so thin.
- For getting the benefit of dual IMU (integrated to the XIAO and the external one), such as knowing the angle of the pen, the IMUs should be placed in parallel on X-Y-Z axises (no need for converting the data to match). This means the external need to placed as the XIAO because XIAO is bigger and have less freedom in placement.

I opened KiCad as, to begin with testing the whole idea, I needed to make a PCB that has XIAO and IMU. The IMU I’m planning to use is named as LSM6DSOXTR and the IMU in KiCad is called LSM6DSV16XTR. There are similar letters but I’m not sure if their pinnings are similar enough to model a PCB on. Comparing their datasheets, they seem to be same enough to be used in KiCad.

LSM6DSOXTR (the one I’m using)

LSM6DSV16XTR (the one on KiCad)

As I didn't know well how the IMU works, I asked Gemini to get me started: I wanted to make it as simple as possible so that the XIAO and IMU work together. The IMU is connected via I2C pins, i.e. SDA (D4 or pin 5 in KiCad) and SCL (D5 or pin 6 in KiCad). Also, Gemini says that there’s a need for decoupling capacitor for the IMU (between 3.3V and GND). This was the initial schematic that I asked for Gemini if it looks correct.

Geminis reply demonstrated that I still didn’t know how the IMU is used on a PCB.

The issues were:
- Vdd pin powers the IMU sensors but SDA and SCL are powered through Vdd_IO pin which I had marked crossed
- CS pin tells the IMU whether to use SPI or I2C, and having it HIGH means I2C is in use. Gemini suggests connecting it to 3.3V to have it HIGH all the time, and I decide to do that (at least for now).
- SDO/SA0 (pin 1) should also be connected to 3.3V or GND to define the address of the chip for the I2C bus. It’s the only address used so it doesn’t matter which one, I set it to GND.

After fixing these issues, the schematic looked like this:

Brought up by Gemini, I came to think if the design should be two boards connected by a cable (eg. FFC) instead of one rigid PCB. Then the structure of the pen relies more on 3D printed shell. Discussing further with Gemini about the cables, the main issue with the connections seems to be intermittent signals and electric noise (which are also why my protoype of a touchpad likely failed a few weeks ago).

This depends also of the directions of the two IMUs in case the IMU needs to be placed in different orientation to the XIAO, eg. turned 90degrees, which would exclude rigid PCB. Asking about the orientation of the two IMUs, Gemini’s answers were hallucinated as I checked the LSM6DSOXTR datasheet which has a clear image of the orientation (below).

It was more difficult to find information about which way the (similar) IMU is placed inside the XIAO. This would be relatively easy to test, but I lost my electronics yesterday by mistake and haven’t yet replaced them.

Here are two picks from the Gemini reply: flexible wires twisted together or flexible PCB.

As the first iteration focuses mostly on testing the whole concept, having the wires twisted together is probably better option than a rigid PCB regarding the noise. For the final design a vinyl cut flexible PCB with a ground copper layer is a good option. It seems the connectors are a likely weak link what comes to using cables.