Final Project

Application programming

In-out Board (ATtiny3216)

In-out board pattern

Potentiometer

  1. BPM - PA4
  2. Mosaic Rate - PA5
  3. Volume(velocity) 2 - PA6
  4. MIDI Note 2 - PA7
  5. Volume(velocity) 1 PB5
  6. MIDI Note 1 PB4

Switch

  1. MIDI Channel 1 - PC1
  2. MIDI Note on 1 - PC0
  3. MIDI Channel 2 - PC2
  4. MIDI Note on 2 - PC3

OLED(I2C)

  • SCL
  • SDA

Check values and format

For analog read(potentiometers) and digital read(switches), I adjusted values to utilize on application.

  • BPM: 30 - 360
  • Mosaic Rate(mosaic pixel per screen height): 2 - 24
  • Volume: 1 - 127
  • MIDI Note: 1 - 108 (As many of upper note number of MIDI channel is 108)
  • MIDI channel: 1 - 16

Serial transmit to Raspberry Pi

Using ArduinoJson library, I made json document assigning items from potentiometers.

  doc["bpm"]=p1; // purple, BPM
  doc["mrt"]=p2; // white, Mosac Rate
  doc["vl2"]=p3; // blue, Volume(velocity) 2
  doc["nt2"]=p4; // green, MIDI Note 2
  doc["vl1"]=p5; // yellow, Volume(velocity) 1
  doc["nt1"]=p6; // red, MIDI Note 1
  
  doc["up1"]=s1; // red, MIDI Channel up 1
  doc["dr1"]=s2; // yellow, MIDI Note on 1
  doc["up2"]=s3; // green, MIDI Channel up 2
  doc["dr2"]=s4; // blue, MIDI Note on 2
  doc["ch1"]=ch1; // (yellow), MIDI Channel 1
  doc["ch2"]=ch2; // (blue), MIDI Channel 2
Example of JSON format data
{'up1': 1, 'nt1': 39, 'vl1': 18, 'vl2': 111, 'nt2': 36, 'dr2': 1, 'dr1': 1, 'img': 1, 'up2': 1, 'bpm': 71, 'mrt': 9, 'ch2': 9, 'ch1': 12}


Graphics

For displaying the note mark of "♪", I created a graphics by 15x16(height x width) pixels.

The hight and pixel match with the size of text and row of OLED.

Display OLED over I2C

The way to setup I2C pin swapping from default and define address is at networking section.

Raspberry Pi

Multi-thread application

Because this project needs to handle image processing and sound generation simlataniously, I need to implement pararel processing system. Main application also needs to receive signals from input devices at the same time. . Raspbian OS on Raspberry Pi can support multi process or multi thread program.

This system needs to handle 3 kinds of processes running seprately as follows:

Receive serial (Communication)

Application needs to receive message over serial from ATtiny3216 In-out board. The interval to receive depends on In-out board. For getting input signal instantly, the setting of "delay" on In-out board is 100 msec interval.

Send MIDI (Sound)

Sending MIDI data to software synthesizer from application depends on tempo while MIDI itself is serial protocol. Tempo is controlled by BPM(Beat per minutes) controlled by potentiometer. Channel 1's BPM is fixed by 360 BPM and Channel 2'is from 30 to 360 BPM. The highest 360 BPM means 6 beat per seconds = 166.6 msec interval

Video stream

Capturing image and streaming video with editing frames require low latency process. Whereas usual animation is 30FPS(Frame per seconds), for PoTone system I configured 20FPS for processing image. 20FPS = 50 msec interval

My idea is to make separated worker threads (executor = concurrent.futures.ThreadPoolExecutor(max_workers=2)) for "Receive serial" and "Send MIDI" from main thread that handle video stream. For communication between separated thread, I used internal FIFO queue(queue.Queue()). This realizes asynchronous processing in real time system. On starting main process, main thread kicks the other following 2 threads by concurrent.future library package. Main thread itself handles video stream after kicks the other threads.

For stopping concurrent threads, send "f" key to standard input on Raspberry Pi's terminal.

Behavior

Receive serial("Communication" worker thread)

  • Check USB serial port by running operational command(bash) from python
  • Open and close Serial communication
  • Receive signals from potentiometers and switches
  • Transmit signals to the other threads (video stream and sound generation) via internal queues

Send MIDI ("Sound" worker thread)

  • Check, open and close MIDI port of Qsynth(software synthesizer)
  • Receive signals from communication thread via internal queue
  • Parse JSON data
  • Make 3 kind of sounds at the same time
    • Channel 1: make sound (using volume 1, midi note 1 and MIDI channel 1) in 360 BPM(BPM is fixed).
    • Channel 2: make sound (using volume 2, midi note 2 and MIDI channel 2) in X BMM(BPM is configured by potentiometer)
    • Image: make sound (using volume 1, midi note (called by image thread) and channel (by image thread) in X BPM (BPM is configured by potentiometer)

Image processing (called by main thread)

  • Open and close video source
  • Capture image from camera
  • Receive signals from communication thread via internal queue
  • Make canny frame (outline of image)
  • Capture motion
  • Invert color and make mosaic pattern
  • Assign mosaic pixels to MIDI note number and channel as following matrix.
    • The height of mosaic represents MIDI note number
    • The width range of mosaic represents MIDI channel (like left: 0(Fantasia), middle 9(Room1), right 15(Koto))
  • Display videos. Video displays are closed by "Esc" key.

Python and library packages versions

  pi@homepi:~/repo/tatoflam/potone/bin $ python3 --version
  Python 3.5.3

Following list is extracted from pip3 list at the same directory

  pi@homepi:~/repo/tatoflam/potone/bin $ pip3 list
  Package               Version    
  --------------------- -----------
  astroid               1.4.9      
  asttokens             1.1.13     
  automationhat         0.2.0      
  blinker               1.3        
  blinkt                0.1.2      
  buttonshim            0.0.2      
  Cap1xxx               0.1.3      
  chardet               2.3.0      
  click                 6.6        
  colorama              0.3.7      
  colorzero             1.1        
  cryptography          1.7.1      
  cycler                0.10.0     
  decorator             4.0.11     
  docutils              0.13.1     
  drumhat               0.1.0      
  envirophat            1.0.0      
  ExplorerHAT           0.4.2      
  Flask                 0.12.1     
  fourletterphat        0.1.0      
  gpiozero              1.5.0      
  h5py                  2.7.0      
  idna                  2.2        
  intelhex              2.2.1      
  isort                 4.2.5      
  itsdangerous          0.24       
  jedi                  0.10.2     
  Jinja2                2.8        
  joblib                0.10.4.dev0
  Keras                 2.2.4      
  Keras-Applications    1.0.7      
  Keras-Preprocessing   1.0.9      
  keyring               10.1       
  keyrings.alt          1.3        
  lazy-object-proxy     1.2.2      
  lxml                  4.2.5      
  MarkupSafe            0.23       
  matplotlib            2.0.0      
  mcpi                  0.1.1      
  microdotphat          0.2.1      
  mido                  1.2.9      // generating MIDI message
  mote                  0.0.4      
  motephat              0.0.2      
  mypy                  0.470      
  nose                  1.3.7      
  numpy                 1.12.1     // dependency of opencv
  oauthlib              2.0.1      
  opencv-contrib-python 4.1.1.26   // image processing
  opencv-python         3.4.4.19   // image processing
  pantilthat            0.0.7      
  pbr                   4.0.3      
  pgzero                1.2        
  phatbeat              0.1.1      
  pianohat              0.1.0      
  picamera              1.13       
  picraft               1.0        
  piglow                1.2.5      
  pigpio                1.38       
  Pillow                4.0.0      
  pip                   20.0.2     
  pretty-midi           0.2.9      
  py                    1.4.32     
  pyasn1                0.1.9      
  pycrypto              2.6.1      
  pygame                1.9.3      // dependency of mido
  Pygments              2.2.0      
  pygobject             3.22.0     
  pyinotify             0.9.6      
  PyJWT                 1.4.2      
  pylint                1.6.5      
  pyOpenSSL             16.2.0     
  pyparsing             2.1.10     
  pyperclip             1.5.27     
  pyserial              3.2.1      // serial communication
  pytest                3.0.6      
  python-apt            1.1.0b5    
  python-dateutil       2.5.3      
  python-rtmidi         1.4.1      // dependency of mido
  pytz                  2016.7     
  pyxdg                 0.25       
  PyYAML                5.1        
  rainbowhat            0.1.0      
  requests              2.12.4     
  requests-oauthlib     0.7.0      
  roman                 2.0.0      
  RPi.GPIO              0.6.5      
  RTIMULib              7.2.1      
  scikit-learn          0.18       
  scipy                 0.18.1     
  scrollphat            0.0.7      
  scrollphathd          1.2.1      
  SecretStorage         2.3.1      
  sense-emu             1.1        
  sense-hat             2.2.0      
  setuptools            33.1.1     
  simplejson            3.10.0     
  six                   1.12.0     
  skywriter             0.0.7      
  sn3218                1.2.7      
  spidev                3.3        
  stevedore             1.28.0     
  Theano                1.0.4      
  thonny                3.1.0      // IDE
  thonny-pi             1.1        // IDE
  touchphat             0.0.1      
  twython               3.4.0      
  typed-ast             0.6.3      
  unicornhathd          0.0.4      
  urllib3               1.19.1     
  virtualenv            16.0.0     
  virtualenv-clone      0.3.0      
  virtualenvwrapper     4.8.2      
  Werkzeug              0.11.15    
  wheel                 0.29.0     
  wrapt                 1.9.0      
  WARNING: You are using pip version 20.0.2; however, version 20.1.1 is available.
  You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Operational command

Behavior

Start service bin/potone.sh start

  • Automatically called on boot Raspberry Pi with displaying LX terminal(added to ~/.config/autostart/potone.desktop)
  • Check if processes have already started
  • Start qjackd process
  • Start Qsynth process
  • Start PoTone application service

Stop service bin/potone.sh stop

  • Check if processes have already started
  • Stop Qsynth process
  • Stop qjackd process
  • Note that PoTone application itself is stopped by key operation ("Esc" key closes video screens and "f" key stops worker threads and main application)

Check USB portsbin/checkUSB_linux.sh

  • Check and dispaly the USB port number and name on Linux
  • Called by python "Communication" thread on application on opening Serial port.
  • Note that python calls redirected command that finds port name and return port number for serial communication (GET_USB_PORT_COMMAND="../bin/checkUSB_linux.sh | awk '/D307RG9Y/{print($1)}'")

Files (Source code)

Arduino (ATTiny3216 In-out board)
Python (Raspberry Pi)
bash (Operational command)

Reference