Week 1 – Principles and Practices, Git & Project Management

Weekly Summary

The first week of FabAcademy! Doing a git refresher, thinking and sketching a potential final project, and building a website. For the site, I choose the long way round, instead of using mkDocs or any other static site generator, I decided to build it from the ground up - both code and design. For the final project I will leave my options open - I don't want to fully commit to fully-defined project yet, but I want to make something that is useful to my, and the bio community in general: one thing that the lockdown made clear, was the lack of home-incubators, that can be small, and integrate more seamlessly into a living environment than a fridge. I have an overall goal of using Sea Sparkle for the the final project, but that depends, if it blooms this year in Kamakura and if I can isolate it.

Individual Assignments

Plan and sketch a potential final project, work through a git tutorial, build a personal site in the class archive describing you and your final project.

Sign the FabAcademy Student Agreement

Signed and committed: FabAcademy Student Agreement

Work through a Git Tutorial

Which Tutorial?

A search on 'git tutorial' brings up the following:

• From the people who brought you git: Git Reference https://git-scm.com/docs/gittutorial
• GitHub Cheat Sheet https://training.github.com
• Interactive, Visual Cheat Sheet https://ndpsoftware.com/git-cheatsheet.html
• Pro Git book by Scott Chacon and Ben Straub and published by Apress https://git-scm.com/book/en/v2
• Resources to learn Git from GitHub https://try.github.io, https://guides.github.com/activities/hello-world/, https://guides.github.com

The Basics

Configure Tooling

• git config --listshell lists a all the user information for your local repositories.
• git config --global user.name "georg.tremmel"
• git config --global user.name "[my_email_address]"

Is that used for login information? No.

• git config --global color.ui auto enables helpful colorization of command line output, I would probably also have to enable colors first.

Creating & Cloning

Create a Local Repository git init

Clone a Remote Repository git clone [url]

Ignore Files

Add Files and Directories to ignore in .gitignore Why ignore files? One the one hand each OS has certain hidden files, that are used for the GUI, but are not needed to be shared. On OSX this is the .DS_Store which exists in each folder. Modules, env, log or editor files should also no be commit to the project: An example of a larger project:

.DS_Store
/vue/node_modules
/vue/dist

#  has REST URL var


# local env files
.env.local
.env.*.local

# Log files
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*

# Editor directories and files
.idea
.vscode
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

Now we have a working repository, let's make and track some changes

Add new content, let's add a file and call it new.txt Git does not know about the content yet - git status always tell you about the current status.

Adding the file with git add new.text

Adding files one by one is a bit tedious, to add all new and changes files in one go, say git add . (Neil rightly warns about the dangers of git add .. Really, really make sure you know what you are adding.)

git commit -m "A very descriptive but not too long message"

This commit the changes to the history. We are still local.

git push pushes your local changes to the server.

In case the repository on the server has changes which are not in your local repository yet, you will get a error, including a nice message asking to solve the conflict.

When trying to do a push before a pull, the terminal is asking for a commit message. This is opened in the default git editor on OSX, which is vi/vim. (Bonus: How to change default editor for git?)

If you have not come across vi it can be slightly frustrating. Writing a message is Vim is not straight-forward, just staring to type is greeted by an error beep, the same goes for pressing enter. Here is how to do it:

• 'i' puts vi into insert mode
• then you can write a message
• the 'esc' key escapes from insert mode
• typing the command ':wq' (command write quit)
• type enter to execute the command

Example workflow:

• update documentation
• git add .
• git commit -m 'Updated Documentation'
• git pull
• git push

Git and Case-sensitive Filenames

On OSX you have the choice of formatting your volumes amongst others with the Apple File System APFS and APFS (Case-sensitive).

I ran into a problem, where I wanted to rename my design files and commit them to git. For example, from wasteless.f3d to WasteLess.f3d.ai

mv wasteless.f3d WasteLess.f3d

git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean

Creating a SSH Key on OSX

Our local SSH key on OSX is located in the ~/.ssh directory. I made a new key, which I will only use for the FabCloud.

Run the following command:

ssh-keygen -t rsa

Give the key a meaningful name:

Generating public/private rsa key pair.
Enter file in which to save the key (/Users/georg/.ssh/id_rsa): georg_fabcloud

You can choose to add a passphrase to the key. No using a passphrase makes connections slightly more convenient, using a passphrase makes it more secure.

Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 

You key has been generated. The Public Key, in this case _georgfabcloud.pub can be shared to proof your identity. Never share your Private Key (_ georgfabcloud)

Your identification has been saved in georg_fabcloud.
Your public key has been saved in georg_fabcloud.pub.
The key fingerprint is:
SHA256:efvk3jYq0n/2I8/qZsZYcacj6caTZ12n42dvpjy/aKXw424OMJmr8Qk georg@local
The key's randomart image is:
+---[RSA 3072]----+
|                 |
|                 |
|                 |
|         . o     |
|        S _*      |
|         . x o   |
|        E.ox@ + .|
|        .=oOoc%OBo|
|        ..o+x%x@O|
+----[SHA256]-----+

Uploading the SSH Public Key to the FabCloud

For the git command to work, we need to provide proof of identity. We can do this by uploading our Public Key to the FabCloud Server.

If everything worked well, your Public Key is shown on the FabCloud.

Working with SSH Keys, Keychain and OSX

For our FabCloud repos we created an SSH Key and uploaded the public key to the server, to create secure communication. This allows us to clone a repo with git clone git@gitlab.fabcloud.org:academany/fabacademy/2021/labs/kamakura/students/georg-tremmel.git

So far so good. If you chose to secure your SSH with a password, you will be asked to enter the password when you do git push, pull clone operations. Not so good.

OSX has an in-build way to securely store keys in the OSX Keychain. (https://developer.apple.com/documentation/security/keychain_services)

Add your key to the OSX Keychain:
ssh-add -K ~/.ssh/fab_id_ed25519

In .ssh/config tell SSH which key to use for which services:

Host *
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/fab_id_ed25519

Command Line Git Tutorials

git also comes with build-in tutorials, an everyday guide, and a glossary:

• git help tutorial https://git-scm.com/docs/gittutorial
• git help tutorial-2 https://git-scm.com/docs/gittutorial-2 (Part 2 of the Tutorial)
• git help everyday https://git-scm.com/docs/everyday
• git help glossary https://git-scm.com/docs/glossary

Something I wasn't aware of: In the OSX Terminal, in git help tutorial, command-clicking on bold text (links?) open a new, yellowish man page in a new, slightly atypical Terminal window.

Sketching a (Potential) Final Project

The final project will be the cumulation of the FabAcademy journey it should show off all the skills learned during the academy, but it should also show creativity and interest.

Milieu Manipulating Machines: Parasite Incubators

• Make an incubator* for N. scintillans (Sea Sparkle, ヤコウチュウ, 夜光虫)
• Catch N. scintillans at Kamakura's Yuigahama Beach
• Grow and multiply N. scintillans in the incubator
• Mechanically and selectively agitate/excite the water and N. scintillans
• Create a volumetric bioluminescence display

Spiraling & Scaling

An incubator is quite a common and basic tool for working and collaborating with micro-organisms. I choose it, because:

• I found myself in situations I could not keep micro-organisms and plants alive, because of the lack of suitable incubator
• It it spiralable and scalable.

Spiraling Opportunities

• Temperature Control
• Light Control
• Camera, Visual Monitoring
• Data Logging, WiFi, Monitoring/Controlling via Internet
• Scale of Incubator (Nano, Micro Incubators?)
• Feeding Mechanism
• Micro-algae growth Montoring
• ...

Background

Climate in Japan

The climate in around Tokyo and Kamakura is not a extreme as in the north, winters can get below 0ºC, summers can be around 40ºC - with high levels of humidity. Houses are following strict earthquake-proofing regulations, mostly fabricated from wood, not much isolation - I assume because of the humidity, and its effect on isolation material. Heating cooling is done with air-conditioners, which have the disadvantage, that as soon as you turn them off, the inside temperature aligns with the outside temperature rather quickly.

Access to Incubators and DIY Bio/BioHacking

Access to incubators in universities and bio-hackers spaces can be difficult during the pandemic. A range of tools, equipment and tactics for and from the BioHacking community exists:

• DIY approaches to Magnetic Stirrers, DIY PCR Machines, Centrifuges, etc
• Affordable Lab equipment like the Bento Lab or Amino Labs
• Second-hand Lab Equipment

Cheap, affordable Incubators are also on the market, cheap ones can be had for ca ~¥10,000 (~$100, ~€85), giving you a basic temperature-controllable environment.

Sometimes these cheap, affordable options are not suitable, after all, it depends on the collaborating organism - and which environment, or milieu, it needs.

Organisms have their own specific environments, here are some common - and more extreme - examples.

• 37ºC, 5% CO₂
• 24ºC, Photo-period of 16h
• 1ºC, High Humidity
• ...

The Incubator(s) - Growing N. scintillans

While I am very clear on the functionality of the incubator, I am less clear about the shape and form of it. Most incubators have fridge-like designs - which take up a lot of space - I would like to explore possible design and sizes.

• Picture Frame Incubator
  Hang your incubator on the wall.

• Muji Shelf Incubator
  Incubator that fits in standard Muji Shelves. 340mm x 340mm x 275mm.

• Mini/Micro Incubators
  How small can the incubator be? Smaller Incubators can scale in number, which in turn enables parallel parameter search for the most suitable growing parameters.

• Parasite Incubators
  Repurpose and re-function already existing boxes and containers. Natural incubators situations within the house/room.

Catching N. scintillans at Yuigahama Beach in Kamakura

The FabLab Kamakura is about 500m from the beach, that's where I observed (for the first time) the amazing and magically spectacle of glowing Sea Sparkle. The picture above was taken in May. Need to observe the most suitable weather conditions of the last blooms and find suitable weather forecasts for upcoming blooms. (Side Project: Make Forecast Site for Sea Sparkle Bloom, can it be seen from satellites?)

Growing N. scintillans

Sea Sparkle grow in Artificial Seawater Medium, with Guillard's (F/2) Marine Water Enrichment Solution, which is for growing Micro-algae, which in turn are consumed by N. scintillans. Suitable temperature are from ca. 10ºC - 24ºC, N. scintillans also has a funky diplontic life cycle, the incubator(s) can also help in finding the right growth conditions for the different life cycle stage.

Many Opportunities for Failing

Ok, now we are getting to the really interesting part. I am aware that this is quite a large and long project, with one big uncontrollable parameters (what if there is no Sea Sparkle bloom in Kamakura this year?) I could try to obtain Sea Sparkle from other sources, but I like the fact that the beach with the organisms and the FabLab are in close, hyper-local distance. Even if I can not obtain and grow the Sea Sparkle, the incubator will functioning and ready. Which is a also a success in itself.

Volumetric Bioluminescence Display

In the best-case scenario, I have the Sea Sparkle, they are growing abundantly, we are now ready for the final experiments. Mechanically disturbing a container of Sea Sparkle light up the container. How can we selectively excise the Sea Sparkle? How can we selectively shake the water? Transparent Nylon wires? Ultra-sound? What is the exact activation energy that Sea Sparkle need to bioluminescent? These questions and experiments can only be done once we can stably and reliably grow and cultivate the organisms.

I am looking forward to the Fab Journey and I am curious what it will uncover and where it might lead!

Project & Time Management

Two things that Neil emphasizes in nearly every lecture are Spiral Development and Supply-Side Time-Management.

Spiral Development

Spiral Development or developing in spirals is the idea to continuously improve a project, to continuously add features, but that the project is always in a working state.

Many small steps, instead of giant leaps of faith.

This echoes the famous MIT motto's of 'Demo or Die' and 'Deploy or Die'. (I assume with the die part the project would die, if it can't be demo'ed or deployed.)

From Spiral to Linear and retour

Neil also has a nice visual metaphor for the FabAcademy process. The yarn ball of dense knowledge is unrolled into a single strand, hopefully taken up by our brains, and then condensed into knowledge once more.

Supply-Side Time-Management

Neil also encourage us to take up Supply-Side Time-Management. Working on problems until they are fixed has the disadvantage of getting lost in seemingly important, but small problems, which don't move the project forward as a whole. The SST approach is to supply a certain amount of time to certain problems and challenges. That way the big picture and overview of the project is kept in sight, the project will be on time, and small problems can always be solved in the next spiral.

Time Management Template

Make a detailed work plan for the coming week.

• I am planning to..
• I have this time (in hours)
• I will do this at that time

Time Management for this Week

Time Budget:

Document as you go:

• Describe failures and successes
• Challenges and solutions
• Take notes during the work for future reference

Build a Personal Website

Environment, Constrains, Notes

The site is hosted at the fabcloud.io, which runs an instance of Gitlab, CI/CD possible, Static Site Generators like MkDocs, Hugo, Jekyll, etc are offered by default. Of course pure HTML is also an option. (See Neil's sites.)

While creating a site with Site Generators and Prefab Themes is of course perfectly valid, I would like to have full knowledge and understanding of the code that is used to generate the site.

Make a (brutalist?) HTML page.

No HTML template, no CSS framework. Build from the ground up. I will allow myself modern CSS and also a modern font. The HTML will be mostly utilitarian, it might come across a bit brutal.

page.html

<!doctype html>
<html>
<head>
  <title>FabAcademy 2021 - Georg Tremmel - FabLab Kamakura</title>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <link href="trembl.css" rel="stylesheet">
  <link rel="stylesheet" href="https://use.typekit.net/mhw5cxw.css">
</head>
<body>
<header>
Navigation
</header>

Content

<footer>
Footer Navigation
</footer >
</body>
<html>

Option 1: Use a Static Site Generator

Now we have 2 options. Use Hugo, MkDocs, VuePress or a similar environment and create a minimal theme based on my HTML.

Option 2: Make your own Static Site Generator

Don't use a site generator, use simple PHP to streamline site maintenance, and use wget to generate static sites from PHP.

While I'd be happy to dive into site generators and make a custom theme, I'd like to stay minimal and not overly rely on frameworks to generate the site. Having said this, when using PHP, a local server is needed

Spiral Development: Sharing Headers & Footers

Wouldn't it be great to not always have to copy/paste the headers and footer? Let's make separate files for shared headers and footers. Move the header and footer section into their own PHP field, the page.php now looks like this:

<?php include('header.php'); ?>
Content
<?php include('footer.php'); ?>

We also need to crawl the site with wget to save the pages and images - and to convert the absolute into relative links:

wget \
  --recursive \
  --page-requisites \
  --no-parent \
  --directory-prefix download \
  --no-host-directories \
  --adjust-extension \
  --convert-links \
  "http://127.0.0.1/fab/"

Condensed version:

wget -rpnp -nH -E -k "http://127.0.0.1/fab/" 

<a href="about">About</a>

File: about.html NG

<a href="about/">About</a>

Folder: about, File: OK

Now we can generate a static website - with nice, clean links - from the PHP files.

Adding Markdown Parser

Markdown simplifies writing documentations, I would like to have it as an option. There are a number of Client- and Server-side Markdown Parser out there. I would like to generate it on the server, and as we are in PHP, I will use Parsedown, a _simple, fast GitHub-flavoured Markdown Parser without dependencies. Which means we can just drop it in, feed it the text - and out comes Markdown.

include('includes/Parsedown.php');
$Parsedown = new Parsedown();

$text = "my markdown text";
echo $Parsedown->text($text);

Spiraling (out of Control): Automatic Image Resizing and Management, Tags, Categories?

Now it's get a little be fuzzy. I could spend more and more time and add more and more features, but this is FabAcademy so I try to stand on the shoulders of giants - and not on their toes (©Neil Gershenfeld).

I integrated my custom code into Wordpress giants, (still not using any plugins or HTML templates/frameworks), wrote a shell script to generate the site with wget, clean the link paths, and copy it to my local git lab directory.

This also allows me to create a filter to set the image quality of imported JPEGs.

function set_jpeg_quality() {
  return 70;
}
add_filter('jpeg_quality', 'set_jpeg_quality');

TL;DR

• Basic, low-level site
• Shared Headers, Footers
• Optional Markdown Parser
• Images resizing and converting to JPEG Quality 70
• Generate Site with wget

The Shell Script, with comments:

#!/bin/zsh

# call script, when content changes

# define target and source directories, don't forget final '/'
TARGET="/Applications/XAMPP/htdocs/georg-tremmel/"
STATIC="/Applications/XAMPP/htdocs/fab/wp-content/static/fab/"

# clear STATIC
# .zsh, set 'setopt rmstarsilent' to allow silent rm *
rm -r ${STATIC}*

# generate static site with wget
#wget \
#  --recursive \
#  --page-requisites \
#  --no-parent \
#  --directory-prefix download \
#  --no-host-directories \
#  --adjust-extension \
#  --convert-links \
#  http://127.0.0.1/fab/

# (multi-line commend don't line spaces after \, so this is NG): wget \     # commend

# same as:
# wget \
#   -r \
#   -p \
#   -np \
#   -P download \
#   -nH \
#   -E \
#   -k \
#   http://127.0.0.1/fab/

# same as:
wget -rpnp -nH -E -k "http://127.0.0.1/fab/" 

# Clean 
# Change to directory, otherwise RE  will also be replaces
cd $STATIC 
LC_CTYPE=C && LANG=C find ./ -type f -exec sed -i '' -e 's///g' {} \;

# Clean Theme Path
mv ${STATIC}* $STATIC
LC_CTYPE=C && LANG=C find ./ -type f -exec sed -i '' -e 's/wp-content\/themes\/fab\///g' {} \;

# Clean Image Path
mv ${STATIC}images/ ${STATIC}images
LC_CTYPE=C && LANG=C find ./ -type f -exec sed -i '' -e 's/wp-content\/uploads/images/g' {} \;
rm -r ${STATIC}/wp-content/

# clear TARGET
# .zsh, set 'setopt rmstarsilent' to allow silent rm *
rm -r ${TARGET}*

# copy to target directory
#-a does not copy invisible files
cp -a $STATIC* $TARGET

cd $TARGET
# check target directory
ls -la

# check sizes of files
# as per suggestion of Neil at Git Recitation
du -sh *

cd $STATIC 
# git push manually, why? push only major updates to the site

Great tip at the Recitation: use du -sh * to check sizes before committing

Workflow Overview

• Edit Site locally
• Run ./generate script
• Push sit to GitLab

Update: Files

Ideally files would reside in sub-folders of the weekly documentation pages, this is not easily achieved owning to my site set-up. I decided to create a separate structure for files:

/files/w1/
/files/w2/
/files/w3/
...

And link directly to the files.

Update: Automatic Sidebar

I populate the sidebar manually in the first couple of weeks. Not as much funs as I thought. I wrote a script, that look for lines starting with # and creates the hierarchically sidebar structure, and also converts the headings into links - so the sidebar links work.

It was also necessary to check, if we are in a code block, and exclude these comments. (which also start with a #).

function header_to_link($h) {
  $h = trim($h, "# \r\n");
  $h = strtolower($h);
  $h = preg_replace("/[^a-z]/", '', $h);
  $h = urlencode($h);
  return $h;
}

Code

Code Repository for this site: https://github.com/trembl/FabAcademy-Static-Theme/