Skip to content

Project Management

  • Build a personal site describing you and your final project
  • Work through a git tutorial
  • Upload it to the class archive

Setup

I’m already familiar with website development since I was working as a Data Scientist before entering the Fab Academy. I know HTML/CSS basics (though it’s been a while), markdown syntax and I daily work with git and GitLab.

This week I plan to setup my documentation to ease my workflow for the next six months to come.

GitLab and SSH keys

When I received my credentials to access all Fab Academy tools, a website was already configured. To work locally, I needed to clone the repository by creating a SSH key and adding it to my new account. GitLab offers a detailed example of this process here.

# Create a new SSH key
ssh-keygen -t ed25519 -C "fabacademy"

# Copy to clipbpard
xclip -sel clip < ~/.ssh/id_ed25519_fab.pub 

As I already have a personal GitLab account, I configured my ~/.ssh/config file to include my new repository (which you can find here).

# Personal Account Identity
Host gitlab.com
  Hostname gitlab.com
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/id_ed25519

# Fab Academy Account Identity
Host gitlab.fabcloud.org
  Hostname gitlab.fabcloud.org
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/id_ed25519_fab

I then verified that I was connected to the right server with ssh -T git@gitlab.fabcloud.org and accepted to add it as a known host.

After everything was set, I cloned the repository on my laptop.

git clone git@gitlab.fabcloud.org:academany/fabacademy/2022/labs/sorbonne/students/chloe-laurent.git

Mkdocs and virtual environments

I decided to use the provided MkDocs as a static site generator: I like to keep my documentation simple, and I know I can customize it with custom CSS and JS. Moreover, there are a lot of new python plugins that I want to explore.

After opening my newly cloned folder, I created a python virtual environment, a good practice that proved itself very valuable over the time.

# Create a virtual environment
python3 -m venv .venv-fab

# Activate
source .venv-fab/bin/activate

# Verify
which python
>>> $REPOSITORY/.venv-fab/bin/python

Warning

Don’t forget to add your newly created virtual environment into your .gitignore file, otherwise you’ll end up with a git instance way too heavy.

Once you have activated your virtual environment, you can install mkdocs and as many plugins as you want without interfering with your main python instance (and believe me, you don’t want to mess up your main python, especially on a linux OS!).

pip install mkdocs-material

You can now build you website locally and start completing your documentation! You will automatically see your changes on your browser (127.0.0.1:8000).

mkdocs serve

>>> INFO     -  Building documentation...
>>> INFO     -  Cleaning site directory
>>> INFO     -  Documentation built in 0.19 seconds
>>> INFO     -  [19:28:36] Serving on http://127.0.0.1:8000/2022/labs/sorbonne/students/chloe-laurent/

Writing the documentation

I will try to keep the mantra document as you go alive during the all Fab Academy. This week it’s quite easy because I’m only working on the computer, but I know that it will become harder when using the machines in the lab. I have with me all the time a little notebook where I note everything. I hope this will help me!

notebook picture

Markdown

I will use mainly markdown, so I made myself a cheatsheet where I’ll keep some useful features.

Visual Studio Code

I used Visual Studio Code as an IDE for a while now and I’m very satisfied with its UI and practicality. It’s open source and available on all platforms if you want to give it a try. I love that there are a lot of extensions in there and that you can write your own if you want.

Here’s a peek of how my screen looks like when I’m working: Screen Peek

Custom theme

I really wanted to pimp my website, even if I used a static site generator. MkDocs is very reliable but not that fancy, so I twisted it a little to obtain a result that I like. I use as a base template Material for MkDocs.

You can change a lot with only the mkdocs.yml file, such as the title, the color palette, the fonts…

Extensions

The Material theme was already configured on my website when I cloned it. I completed the basic informations on mkdocs.yml and added some plugins and markdown extensions that I found on the Material documentation.

These are extensions that I didn’t know and found very useful:

  • pymdownx.keys

    Add keycaps on your doc Ctrl+Alt+T

  • pymdownx.tasklist

    • Add task lists to your assignments
  • attr_list

    Add HTML/CSS tags directly to markdown syntax: ![img](../images/img_web.jpg){: .width=50%}

  • pymdownx.highlight and pymdownx.superfences

    Add highlights on code blocks

Custom stylesheets (CSS and JS)

You can also write HTML directly in your markdown files, but not only. It’s possible to write custom CSS and JS to customize your website even more.

extra_css:
  - stylesheets/extra.css

extra_javascript:
  - javascript/extra.js

To help me build a beautiful website, I used the Fab Academy 2022 color palette to overwrite default colors of mkdocs.

Color palette

I also added a radius to the border of images, and a gallery on the home page.

img {
    border-radius: 6px;
}

.gallery {
    display: flex;
    flex-direction: row;
}

.gallery-link {
    width: 50%;
    margin: 3px 4px;
}

.gallery-link a {
    width: 100%;
    height: 100%;
    display: block;
    text-align: center;
    color: #f04260;
}

.gallery-link div {
    display: flex;
    flex-direction: column;
    justify-content: center;
    align-items: center;
    text-align: center;
}

I found a useful zoom feature hack, it allows to scale and outscale an image.

/* CSS */
.zoom {
    transition: transform ease-in-out 0.5s;
    cursor: zoom-in;
}

.image-zoom-large {
    transform: scale(1.5);
    cursor: zoom-out;
    position: relative;
}
// JS
document.querySelectorAll('.zoom').forEach(item => {
    item.addEventListener('click', function () {
        this.classList.toggle('image-zoom-large');
    })
});

For the final touch, I added the Fab Academy 2022 assets for the favicon and the home button. Ready for six months of documentation!

fabac logo

You can now see the differences between the original website and my version:

Fab Academy student website template After tweaks

Storage optimization

Note

Before pushing my work to the cloud, I would like to write some scripts to optimize the size of my repository. Ideally, I want to automate the process and execute it just before the deployement.

I can write a python script, but maybe there is a more efficient way ? Here are some articles I found during my research:

What is the size of my GitLab repository?

I started by checking my repository size with the use of git-sizer --verbose and obtained the following output:

Processing blobs: 103                        
Processing trees: 98                        
Processing commits: 39                        
Matching commits to trees: 39                        
Processing annotated tags: 0                        
Processing references: 3                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size      |           |                                |
| * Commits                    |           |                                |
|   * Count                    |    39     |                                |
|   * Total size               |  9.08 KiB |                                |
| * Trees                      |           |                                |
|   * Count                    |    98     |                                |
|   * Total size               |  16.8 KiB |                                |
|   * Total tree entries       |   456     |                                |
| * Blobs                      |           |                                |
|   * Count                    |   103     |                                |
|   * Total size               |   587 KiB |                                |
| * Annotated tags             |           |                                |
|   * Count                    |     0     |                                |
| * References                 |           |                                |
|   * Count                    |     3     |                                |

Nothing to worry about for now, but I can easily imagine how this could look like after a few weeks of documenting Fab Academy: large images and blob files all over the place. I will keep this command in my CLI cheatsheet.

Dealing with large files

I would like to use Git LFS to track my most large files (3D files for example). However, this seems to be not applicable for GitLab pages, so I won’t be using this for images and videos.

Warning

This must be done before committing the files to the repository.

Luckily, I have not commited anything large for now (mainly .md files). I can configure my storage rules from scratch without having to migrate old files.

  1. Install git-lfs

    • On your local machine: sudo apt-get install git-lfs
    • On your repository: git lfs install --local
  2. Track your large files

    • git lfs track "$FILENAME"
    • git add .gitattributes
  3. Check which files are tracked

    • git lfs ls-files

After these steps, I have nothing more to worry about, all my files in .gitattributes will be tracked with Git LFS.

Success

I tested it during the Computer Aided Design week and it worked perfectly. My files are stored with Git-LFS and are accessibles in the website.

lfs

Pruning a git tree

Warning

This can damage the repository, so I’ll just put a link of the procedure here as a reminder, I won’t do it as it is quite unnecessary for the moment.

Image and video compression

Images

When I look at this week images folder (with ncdu), I can see that they take a lot of place, especially the photo I took of my notebook:

    1,7 MiB [ 65,4%]  notebook.jpg
  440,0 KiB [ 16,7%]  screen_peek.png
  192,0 KiB [  7,3%]  website-before.png
  160,0 KiB [  6,1%]  website-after.png
   64,0 KiB [  2,4%]  fab-color-palette.png
   52,0 KiB [  2,0%]  notebook-size.png

This is completely useless, because if I take a closer look to my generated website, I can see that my images are reduced to a sixth of their sizes (in the case of JPG) ! screenshot

By using python and OpenCV, I wrote a script to resize automatically images that are way too big for this usage.

pip install opencv-python
def resize_image(img_path: str, width: int) -> object:
    '''
        Resize images bigger than 800px width
    '''
    image = cv2.imread(img_path, cv2.IMREAD_UNCHANGED)
    (img_height, img_width) = image.shape[:2]

    if img_width > width:
        ratio = width / float(img_width)
        dim = (width, int(img_height * ratio))

        resized = cv2.resize(image, dim, interpolation=cv2.INTER_AREA)
        return resized

    return image
"""
    Python script to optimize images and video sizes for web
"""

from typing import List
import argparse
import os
import cv2


def parse_arguments():
    """
        Argument parser
    """
    parser = argparse.ArgumentParser(
        description="Usage:  python compression.py --path $FOLDER_PATH"
    )
    parser.add_argument("--path", type=str, help="Path to the folder.")
    parser.add_argument("--width", type=int,
                        default=800, help="Maximum px width")
    parser.add_argument("--quality", type=int,
                        default=70, help="JPEG quality")
    return parser.parse_known_args()


def list_images(path) -> List[str]:
    """
        Returns a list of JPG and PNG images
        in the folder directory and subdirectories
    """
    img_list = list()

    for root, _, files in os.walk(path):
        for file in files:
            if ".svg" in file or ".mp4" in file:
                print(f"File is not JPG or PNG: {file}")
            elif "_web.jpg" in file:
                print(f"File already optimized for web: {file}")
            else:
                img_list.append(os.path.join(root, file))

    return img_list


def resize_image(img_path: str, width: int) -> object:
    '''
        Resize images bigger than 800px width
    '''
    image = cv2.imread(img_path, cv2.IMREAD_UNCHANGED)
    (img_height, img_width) = image.shape[:2]

    if img_width > width:
        ratio = width / float(img_width)
        dim = (width, int(img_height * ratio))

        resized = cv2.resize(image, dim, interpolation=cv2.INTER_AREA)
        print(f"File resized for web: {img_path}")
        return resized

    print(f"File not resized: {img_path}")
    return image


def compress_images():
    """
        Main function
        Resizes images and rewrites the folder
    """
    args, _ = parse_arguments()
    img_list = list_images(args.path)

    for img in img_list:
        web_img = resize_image(img, args.width)

        path_in, _ = os.path.splitext(img)
        path_out = f"{path_in}_web.jpg"

        cv2.imwrite(
            path_out, web_img, [cv2.IMWRITE_JPEG_QUALITY, args.quality]
        )
        os.remove(img)


if __name__ == "__main__":
    compress_images()

I chose a default width size of 800px (thanks to that article), but it can be changed to match a specific case.

# Blog images
python compression.py --path ./docs/images/week02/

# Home gallery
python compression.py --path ./docs/images/gallery/ --width=400

We can see the size was reduced to an acceptable web size, without loosing ratio or quality.

52,0 KiB [ 20,6%]  website-before_web.jpg
52,0 KiB [ 20,6%]  screen_peek_web.jpg
40,0 KiB [ 15,9%]  fab-color-palette_web.jpg
40,0 KiB [ 15,9%]  website-after_web.jpg
40,0 KiB [ 15,9%]  notebook_web.jpg
24,0 KiB [  9,5%]  notebook-size_web.jpg

Videos

I will use ffmpeg to compress my videos later on.

Success

During the Electonic production week, I made some explanatory videos and needed to compress them (they were unnecessary long and loudy).

I used this command to compress them without the sound:

ffmpeg -i input_video.mp4 -vcodec libx264 -b:v 1000k -vf scale=-2:1080 -an output_video.mp4

And this one to cut the ten first seconds of it:

ffmpeg -i input_video.mp4 -vcodec libx264 -b:v 1000k -an -ss 00:00:00 -t 00:00:10 output_video.mp4

To insert a video into my documentation:

<video width="400" height="200" controls autoplay>
    <source src="../../files/video.mp4" type="video/mp4">
</video>

Thumbnails

For the thumbnails on the home page, I need to keep the same ratio. I can use gThumb to resize my gallery as I want. The ideal ratio is 388*218.

CI/CD

I am now ready to push my work to the class archive with a simple git push after committed all my modifications.

Locking support detected on remote "origin". Consider enabling it with:
  $ git config lfs.https://gitlab.fabcloud.org/academany/fabacademy/2022/labs/sorbonne/students/chloe-laurent.git/info/lfs.locksverify true
Enumerating objects: 105, done.
Counting objects: 100% (105/105), done.
Delta compression using up to 8 threads
Compressing objects: 100% (79/79), done.
Writing objects: 100% (91/91), 255.36 KiB | 18.24 MiB/s, done.
Total 91 (delta 26), reused 1 (delta 0)
To gitlab.fabcloud.org:academany/fabacademy/2022/labs/sorbonne/students/chloe-laurent.git
   5b3835a..49777d1  master -> master

pipeline

The pipeline deployed the website without any issue.

Note

That’s it for this week, I’m glad I did my documentation along with my modifications, it allowed me to keep track of everything.


Last update: March 1, 2022
Back to top