2. Project management¶
For this weeks assigment we needed to upload our personal information on github and to create a own website on it. Being a complete novice on the subject i needed to learn the basics of all the programs necessary to be able to create my own site. First i like to explain about the different programs and methods i needed to be able to make the site and a small overview of what they do. My OS system is currently windows 10 and for this weeks assigment i installed a Ubuntu terminal on my laptop to be able to use the commands. I would consider myself as a novice when it comes to computer set ups. Along the way i learned to work with different programs but when it comes to settings and preferences of the operations system there is a lot of improvement possible. Here a small summary of the different programs and languages used.
Git: Git is a distributed version control system which enables online collaboration between designers, programmers and developers. Basicly it enable to work and share information between a group of people. You have a online repositary and a local repository. You can have acces to the remote repository by dragging the information to your local one. You can adjust information and share it to the online repository based in our case on gitlab.
git environment where i will be posting my work
ssh To be able to link your local account to gitlab you need to have ssh. This provides a secure connection between your computer and remote repository. To connect you have to fill in a private or public key. This program enables you to have a secure and encrypted connection.
texteditor: In the text editor you can work ofline on the project. here you can adjust files, upload files, or write text. By using Markdown symbols you can prepare the files in such a way that when you share the information on the online repository it gives a website feeling. Further the text editor can be used to share links, upload pictures, highlight sentences, etc. For now i choose to work with sublime text editor which have some good functions for gitlab use.
Markdown: Markdown convert text by using symbols and translate the text in a ready html file. Using markdown symbols is a simplified way for writing in html. Git will convert text written with markdown symbols and safed as .MD automaticly as a html file for online viewing.
css: Where markdown focus on text, css is foccused on the look of the site. It decided colors, location of the file and it is used to alter the looks of a document online. It enables you to create a stunning look of your online repository. I will not be using this for the time being since i would like to put my focus on mastering the other skills.
terminal: The terminal shows a simple display where you can acces your computer by writing the commands. It gives a good overview of the files on the computer and allows you to alter each individual file. The terminal we use to install some requererd programs but also combine your local repository with the online repository. In order to create a safe connection with gitlab we need to connect git with your computer with a ssh key, which recognize the remote user and send information encrypted to gitlab.
imagemagick: This is a tool to downsize your pictures before uploading it to the git server. Uncompressed pictures requere lots of date. To ensure a smooth running interface it is important to downsize the files. Further it has options to resize the picture in the desired size. To change pictures by imagemagick u have to use the terminal and write the changes you would like to commit.
One of the first thing requered was installing a ssh key on your computer. I followed simple instructions to open my terminal to create and copy the ssh key to my gitlab account. I was able to create a ssh key on windows 10 command prompt but not able to copy. It did not recognize the commands. My instructor recommanded installing a ubuntu terminal on my computer to work better with the different commands.
To install the ubuntu server it requere me to take a few steps to do so. First i had to open my taskbar and open windows powershell as administrator. That brought me to a terminal where i had to copy and paste
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
After pressing enter i had to comfirm my command and restart my computer. The next step was installing the ubuntu server on my laptop. This could be done by downloading the ubuntu app in windows store. Once installed you have acces to the ubuntu server. Next step is to set up a username and password and then you are ready to install the git username and the ssh key.
command for git username
git config –-global user.name “YOUR_USERNAME”
command for using your email for uploading git files
git config -–global user.email “email@example.com”
In Command promp i had tried before to install the ssh key and it was present in the terminal using the command:
sh-keygen -t rsa -C "$your_email"
Now all i had to do was copy the ssh file in my gitlab file on https://gitlab.fabcloud.org/profile/keys
Being in a new environment not understanding how it had to be done i received some help getting the ssh key copied to my gitlab account. With the following command i could see my ssh key and copied this to my account.
Next step is cloning your repository from gitlab in to your local repository. A file under the name roomkes/fabacademy2019 was used to send the repository to.
git clone http://firstname.lastname@example.org:academany/fabacademy/2019/labs/waag/students/rutger-oomkes.git fabacademy2019
Chosing a Editor¶
Now the local and online repository are connected you can work local in documenting your projects. A text editor enables you to write on your local repository. Since my lack of knowledge regarding text editors my instructor adviced using sublime text 3. A text editor which have a more friendly interface and can recognize Markdown documents. I might change the editor later in the fabacademy when i get more acquainted to editing. For now this program seems good enough to start documenting.
workflow preview on sublime text editor
Lots of the commands i have to perform to use git and to control my images i make use of the the terminal. Also is the terminal a good place for controlling settings and documents. Here is a list of common used terminal commands.
|ls -la||List of files in directory with detailed information|
|ls -a||List of files including hidden files|
|pwd||Shows current location on the drive|
|cd||This command direct you to a directory. Just using cd returns to home directory. With variation you can move up and down your directories|
|rm (file)||Removes file|
|mv (file)||Move a file|
|mkdir||Make a directory|
|rmdir||remove a directory|
|sudo||Making changes with admistrator rights|
|help||Shows all possible commands in the directory. This you can specify when you want to know about one specific option|
|touch (name)||Make a new directory|
playing in terminal using different commands
When you change anything in the editor, these changes will be visibible in the terminal. When you want to share changes with the online repository you need to use the terminal by using different git commands. to get used to GIT envirement i watched several tutorials about GIT
basic keycomands https://www.youtube.com/watch?v=HVsySz-h9r4 overview invirement < https://www.youtube.com/watch?v=LgY8XuoM8aU&t=428s> common mistakes https://www.youtube.com/watch?v=FdZecVxzJbk&t=105s
|Here is a list of most commonly used commands for git|
|git status||Show the status of the different files. tracked/untracked|
|git -dif||shows changes made inside the file|
|git add (file)||Ads a file to the commit stage|
|git add .||Add all files to commit stage|
|git commit -m “message”||Commiting all the changes to staging area. Add messages with it to be able to check your updates|
|Git push||pushing your local repository with the remote repository. Always made sure there has not been changes in the remote. This result in a conflict|
|gitt pull||Update local repository with remote repository|
|gitt log||Shows changes made with the file|
|git –help||Shows different commands possible in git environment|
|git add –help||Shows the different commands in this particular command option|
|pwd||Shows origin directory|
|git init[project name]||Create new repository|
|git rm (file)||remove file from directory|
|git reset (file)||Retrack a file from the staging area|
When you make changes in your local repository you want to share them with the remote repository. This is the moment you starting using these commands in the teminal. When i add for example a new chapter to my fab academy course with the use of a texteditor (sublime) i like to share this to gitlab. First thing after saving the file i go to my local repository in the terminal and use the git status command. Now you see all the tracked and untracked files. Now you can make a selecting in which files you like to update. Using the command gitt add . you ad all the tracked and untracked files to the commiting stage. If you now press git status all files should be green, meaning the files are now tracked. With the use of git commit -m “message” you put all the files in the staging area. The message is used to clarify your commits. When realising a file shouldnot be in the staging area you can use the command git reset (file) to remove that partulair file from the staging area. Last step is to push your files to the remote repository. this you do with the command git push. When you are working alone on a project these procedure is good. When working with others you first should apply the git pull command. This way you sync the local with the remote repository. After that you can push the changes.
Common procedure when pushing local repository to remote
Mkdocs is a static site generator well suited for documentation. On the fabacademy this site generator is already installed. When i make alterations in my sublime text editor and upload it to the git repository it will shown in the mkdocs static website. I can therefore choose to create a extra page by simple adding a extra folder in my docs. I added a extra folder called contact, which should be visible on my website. http://fabacademy.org/2019/labs/waag/students/rutger-oomkes/ In the text editor you can adjust the looks of the mkdocs interface. In the directory mkdocs.yml. To use certain themes you can upload mkdocs themes in the terminal. You need also to install python pip to use the mkdocs themes. https://www.mkdocs.org/
For now i am happy with mkdocs to show my work. It is a simple template which automatic change when i add new folders in my repository. At one point i might change to a more appealing format.
For good documentation using pictures is essential. With terminal commands you can adjust, identify, quality for picture and resize pictures to your repository.
|Here is a list of most commonly used commands for imagemagick|
|adjust extension||convert rutger.jpg rutgernew.png|
|resize||convert rutgernew.png -resize 50% rutgernew1.png|
|quality||convert rutgernew1.png -quality 85% rutgernew2.png|
|identify||convert rutgernew1.png -quality 85% rutgernew2.png|
|convert, resize, adjus||convert -quality 85% -resize 60% rutger.jpg rutgertest.png|
|convert all||magick mogrify -format png *.jpg|
As a test i used the command on http://academy.cba.mit.edu/classes/computer_design/image.html my own excisting files in images. Made a mess of my image folder. Most of the commands are for making the pictures more suitable for website content. All the pictures are uploaded on my gitlab repository.
A few weeks later i came to the realization that my png pictures were to big. I was number two in filesize on gitlab. Second place sounds appealing but in this case it is not. Your suposed to upload small files on gitlab to keep the website running smoothly. After some trouble seeking i found a way to convert all my images without to much qualityloss with this formula.
magick mogrify -strip -interlace Plane -resize 665x332 -quality 85% *.png
This convert all images at the same time. It will replace the original files When you just want to update one picture you should type.
convert -strip -interlace Plane -resize 665x332-quality 85% source.jpg result.jp
source: [Stack overflow]https://stackoverflow.com/questions/7261855/recommendation-for-compressing-jpg-files-with-imagemagick After bit more research i found that -strip is better not to be used. It removes the copyright from the image. Here is the new formula:
mogrify -interlace Plane -resize 665x332 -quality 85% *.png
Having changed my terminal from ubuntu to windows bash the command i need to use changed. instead of
convert i need to use
magick mogrify -format jpg *.png
Still have to learn all the possible markdown symbols and shortcut. So far i put my focus on the most common use of Markdown. In case of documenting your work i like to highlight a few used commands.
- Highlight code:
you can place 3 ` in front and after sentences, this result in a grey indicated block. (often used in code)
- Size title:
You can use # to decide which size title you like to see, One # is the biggest size, ## sligly smaller, 6# smallest title
- Create a Link:
When you have a website you like to link to you can put the url between <url>, this result in a clickable link
- A link button using a specific word/sentence to a website can be created by
`[specific word](url from specific website)
- bold or italic text:
When you like to emphasize something, you can add *emphasize* around the text for italic and to **emphasize** in bold
- Adding pictures:
![text for image](directory file where image is placed )
Adding pictures from computer to local repository¶
One struggle i encountered was adding new pictures to my repository. First i tried dragging files in my text editor file. It turned out that when i do it like this i have no permission to gitt add (file) to commit fase. I had to give permission manually. A way that seems to work for now is to upload the picture in my gitlab environment and by using gitt pull dragging it in my local repository. Works for now but keen on finding a more efficient way for doing this.
Update on using my terminal¶
I kept having issues with ubuntu terminal installed on windows 10. Often when i work on the terminal i kept recieving permission denied messages. This was frustrating since i like to continue on my new assigment and it did not alowed me to push my work in to gitlab. I really wanted to fix these recuring issues and start searching for a perminant solution. In my limited knowlegde i asumed the problem was with a ubuntu terminal in a windows workplace. I searched if it was possible to use a windows terminal to push my git project further. I found a good tutorial on youtube how to install git bash and set up the new settings. https://www.youtube.com/watch?v=SzOa1TFltaY&index=4&list=PLdYQMTciVWO-J8Qj3G6YVwUmQ8vkkgTZd. Following the different tutorials i was able to create a new terminal setting. To get acces to my sublime documents i had to go in many subdirectories. I was affraid to have to do these steps everytime i open the new git bash terminal and decided to replace the whole directory in a more accesable folder. When i go into the terminal it indicate Master and using git status i could see my repository. I did all the steps before when installing ubuntu to set up my account.
git config --global user.name "Roomkes"
git config --global user.email "email@example.com"
Hoping to be able to push now i tried the git add . command.
$ git add . fatal: Unable to create 'C:/Users/Ruter Oomkes/Desktop/fabacademy2019/rutger-oomkes/.git/index.lock': File exists. Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes are terminated then try again. If it still fails, a git process may have crashed in this repository earlier: remove the file manually to continue.
After search the error i found online the solution.
rm -f .git/index.lock
This solved the issue and i could commit changes. I recieved the folowing error trying to push.
$ git push The authenticity of host 'gitlab.fabcloud.org (188.8.131.52)' can't be established. ECDSA key fingerprint is SHA256:Rphnjz211wPLjTJcyAip0xozt0hOoFgT9hl5eZfxzd0. Are you sure you want to continue connecting (yes/no)? y Please type 'yes' or 'no': yes Warning: Permanently added 'gitlab.fabcloud.org,184.108.40.206' (ECDSA) to the list of known hosts. firstname.lastname@example.org: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.
In response to this i recreated a new ssh key. After this i used the clip command to copy the ssh key.
$ clip < ~/.ssh/id_rsa.pub
This i pasted again on my ssh key settings in gitlab acount. Now i was able to push my local repository in to my git lab account. I hope i now achieved a more stable git environment. I do receive a message everytime i push.
$ git add docs/assignments/week03.md warning: LF will be replaced by CRLF in docs/assignments/week03.md. The file will have its original line endings in your working directory
I looked on line what are the consequences of the comment and it seemed not to be important. https://stackoverflow.com/questions/5834014/lf-will-be-replaced-by-crlf-in-git-what-is-that-and-is-it-importantThere is a way to hide these comments but i first like to verify that it is indeed not relevant for me. This was the possible command to hide this comment.
git config --global core.autocrlf false
I really hope these changes will make the work flow better and that i don’t have to spend lots of time on problem solving. Lets see how it will go. Otherwise i keep updating the process here.
Trying to clean my repository¶
In the first weeks of the fab academy i have been uploading large png files to my git repository. In result of this i ended up with a way to big repository. After some troubleshooting i decided to clean my git account with the use of bfg repo cleaner. https://rtyley.github.io/bfg-repo-cleaner/ This program is used to clean unwated or big files from the repository. It gives a clear overview in how to archieve this. As mentioned earlier i am by far a expert in using git or working in the terminal, so this simple tutorial turned out quite a challenge. I mostly was worried to erase my complete repository or delete all my commits in the past.
First step was to clone my repository which i did with the following command.
$ git clone --mirror git://example.com/some-big-repo.git
For the git i used my link in gitlab. After this i had to install java runtime environment to be able to use the bfg repo cleaner. When i tried to use the following command it did not recognized my command.
$ java -jar bfg.jar --strip-blobs-bigger-than 100M some-big-repo.git
Trying to check where the mistake was i tried different directories to work with.
$ cd some-big-repo.git
I went to rutger-oomkes.git directory thinking this was the right step. Using command:
$ git reflog expire --expire=now --all && git gc --prune=now --aggressive
After which i used git push to my remote repository.
Curious to see what effect this had on the size of the repository i checked on gitlab my account. It did change indeed. Only the opposite way. My new size was almost twice as big as the previous version. I my lack of skills in git and terminal i posted my mirror clone and my local repository to gitlab. Now having my history posted twice on gitlab…
I the flow of trying to clean the repository my size doubled again to a a shoking 850 mb. In despiration of how to fix tis i recieved some help of my instructor. After the review of Neil Gershenfeld where he explain how to get rid of the history in the repository we managed to get rid of the history. https://vimeo.com/320588564
In the terminal you can get rid of the history. I explain the command used for doing this. When you are in your masterbranch in the terminal you can type
ls -a .git
This shows you the hidden folders including the history. Even when removing files in your repository the will remain in the history. Resulting in enlarging the repository. To remove history you clone your remote server repository. For this you direct to the remote server.
cd site (repository) git clone --bare sit server
Neil explaining how to erase history
When go to the log of the server you see all the files stored in the history. To remove the history you remove the whole git directory.
rm -rf .git
Now you create a new git repository.
Now when you check:
You see a empty directory. On the server it still has all the history. Now with command. Change the server master to your specific server
git push --set-upstream ../server master
When this command is exuceted you recieve a warningand the command is not done. By using –force you overwrite the warning and are able to push.
git push --force --set-upstream ../server master
This way you able to push and get rid of the hidden directory. Not your files. Now the repository is cleaned from the history and only contains your last push. With this procedure you are able to downsize your repository from old files.
What i learned (and failed) on this weeks assignment¶
This weeks assignment was very learning full in teaching me the essentials for good documentations and use of all the different programs to share your local repository with the gitlab repository. I have now a understanding of the purpose of the different facets in documenting. This being said i realize i still have big steps to make to master all the different skills. Most hard in this week assignment i found using git in the terminal. Both were alien to me before i started out and i spend allot of time figuring out problems that occurred. From files that were not recognized to permission denied. Working on windows 10 with a ubuntu server also resulted in not being able to use imported pictures in the local repository. One of the reasons for this that i haven’t optimized the settings between ubuntu and windows 10. I will continue to work on this the coming weeks to create a site more appealing to the eye.
This weeks assigment keeps me occupied during the course since this is a interface i have to keep working with the whole fab academy. I changed my terminal in the mean time to git bash which made my work much more efficient. I did mess around in the beginning making a huge mesh of my repository resulting in double files, weird directories, huge sized git repository. Now my repository size is modest again and i will try my best of keeping it this way.Till the next problem accurs.. Well.. Thats life.
The latest update