As for this week's assignment, I needed to make an interface and application programming. So I decided to use this opportunity to make the hologram video for my final project. The main goal is to take the feed from the webcam, understand the object, blur / remove the background and rotate it in four directions to make it a video for projecting on the hologram.

I am completely new to programming and was stuck not knowing what to choose. As suggested by everyone processing seemed like the easiest option, but since my project required a lot of video analysis, I came across OpenCV(Open Source Computer Vision Library) that would help me in the process. I found a lot of tutorials for it online and mostly using python language. So I started with my basic research on python.

Stack Overflow boiled it down to one main reason: the rise of Python can be connected to the rise of interest in data science. Additionally, Python has become a go-to language for data analysis. With data-focused libraries like pandas, NumPy, and matplotlib, anyone familiar with Python’s syntax and rules can deploy it as a powerful tool to process, manipulate, and visualize data.

As a beginner, I did not understand much of it but seemed like a good language to learn and start with. Furthermore, the internet is filled with tutorials for beginners to learn python.

Downloading and Installing Python

The following link leads to the python download page.

But once in the download page, you are faced with another question. Python 2.x or 3.x. I didn't know it mattered much. As usual, I thought the bigger the version number more the improvements. In case of python, that wasn't true. There is a whole debate on the internet re3garding this matter, and here is a link to one such research done by a beginner as I am. Also a video by an expert and his reasons.

Of both the research and arguments what stuck to me was "Python 2.x is legacy, Python 3.x is the present and future of the language." So I concluded and decided to use python 3. I still don't understand how it's going to affect me but seems like the better option.

From there installing the application was straightforward. Opening up the .exe file brings up the dialogue box for installation.

Using Python

IDE

An integrated development environment (IDE) is a software suite that consolidates the basic tools developers need to write and test software. Typically, an IDE contains a code editor, a compiler or interpreter and a debugger that the developer accesses through a single graphical user interface (GUI). An IDE may be a standalone application, or it may be included as part of one or more existing and compatible applications.

There are several IDE platforms with different features. Some come with GUI and others don’t. The following link gives an idea of IDE

Python Comes with IDLE that's quite boring.

I wanted to go with something that had GUI. So I went with ATOM

Atom can be downloaded from the above link. Installing atom is quite simple, but atom does not come with a compiler to debug and run the program. That has to be installed manually.

Go to file-setting

Click on the install tab and type in the script in the search box

Click on install

Once installed the script can be accessed in the packages menu. The shortcut to run the script is Ctrl+Shift+B

Learning to Code

I found this really good tutorial page on youtube for beginners. The video explains all about Python, basics, operations, and functions.

Furthermore, the following link explains the same video with sample codes that can be tried out.

I went through some of the tutorials to understand the basics of python. It is not possible to learn an entire language and then trying to code the Hologram video. It was actually quite difficult to follow because the whole process of learning a language was new. I tried to understand how much I could and then shifted to exploring openCV

Installing

Installing OpenCV was easy. You can download and install from the following link

Once the .exe file is opened, the program self-extracts and installs all the necessary files.

As I had mentioned earlier, OpenCV requires some libraries such as Numpy and Matplotlib to run. So installing them is necessary. Installing libraries is not the same as we are used to in windows. To install libraries something called pip has to be installed first.

PIP

pip is a package management system used to install and manage software packages written in Python. Many packages can be found in the default source for packages and their dependencies — Python Package Index (PyPI).

To install pip open up cmd prompt in windows and run the following command

pip install some-package-name

This will automatically download and install pip

In my case after installation, it prompted that a higher version was available, hence I updated my version.

Numpy

NumPy is a library for the Python programming language, adding support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.

To install numpy open up cmd prompt in windows and run the following command

pip install numpy

Matplotlib

Matplotlib is a plotting library for the Python programming language and its numerical mathematics extension NumPy. It provides an object-oriented API for embedding plots into applications using general-purpose GUI toolkits like Tkinter, wxPython, Qt, or GTK+. There is also a procedural "pylab" interface based on a state machine (like OpenGL), designed to closely resemble that of MATLAB, though its use is discouraged.SciPy makes use of matplotlib.

To install matplotlib open up cmd prompt in windows and run the following command

pip install matplotlib

To check if all the libraries are loaded correctly, run python and type in:

import cv2

import matplotlib

import numpy

If there are no errors then everything is installed correctly.

Understanding and Learning to Code

To understand about OpenCV I followed these two tutorials and followed the examples codes to build my code:

 This page explains what OpenCV is capable of, its features and how to code them with examples.

This page provides video and code examples of what can be done using the code. The explanations were quite simple however it was not very easy to understand what was happening. It took some time to first comprehend the syntax.

Importing image

The first code I learnt was how to import an image. The code is as follows.

I'll break down the code to explain it.

import cv2

import numpy as np

from matplotlib import pyplot as plt

By writing this we are importing OpenCV, Matplotlib and numpy libraries into the program and also denoting how they are going to call in the course of the program

It is easy to convert any image to grayscale and perform analysis, it takes less processing time and memory.

img = cv2.imread('sunset.jpg',cv2.IMREAD_GRAYSCALE)

This line is to basically read the image that is specified, in this case, ‘sunset.jpg' and converts it to grayscale.

cv2.imshow('image',img)

cv2.waitKey(0)

cv2.destroyAllWindows()

This is to show the image that is read and to close the preview window when any key is pressed.

Importing Videos:

The next part is to import video into python. Video files are also read like images files in python. Since a video is made up of frames, python reads each frame as individual pictures to process them. However, the coding is slightly different.

Now let's break down the code

import numpy as np-

import cv2

cap = cv2.VideoCapture(0)

First, we import numpy and cv2, nothing fancy there. Next, we cay cap = cv2.VideoCapture(0). This will return video from the first webcam on your computer. The value 0 denotes the equipment in use. The default laptop cam is 0 and any externally connected camera will be 1,2.. etc

while(True):

ret, frame = cap.read()

This code initiates an infinite loop (to be broken later by a break statement), where we have ret and frame being defined as the cap.read(). Basically, ret is a boolean regarding whether or not there was a return at all, at the frame is each frame that is returned. If there is no frame, you won't get an error, you will get None.

gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

Here, we define a new variable, gray, as the frame, converted to gray. Notice this says BGR2GRAY. It is important to note that OpenCV reads colors as BGR (Blue Green Red), where most computer applications read as RGB (Red Green Blue). Remember this.

   cv2.imshow('frame',gray)

Notice that, despite being a video stream, we still use imshow. Here, we're showing the converted-to-gray feed. If you wish to show both at the same time, you can do imshow for the original frame, and imshow for the gray and two windows will appear.

  if cv2.waitKey(1) & 0xFF == ord('q'):

        Break

This statement just runs once per frame. Basically, if we get a key, and that key is a q, we will exit the while loop with a break, which then runs:

cap.release()

cv2.destroyAllWindows()

This releases the webcam, then closes all of the imshow() windows.

As mentioned earlier the following tutorial is what I followed to learn all this. The documentation and the explanations are great.

Background Subtraction

As explained I want the program to detect the human and subtract the background and rotate the video in four directions. Background subtraction can be done in multiple ways. The most common one is using motion to do so. There are certain algorithms within open cv for this like the BackgroundSubtractorMOG

BackgroundSubtractorMOG2

BackgroundSubtractorGMG

These algorithms were based on a paper by different people which were adapted for this purpose.

I tried edge detection and background subtraction through varying hue saturation and value.

Edge detection

The following link explains edge detection using Canny edge detection

Canny edge detection does

-Noise reduction

-Finding Intensity Gradient of the Image

-Non-maximum Suppression

-Hysteresis Thresholding

OpenCV puts all the above in single function, cv.Canny().

To explain the code, all other parts remain the same as importing a video and displaying it. The difference is here

edge = cv2.Canny(frame, 100,200)

edge2 = cv2.Canny(frame, 100,100)

cv2.imshow('sobely',edge)

 cv2.imshow('sobely2',edge2)

Where cv2.Canny(image, threshold, threshold)

So to compare different thresholds I had given 2 outputs. The one with the lesser threshold shows more grains.

BackgroundSubtractorMOG2

As explained already MOG2 is Gaussian Mixture-based Background/Foreground Segmentation Algorithm. The code is very simple

We first define the function by adding

fgbg = cv2.createBackgroundSubtractorMOG2()

And then creating a mask for it

mask = fgbg.apply(frame)

 To make things interesting I added this piece of code to it

res = cv2.bitwise_and(frame, frame, mask = mask)

 What it basically does is applies the mask to the actual frame which is the original image. The result was that the with the mask output I get to apply that on the real image as well.

Background Subtraction through HSV

To work on background subtraction by this method I used the following tutorial and code, modified it to suit my needs.

To work on background subtraction by this method I used the following tutorial and code, modified it to suit my needs.

Apart from using HSV to isolate the image from the background, slider bars have been introduced to adjust the threshold of the image. Here is the break-up of the code.

import cv2

import numpy as np

cap = cv2.VideoCapture(0)

panel = np.zeros([100,700,3],np.uint8)

cv2.namedWindow('panel')

This part is the same but introduced a panel using numpy where the sliders are going to sit.

def nothing(x):

    return x

cv2.createTrackbar('L - h', 'panel', 0,179,nothing)

cv2.createTrackbar('U - h', 'panel', 179,179,nothing)

cv2.createTrackbar('L - s', 'panel', 0,255,nothing)

cv2.createTrackbar('U - s', 'panel', 255,255,nothing)

cv2.createTrackbar('L - v', 'panel', 0,255,nothing)

cv2.createTrackbar('U - v', 'panel', 255,255,nothing)

Here I created the track bar using cv2.createTrackmar(). The inside parameters talk about the heading displayed wherein the first case L is lower h is Hue, then panel, the number is the range of the hue. The same thing is repeated for HSV values.

while True:

    ret, frame = cap.read()

    crop = frame[0:640,0:860]

    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

    l_h = cv2.getTrackbarPos('L - h', 'panel')

    u_h = cv2.getTrackbarPos('U - h', 'panel')

    l_s = cv2.getTrackbarPos('L - s', 'panel')

    u_s = cv2.getTrackbarPos('U - s', 'panel')

    l_v = cv2.getTrackbarPos('L - v', 'panel')

    u_v = cv2.getTrackbarPos('U - v', 'panel')

    lower_red = np.array([l_h,l_s,l_v])

    upper_red = np.array([u_h,u_s,u_v])

    mask = cv2.inRange(hsv, lower_red, upper_red)

    mask_inv = cv2.bitwise_not(mask)

    bg = cv2.bitwise_and(crop, crop, mask=mask)

fg = cv2.bitwise_and(crop, crop, mask=mask_inv)

Here we are reading the video data, and converting the BGR(in OpenCV RGB is read as BGR) into HSV values. Then we are asking the program to obtain those values form the slider that has been created. The cv2.bitwise_and command Calculates the per-element bit-wise conjunction of two arrays or an array and a scalar. The syntax is

cv2.bitwise_and(src1, src2[, dst[, mask]]) → dst

cv2.imshow('bg',bg)

    cv2.imshow('fg', fg)

    cv2.imshow('panel',panel)

    if cv2.waitKey(1) & 0xFF == ord('q'):

        break

cap.release()

cv2.destroyAllWindows()

The last part of the code is to display the results as videos, display the panel and to release the camera when the wait key is pressed which in this case is ‘q’.

Of all the methods compared I think the HSV is better as I got cleaner output. The only issue is setting the right threshold for it. Once the lighting conditions are finalized and set I can work on this program to improve its subtraction capabilities.

Video Manipulation

This part was quite tricky, and as a beginner to python, there were no proper video or tutorial to explain how to rotate videos. After googling for a long time the only thing I was certain of is to use cv2.getrotationmatrix2D() for such types. The functions in this section perform various geometrical transformations of 2D images. They do not change the image content but deform the pixel grid and map this deformed grid to the destination image. In fact, to avoid sampling artifacts, the mapping is done in the reverse order, from destination to the source.

The explanation for this function can be found in the following link

But the examples provided were for 2d images and not videos. I tried to modify those codes to suit my needs but was unsuccessful. Then I came across this file in Github about video generator for holograms. I started modifying the code.

Understanding the code.

SIZE = 600

cap = cv2.VideoCapture(0)

cap.set(cv2.CAP_PROP_FRAME_WIDTH, SIZE)

cap.set(cv2.CAP_PROP_FRAME_HEIGHT, SIZE / 2)

img = np.zeros((SIZE, SIZE, 3), dtype=np.uint8)

The size denotes the size of the video frame. Cap.set() is used to change the certain properties of the video stream. In this case the height and width of the video.

CV_CAP_PROP_FRAME_WIDTH Width of the frames in the video stream.

CV_CAP_PROP_FRAME_HEIGHT Height of the frames in the video stream.

The size denotes the size of the video frame. Cap.set() is used to change the certain properties of the video stream

np.zeros(shape, dtype = None, order = ‘C’) : Return a new array of given shape and type, with zeros.

    ret, frame = cap.read()

    if ret:

        for i in range(SIZE // 2):

            row = frame[i, i:SIZE - i - 1]

            rowFlipped = cv2.flip(row, 0)

            np.copyto(img[i, i:SIZE - i - 1], row)

            np.copyto(img[i:SIZE - i - 1, i], rowFlipped)

            np.copyto(img[SIZE - i - 1, i:SIZE - i - 1], rowFlipped)

            np.copyto(img[i:SIZE - i - 1, SIZE - i - 1], row)

Here 'i' is the range which is size divided by 2. Then what a row is defined. Np.copyto() Copies values from one array to another, broadcasting as necessary. So the array is copying the video stream to four points and rotating it at the same time.

  cv2.imshow('frame', img)

        if cv2.waitKey(1) & 0xFF == ord('q'):

            break

    else:

        break

cap.release()

cv2.destroyAllWindows()

This is to display the result and ‘q’ key to terminate the operation.

I've was successful in doing two different processes, the next step is to combine the two to give a final output. I will be doing that in teh following weeks.

Meanwhile having fun with manipulated video..=D