An Apache Server hosted on a Raspberry Pi, for plotting data!

Note: All the files here

Contents

Concept

In this week assignment I will try to configure a local webserver on a Raspberry Pi in order to display the data acquired by my final project’s sensors. I don’t have much idea about how to do this, but through the help of the Raspberry Pi’s documentation and some documentation review online, I got some insights on the process. In the Raspberry Pi’s docs, one of the ways to do this is via the Apache Web Server, which is the way I will be using.

Configure the Server

So now, that we got some information about what we are doing, let’s set up the Raspberry Pi to host the server via Apache. In order to install Apache:

sudo apt-get install apache2 -y

For us to find the Raspberry Pi’s address from a computer in the same network, we can run this command in the terminal (seen in the previous assignment):

MY_RANGE=$(ip addr | grep "UP" -A3 | grep '192' -A0 | awk '{print $2}') && nmap -sn $MY_RANGE && arp -na | grep b8:27:eb

Apache will then be hosting a test HTML in the web folder. If we navigate to the Pi’s IP address in whichever browser we like we should see something like this:


Enable CGI (Common Gateway Interface)

In order to run external content-generating programs, CGI programs / scripts define a way for the web server to do so. It is a simple way to put dynamic content on a web site, using whatever programming language we want, for example Python, Perl… I will be using Python in order to interact with the sensors’ I2C protocol of my installation and then try to create a dynamic plot with this data.

The way Apache looks for the CGI content is through the 000-default.conf file located under /etc/apache2/sites-enabled:

pi@raspberrypi: /etc/apache2/sites-enabled $ ls
000-default.conf
pi@raspberrypi: /etc/apache2/sites-enabled $ sudo geany 000-default.conf

A bit more into detail, we need to:

  • Set up a ScriptAlias which will look into /usr/lib/cgi-bin/ for Scripts defined. This is done through the command below in the .conf file:
ScriptAlias "/cgi-bin/" "/usr/lib/cgi-bin/"
  • Configure the <Directory "/usr/lib/cgi-bin">... </Directory> to run CGI, allowing everyone to do so and to allow python scripts to run:
<Directory "/usr/lib/cgi-bin">

  AllowOverride None

  Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch

  # For Previous versions
  Order allow,deny

  Allow from all
  
  # For Apache 2.4.7
  Require all granted
  
  # Add the handler for the script (python)
  AddHandler cgi-script .py

  </Directory>

There is a particularity for Apache 2.4.7 that is mentioned in this website, which concerns the permissions to run the code.

Altogether, the .conf file looks like:

<VirtualHost *:80>
  # The ServerName directive sets the request scheme, hostname and port that
  # the server uses to identify itself. This is used when creating
  # redirection URLs. In the context of virtual hosts, the ServerName
  # specifies what hostname must appear in the request's Host: header to
  # match this virtual host. For the default virtual host (this file) this
  # value is not decisive as it is used as a last resort host regardless.
  # However, you must set it for any further virtual host explicitly.
  #ServerName www.example.com

  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/html

  # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
  # error, crit, alert, emerg.
  # It is also possible to configure the loglevel for particular
  # modules, e.g.
  #LogLevel info ssl:warn
  
  ScriptAlias "/cgi-bin/" "/usr/lib/cgi-bin/"

  <Directory "/usr/lib/cgi-bin">

  AllowOverride None

  Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch

  # For Previous versions
  Order allow,deny

  Allow from all
  
  # For Apache 2.4.7
  Require all granted
  
  # Add the handler for the script (python)
  AddHandler cgi-script .py

  </Directory>

  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  # For most configuration files from conf-available/, which are
  # enabled or disabled at a global level, it is possible to
  # include a line for only one particular virtual host. For example the
  # following line enables the CGI configuration for this host only
  # after it has been globally disabled with "a2disconf".
  #Include conf-available/serve-cgi-bin.conf
</VirtualHost>

With this file, we should be able to place Python scripts in the /usr/lib/cgi-bin and be able to see them from another computer in the same network.

Finally, in order to allow CGI, we need to do:

pi@raspberrypi: sudo a2enmod cgi

And then restart apache:

pi@raspberrypi: sudo systemctl restart apache2

Test Python Script

In the Directory of our choice (the same as above), we should now set up a Python Script with some particularities. First, a shebang marker needs to be put at the beginning of the file for it to be executed by Python:

#!/usr/bin/python

Note: In order to find where python is installed in the Raspberry Pi:

pi@raspberrypi:~ $ which python
/usr/bin/python

Next, we need to tell the contents of the file are going to be set as HTML. This won’t be shown in the actual website, but are needed for the CGI to undertand what we are printing:

print 'Content-type: text/html\n\n'

With this, and with some additions to print something (in HTML mode), my test file looks like:

#!/usr/bin/python
#~ Shebang configuration for running the script in python (the first line must be the shebang)

#~ Print Content-type as html
print 'Content-type: text/html\n\n'

#~ Print Stuff for testing
print '<h1>Hello World</h1>'
print '<h2> This is a test site </h2>'

Finally, the Python file needs to be set as an executable and we need to check it’s able to print stuff in HTML:

pi@raspberrypi: sudo chmod a+x hello.py
pi@raspberrypi: /usr/lib/cgi-bin $ ls -la
total 16
drwxr-xr-x  2 root root 4096 abr 22 19:47 .
drwxr-xr-x 87 root root 4096 abr 22 18:54 ..
-rwxr-xr-x  1 root root   87 abr 22 19:47 hello.py
pi@raspberrypi:/usr/lib/cgi-bin $ ./hello.py
Content-type: text/html

<h1>Hello World</h1>
<h2> This is a test site </h2>

And after a lot of effort, I saw this:


Note: some websites that helped me in the process are this one for when the content was being denied by the server and this tutorial.

It also helps to have a look a the error.log from Apache:

pi@raspberrypi:/var/log/apache2 $ ls
access.log  error.log  other_vhosts_access.log
pi@raspberrypi:/var/log/apache2 $ less error.log

...
Sun Apr 22 21:34:00.888412 2018] [cgid:error] [pid 6803:tid 1970271280] [client 192.168.1.44:62328] End of script output before headers: hello.py
[Sun Apr 22 21:34:23.869698 2018] [cgid:error] [pid 8864:tid 1995599872] (8)Exec format error: AH01241: exec of '/usr/lib/cgi-bin/hello.py' failed
[Sun Apr 22 21:34:23.872062 2018] [cgid:error] [pid 6803:tid 1978659888] [client 192.168.1.44:62353] End of script output before headers: hello.py
[Sun Apr 22 21:34:51.197570 2018] [cgid:error] [pid 8881:tid 1995599872] (8)Exec format error: AH01241: exec of '/usr/lib/cgi-bin/hello.py' failed
[Sun Apr 22 21:34:51.200081 2018] [cgid:error] [pid 6802:tid 1978659888] [client 192.168.1.44:62356] End of script output before headers: hello.py
[Sun Apr 22 21:35:03.003928 2018] [mpm_event:notice] [pid 6800:tid 1995599872] AH00491: caught SIGTERM, shutting down
[Sun Apr 22 21:35:03.169589 2018] [mpm_event:notice] [pid 8910:tid 1996201984] AH00489: Apache/2.4.25 (Raspbian) configured -- resuming normal operations
[Sun Apr 22 21:35:03.169950 2018] [core:notice] [pid 8910:tid 1996201984] AH00094: Command line: '/usr/sbin/apache2'
[Sun Apr 22 21:35:06.233389 2018] [cgid:error] [pid 8980:tid 1996201984] (8)Exec format error: AH01241: exec of '/usr/lib/cgi-bin/hello.py' failed
[Sun Apr 22 21:35:06.235925 2018] [cgid:error] [pid 8914:tid 1971319856] [client 192.168.1.44:62359] End of script output before headers: hello.py
[Sun Apr 22 21:35:08.014193 2018] [cgid:error] [pid 8981:tid 1996201984] (8)Exec format error: AH01241: exec of '/usr/lib/cgi-bin/hello.py' failed
[Sun Apr 22 21:35:08.016726 2018] [cgid:error] [pid 8913:tid 1971319856] [client 192.168.1.44:62361] End of script output before headers: hello.py

Creating plots

Using Matplotlib

In order to create a basic plot, I first set up a python script using matplotlib, which is a broadly used library for generating plots. The set-up however, changed a bit, by following this tutorial.

  • Create a python script that outputs a matplotlib graph as a png
  • Embed that as an image in the html code of our page

The python script could look like this:

#!/usr/bin/python
#~ Shebang configuration for running the script in python

#~ Import and enable cgi debugging
import cgi

import io
import numpy as np

import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

# Create a random vector
num_items = 100
tempdata_X = range(num_items)
tempdata_Y= np.random.rand(num_items)

# Set up the plot
fig, ax = plt.subplots(figsize=(6,5))
ax.plot(tempdata_X, tempdata_Y, ls='-', color='red' )

# Finish plot
ax.set_ylabel('Temperature F')

# Save the image to buffer
buf = io.BytesIO()
fig.savefig(buf, format='png')
out = buf.getvalue()
buf.close()

#~ Print PNG
print 'Content-Type: image/png\n'
print out

And the HTML (calling the python application that outputs the image):

pi@raspberrypi:/var/www/html $ cat index.html
<html>
 <head>
<title>Test Graph</title>
<body>
<p>Test<br>
<img src="cgi-bin/app.py"></p>
</body>
</html>

This generates the following site (accesible via: $PI_ADDRESS/cgi-bin):

Which is okayish! A nice success, but not a very nice graph.

Using Interactive content

I decided then to explore further options with some other plotting libraries. One of the most interesting ones I found is called plot.ly. It is an open source service, which can simply provide with plotting tools offline, or for a very reasonable price (only 9,000$… meh!) provide support and data hosting. Of course I didn’t pay that much, but I dig into their API in order to see what they can do as Live Streamming of data:

Ups! Not discouraged by this, I went further into the exploration of the other API called dash. This again, can be paid at a similar price, or one could try to explore it’s on way into it. To get started, I pressed on the get started button and began with the installation process (all on the Pi):

pip install dash==0.21.0  # The core dash backend
pip install dash-renderer==0.12.1  # The dash front-end
pip install dash-html-components==0.10.0  # HTML components
pip install dash-core-components==0.22.1  # Supercharged components
pip install plotly --upgrade  # Plotly graphing library used in examples

I then followed the simple tutorial below that (which I won’t detail here because it’s on the site), plotting a set of numpy vectors in a bar chart, in order to understand how they generate the content on the site. Basicly, to remember is that (I have no relationship with Dash or Plotly, but the plots are nice):

  • The application is written in Python and can import python libraries such as pandas, numpy or any other we like
  • Dash provides with two main components: HTMLand Core components: HTML created html-elements and the core creates higher level object such as graphs, dropdowns, buttons and so on, based on javascript and react.js.
  • It supports Markdown as part of the core components content generation and it does it through the same way it does the graphs and other higher level components

Now, it’s time to get the hands on the plot!

Plotting actual data

For the shake of clarity, let’s try to make a sketch of what the plotting / sensor acquisition will look like. I based this on this site and built on top of it the rest of items I have related above:

  • The Pi will be reading via I2C a pressure sensor I used on the Input devices week, using a python code to output the data via print, with a timestamp
  • A crontab job will be running this script and redirecting it to a log.dat file (more on this below)
  • The apache server will use a python application via mod_wsgi (more on this below) to read the log.dat file and plot it using Dash

The Python-I2C code

This code can be wherever we want it to be. I have it in a personal folder on the Pi and looks like this (with a shebang! at the beginning):

#!/usr/bin/python

# Import packages
import smbus
import datetime
import glob
import sys
from sys import stdout

# Numpy
import numpy as np

# I2C device definition
bus = smbus.SMBus(1) # Indicates /dev/i2c-1
address = 0x13
packet_size = 4

def ReadSensor(_address):
  
  i = 0
  _value = 0
  
  while (i < packet_size):

    _measure = bus.read_i2c_block_data(_address, 0, 1)
    #~ print "Measure"
    #~ print _measure
    
    _value |= _measure[0] << (8*(packet_size-(1+i)))
    i+=1
    
  return _value

# Create the reading
timestamp = datetime.datetime.now()
reading = ReadSensor(address)
#~ print reading
    
# Print it and flush standard output
print "{},{}".format(timestamp,reading)
sys.stdout.flush()

We have to change the script settings for it to be executable:

pi@raspberrypi:/path/to/script/ $ sudo chmod +x sensorRead.py

And the, when running this script, we should get something like:

pi@raspberrypi:/path/to/script/ $ ./sensorRead.py
2018-04-24 20:28:02.146216,10321

The crontab job

This is a way to create a scheduled job in linux, by setting a shell command with a certain timing that we define and we write in a certain format in a text file. The line definition looks like:

Where each star means:

  • m: interval in minutes
  • h: interval in hours
  • dow: day of the week
  • mon: month
  • dom: day of the month

And in order to execute it:

pi@raspberrypi:/path/to/script/ $ sudo crontab -e

Which opens up a file (edited with nano text editor):

# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
*/1 * * * * /home/pi/path/to/script/sensorRead.py >> /home/pi/path/to/script/log.dat

My line there means: every minute (minimum timing), run the sensorRead.py and redirect it’s standart output to a .dat file in the same directory.

This log.dat file, after a while, will look like:

pi@raspberrypi:~/path/to/script $ ls
log.dat  sensorRead.py  test.dat
pi@raspberrypi:~/path/to/script $ tail log.dat
2018-04-24 20:19:01.542546,11240
2018-04-24 20:20:01.961694,8979
2018-04-24 20:21:01.359398,10469
2018-04-24 20:22:01.729572,10363
2018-04-24 20:23:02.203438,10484
2018-04-24 20:24:01.535844,10425
2018-04-24 20:25:01.953487,10320
2018-04-24 20:26:01.337837,10463
2018-04-24 20:27:01.745948,10464
2018-04-24 20:28:02.146216,10321

The apache server

Here, we will be using a different mode of operating the python code. Instead of using CGI, as above, I will be using Flask mod_wsgi, in order to interact with the python application. WSGI sets an specification of a generic API for mapping between an underlying web server and a Python web application. This sets up a more compatible way of using the python packages and the dash application mentioned above. In this site, it is explained how to create as similar concept as the ScriptAlias in CGI, but with some internal WSGI particularities. I detail below the result of my research and the result on the apache conf file:

pi@raspberrypi:/etc/apache2/sites-enabled $ cat 000-default.conf 
<VirtualHost *:80>
  # The ServerName directive sets the request scheme, hostname and port that
  # the server uses to identify itself. This is used when creating
  # redirection URLs. In the context of virtual hosts, the ServerName
  # specifies what hostname must appear in the request's Host: header to
  # match this virtual host. For the default virtual host (this file) this
  # value is not decisive as it is used as a last resort host regardless.
  # However, you must set it for any further virtual host explicitly.
  #ServerName www.example.com

  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/html

  # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
  # error, crit, alert, emerg.
  # It is also possible to configure the loglevel for particular
  # modules, e.g.
  #LogLevel info ssl:warn

    WSGIDaemonProcess plotApp processes=2 threads=15 user=pi group=pi home=/home/pi
    WSGIProcessGroup plotApp
    WSGIScriptAlias / /var/www/wsgi-scripts/plotApp.wsgi
  
  <Directory /var/www/wsgi-scripts>
  
        WSGIProcessGroup plotApp
        WSGIApplicationGroup %{GLOBAL}
        WSGIScriptReloading On
        Require all granted
        
    </Directory>

  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  # For most configuration files from conf-available/, which are
  # enabled or disabled at a global level, it is possible to
  # include a line for only one particular virtual host. For example the
  # following line enables the CGI configuration for this host only
  # after it has been globally disabled with "a2disconf".
  #Include conf-available/serve-cgi-bin.conf
</VirtualHost>

To highlight from there:

  • We need to create a DaemonProcess to put an eye on the application (in my case plotApp)
  • The ScriptAlias are now called WSGIScriptAlias. If we want the alias to be executed in the root directory of the server, we can just put / in the first part of the alias
  • We need a * .wsgi file that I will detail below, interacting with the actual python application
  • The permissions are granted as in the CGI ScriptAlias for apache 2.4.7

Now, this mod_wsgi configuration will be interfacing with the python code via a * .wsgi file that we have declared. This file is indeed a python file that in my case looks like this (inherited from Dash forum):

pi@raspberrypi:/var/www/wsgi-scripts $ cat plotApp.wsgi 
#!/usr/bin/python
import sys

sys.path.insert(0,"/usr/lib/cgi-bin/")

from plotApp import server as application

Which basicly is looking for the real-python application that will be plotting the data read from the log.dat file above and is setting it as an application in the mod_wsgi framework. This application needs to be called as such, and cannot be changed. Note that we need to create this file as an executable:

pi@raspberrypi:/var/www/wsgi-scripts $ sudo chmod +x plotApp.wsgi

The dash application

Finally, the fun part! After all this set up, we get to plot some of the data with dash. For this, we create a python code in the folder we specified in the .wsgi above that reads the data and creates the plots.

In order to read the data, I will be using pandas and numpy. For now, I did not manage to install pandas via pip on the Pi, but it’ll be a matter of time and I’ll stick to numpy for the moment (although I won’t be able to import the strings from the datestamp of the log.dat file):

# -*- coding: utf-8 -*-

#~ Numpy and data treatment
import numpy as np
import io
#~ import pandas as pd
#~ from padas_datareader.data import DataReader
import time
import datetime as dt

dataVal=np.genfromtxt('/home/pi/Documents/Oscar/Fabacademy/SensorLogPlot/log.dat',delimiter=',')
xvalues = range(len(dataVal)) # workaround for the datestamp
yvalues = dataVal[:,1]

For the data plotting, using dash and some of the examples from this site as a reference, the basic chart looks like this:

import dash
import dash_core_components as dcc
import dash_html_components as html
import plotly.graph_objs as go
from dash.dependencies import Input, Output

#~ Dash 
app = dash.Dash()
server = app.server

app.layout = html.Div(children=[
  dcc.Graph(id='pressure-graph',
    figure={'data': [
                {'x': xvalues 'y': yvalues, 'type': 'scatter', 'name': 'Pressure'},
            ],
            'layout': {
                'title': 'Dash Data Visualization'
            }
        }),
])

if __name__ == '__main__':
  app.run_server(debug=True, host='0.0.0.0', port=8082, threaded=True)

In order to add some text, I added some Markdown:

markdown_text = '''
## Pressure Sensor Reading

### Concept 
This graph shows the pressure evolution with the input devices assignment, based on a custom made ADC with an Attiny84.
The Raspberry Pi creates a webserver with apache2, which, through mod_wsgi, runs a plotting interface based on plot.ly / dash.
The plot reads the data from a log.dat file, which is run with a crontab work every minute, in the Pi itself.

'''
app.layout = html.Div(children=[
  dcc.Markdown(children=markdown_text),
])

And formatted it with generic styles:

app.css.append_css({"external_url": "https://codepen.io/chriddyp/pen/bWLwgP.css"})

Finally, in order to make it refresh, I followed this site in order to create a callback based on a timer (from dash API itself):

app.layout = html.Div(children=[
  dcc.Interval(
    id='interval-component',
    interval=1* 1000,
    n_intervals=0
  )
])

This calls every 1000ms the app callback, which uses as input the n_intervals:

@app.callback(Output('pressure-graph','figure'),
  [Input('interval-component','n_intervals')])
def update_graph_live(n):
  # Do stuff here
  return the stuff

Summing up, the whole application, with some changes, sets up a callback every 1000ms, reads the log.dat data and updates the plot. Additionally it puts some nice css’ed text based on Markdown:

# -*- coding: utf-8 -*-
import dash
import dash_core_components as dcc
import dash_html_components as html
import plotly.graph_objs as go
from dash.dependencies import Input, Output

#~ Numpy and data treatment
import numpy as np
import io
#~ import pandas as pd
#~ from padas_datareader.data import DataReader
import time
import datetime as dt

#~ Dash 
app = dash.Dash()
server = app.server

markdown_text = '''
## Pressure Sensor Reading

### Concept 
This graph shows the pressure evolution with the input devices assignment, based on a custom made ADC with an Attiny84.
The Raspberry Pi creates a webserver with apache2, which, through mod_wsgi, runs a plotting interface based on plot.ly / dash.
The plot reads the data from a log.dat file, which is run with a crontab work every minute, in the Pi itself.

'''

app.layout = html.Div(children=[
  dcc.Markdown(children=markdown_text),
  dcc.Graph(id='pressure-graph'),
  dcc.Interval(
    id='interval-component',
    interval=1*1000,
    n_intervals=0
  )
])

# Callback
@app.callback(Output('pressure-graph','figure'),
  [Input('interval-component','n_intervals')])
def update_graph_live(n):
  dataVal=np.genfromtxt('/home/pi/path/to/script/log.dat',delimiter=',')
  xvalues = range(len(dataVal))
  yvalues = dataVal[:,1]
  
  fig = go.Figure(
    data=[
      go.Scatter(
        x=xvalues,
        y=yvalues,
        name='Pressure',
        marker=go.Marker(
          color='rgb(255, 0, 0)'
        )
      ),
    ],
    layout=go.Layout(
      title='Pressure Sensor Reading',
      showlegend=True,
      legend=go.Legend(
        x=0,
        y=1.0
      ),
      margin=go.Margin(l=80, r=40, t=40, b=80, pad=10),
      autosize=True,
      xaxis=dict(title='Date'),
      yaxis=dict(title='Pressure (kPa)')
    )
  )
  
  return fig
  
app.css.append_css({"external_url": "https://codepen.io/chriddyp/pen/bWLwgP.css"})

if __name__ == '__main__':
  app.run_server(debug=True, host='0.0.0.0', port=8082, threaded=True)

And the plot, finally, after all this boring stuff looks like:

As a bonus, for those who like old-styled-times-new-roman:

Note on live updates

I notices that this method updates the whole graph, and provokes the interactivity to be reset every second, not allowing to zoom in or pan, among others. The callback above can be better changed into:

@app.callback(Output('pressure-graph','figure'),
  [Input('interval-component','n_intervals')])
def update_graph_live(n):
  dataVal=np.genfromtxt('/home/pi/Documents/Oscar/Fabacademy/SensorLogPlot/log.dat',delimiter=',')
  xvalues = range(len(dataVal))
  yvalues = dataVal[:,1]
  
  traces = list()
  traces.append(go.Scatter(
    x=xvalues,
    y=yvalues,
    name='Pressure'
    ))
  return {'data': traces}

Which, by the way, is a much simpler code.