# Build a statistics dashboard for COVID-19
Use Vue.js, Express, and Python to build a statistics dashboard for COVID-19.Specifically, you will build a line chart to display the COVID-19 cases in a country as the time progresses. This tutorial was inspired by the winning entry of the MariHacks 2021 hackathon ([iCovid](https://hackmd.io/dWTFeP8SQs-U8ZD7Dg1mfQ?view#iCovid---Marihacks-2021)).

# Introduction

## Tutorial Dependencies
*Note: Before starting the tutorial, you will need basic knowledge in HTML, CSS, Javascript, Vue (optional) and Python.*

## Understanding the Process
Here is a brief overview of the overarching steps needed to make our application work and a small explanation of the technologies that we will be using!

### 1. **Gathering the Data**
We will retrieve the data of the COVID19 cases from websites.

### 2. **Processing the Data**
The processing of data will be done using **python 3**. We will be using the powerful module named ```pandas``` to process our data.

The data processing can be done using a simple text editor or using a [jupyter notebook](https://jupyter.org).

### 3. **Building the API**
To display the chart data, we will first fetch and treat our data from the backend. We will then send the data to our frontend that will display it using visual charts. We will be using `Node.js`, more specifically `express` for the backend and `Vue.js` for the frontend for this tutorial. We will use `axios` to communicate with the backend server. We will also use `vue-chartjs` to build our charts!

## Setup

### Data Analysis Part
For this tutorial you need to have [Python](https://www.python.org/downloads/) installed. With that in mind, the module required to process the data will be `pandas`.

pip install pandas

Note: If you are on a different platform than windows, you may have to replace `pip` with `pip3`.

### API Part
We will begin with installing Node and npm.
Download Node.js from [here](https://nodejs.org/en/download/).
Once you are finished, you should be able to use the npm package manager. To confirm you have installed node and npm, run these commands:

node -v
npm -v

Make a new directory and move into it:

mkdir myapp
cd myapp

Initialize npm:

npm init

This will prompt you multiple questions, such as specifying the package name, the author, and the keywords describing your application. Just fill it out according to your needs, or just press enter to enter the default options.

We can then use it to install `express`, `body-parser`, `moment`, and `request-promise`:

npm install express body-parser moment request-promise

We can also install globally the `vue` package:

npm install -g @vue/cli 

We can then use this package to create a new client app inside of the root directory of our project:

vue create client

Again, this will prompt you with questions regarding the project's settings. Just enter the default or set it to your needs.

Then we can start our backend with a simple:

npm start

This will run at [http://localhost:8081](http://localhost:8081)
It should display a blank page!

To start our frontend, we can run:

cd client
npm run serve

This will run at [http://localhost:8082](http://localhost:8082)
It should display a welcome page from Vue!

---

# 1. Gathering data

We will need first to retrieve the COVID19 cases data from multiple websites. Then, we can use the data to display it in the format of charts.

The first block of data comes from a Github repository containing COVID19 cases over time. To download it, go to this link to get the text data: https://pomber.github.io/covid19/timeseries.json

The second block of data comes an online dataset containing reports of COVID19 cases in the city of Montreal. The link to get the data: https://resources-covid19canada.hub.arcgis.com/datasets/provincial-daily-totals/data

---

# 2. Processing the data

The first block of data will be used to get current data for the current cases. This will be used later on for the API.

The second block of data is more interesting since we can extract features and generate statistics for the front end.

Saving the file as ```canada_region_cases.csv```, you can run the following blocks of python code in the terminal or in a Jupyter Notebook.

## 2.1 Loading Data

# Imports
import pandas as pd

# Reading the data from csv
df = pd.read_csv('canada_region_cases.csv')

## 2.2 Visualization Of The Age Groups
To get the distribution of the age groups, we use a function nammed `value_counts` to get how many labels do we have for each category (Ex: <20: 200, 20-29: 400, etc). The normalize parameter is to scale down everything between 0 and 1.

The reindex function is used to reorder the indexes to show the distrubtion across age and not from most to least.

# Getting data about the age groups
age_group = df['age_group']
indexes = ['<20', '20-29', '30-39', '40-49', '50-59', '60-69', '70-79', '80+']
print((age_group.value_counts(normalize=True) * 100).reindex(indexes))

Output:

<20       8.950015
20-29    12.170860
30-39    10.072569
40-49     8.764865
50-59     8.159194
60-69     5.185605
70-79     2.737404
80+       1.904169

## 2.3 Visualization Of The Gender
For visualizing the genders, we first need to remove any value that is `Not Reported` since it does not give us any info on which gender got affected. Then we use the same function as mentionned above to get the percentage of Male and Female with cases.

# Getting data about the gender
gender = df['gender']
gender = gender[gender != 'Not Reported']
print(gender.value_counts(normalize=True) * 100)

Output:

Female    50.156479
Male      49.843521

## 2.4 Visualization Of The Province
For visualizing the province, we use the same function specified above since we want to get the distribution of the cases across the provinces and territories.

# Getting data about the provinces in Canada
province = df['province']
print(province.value_counts(normalize=True) * 100)

Output:

Ontario             34.642668
Quebec              33.246642
Alberta             15.430972
British Columbia     9.166975
Manitoba             3.688673
Saskatchewan         3.284814
Nova Scotia          0.189228
New Brunswick        0.166274
NL                   0.113374
Nunavut              0.041365
PEI                  0.013633
Yukon                0.008389
NWT                  0.005476
Repatriated          0.001515

*Note: In this tutorial, we will only be displaying the COVID19 cases per country. You can have fun integrating the other data (age group, gender, provinces) in charts after the tutorial!*

---
# 3. Building the API

#### What is an API?
API stands for an [application programming interface (API)](https://en.wikipedia.org/wiki/API) and enables software to communicate with each other. In our case, the API enables the frontend (client) to communicate with the backend (server) in a meaningful way.

## 3.1 Starting our Backend

We will start with the initial portion of the app:

// File: ./app.js
/* Getting our main packages */
const express = require('express'); // required
const cors = require("cors"); // optional
const morgan = require('morgan'); // optional
const bodyParser = require('body-parser'); // required
const app = express();

const corsOptions = {
    optionsSuccessStatus: 200,
}
// Setting up our middlewares
app.use(cors(corsOptions));
app.use(morgan('dev'));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));

// Main routing
require('./routes')(app);

// Running the app on port 8081
const port = process.env.PORT || 8081;
app.listen(port, () => {
  console.log("App is running.");
});

Using Express, we can create a backend webserver to act as our API that will run independently from the frontend. It will manage any incoming server-related request containing an input dictionary parsed by `body-parser`. Then it will respond to the request with another dictionary containing the processed output.

*Note: Here, the app is running on port 8081, but this value is, of course, arbitrary.*
## 3.2 Routing the Backend
For the API to work properly with the frontend, it is necessary to organize a well structured routing model in `./routes`.

// File: ./routes/index.js
const {simulation, predict_cases, generate_simulation} = require("../middlewares"); // Middlewares
const path = require('path');

module.exports = (app) => {
    app.get('/', (req, res) => res.status(200).send({
        message: 'Welcome to the users API!',
    }));

    ...
    
    app.post('/stats', simulation.getCovidStats);
}

Once created, this will enable us to properly request specific data for the client by extracting and processing the data in `./middlewares`.

#### Why use the POST method?
With HTTP requests, we can use both the GET and POST methods to enable the user to transfer data to our server. Although they have the same function, they do not do the same things. GET uses the parameters integrated with the URL, while POST creates a separate message body. Hence, it is generally accepted that POST is a more secure method for our type of application. This will also enable us to transfer more complex data structures such as arrays.

## 3.3 Getting data and treating it
Once we have created a routing system, we can access individual routes and create appropriate middlewares to manage, process, and report the requests. For this reason, the following section is the most crucial part of our API as it does all the work.

In the following examples, we will evaluate the `/stats` request, which uses the `getCovidStats` middleware function.

// File: ./middlewares/simulation.js
const rp = require('request-promise'); // Request GET data online
const moment = require('moment'); // Time manipulation lib

module.exports = {
    async getCovidStats(req, res) { // Data for client chart covid 19 cases
        ...
    },
}

### 3.3.1 User input data

async getCovidStats(req, res) { // Data for client chart covid 19 cases
    const country = req.body.country || "Canada"; // Requested country
    const startDate = moment(req.body.startDate, 'YYYY-MM-DD') || moment().format('YYYY-MM-DD'); // Convert string date to moment object date
    const endDate = moment(req.body.endDate, 'YYYY-MM-DD') || moment().format('YYYY-MM-DD');
    const subject = req.body.subject || "confirmed"; // Desired data (confirmed, deaths, recovered)
    ...   
},

We create a new async function named `getCovidStats` that is going to handle our request. To do so, we will need to forward `req` and `res` as parameters. Only `req` (request) is of interest as it contains the dictionary with the user input. More specifically, the data is stored in `req.body`, where we can expect the following variables inside:
* `country`: Country concerned for data extraction
* `startDate`: Start date from which to begin data extraction
* `endDate`: End date from which to stop data extraction
* `subject`: Type of data desired (confirmed, recovered, and deaths)

We are also using [Moment.js](https://www.npmjs.com/package/moment/v/1.1.0) to manage and format dates. By default, if no value is specified, the present date is used.

*Note: The presence of the operator || means "or" in Javascript. Here, it creates a default value for the variable if the user does not specify any value (i.e. the variable is equivalent to false).*

### 3.3.2 Data scraping

// File: ./middlewares/simulation.js
rp.get({uri: "https://pomber.github.io/covid19/timeseries.json", json: true}).then(data => { // Request live data from github repository
    ...    
}).catch(err => {
    console.log(err);
    return res.send({ data: [], msg: "API error." }); // Could not access data online
});

To extract the needed data on COVID-19 cases worldwide, we will need to create a GET request to this [repository](https://pomber.github.io/covid19/timeseries.json). We are using `request-promise`, where we input the URL and specify that the data is in JSON format and needs to be parsed (this will save us another step). Once done, `data` will contain everything required.

*Note: In this case, because the repository is only updated once a day, a potential optimization of this task would be to save the file locally so it can be quickly accessed at each new request. This file would only need to be updated once a day.*
### 3.3.3 Data filtering

// File: ./middlewares/simulation.js
var countryData = data[country] || []; // Get array data only relevant to country
if (countryData.length > 0) {
    var returnData = [];
    for (var d = 0; d < countryData.length; d++) {
        if (moment(countryData[d].date, 'YYYY-MM-DD').isBetween(startDate, endDate)) { // If date is between range
            returnData.push({ date: countryData[d].date, value: countryData[d][subject] }); // Add valid data to array
        }
    }
    return res.send({ data: returnData, msg: "Found available data. "}); // Return results
} else {
    return res.send({ data: countryData, msg: "No data is available for that country." }); // No data for specified country was found
}

*Note: The following part depends on the chosen data repository and may not be the same for you.*

Great, we have extracted the raw data, but we still have to filter it to obtain only the user's specific requested data. To do so, we need to identify the structure of our parsed JSON dictionary.

// File: https://pomber.github.io/covid19/timeseries.json
{
  "Canada": [
   {
	"date": "2020-1-22",
	"confirmed": 0,
	"deaths": 0,
	"recovered": 0
   },
	... (Another date for the same country)
   ],
	... (Another country)
}

In this case, we have a list of countries that contain another list of dates with the statistics of that respective day in that specific country.

When filtering our data, we can start by isolating our country from the list. To do so, we use:

var countryData = data[country] || []; // Get array data only relevant to country

We then verify the array's size to see if any data is available for the requested country. If there is nothing available, we can return an empty array with an error message. Otherwise, if data is available, we can loop inside the list of dates to verify with a [Moment.js](https://www.npmjs.com/package/moment/v/1.1.0) date object if each respective date is in the requested interval of time (between `startDate` and `endDate`). If the date respects our criteria, we can extract the subject of interest from it to then append it to a response array (`returnData`) sent back to the user.

var returnData = [];
    for (var d = 0; d < countryData.length; d++) {
        if (moment(countryData[d].date, 'YYYY-MM-DD').isBetween(startDate, endDate)) { // If date is between range
            returnData.push({ date: countryData[d].date, value: countryData[d][subject] }); // Add valid data to array
        }
    }

Now, we can finally return the response array `returnData` back to the user. For that, we can use the `res` (response) object to return a response:

return res.send({ data: returnData, msg: "Found available data. "}); // Return results

Congratulations, you have successfully managed to create an API for the backend.

*Note: It is important to use `return` for the response to be properly sent. Otherwise, you could get a "headers already sent" error message.*

## 3.4 Using Axios to communicate with the backend
This section will retrieve the data from the backend server and use it for our charts in the front end part of our application. First, ensure that you have `axios` installed inside the `client` directory.

#### What is Axios?
Axios is a library that we will use on the client-side to make HTTP requests (using both the GET and POST methods). A big advantage that it offers is its use of [Javascript promises](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise) that we previously used for the backend but did not mention.

cd client
npm install axios

For the sake of this simple tutorial, most of the code will be written inside the `App.vue` file in the `src` directory of the `client` directory. We will not be touching other files unless specified.

Before working with the front end of our application, we will need to understand how Vue works. [**Vue**](https://vuejs.org) is a front-end framework that is **component-based**. This means that the web application can be broken down into several components. For instance, we could have a navbar component, a blog post, and a footer. These components can then be reused in other components by exporting them and importing them!

A Vue file (which has a `.vue` extension) can be broken down in 3 distinct parts:
* `html`
* Is inside the`template` tag
* There is a single `div` inside the `template` tag. Everything else has to be nested inside that `div` tag!
* `css`
* Is inside the `style` tag
* We can style everything from the `template` tag.
* `javascript`
* Is inside the `script` tag
* Will typically include imports at the beginning of any other components we are using
* Typically has an `export module {}`, containing everything pertaining to the component itself!

A typical Vue file will look like this:

<template>
    <div>
        <!-- Inside this single div will be the html structure of the component -->
    </div>
</template>
<script>
// Any import of components are done here
import 'myImport' from 'myImport';

export module {
    name: "NameOfTheComponent",
    components: {
        // Include any components
        myImport,
    },
    // Stores data of the component
    data() {
        return {
            // Example
            myData: 'Hello World!',
            // We can access this variable in other parts of the export module 
            // by using this.myData
        };
    },
    // This will contain the methods used to process and treat the data!
    methods: {
        // Example
        methodName() {
            // do something
        },
    }
}
</script>
<style>
/* Make your component pretty here! */
</style>

We will also need to have a general understanding of the [**lifecycle of a Vue instance**](https://vuejs.org/v2/guide/instance#Lifecycle-Diagram) (from when the component is created to it being destroyed).

It usually goes as following:
* BeforeCreate `beforeCreate()`
* Created `created()`
* BeforeMount `beforeMount()`
* Mounted `mounted()`
* BeforeDestroy `beforeDestroy()`
* Destroyed `destroy()`

Vue will call these functions in the order presented above. We can specify what we want to be executed during these stages by overwriting these functions inside the `export module {}` section.

Now that we know the general theory behind this, we can begin by importing `axios` library inside `App.vue`:

import axios from 'axios';

This line will be inserted at the beginning of the `script` tag.

Then, we can use axios to send a `post` request to retrieve the data from our backend server! Since the `post` request will take time to fetch the data, we will call this `post` method in the `mounted()` stage of the lifecycle. This will allow the rest of the page to be rendered while `post` fetches the data.

We will pass in the `post` method the url that the backend server will use to process our request and send us the appropriate information (in this case `/stats` which we have defined in the previous section, in the router), and the parameters of the data that we want to fetch (`country`, `startDate`, `endDate`, `subject`).

axios.post('/stats', {
    country: 'Canada',
    startDate: '2021-01-25',
    endDate: '2021-02-26',
    subject: 'confirmed',
}).then((response) => {
    ...
})

We can then access the data from the response inside the `.then((response) =>{})` by typing:

response.data.data

The data will be under the form of an array of several objects that have a `value` and a `date` property. They will each respectively correspond to the y and x axis of the line chart that we will be creating. We will thus have to loop through each object to store these data inside of our component:

let data = response.data.data;
for(var i = 0; i < data.length; i++) {
   this.dates[i] = data[i].date;
   this.numbers[i] = data[i].value;
}

We will have to create these variables in the `data` method, of course, to use them in other methods:

return {
    dates: [],
    numbers: [],
    datacollection: [], // This will be used to display the chart, we will see it later in the section.
    // It will be empty for now. We will fill them up with data after!
}

And that is it! We have successfully retrieved the data from the backend server! :confetti_ball:

If we want to check whether our data has been properly loaded, we can do a `console.log` of the data at the end of the `mounted()` function:

console.log(this.numbers + this.dates);

## 3.5 Displaying the data using nice charts
To display charts, we will use the [vue-chartjs](https://vue-chartjs.org) library, which is the *Vue* version of [chart.js](https://www.chartjs.org/)! Let's install it in the frontend part of our application:

cd client
npm install vue-chartjs

Then, we can use this library to generate charts to display our data graphically!

We will start by building a Line chart!

To do this, we will first create the `LineChart` component to integrate it into our Vue file. Create `LineChart.js` in the `src` directory of the `client` folder.

Inside this file, write the following:

import { Line, mixins } from 'vue-chartjs'
const { reactiveProp } = mixins
// The mixins reactiveProp is used for updating the Chart dynamically. 
// It won't be used in this tutorial, but you can have fun with it!

export default {
  extends: Line,
  mixins: [reactiveProp],
  props: ['options'],
  mounted () {
    // this.chartData is created in the mixin.
    // If you want to pass options please create a local options object
    this.renderChart(this.chartData, this.options)
  }
}

To integrate the LineChart component we created earlier on, we will first have to import it inside the `App.vue` file:

import LineChart from 'LineChart.js';

Then, we can state that the `App.vue` component will be using the `LineChart` component by adding it to the `export module {}`:

components: {
    LineChart,
},

We can then display it in the `template` tag with

<div class="size">
  <line-chart :chart-data="datacollection"></line-chart>
</div>

Remember to have a single `div` inside of your `template` tag! Here, we are displaying the `LineChart` component using the `line-chart` tag generated by Vue and wrapping it with a `div` tag with the `size` class. We will use this `div` wrapper to display the chart in the desired size.

You can also notice that the `line-chart` tag has a `:chart-data` attribute with the value of `datacollection`. This binds the data `datacollection` that we have previously created in 3.4 to the `chart-data` attribute. This chart data will then be used to render the actual chart (Remember `this.chartData` in `LineChart.js`!).

We can then style the `size` class in the `<style>` section to set the appropriate size of the chart:

.size {
    max-width: 500px;
    margin:  50px auto;
 }

You can set the `max-width` and `margin` to whatever size you prefer your chart to be.

After this, we will need to display the data in the chart. Create the `fillChart` method inside the `methods` portion of `export module` {}:

fillChart() {
    // We are filling up this.datacollection variable for it to be displayed in the chart
   this.datacollection = {
      labels: this.dates, // This is the x-axis
      datasets: [
        {
          // Explains the significance of the y-axis value
          label: 'Total confirmed cases in Canada for COVID 19',
          backgroundColor: '#1E88FF', // Will display the line and the background in blue
          data: this.numbers, // This is the y-axis
        },
   ]};
}

Call this method at the end of the `mounted` function to load the chart once the data has been retrieved from the `post` request.

And voila! If you go to [http://localhost:8082](http://localhost:8082), you will see a chart being displayed just like magic! :confetti_ball:


# Project summary
We learned how to build a statistics dashboard for COVID-19 using Vue, Express, and Python. Specificially, we built a line chart displaying the number of confirmed COVID-19 cases in Canada as the time progress. Congrats! :tada:

# Take this project further!
* Enable the feature to update your graph dynamically from user input!
* Display a graph that compares the progress of multiple countries.
* Use other types of displays such as maps and charts from the `Chart.js` library to display the COVID19 cases!
* Predict the progress of a country with machine learning.