---
# System prepended metadata

title: Introduction to High Performance Computing

---

# Introduction to High Performance Computing
### Dr. Anirban Pal, Asst. Professor of Mechanical Engineering 

Time, Date: 4:15-5:00 CT, 04/08/2022
Location: ECS 142
Pre-requisites: Ability to read and type
Recommended: Familiarity with Linux command line and Python3

## 0. What is High Performance Computing (HPC)? (4:15-4:30)

HPC or High Performance Computing is the use of a numerous processing elements (CPU/GPU) to solve calculation-intensive and/or data-intensive tasks. Tasks include but not limited to scientific/engineering problem solving, data-analysis, visualization. History can be traced to supercomputing. In fact, supercomputing and HPC is often used synonymously. Computing on your laptop/desktop may be inadequate and “You are going to need a bigger boat!”

Slides here: [Introduction to HPC](https://wtamu0-my.sharepoint.com/:b:/g/personal/apal_wtamu_edu/Eaa7xN4MHAhEvLUgeuxYlP0B4b2LFc_TueWm5s_YfMrPcw?e=CssIgh)

## 1. Logging into WTAMU HPC (4:30-4:35)

STEP 1: Open command line terminal (bash) and type+enter (buff** is your HPC username)
```
ssh buff**@hpcjump.wtamu.edu
```
STEP 2: Enter the HPC password provided to you.
```
apal@ecsl-44:~$ ssh buff50@hpcjump.wtamu.edu
+------------------------------------------------------------------------------+
| INFORMATION RESOURCES ACCESS POLICY                                          |
+------------------------------------------------------------------------------+
| This computer system is the property of West Texas A&M University. Only      |
| authorized users may login to this computer system. All unauthorized use is  |
| strictly prohibited and subject to local, state, and/or federal laws.        |
| Therefore, this system is subject to security testing and monitoring. Misuse |
| is subject to criminal prosecution.                                          |
|                                                                              |
| If you proceed to log into this system, you acknowledge compliance with all  |
| related TAMU System Security Standards and University Security Controls      |
| located at https://www.wtamu.edu/rules/ . There should be no expectation of  |
| privacy except as otherwise provided by applicable privacy laws.             |
+------------------------------------------------------------------------------+

buff50@hpcjump.wtamu.edu's password:
```
STEP 3: You should be in the **hpcjump** jump server now. Log into the login node. Enter **yes** if prompted.
```
[buff50@hpcjump ~]$ ssh L01
The authenticity of host 'l01 (10.10.55.21)' can't be established.
ECDSA key fingerprint is SHA256:8bmTckGZJdagIq64oIPNJMyaA3K8FisXMnaHpi8Y3sE.
ECDSA key fingerprint is MD5:19:f6:c5:3c:e7:5d:33:39:49:87:71:68:2b:e4:bb:e1.
Are you sure you want to continue connecting (yes/no)? yes
```
You may need to reenter the HPC password.
```
Warning: Permanently added 'l01,10.10.55.21' (ECDSA) to the list of known hosts.
+------------------------------------------------------------------------------+
| WTAMU COMPUTER ACCESS POLICY                                                 |
+------------------------------------------------------------------------------+
| This computer system is the property of West Texas A&M University.  Only     |
| authorized users may login to this computer system.  All unauthorized use is |
| strictly prohibited and subject to local, state, and/or federal laws.        |
| Therefore, this system is subject to security testing and monitoring.        |
| Misuse is subject to criminal prosecution.                                   |
|                                                                              |
| If you proceed to log into this system, you acknowledge compliance with      |
| University Rule 24.99.99.W1 -- Security of Electronic Information Resources  |
| and all related University security polices located at                       |
| http://www.wtamu.edu/informationtechnology/university-saps-and-rules.aspx    |
| and University Rule 33.04.99.W1 -- Rules for Responsible Information         |
| Technology Usage.  There should be no expectation of privacy except as       |
| otherwise provided by applicable privacy laws.                               |
+------------------------------------------------------------------------------+
buff50@l01's password: 
Creating ECDSA key for ssh
[buff50@L01 ~]$
```
Now you are in the login node and ready to submit jobs!

## 2. Submitting a simple python job on the HPC (4:35-4:40)

An HPC job typically needs three things: 

>(1) **input data** to be processed (optional)
>(2) **program** to do something, eg. process the data (necessary)
>(3) **job script** that informs the HPC how the job will be run. (necessary) 

and produces:
>(1) **output data** containing some information/data.

### 2.1 Preparing and studying the files
Files for this following section can be obtained by
```
[buff50@L01 ~]$ module load spack/git/2.31.1 
[buff50@L01 ~]$ git clone https://github.com/anirban-pal/hackhpcworkshop.git
Cloning into 'hackhpcworkshop'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (5/5), done.
[buff50@L01 ~]$ cd hackhpcworkshop/simple/
[buff50@L01 simple]$ 
```
Here we will run a very simple python program *simple.py*. Note that this program has no input data and simply prints the message "Hello World".
```
[buff50@L01 simple]$ cat simple.py
print("Hello World")
```
To run this program on an HPC, we need to create a **slurm job script** that will contain instructions on how to run the program. The slurm job script *sb.simple* contains the following.
```
[buff50@L01 simple]$ cat sb.simple
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --partition=compute-cpu
#SBATCH --output=log.slurm.out
#SBATCH --error=log.slurm.err
#SBATCH --time=10:00:00

module load slurm/18.08.9
module load spack/python/3.8.12

srun python simple.py
```
Let us spend some time understanding what all this means.
>#!/bin/bash

This line indicates that this is a bash script. Bash is the language of the command line.

>#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --partition=compute-cpu
#SBATCH --output=log.slurm.out
#SBATCH --error=log.slurm.err
#SBATCH --time=10:00:00

These lines provide all the information on how the program will be run, including: how many nodes are requested, how many processors/node are requested, which partition of the cluster to run the program on, filenames where the output and error information will be stored, and the upper time-limit within which the program should complete/terminate.

>module load slurm/18.08.9
module load spack/python/3.8.12

These commands basically tells the HPC where the **python** and **srun** programs are located. If you do not provide this, the HPC will not know what the word **python** and **srun** (used in the next line) means.

>srun python simple.py

Finally, this line indicates the program **simple.py** will be executed using the **python** interpreter and **slurm (srun)** scheduler.

### 2.2 Running the job
Once the files *simple.py* and *sb.simple* are ready, they can be executed using:
```
[buff50@L01 simple]$ sbatch sb.simple
Submitted batch job 27968
[buff50@L01 simple]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             27968 compute-g sb.simpl     buff50 PD       0:00      1 (None)
```
### 2.3 Viewing results
The output information is stored in *log.slurm.out* as per the *sb.simple* file.
```
[buff50@L01 simple]$ cat log.slurm.out
Hello World
```
Now try setting the number of processors to 16 or 64 in *sb.simple* and re-view the output.
>#SBATCH --tasks-per-node=16

## 3. Submitting a less simple python job on the HPC (using single CPU, multiple CPUs,  GPU)

### 3.1 Single CPU implementation (serial) (4:40-4:45)
Now let us look at a  less simple program **color2gray.py**. This program will run on 1 cpu and convert a set of images from color (rgb) to grayscale.
```
[buff50@L01 simple]$ cd ../serial/
[buff50@L01 serial]$ cat color2gray_serial.py 
import numpy as np
from matplotlib import pyplot as plt
from time import time

import os
try:
    os.mkdir("grayscale")
except OSError as error:
    pass

start_time = time()

num_imgs = 800
for i in range(1,num_imgs+1):
    
    # Read Images
    infile = "/cm/shared/data/DIV2K_train_HR/{:04d}.png".format(i)
    img = plt.imread(infile)

    # Convert to grayscale
    orig = np.asarray(img)
    gray = (0.2989 * orig[:,:,0] + 0.5870 * orig[:,:,1] + 0.1140 * orig[:,:,2])*255

    # Output Images
    outfile = "grayscale/{:04d}.png".format(i)
    plt.imsave(outfile, gray, cmap="gray")
    print(outfile)

finish_time = time()
elapsed_time = finish_time - start_time
print("Time taken (seconds):", elapsed_time)
```
If you are familiar with python, you might be able to figure out what is going on. If not, we can study this program more closely. The key portion of the program is the loop.
```python=
num_imgs = 800
for i in range(1,num_imgs+1):
    
    # Read Images
    infile = "/cm/shared/data/DIV2K_train_HR/{:04d}.png".format(i)
    img = plt.imread(infile)

    # Convert to grayscale
    orig = np.asarray(img)
    gray = (0.2989 * orig[:,:,0] + 0.5870 * orig[:,:,1] + 0.1140 * orig[:,:,2])*255

    # Output Images
    outfile = "grayscale/{:04d}.png".format(i)
    plt.imsave(outfile, gray, cmap="gray")
    print(outfile)
```
The first part of the loop reads an image (indexed by i, the loop variable) into an 3D array **img** which is then converted to a 3D numpy array **orig** in the second part. The three color channels (red, green, blue) of the image are the 2d matrices **orig[:,:,0]**,**orig[:,:,1]**, and **orig[:,:,2]** respectively, which are weight-averaged to create a grayscale image matrix **gray**. The third part writes the image to a file.

### 3.2 Multi-CPU implementation (mpi) (4:45-4:55)

Now let us run the above program in parallel. The easiest way to parallelize the above code is to divide the set of images equaly among the various processors.
```
[buff50@L01 serial]$ cd ../mpi/
[buff50@L01 mpi]$ cat color2gray_mpi.py 
import numpy as np
from matplotlib import pyplot as plt
from time import time
from mpi4py import MPI

import os
try:
    os.mkdir("grayscale")
except OSError as error:
    pass

start_time = time()

world_comm = MPI.COMM_WORLD
world_size = world_comm.Get_size()
my_rank = world_comm.Get_rank()

num_imgs = 800
for i in range(1,num_imgs+1):
    
	if (i % world_size == my_rank):
		
		# Read Images
		infile = "/cm/shared/data/DIV2K_train_HR/{:04d}.png".format(i)
		img = plt.imread(infile)
		
		# Convert image to grayscale
		orig = np.asarray(img)
		gray = (0.2989 * orig[:,:,0] + 0.5870 * orig[:,:,1] + 0.1140 * orig[:,:,2])*255
		
		# Output Images
		outfile = "grayscale/{:04d}.png".format(i)
		plt.imsave(outfile, gray, cmap="gray")
		print(outfile, "processor ", my_rank)

world_comm.Barrier()

if (my_rank == 0):
	finish_time = time()
	elapsed_time = finish_time - start_time
	print("Time taken (seconds):", elapsed_time, " with ",world_size, " processors." )
```
We have had to make a few changes. See if you can spot what has changed. What do the changes mean?
```python=
world_comm = MPI.COMM_WORLD #
world_size = world_comm.Get_size() #
my_rank = world_comm.Get_rank() #

num_imgs = 800
for i in range(1,num_imgs+1):
    
	if (i % world_size == my_rank): #
		
		# Read Images
		...
		...
		...
```
Now submit the mpi job. 
```
[buff50@L01 mpi]$ sbatch sb.mpi
Submitted batch job 28103
[buff50@L01 mpi]$ cat log.slurm.out | grep "Time"
Time taken (seconds): 92.5342857837677  with  8  processors.
```
Change the number of processors used (--tasks-per-node) to 16 and 64 and resubmit the job to see how the *Time taken* changes. You should get faster (shorter times) with increasing number of processors.

### 3.3 GPU implementation (cuda) (4:55-5:00)

Running code on gpu can be fast if there are a lot of matrix operations involved, compared to I/O operations. Here we shall look at a different program **cupy_numpy.py** to see the advantage of using GPUs.

This program has three tasks.

(Task 1) Creating a 1000x1000x1000 array of ones.
```python=
### Numpy and CPU
s = time.time()
x_cpu = np.ones((1000,1000,1000))
e = time.time()
print("(Task 1) Time with Numpy+CPU: ",e - s)
### CuPy and GPU
s = time.time()
x_gpu = cp.ones((1000,1000,1000))
cp.cuda.Stream.null.synchronize()
e = time.time()
print("(Task 1) Time with Cupy+GPU: ",e - s)
```
(Task 2) Multiply entire array by 5
```python=
### Numpy and CPU
s = time.time()
x_cpu *= 5
e = time.time()
print("(Task 2) Time with Numpy+CPU: ",e - s)

### CuPy and GPU
s = time.time()
x_gpu *= 5
cp.cuda.Stream.null.synchronize()
e = time.time()
print("(Task 2) Time with Cupy+GPU: ",e - s)
```
(Task 3) Multiply the array by 5, then multiply the array by itself, and then add the array to itself.

```python=
### Numpy and CPU
s = time.time()
x_cpu *= 5
x_cpu *= x_cpu
x_cpu += x_cpu
e = time.time()
print("(Task 3) Time with Numpy+CPU: ",e - s)

### CuPy and GPU
s = time.time()
x_gpu *= 5
x_gpu *= x_gpu
x_gpu += x_gpu
cp.cuda.Stream.null.synchronize()
e = time.time()
print("(Task 3) Time with Cupy+GPU: ",e - s)
```
Submitting the program will show us the advantage of using GPUs.
```
[buff50@L01 mpi]$ cd ../cuda/
[buff50@L01 cuda]$ sbatch sb.cuda 
Submitted batch job 28114
[buff50@L01 cuda]$ cat log.slurm.out 
(Task 1) Time with Numpy+CPU:  0.8523945808410645
(Task 1) Time with Cupy+GPU:  0.31257200241088867
(Task 2) Time with Numpy+CPU:  0.590524435043335
(Task 2) Time with Cupy+GPU:  0.04698443412780762
(Task 3) Time with Numpy+CPU:  1.8139195442199707
(Task 3) Time with Cupy+GPU:  0.07738733291625977
```
## 4. Conclusion
HPC resources can be really useful when the task at hand is too big for a workstation to handle (eg. data science, computational science and engineering). In this case, one must either develop a parallel program (with potentially parallel I/O) or use a parallelized code for the task. Developing a good parallel code requires extensive knowledge of computer architecture (processor speeds, cache sizes, memory, network latency and bandwidth, etc). Maintaining and administering such HPC systems is also quite a challenge considering reliability, cybersecurity, and package/library management). Both present significant career opportunities.