owned this note
owned this note
Published
Linked with GitHub
###### tags: `Tech Article`
# Data Computing with Python
In my [previous blog article](https://www.osedea.com/en/blog/data-cleaning-with-python), we've learned how to clean data with Python Pandas. Today we are going to learn one of the key tools that has been widely used for data processing and data intensive computing.
Python is one of the most popular languages for data analysis and development, because it's fast for testing and coding. However, this interpreted, dynamically-typed, high-level language is somewhat slow for advanced/complex computing tasks.
To get the best of both worlds (fast code development time and fast code execution time), **Numpy** comes in.
In this blog, we'll be going through some basic Numpy functions and how to use them effectively when working with large dataset. We will also take a look at the difference between Pure Python and Numpy, and lastly we will learn some strategies that can help us speed up code with Numpy.
Here's the topics that we will cover:
* [Introduction to Numpy](#Introduction-to-Numpy)
* [What is Numpy?](#What-is-Numpy?)
* [Quick start to Numpy](#Quick-start-to-Numpy)
* [Python Lists vs. Numpy Arrays](#Python-Lists-vs.-Numpy-Arrays)
* [Strategies for speeding up code with Numpy](#Strategies-for-speeding-up-code-with-Numpy)
* [Ufuncs](#Ufuncs)
* [Aggregations](#Aggregations)
* [Slicing, Masking and Fancy indexing](#Slicing,-Masking-and-Fancy-indexing)
---
## Introduction to Numpy
### What is Numpy?
[NumPy](https://numpy.org/doc/stable/index.html) is a Python library used for working with **multi-dimensional arrays**. Numpy stands for "Numerical Python", it provides a high-performance multidimensional array object, and tools for working with these arrays.
Numpy is a core Python library for scientific computing. With Numpy, we can perform complex mathematical and logical operations faster and easier, such as:
* Vector-Vector multiplication
* Matrix-Matrix and Matrix-Vector multiplication
* Element-wise operations on vectors and matrices (i.e., adding, subtracting, multiplying, and dividing by a number )
* Element-wise or array-wise comparisons
* Applying functions element-wise to a vector/matrix ( like pow, log, and exp)
* A whole lot of Linear Algebra operations can be found in NumPy.linalg
* Reduction, statistics, and much more.
### Quick start to Numpy
Let's take a look at a few examples to understand better.
**1. Creating Arrays**
With Numpy, we can create multi-dimensional arrays:
```python=
import numpy as np
# Create a 1-D Array
arr = np.array([1, 2, 3, 4, 5])
print(arr)
# Create a 2-D Array
arr = np.array([[1, 2, 3], [4, 5, 6]])
print(arr)
# Create a 3-D Array
arr = np.array([[[1, 2, 3], [4, 5, 6]], [[1, 2, 3], [4, 5, 6]]])
print(arr)
```
```python
array([1, 2, 3, 4, 5])
---
array([[1, 2, 3],
[4, 5, 6]])
---
array([[[1, 2, 3],
[4, 5, 6]],
[[1, 2, 3],
[4, 5, 6]]])
```
**2. Array Indexing**
Indexing arrays is similar to Python lists:
```python=
arr = np.array([1, 2, 3, 4])
# Get the first element from the array
print(arr[0])
# Get third and fourth elements from the array and sum them.
print(arr[2] + arr[3])
```
```python
1
---
7
```
**3. Array Slicing**
We can get a slice of an array like this: `[start:end]`.
```python=
arr = np.array([1, 2, 3, 4, 5, 6, 7])
# Slice elements from index 1 to index 5 from the array
arr[1:5]
```
```python
array([2, 3, 4, 5])
```
**4. Array Shape**
The shape of an array is the number of elements in each dimension. We can get the current shape of an array like this:
```python=
# Create a 2-D array
arr = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
print(arr)
# Get the shape
print(arr.shape)
```
```python
array([[1, 2, 3, 4],
[5, 6, 7, 8]])
---
(2, 4)
```
(2, 4) means that the array has two dimensions, the first dimension has two elements and the second has four.
**Reshaping arrays**
- Reshape an array from 1-D to 2-D
```python=
# Create a 1-D array
arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
print(arr)
# Convert the 1-D array with 12 elements into a 2-D array. The outermost dimension will have 4 arrays, each with 3 elements
new_arr = arr.reshape(4, 3)
print(new_arr)
```
```python
array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
---
array([[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9],
[10, 11, 12]])
```
For more details, see [NumPy Documentation](https://numpy.org/doc/stable/).
## Python Lists vs. Numpy Arrays
A common question is what is the real difference between Python lists and Numpy arrays. The answer is **performance**.
#### Why is Numpy faster?
Python speed suffers from small overhead, such as type checking and reference counting, which builds up for each operation and it could become especially significant in a large loop.
What Numpy does is that it pushes those loop operations down into compiled code, so the type checking only has to happen once for the entire loop therefore they can be done quickly.
**Overall, Numpy performs better in:**
* **Size** - NumPy uses much less memory to store data.
* **Performance** - NumPy arrays are faster than Python lists.
* **Functionality** - NumPy has optimized mathematical functions built-in that are convenient to use.
#### Memory consumption
To allocate a list of 1000 elements, Python takes 48000 bytes of the memory, whereas Numpy only needs 8000 bytes, what a significant improvement!
```python=
import numpy as np
import sys
# Declaring a Python list of 1000 elements
ls = range(1000)
print(f"Size of each element of the list: {sys.getsizeof(ls)} bytes.")
print(f"Size of the list: {sys.getsizeof(ls)*len(ls)} bytes.")
# Declaring a Numpy array of 1000 elements
arr = np.arange(1000)
print(f"Size of each element of the array: {arr.itemsize} bytes.")
print(f"Size of the array: {arr.size*arr.itemsize} bytes.")
```
```python
Size of each element of the list: 48 bytes.
Size of the list: 48000 bytes.
Size of each element of the array: 8 bytes.
Size of the array: 8000 bytes.
```
#### Performance
There's a huge difference in performance. If we add 5 to each element of a list of 100k values:
```python=
# Python way
ls = list(range(100000))
%timeit [val + 5 for val in ls]
# Numpy way
arr = np.array(ls)
%timeit arr + 5
```
```python
6.17 ms ± 35.9 µs per loop
---
34.8 µs ± 340 ns per loop
```
It turned out using about 6 milliseconds (could be varied depending on different machines) for Python to process per loop while Numpy only took 34 microseconds per loop. We have about **nearly 200x speed up** by taking away the loops in python!
## Strategies for speeding up code with Numpy
Now we've learned that Numpy is a powerful tool for working with large arrays. However, it's not only useful for scientific computing, but can also be handy for some software/services development tasks that involve complex data processing with a large dataset. With pure Python, we would normally loop over the dataset and do it raw and that's... not so optimal right?😅
In order to achieve that effectively, let's see what Numpy is capable of and how we can use it to speed our code, by applying the following strategies. Let's get rid of those slow loops!
* [Ufuncs](#Ufuncs)
* [Aggregations](#Aggregations)
* [Slicing, Masking and Fancy indexing](#Slicing,-Masking-and-Fancy-indexing)
### Ufuncs
Ufuncs, short for **universal functions**, offers a bunch of operators that **operate element by element on entire arrays**.
(See [universal functions basics](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-basics) for more details about ufuncs.)
For example, we want to add the elements of list `a` and `b`. Without Numpy, we can use the built-in `zip()` method:
```python=
a = list(range(1001)) # [0, 1, 2, 3, ..., 999, 1000]
b = list(reversed(range(1001))) # [1000, 999, 998, ..., 1, 0]
c = [i+j for i, j in zip(a, b)] # returns [1000, 1000, ..., 1000, 1000]
```
Instead of looping the list, we can simply use the `add()` from Numpy ufunc:
```python=
import numpy as np
a = np.array(list(range(1001)))
b = np.array(list(reversed(range(1001))))
c = np.add(a, b) #returns array([1000, 1000, ..., 1000, 1000])
```
Let's compare the execution time:
```python=
%timeit [i+j for i, j in zip(a, b)]
%timeit np.add(a, b)
```
```python
61 µs ± 4.11 µs per loop
---
811 ns ± 29.8 ns per loop
```
We got 75x faster with Numpy version and the code looks much cleaner! Isn't that awesome? 😎
There are many other [ufuncs](https://numpy.org/doc/stable/reference/ufuncs.html) available:
* Arithmetic: `+`, `-`, `*`, `/`, `//`, `%`…
* Comparison: `<`, `>`, `==`, `<=`, `>=`, `!=`…
* Bitwise: `&`, `|`, `~`, `^`...
* Trig family: `np.sin`, `np.cos`, `np.tan`…
* And more…
### Aggregations
Aggregations are functions which summarize the values in an array. (e.g. min, max, sum, mean, etc.)
Let's see how long it takes a Python loop to get the mininum value of a list of 100k random values.
With Python built-in `min()`:
```python=
from random import random
ls = [random() for i in range(100000)]
%timeit min(ls)
```
With Numpy aggregate functions `min()`:
```python=
arr = np.array(ls)
%timeit arr.min()
```
```python
1.3 ms ± 7.59 µs per loop
---
33.6 µs ± 248 ns per loop
```
Again, we got about 40x speedup without writing a single loop with Numpy!
In addition, aggregations can also work on multi-dimensional arrays:
```python=
# Create a 3x5 array with random numbers
arr = np.random.randint(0, 10, (3, 5))
print(arr)
# Get the sum of the entire array
print(arr.sum())
# Get the sum of all the columns in the array
print(arr.sum(axis=0))
# Get the sum of all the rows in the array
print(arr.sum(axis=1))
```
```python
array([[7, 7, 5, 1, 9],
[8, 7, 0, 2, 4],
[0, 5, 6, 4, 0]])
---
65
---
array([15, 19, 11, 7, 13])
---
array([29, 21, 15])
```
There are more aggregate functions:
* sum
* mean
* product
* median
* variance
* argmin
* argmax
* And more
### Slicing, Masking and Fancy indexing
We just saw how to slice and index Numpy arrays earlier. In fact, there are some other faster and more convenient ways that Numpy offers:
#### Masking
With masks, we can index an array with another array. For example, we want to index an array `a` with a boolean mask. What we get out, is that **only the elements that line up with True in that mask will be returned**:
```python=
# Create an array
arr = np.array([1, 2, 3, 4, 5])
print(arr)
# Create a boolean mask
mask = np.array([False, False, True, True, True])
print(arr[mask])
```
```python
array([1, 2, 3, 4, 5])
---
array([3, 4, 5])
```
Here we only get `[3 4 5]`, because these values line up with True in the mask.
Now you might be a little confused why we would ever need that? Where masks become useful, is when they are combined with ufuncs. For example, we want to get the elements from an array based on some conditions. We can index that array using a mask with the conditions:
```python=
# Create an array
arr = np.array([1, 2, 3, 4, 5])
print(arr)
# Create a mask
mask = (arr % 2 ==0) | (arr > 4)
print(arr[mask])
```
```python
array([1, 2, 3, 4, 5])
---
array([2, 4, 5])
```
And the values that meet the criteria will be returned.
#### Fancy indexing
The idea of fancy indexing is very simple. It allows us to access multiple array elements at once by passing an array of indices.
For example, we get the 0th and 1st element from array `a` by passing another array of the indices:
```python=
arr = np.array([1, 2, 3, 4, 5])
index_to_select = [0, 1]
print(arr[index_to_select])
```
```python
array([1, 2])
```
#### Let's combine all these together:
* Access multi-dimensional arrays by `[row, column]`:
```python=
# Create a 2x3 array
arr = np.arange(6).reshape(2, 3)
print(arr)
# Get row 0 and column 1
print(arr[0, 1])
# Get all rows in column 1 (Mixing slices and indices)
arr[:, 1]
```
```python
array([[0, 1, 2],
[3, 4, 5]])
---
1
---
array([1, 4])
```
* Masking multi-dimensional arrays:
```python=
# Create a 2x3 array
arr = np.arange(6).reshape(2, 3)
print(arr)
# Masking the entire array
mask = abs(arr-3)<2
print(arr[mask])
```
```python
array([[0, 1, 2],
[3, 4, 5]])
---
array([2, 3, 4])
```
* Mixing masking and slicing:
```python=
arr = np.arange(6).reshape(2, 3)
print(arr)
mask = arr.sum(axis=1) > 4
print(arr[mask, 1:])
```
```python
array([[0, 1, 2],
[3, 4, 5]])
---
array([[4, 5]])
```
**All of these Numpy operations can be combined in nearly limitless ways!**
## Conclusion
No matter how large is your dataset, or how complex are the operations you need to perform with your data, Numpy functions are there to make it easier and faster!
Let's summarize what we've covered:
* Writing Python is fast, but loops are slow for large dataset computing.
* Numpy is a greate tool to use! It pushes loops into its compiled layer, so we get:
* Faster development time, and
* Faster execution time.
* Strategies for speeding up your code:
* **Ufuncs** - For element wise operations.
* **Aggregations** - For array summarization.
* **Slicing, masking and fancy indexing** - For quickly selecting and operating on arrays.
---
Find the final source code [here](https://github.com/Osedea/osedea_training_data-computing).👈
References:
* Python Course - [NumPy Tutorial](https://webcourses.ucf.edu/courses/1249560/pages/python-lists-vs-numpy-arrays-what-is-the-difference)
* [NumPy Documentation](https://numpy.org/doc/stable/)
