---
# System prepended metadata

title: 9/6/20 Sunday Demos 10-11

---

# 9/6/20 Sunday Demos 10-11
Yanzhi - example problems.  Standard format, pytest stuff.
Jane - Dask
Jane & Liam - tidy-data
Liam - SQL practice, Data-Viz
Will - Web Scraping
Da - Wikipedia Demo, Tuturial Project

# Course Outline

## Software Development Setup and Skills 101 (4 weeks)


### Data Structures and Algorithms I (Yanzhi)
[Detailed Outline](https://hackmd.io/JdPkiyP9R66leyw63G3rkw)  

3 lectures:
1. VSCode & Docker or GCP Setup
3. Test First Development (A)
5. Idiomatic Python (A - Brett Slatkin)  
[Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)
7. Recursion (A)

6 lectures(Assignments for each Dan and Yanzhi):
1. ADTs and Classes
1. Data Structures
1. Time and Space - Elementary Operations & Memory
1. Complexity Analysis - Theoretical
1. Complexity Analysis - Practical
1. Logarithms
1. Arrays
1. Linked-Lists
1. Hash Tables
1. Stacks and Queues
1. Strings
1. Graphs
1. Trees

### Tidy-Data (Jane & Liam)
Students will connect to a remote postgres database and practice SQL and pandas using tidy-data principles. 

SQL 3 assignments

#### SQL (Liam)
1. [SQL first steps / tutorial](https://hackmd.io/_2bZWoB5QbeQDTSi9tLGmg?both) for students to follow along in pgAdmin
1. SQL assignment using MIMIC data

*TODO:* add info on star schemas


#### Python (Jane)
1. Python notebook for pandas practice
1. First Step tutorials for connecting to PostgreSQL in Jupyter Notebook and connecting to BigQuery in Tableau
1. projects/assignments

Pandas with Databases Assignment?

### Big Local Data (Jane)
Vaex, Dask versus Plain Pandas
First Steps
An Example
Modifcation Assignment
Deliverable:
Jupyter Notebook Tutorial and Excercise

Review example, maybe Vaex as well
1 asssignment where they work with Big Local Data


### Tableaux/Looker (Jane + ?)
First Steps
An Example
Dashboard Examples
Modification Assignment

### Static Site (Jane)
Deliverables:
3 tutorials: HTML, Hugo, Hosting on Github
yangyuegly.github.io
First Step
Example (1. Plain HTML+CSS, 2. HUGO)
Modification Assignments
1. Something local?
2. Students own GitHub.io pages

Give them a screen shoot and have them build the page (start with a template) (Will)
Assignment: Setup your own webpage on github.io



## Accessing Live Data (Will)
### MongoDB
Deliverable:
MongoDB Training Tutorials
First Steps
Example
Modification Assignment

Mongodb - Demo Atlas Account?
give each student own db w r/w - allow r access to every db

## Distributed Computing (3 weeks)

### SageMaker (+ Spark)

#### Deliverables (Desmond)

1. A demo of a K-means model trained on MNIST to read handwritten digits. Host an instance on SageMaker running this model as set up in this [tutorial](https://docs.aws.amazon.com/sagemaker/latest/dg/apache-spark-example1.html)
2. First Steps for SageMaker (+S3? Or AWS in general)?

### Spark

The demo for SageMaker mostly demonstrates the capabilities of SageMaker along with Spark's ability to provide a distributed dataset, however it doesn't illustrate how Spark can be used for large-scale data processsing i.e. via MapReduce.

To address this, we can modify the SageMaker demo to rely on our own distributed K-means clustering algorithm. We can then use this distributed computation to illustrate various concepts about distributed computing via Spark. More details on the distributed K-means algorithm can be found [here](https://stanford.edu/~rezab/classes/cme323/S16/projects_reports/bodoia.pdf)

Topics to cover:

* MapReduce
* Resilient Distributed Datasets
* Lazy evaluation
* Caching (find a way to include this?)
* Running on a cluster

#### Deliverables (Desmond)

1. The same K-means model to read handwritten digits, but with a distributed K-means algorithm coded using Spark.
2. First Steps for Spark?
3. First Steps for whatever platform they are running Spark on (e.g., Sagemake First Steps)

### Kafka

#### Deliverables
First Steps Kafka Config
Tutorials 
Exercises

Example: Wikipedia Example
Modification Exercise

1. Host Kafka service in cloud (AWS MSK)
2. Demo Kafka project with wikipedia (https://docs.confluent.io/current/tutorials/cp-demo/docs/index.html?utm_source=github&utm_medium=demo&utm_campaign=ch.cp-demo_type.community_content.cp-demo)

topics to cover:
1. log commit
2. producer
3. consumer
4. Connect API
5. ksql ?

### Neo4j (Dan)
First Steps (Dan)
Tutorials (Dan)
Exercises (Dan)
Example Application (Dan)
Modification Exercise (Dan)

## Project Examples (3 weeks - detailed walk-through of each)

