Jupyter / Cloud deployments working group
===

[30 minute plenary slides here](
https://docs.google.com/presentation/d/15a6ORNJvavmtdLXW3IZzlVIhuxb81AWG8VPCxh1eEv4/edit?usp=sharing)

:::info
- **Location:** Alder 107
- **Date:** August 20 10-11 AM (PST)
- **Agenda**
1. Create a wishlist`15min`
> prioritized wishlist of features that are lacking (sky is the limit!)
2. Discussion `30min`
3. Recap `15min`
- **Participants (please add name):**
- - YOUR NAME (affiliation)
- Scott Henderson (UW eScience)
- Yuvi Panda (Project Jupyter)
- James Munroe (MUN)
- Scott Black (UWRL)
- Matt! Rocklin (NVIDIA)
- Rodrigo Manzanas (IPCC)
- Hillary Scannell (UW)
- Shreyas Cholia (LBL)
- Tim Crone (Lamont-Doherty Earth Observatory)
- Mary Romelfanger (STScI Space Telescope Science Institute)
- Friedrich Knuth (UW)
- Kirstie Haynie (USGS)
- Nicholas Sofroniew (CZI)
- Jacob Matuskey (STScI Space Telescope Science Institute)

- **Contact:** Scott <[e-mail]>

:::

## Wishlist

### User convenience

- Possibility to execute line by line within the same cell?
- Secure push / pull integration with GitHub / GitLab
- SSH / SCP / FTP access to your home directory
- Propagate user environments from notebooks to dask workers
- Submit batch jobs rather than interactive notebooks
- Ability for users to have a single S3 bucket that is accessible only to them and a common bucket accessible from the Hub and non-hub machines (e.g. HPCs, laptops, lab servers)

### Performance

- Continued development of mixing on demand and spot instances for notebook servers and dask workers.
- Faster spinup/teardown of workers

### IT Control

- Secure PVC / home directories
- Expand community of operations folks working in this space
- tutorials on operationalizing your JupyterHub
- Avoid giving users advanced permissions on Kubernetes
- Quotas on User home directories (we're currently giving unlimited NFS storage space!)
- Segregate git-crypt keys or use another system
- Remove staging and test deployment to temporary deployment/release

### Generalize

- Defining Pangeo for X - what pieces of pangeo would make sense in other domain sciences
- Abstract out a generic pangeo-cloud deployment from specific deployments
- More turnkey deployments on traditional HPC environments (may need some coordination for HPC centers for deploying hubs etc.)
- Using images built for cloud deployments on HPC, and vice versa
- The Littlest Pangeo JupyterHub
- The Littles BinderHub

### Diagnostics
- JupyterLab widgets to track Kubernetes pods
- Log usage statistics and have system to visualize. Track costs at user level.

### Now prioritize the above list
1. Top priority
2. Second priority

## Discussion Notes

### prompts
- What are the most effective ways to communicate and collaborate among open-source projects that share depencies (Discourse, GitHub - issues, pull requests, project boards), Gitter, Slack)?
- How to “operationalize” the hubs? User management. Track usage at cluster, hub, and user levels.
- How to prevent disruptions between staging and production environments (current issues with pangeo cloud federation)
- Should we focus on making hubs easy to deploy in each cloud region, or focus on a single hub deployment and then tools that can operated on data in different regions (e.g. dask distributed refactor, submitting batch jobs)
- What is the long-term vision for sustaining projects that are currently subsidized by Cloud credits and short-term grants that support administrator/developer salary