---
# System prepended metadata

title: TTT4HPC - Interactive episode
tags: [TTT4HPC, Training]

---

# TTT4HPC - Interactive episode

###### tags: `Training` `TTT4HPC`

Day 2 of TTT4HPC - Tue 23/April/2024: 10-12, 13-14.30 EEST

## Instructors

EG, RD, HF, SW, JR ...?

## Materials

https://github.com/coderefinery/TTT4HPC_Interactive
-> https://coderefinery.github.io/TTT4HPC_Interactive/


## Things to discuss and decide

- :ok_hand: **Setup of the day:**
    - Use morning for lectures (on stream), afternoon for exercises (all in zoom), only demos in morning

- :ok_hand: How to make this session not cluster/tool specific?
    - First exercises could be "find out how to do X on your cluster" -> link to docs pages
    - Tabs with different clusters/tools
    - focus on command line tools to show the concept?, it is easy to switch to OOD/GUI tools afterwards?
    - RD: some of the presentations *might* be, or might not be that easy to reuse, but it's the talking that has to make sure to provide the bigger picture.  Have co-instructors from other clusters who can comment "oh, but is that really general?  How would someone else do that?" and so on
- **Goals/topics of this day**
    - About "geography of things", 
    - developing with the cluster (in mind)
    - **moving data** and code
        - RD: biggest unsolved problems I know of (at the early phase) are moving data around, debugging ("but can I run vscode on it?" - they don't know about cmd line debugging), can only work with "run" button in IDE, Jupyter→ Batch jobs, and *data transfer (most important!)*
- **Prerequisites**
    - Anyone can follow. 
    - If you want to apply it: have access to cluster, basics of CLI 
    - 
- Check timetable below, add your name, check and adjust timing if needed



#### Timing for the day

1) Live stream on TwitchTV 2h

- 00:00 - 00:05 Introduction and practicalities (EG, RD, JR)
    - Welcome
    - Connection to previous and next days in TTT4HPC
    - Setup of the day
    - Collaborative Doc
    - Icebreaker
- 00:05 - 00:15 Motivation (EG, RD, JR)
    - Today is about basic ways to do development work on the cluster
    - Part of that is how to do the initial development and debugging on cluster (stuff that is easy locally but hard in batch jobs)
    - What are ways you do development? (only on laptop?  on login node?  submitting jobs and waiting?  `srun --pty` ?)
    - Reasons you may need interactive work
        - You need something visual with a GPU
        - Initial development (testing and debugging) more quickly than submit-wait-fix-repeat
        - Debugging (fire up debugger on live job)
    - Next: we show how to do it with various options (terminal, screen, sinteractive, srun tty, sshfs, gui tools ood vscode) 
    - ~~start broad, what to do when getting access to a cluster -> "a day in the life with HPC"~~
    - ~~story: maybe run a simple script that makes a figure ?~~
    - ~~geography of things: different homes in different places, different speed of access to data~~

- 00:15 - 00:20 Good project arrangement (RD, EG)
    - Why this is important to start with
    - Examples
        - No live example but show a project arrangement in the lesson.
- 00:20 - 00:40 Show and tell - syncing data (RD, JR) 
    - Make a data plan
    - Example plan for distributed project
        - Demo: show example in lesson
    - Transferring data (rsync)
        - Demo: show rsync
    - Syncing data two-ways (unison)
        - Demo: simple unison demo
    - git-annex (mention only)
    - Mounting data (sshfs)
        - Demo: show sshfs, open up a file locally
- 00:40 - 00:50 Show and tell - syncing code (JR, RD)
    - Basics of using git on two ends (commit, push, pull, run, edit remotely, commit, etc)
        - Demo: doing this
    - Discussion about syncing workflows: is this too slow?  what if you want to work locally? etc.
- 00:50 - 01:00 Working interactively - GUI (JR, RD)
    - There *are* GUIs avaliable
    - May be good for editing and light analysis.
    - Many start with GUI
    - Demos
        - Show RStudio via OOD
        - Difference in speed between X forwarding and OOD/vnc/etc
    - (Ways this is possible on HPC too)
    - (RStudio/Jupyter/VSCode)
    - (GUI software via OOD or -X)
- 01:00 - 01:10 Break
- 01:10 - 01:30 Working interactively - CLI (RD,JR)
    - Interactive working from command line - possibilities
    - When do and when is it time to move to batchjobs?
    - Demos
        - srun --pty bash
        - screen/tmux
        - cli debugging (pdb)
        - shell tips and tricks (common aliases and bashrc config)
- 01:30 - 01:50 Show and tell - VSCode, remote SSH (HF, RD)
    - "one foot on cluster, but still comfortable on laptop" 
    - other tools that provide this option: Matlab, ...?
    - How to setup?
    - Common problems/issues
- 01:50 - 02:00 Summary, future direction, what we will do in the exercises (EG, RD, JR)

2) Lunch 1h Go eat

3) Zoom 1.5h

- () Introduction to exercises

- Exercises, add your name for preparation
    - We could provide a bunch of exercises and people can move into breakoutrooms depending on which exercise they want to discuss? Some cluster dependent breakoutrooms (if available) E.g.


- (SW) Docs: Find out what options your cluster has for
    - data storage and transfer
    - running GUI tools
    - VSCode SSH remote support
- (RD) Try out syncing some data
    - GUI
    - web interface
    - CLI
- (JR) Try out syncing code
    - git
- (JR) Try out running some GUI interactively 
- (EG) Try out running some CL tool interactively
- (HF) Try out SSH remote for VSCode 

- () Summary, wrap up, where to go from here


#### Exercises



---
## From earlier, partly outdated

### User stories

* 2: I develop on my laptop/workstation and also on the HPC cluster, how can I keep things in sync with minimum effort? 
    * remote vs local editors. Git push/pull.
    * rsync of data? (git annex? simple rsync?)
* 5: How can I work interactively (non-gui or gui) on a cluster?
    - non graphical
        - sinteractive?
        - srun + tty
        - screen/tmux
        - vscode?
        - Matlab?
    - graphical
        - OpenOnDemand (not available everywhere)
        - VDI tools?
        - sinteractive +X ?
        - Comsol
        - Spyder
        - Matlab
        - Jupyter
        - -> how to bypass the need of graphical stuff by scripting things 
    - How can one find the interactive part of their workflow and optimize for that? Usually only some part of the workflow needs interactivity.

    - A good example where interactive use is necessary?
        - When there is too much data and has to stay on the cluster but you are still prototpyping and/or visualizing and/or debugging
        - Some tools require interaction to proceed e.g. doing QC of pictures
        - RD: sometimes I had to open a dataset with large memory and do a one-time exploration + processing.  this was simplest in IPython+interactive, rather than debugging by submitting many jobs.


### Plan from old planning doc

* ** Motivation**
    * geography of things

* **From laptop to cluster: Syncing code (and data)** (45 min) 
    * User stories:2+5
    * Three points
        * Arranging projects well and using version control makes it possible to work between laptop and cluster, and this is important
        * Understand example of a good project arrangement
        * How this works with multiple people
    * Motivating example/exercise: 

* **Side episode: sshfs, short demo** (10 min) 
    * Three points:
        * Mounting filesystem is distinct from copying
        * Sshfs works on "anything" you can ssh to
        * Performance limits
    * Motiving example: creating some figures (mention headless rendering)

* **Interactively working on a cluster** (45 min) 
    * user stories 2 + 5 (or keep them separated, or sequential)
    * Three points
        * A cluster is not just batch: interactive is possible and good for debugging
        * Basic interactive jobs from terminal
        * Interactive with graphical interfaces
    * Motivating exammple/exercise: 

* **Remote interactive example: vscode** (25 min) 
    * Three points
        * GUI tools like Spyder/vscode can run on cluster, from your computer, but isn't always a good idea
        * How to setup?
        * Common problems/issues
    * Motivating example/exercise:
        * Remote Debugging

--- 

