owned this note
owned this note
Published
Linked with GitHub
# Bianca intro at UPPMAX
:::warning
**Old page**
:::
## Workshop name:
Introduction to Bianca: Handling Sensitive Research Data
Material: https://uppmax.github.io/bianca_workshop/
Source: https://github.com/UPPMAX/bianca_workshop/tree/main/docs
:::info
- [new planning page for intro course](https://hackmd.io/665LX1RTQtOZTRqmuDwafA?both)
- [new planning page for intermediate course](https://hackmd.io/@UPPMAX/Bianca-intermediate-planning)
:::
## Spring workshops
- Workshop date(s)
- April
- [x] Basic: Fri 19 April
- [x] on-boarding Tue 16 April at 15.30-16.00
- May
- [x] intermediate Hackathon: Friday 24 May (NEW time)
### Next meeting at an unplanned moment
- [discuss] Richel suggested a schedule for the Bianca workshop at <https://uppmax.github.io/bianca_workshop/intermediate/intro/>. Discuss until agree
- BC: Looks OK
- [discuss] start_interactive_node.md is part of the session menu
- BC: suggests rm from there since it is not an own session
### Meeting Fri 15 March 11:00-12:00
- Discuss
- teachers: filled in!
- learnings from reflections and evaluations.
- Where is the old evaluation? Unknown for now
- all sessions should have and take enough time for exercises. It should be clear what is follow-along or type-along.
- Richel: agreed!
- TODOs
- [x] add teachers for sessions on the course materials
- Teachers + timings
- basics
- introduction to workshop 15 (Richel)
- login 30 (Richel)
- thinlinc
- break
- command line 45 (Richel)
- modules 45 (Richel)
- Lunch
- Transferring files 45 (Pavlin)
- gui
- rsync <---- NEW
- Exercise: create file on rackham and mv to wharf
- basic SLURM approx (Björn 45m)
- Legal aspects of sensitive data 30m (Lars/Marcus)
- summary 5 (Richel)
- eval 10m (Richel)
- Q/A 15m (All)
- intermediate
- introduction to workshop
- NAISS-SENS summary 5-10 (Lars)
- Transferring files to and from Bianca (Pavlin--> Richel) 40+20
- rsync etc...
- Slurm jobs part 2 (Diana) 25+45
- Lunch
- Software and packages installation on Bianca (Björn)
- c. 50 (flex)
- IDE:s on Bianca (new) 45 (flex) Richel + Björn C<---- 45 mins is new
- (mention the desktop env: XFCE,gnome etc)
1. rstudio (expand?)
2. jupyter (expand) BC
3. vs code on bianca (new)
- Votes :+1:
- R discuss all three equally (series)
- . discuss all three equally (parallel)
- BD discuss 2 of them (learners vote), and briefly 1
- . same but ignore the 3rd
- B same but ignore the 3rd (parallel)
- **DRP focus on one** (learners vote, briefly on the other 2) <--- WINNER
- R focus on only 1 (learners vote)
- Summary 5
- Q/A 40m
## December course
- Workshop date(s)
- Dec
- [x] Basic: Mo 11
- [x] intermediate: Thu 14
- On-boarding
- the week before
- Thu 7 at 11
### Short meeting 19 Dec
- What to teach in intro
- terminal?
- yes
- mention the desktop and for easy things, but hooks
- transfer
- rsync
- transit
- git clone / wget
- gui
- no ides
- slurm
- cut talking
- interactive
- list the IDES
### Planning meeting WED 22 Nov at 9
- Status students
- 16-17 learners
- 1 with no bianca project?
- Status
- All working with their parts!
- What do learners want?
- be prepared for some questions
- summary
- sensitive data
- Handling of Sensitive Research Data(storing, consuming, transfering)
- transfer
- How to transferring sensitive data from external server, i.e., GDC NIH sensitive data to Bianca.
- slurm
- Run slurm jobs more efficiently
- more effectively use R studio on bianca, when to use batch scripts and when to execute scripts in Rstudio, etc
- or in RStudio session?
- perform data analysis in Bianca
- improve productivity of data analysis
- core node mem & pipeline for RNAseq
- install
- R packages
- using containers,
- version controlling (git-like function) within BIANCA environment
- external programs
- R/python packages which is not present in Bianca.
- git integration
- IDE
- Efficient software development on Bianca that is not via ``vi/emacs/nano`` but still has copy&paste unlike thinlink, i.e how to use VSCode on bianca from home system (if possible)
- how to use Rstudio, doom emacs, git integration,
- Extra?
- removing second password authentication
- see below for details
- To discuss
- overview page again?
- Things left to fix
- Björn
- [ ] testing pip
- [ ] testing julia
- [ ] reducing text in the extra material
### Bianca Hackathon: What would you like to learn during the workshop?
**Answers:**
- improving installation of R packages, using containers, and if possible, version controlling (git-like function) within BIANCA environment
- Efficient software development on Bianca that is not via vi/emacs/nano but still has copy&paste unlike thinlink, i.e how to use VSCode on bianca from home system (if possible)
- Run slurm jobs more efficiently
- How to improve productivity of data analysis on Bianca
- Handling of Sensitive Research Data(storing, consuming, transfering)
- how to more effectively use R studio on bianca, when to use batch scripts and when to execute scripts in Rstudio, etc
- how to use Rstudio,doom emacs, git integration, removing second password authentication
- perform data analysis in Bianca
- Improve My Handling of Sensitive Research Data
- How to install external programme, transferring sensitive data from external server, i.e., GDC NIH sensitive data to Bianca. How to use/install R/python packages which is not present in Bianca.
- To use BIANCA more efficiently and take advantage of its full range of possibilities.
- Gain more general knowledge about handling of sensitive data in bianca
- I'm very much looking forward to learning about installing software on Bianca, and also IDEs. I have experience working on BIANCA but would like to learn more. I will also be working on BIANCA for my master thesis where I will need to install software/packages.
- How to handle Sensitive data in Bianca and how to work on Bianca.
- core node mem & pipeline for RNAseq
- job submission
- Wants to learn more about bianca
## Bianca intro: What would you like to learn during the workshop?
Answers:
* how to handle with sensetive data
* transfering files to and from Bianca
* Using Bianca to analyse research data without compromising the confidentiality of the data
* I would like to learn how to store data on Bianca apart from the wharf, best pactices to handle data, how to use R
* Things about how to use Bianca
* Handling Sensitive Research Data, if possible focused on imaging
* Introduction to Bianca: Handling Sensitive Research Data
* how to work with sensitive data on hpc ressources
* Login, transferring files, slurm, using (un)avaliable software.
* The necessary basics
* Beginner
* Beginners introduction to Uppmax, Bianca, servers, demultiplexing data and data transfer
* How to transfer files and vasics of SLURM
* If there is anything special in handling sensitive data in Bianca
* hur man hanterar helgenom sekvenseringsdata
* How to use Bianca for sensitive data handling
* how to work with Bianca
* I would like to learn about data upload to Bianca, data management, and running computations. I am familiar with the SLURM system. I have used L-SENS at LUNARC.
* Basics. Login, get around and understand the construction of it.
* The very basics of functions and navigation in Bianca (and exporting/importing files)
* I am especially interested in file transfers
* I would like to learn what possibilities there are to use BIANCA, how other universities may connect with it and it's functionality
* Learn how to work with Bianca cluster and how to handle sensititve data.
* slurm
* Learn to use Bianca
* Submitting jobs, retrieving results, utilizing workflows and pipelines, and other data analysis-related stuff.
* How to use Bianca to run proteomics analyses in MaxQuant
### Planning meeting WED 22 Nov at 9
- Teachers (2-3 per day) + timings
- basics
- introduction to workshop 15 (Richel)
- NAISS-SENS 15m (LARS)
- login 20 (Richel)
- thinlinc
- break
- linux 45 (Richel)
- modules 45 (Richel)
- Lunch
- Transferring files 45-55 (Pavlin)
- gui
- transit
- basic SLURM approx (Björn 45m)
- summary 5
- Q/A 40m
- inform about eval before end (link)
- (eval only after, email)
- intermediate
- introduction to workshop
- NAISS-SENS summary 5-10 (Lars)
- Transferring files to and from Bianca (Pavlin) 40+20
- rsync etc...
- Slurm jobs part 2 (Diana) 25+45
- Lunch
- Software and packages installation on Bianca (Björn)
- c. 50 (flex)
- IDE:s on Bianca (new) 55 (flex) Pavlin+Richel
- (mention the desktop env: gnome etc)
1. rstudio (expand?)
2. jupyter (expand)
3. vs code on bianca (new)
- Summary 5
- Q/A 40m
- Materials developer (same)
- basics
- sens
- login+linux+modules
- Transferring files
- basic SLURM
- intermediate
- NAISS-SENS summary
- IDE:s on Bianca
- Transferring files part2
- Slurm jobs part2
- Software and packages installation
### To do 9 Nov
To do:
- [x] advertisement text
- [x] intro
- [x] hackathon
- [x] registration forms
- [x] intro: https://docs.google.com/forms/d/1d7lsiRuvS-INR49QexKjN58sQxEnwHHQ7RhSe3wPyNk/edit
- [x] hackathon: https://docs.google.com/forms/d/1Fux1vSV9hllF0lAUaK8SzzIcZl5obq8n7XfXNvcXNt0/edit
* R: Suggest to change from:
```
Do you have an account for Bianca? *
A Bianca account is required in order to give you access to the course material and to be able to follow along.
```
To the -I feel- more helpful and correct:
```
Do you have a SUPR user account? *
A SUPR user account is required in order to give you access to the course material and to be able to follow along.
If not, follow the steps at https://www.uppmax.uu.se/support/getting-started/applying-for-a-user-account/
```
- [x] fix webpage, including schedule
- [x] intro: https://www.uppmax.uu.se/support/courses-and-workshops/bianca-intro-2023
- [x] hackathon: https://www.uppmax.uu.se/support/courses-and-workshops/bianca-hackathon-2023
#### Bianca intro (beginner)
Online workshop: Introduction to Bianca: Handling Sensitive Research Data, 11 December 2023
Are you just beginning to work with sensitive data in your research? If yes, welcome to a 1-day introduction to handling sensitive data on the UPPMAX cluster, Bianca. We will tell you about NAISS-SENS, how to login to Bianca, transfer files via wharf, basics of the SLURM workload manager and the module system.
The workshop is intended for beginner users of Bianca.
You do not need to be a member of a NAISS-SENS project in order to join the workshop. A SUPR course project will be available to all participants. The workshop will consist of both lectures and exercise sessions.
When: Monday December 11
Where: online via Zoom
For more information and registration please visit: https://www.uppmax.uu.se/support/courses-and-workshops/bianca-intro-2023.
#### Bianca hackathon (intermediate)
Online hackathon: Bianca In-Depth: Improve Your Handling of Sensitive Research Data, 14 December 2023
Are you already working with sensitive data in your research and feel that your workflows can be improved? If yes, welcome to 1 day hackathon where you'll learn smarter ways of working on the Bianca cluster. We will tell you how to do file transfer from a terminal, advanced SLURM, using IDEs (i.e. RStudio and/or VSCode), and installing custom software and packages.
The workshop will consist of short lectures and the content will be driven by questions from the participants.
To attend this course, we expect you to be able to login to Bianca, submit a Slurm bash script, and know how to transfer files.
You do not need to be a member of a NAISS-SENS project in order to join the workshop. A SUPR course project will be available to all participants.
When: Thursday December 14
Where: online via Zoom
For more information and registration please visit: https://www.uppmax.uu.se/support/courses-and-workshops/bianca-hackathon-2023.
#### older text
Online workshop: Introduction to Bianca: Handling Sensitive Research Data, 11 September 2023
Are you working with your sensitive data in your research? If yes, welcome to a full day introduction to handling sensitive data on the UPPMAX cluster, Bianca. We will tell you about NAISS-SENS, how to login to Bianca, transfer files via wharf, Slurm workload manager and the module system, and how to work with Conda packages.
You do not need to be a member of a NAISS-SENS project in order to join the workshop. A SUPR course project will be available to all participants. The workshop will consist of both lectures and exercise sessions.
When: September 11
Where: online via Zoom
For more information and registration please visit: https://www.uppmax.uu.se/support/courses-and-workshops/bianca-hackathon-2023.
### Planning meeting Nov X, at XX
- date (range Thu 7 - Fr 15 Dec)
- frames
- 5-6 Linköping NUF
- 7 Björn occupied in afteroon?
- Dec 7 at 14:00: preliminary date for NAISS Zoom-in
- other events?
- marcus (Mo, Fr)
- 18-22 Last week before Christmas
- Richel: no Dec 8th afternoon, no Dec 11 morning, no Dec 13th morning
- content (classification)
- basic (lectures+hands-ons)
- name: "Basic Introduction to Bianca"
- login
- ThinLinc (only?)
- no ssh
- linux
- also uquota
- projinfo
- editors
- modules
-
- Transferring files
- part 1, e.g. GUI tools (exercise)
- GUI: including thinlinc
- basic SLURM:
- interactive session
- one IDE, e.g. RStudio or VSCode or Jupyter notebook
- SLURM assist in VSCode?
- basic workflow:
- sbatch, squeue, scancel
- hands-on-p, -n
- flags to introduce: -t, -A, -p, -n/N (but briefly)
- sens projects
- 15 min (Lars states this enough!) intro, at start/before login
- [name=Lars] teaches
- whats and whys!
- questions to "support"?
- seminar for PI:s only?
- Evaluations in BO:s
- Q/A: first let learners anonymously evaluate in breakout rooms, they come back if tehre are questions
- intermediate/hackathon
- Short intro (5 mins, Lars states this is enough!, reminder) to sensitive projects
- Requirements: can login, start a bash script, upload/download files
- name:
- "Bianca Hackathon (Q&A-based, intermediate level)"
- "Intermediate Bianca course"
- Overall: intro plus much time for Q/A
- "Blocked hours"
- Transfer files, part 2
- [R: Y] rsync
- [R: N] scp
- [R: N] transit
- more SLURM? [R: e.g dependent jobs?]
- [R: Y] GPU?
- [R: Y] jobstats
- installations
- [R: Y] packages
- [R: ?] tools
- [R: ?] development
- [R: Y] Run container?
- IDE:s
- [R: N] RStudio
- [R: N] jupyter
- [R: Y] VSCode
- [Tip from P about transfer][R: Y] Q&A
- 10 min intro then Q/A
Registration:
- questions from participants
Expected to know for the hackathon:
- you can login to Bianca
- basic Slurm (sbatch & interactive)
- you
-
To do: sens course project
#### Old schedule
09:00 - 09:15 Introduction
09:15 - 9.45 Login: ssh vs. Thinlinc
09:45 - 10:00 Coffee break
10:00 - 10:30 Command line intro specific to Bianca
10:30 - 11:00 Module system
11:05 - 12:00 NAISS-SENS
12:00 - 13:15 Lunch break
13:00 - 14:00 Transfering files to and from Bianca
14:05 - 14:25 Compute nodes and slurm
14:25 - 14:45 Software installation without internet connection
14:45 - 15:00 Summary
15:00 - 15:15 Coffee break
15:15 - 16:00 Q/A and extra material
#### Schedule intro
09:00 - 09:15 Introduction
09:15 - 9.45 Login: ssh vs. Thinlinc
09:45 - 10:00 Coffee break
10:00 - 10:30 Command line intro specific to Bianca
10:30 - 11:00 Module system
11:05 - 12:00 NAISS-SENS
12:00 - 13:15 Lunch break
13:00 - 14:00 Transfering files to and from Bianca
14:05 - 14:25 Compute nodes and slurm
14:25 - 14:45 Software installation without internet connection
14:45 - 15:00 Summary
15:00 - 15:15 Coffee break
15:15 - 16:00 Q/A and extra material
#### Schedule hackathon
09:00 - 09:15 Introduction
09:15 - 9.45 Login: ssh vs. Thinlinc
09:45 - 10:00 Coffee break
10:00 - 10:30 Command line intro specific to Bianca
10:30 - 11:00 Module system
11:05 - 12:00 NAISS-SENS
12:00 - 13:15 Lunch break
13:00 - 14:00 Transfering files to and from Bianca
14:05 - 14:25 Compute nodes and slurm
14:25 - 14:45 Software installation without internet connection
14:45 - 15:00 Summary
15:00 - 15:15 Coffee break
15:15 - 16:00 Q/A and extra material
teachers
- Björn
-
- Evaluation
- [evaluation results on github](https://github.com/UPPMAX/bianca_workshop/tree/main/docs/evaluations/20230911)
- [The evaluation with highlighted points](
https://docs.google.com/document/d/1NWODAsGJUU3kKNLkz-rillzNrO8Rx-weaMTWrWjPqC8/edit)
- Learnings
- less time = basic?
- login
- linux
- sens
- more time = advanced?
- Q/A
- slurm
- install
- Two workshops
- basic
- advanced
- intro example:
- summarize by asking around what experience do people have with Uppmax, and reducing the content
- [DISCUSS] Remove starting an interactive session in the first hour on login from the early course material; it will be discussed later
- [DISCUSS] Using mixed groups (i.e. putting experts in rooms with beginners) may be a useful tool
- [DISCUSS] Is parallel sesion worth it?
### Email to participants
- Two invitations
- Forms
- include questions
## September course
- [x] Login Day:
- Monday September 4th at 13-14
- [name=Björn C, Pavlin, Diana]
- Workshop day
- Monday 11 Sep 9.00-16.00
- [name=Björn C]
-
- Fully remote
### Meeting Sep
#### To discuss
- evaluation
- teachers
- material
### Most important learnings
**Course evaluation:
https://uppmax.github.io/bianca_workshop/**
- more effort to transfer, jobs and install
- first basics not needed for some
- have parallel session for active users also for command-line intro?
- Hands-on
- also for installation and submit jobs
- example today script of Doug's steps
- jobstats example
- Too many teachers with different
- In pre-info:
- course given twice per semester
- VPN outside SUNET
#### Richel's notes
* (2 min) Vote: put evaluations online yes/no?
* yes
* Should be able to "clean" results
* (3 min per person) What is the goal of the course?
* fix first page
* (2 min) Vote: Except for getting logged in, do we think interaction is important?
* Vote: suggest to strongly encourage cameras
and -as teachers- actually interact with the students [1] yes/no
* cameras on, except for exceptions
* (2 min) Vote: For file transfer part: most Pavlin talked about is rsyc. However, he did not show it, as he sticked to the course
material instead. I suggest a rewrite/addition of that course material, to include rsync. Accept/reject?
- fixed
* (2 min) There was an exercise about the legal things regarding sensitive data.
There was no incentive to do so (e.g. 'I will check a random student afterwards and discuss what is written').
This could be because of the lack of interaction.
Else, I suggest to change the course material to include a check.
Accept/reject?
- good idea
- quiz instead
- "threaten" to call on people during facit
Notes
* [1] It is known that for online teaching, engagement is the most important facet in keeping students learning
### Teachers
**Too many teachers!!!**
- Björn
- Diana (can leave my part to someone else)
- Richel
- Douglas (cannot)
- Pavlin
- Lars
- Marcus
### Schedule
- 09:00 - 09:15 Syllabus
- [name=Richèl]
- Goals:
- Students understand how the course is taught
- 09:15 - 09:45 Login: ssh vs. Thinlinc
- [name=Richèl]
- Goals:
- having logged in with SSH and ThinLinc
- experience the difference between SSH and ThinLinc
- Non-goal:
- helping students activate accounts, etc.: they are too late
- 09:15 - 09:45 Parallel session in BO for active users
- [name=Marcus]
- Keep!
- 15 min Coffee
- 10:00 - 10:15 Command line intro specific to Bianca
- [name=Richèl]
- Goals:
- being able to view/create/move/delete files
- create an executable bash script
- 10:15 - 10:45 Module system
- [name=Richèl]
- Goals:
- being able to search/load/unload modules
- Create an executable bash script that uses a module, without SLURM
- 15 min break
- 11:00 - 12:00 Legal and administrative aspects [name=Lars] & [name=Marcus]
- add quiz!!
- (space for QA)
- 12:00 - 13:00 Lunch break
- 13:00 - 13:45 Transferring files to and from Bianca [name=Pavlin]
- is restructured and more hands-on!!
- 15 min break
- 14:00 - 14:45 Compute nodes and SLURM + extra Q/A [name=Björn]
- add effectiveness and more time?
- 14:45 - 15:00 COFFEE BREAK
- 15:00 - 15:35 Software installation without internet + extra Q/A [name=Björn]
- add exercise and more time?
- 15:35 - 15:40 Summary [name=Richel]
- 15:40 - 16:00 Q/A with demos common
### Details
- HackMD is not used nowadays. Skip, and make people more interactive?
- Vote result: no use of HackMD anymore, use chat instead :skull_and_crossbones:
- Link to Zoom link in course project folder. We did not (yet) this year
- :construction: Richel makes a YouTube video how to do this.
- Need a sysadmin from 8:30-9:30 for the people that registered late and/or run into trouble. Diana hunts for one :-)
- All check: do we have the correct access right
## June course
- Wed 14 June 9-16
- Fully remote
### Teachers
- Diana
- Richèl
- Douglas
- Pavlin
- Lars
- Björn
### Cancelled wrap-up meeting June 22nd
Course evaluation:
https://uppmax.github.io/bianca_workshop/
These are Richel's notes, which he hopes will be discussed at a next meeting.
* (5 min) What are the results of the evaluation?
* (2 min) Vote: put evaluations online yes/no?
* (3 min per person) What is the goal of the course?
* (2 min) Vote: Except for getting logged in, do we think interaction is important?
* If no: Vote: replace all by videos yes/no?
* If yes: Vote: suggest to strongly encourage cameras
and -as teachers- actually interact with the students [1] yes/no
* (2 min) Vote: For file transfer part: most Pavlin talked about is rsyc. However, he did not show it, as he sticked to the course
material instead. I suggest a rewrite/addition of that course material, to include rsync. Accept/reject?
* (2 min) There was an exercise about the legal things regarding sensitive data.
There was no incentive to do so (e.g. 'I will check a random student afterwards and discuss what is written').
This could be because of the lack of interaction.
Else, I suggest to change the course material to include a check.
Accept/reject?
Notes
* [1] It is known that for online teaching, engagement is the most important facet in keeping students learning
### Next meeting June 13 at 9:00
- Status
- under control!?
- HackMD page
- last updates?
- Available helpers
- [name=Björn C]: All day, except 11-12 (incl teach at 14.25)
- [name=Björn V]: All day
- [name=Richel]: All day
- [name=Pavlin]: 08:45 - 13:00
#### Extra material, teachers:
- More about Slurm
- Diana
- Conda on Bianca
- Björn C (& ...)
- Install with pip to Bianca
- Björn C (& ...)
- Installing R packages on Bianca
- Douglas or Björn
- Using Julia packages on Bianca
- Björn C
- Containers
- Björn V?
- Own development and Git on Bianca
- Include or not?
- Running Jupyter on Bianca
- Björn (& ...)
### Meeting June 8 t 9:00
- Status
- Same teachers on the same material except
- Marcus --> Richèl
- Lars takes whole SENS session (1 hr)
- How many applicants?
- 30+
- Shared document?
- HackMD link: https://hackmd.io/@dianai/bianca-workshop/edit
- Material?
- Rendered: https://uppmax.github.io/bianca_workshop/
- Edit: https://github.com/UPPMAX/bianca_workshop/tree/main/docs
- can you all edit?
- Check the files in the project folder
- up to dates
- and "reset/clean up" ``wharf``...?
- Changes
- More than 20 min for
- compute nodes and slurm?
- installations?
- Type-along with R packages????
- Less than 55 min for Q/A?
- BO--> common QA with demos
- R: Douglas
- Python/Conda:
- singularity?
### Tentative schedule
- 09:00 - 09:15 Syllabus
- 09:15 - 09:45 Login: ssh vs. Thinlinc [name=Diana]
- inform about the benefits of different logins and inform about parallel session
- 09:15 - 09:45 Parallel session in BO for active users [name=Marcus]
- 15 min Coffee
- 10:00 - 10:30 Command line intro specific to Bianca [name=Richèl]
- 10:30 - 11:00 Module system [name=Douglas]
- 5 min break
- 11:05 - 12:00 Transfering files to and from Bianca [name=Pavlin] [color=red]
- (space for QA)
- 12:00 - 13:00 Lunch break
- 13:00 - 14:00 Legal and administrative aspects [name=Lars]
- 5 min break
- 14:05 - 14:25 Compute nodes and Slurm [name=Diana]
- 14:25 - 14:45 Software installation without internet connection [name=Björn]
~~- 14:40 - 15:05 Hands-on of installations~~
- 14:45 - 15:00 Summary [name=?]
- 15 min Coffee break
- 15:15 - 16:00 Q/A with demos common
### Todo
- ask Marcus about material in other form
- [x] command line material is [here](https://uppmax.github.io/bianca_workshop/commandline/)
- [x] Björn speaks with Doug
- [x] talk to sysexp:s Björn
- [ ] Marcus will check Lars can teach 13:00 - 14:00 NAISS-SENS
- [x] Diana makes a HackMD doc for Q&A
_____________________________________________________
## Date
- Wednesday 15 March
- Length: 1 day, 9:15-12:00, 13:15-16:00
- Fully remote
## Course website
https://www.uppmax.uu.se/support/courses-and-workshops/bianca-workshop-2023
## Email to participants
Hi!
Welcome to the workshop Introduction to Bianca: Handling Sensitive Research Data, to be held March 15.
The lectures are scheduled 09:00-12.00 and 13:00-16:00.
You may find the tentative schedule on the workshop webpage:
https://www.uppmax.uu.se/support/courses-and-workshops/bianca-workshop-2023/.
The course will be held via Zoom: https://uu-se.zoom.us/j/62914645776
Password to join the meeting: Bianca
Important: You need to have an account on Bianca and 2FA enabled before the workshop starts. Please follow the instructions in the Google document a couple of days before the workshop starts:
https://docs.google.com/document/d/1yEwICcGTrqm663QinYkbUsTHn7nMKxna-gOcXQtCvD4.
If you are new to command-line, it is strongly encouraged to go through part of the material taught during the UPPMAX introductory course: https://uppmax.github.io/uppmax_intro/linux_basics.html.
If you encounter any issues in setting up your account, please contact UPPMAX support via https://supr.naiss.se/support/ or email support@uppmax.uu.se. For general questions about the workshop, please email diana.iusan@uppmax.uu.se.
Best regards,
Diana
## Tentative schedule
09:00 - 10:00 Login: ssh vs. Thinlinc
10:00 - 10:30 Command line intro specific to Bianca
10:30 - 11:00 Module system
11:00 - 12:00 Transfering files to and from Bianca
12:00 - 13:15 Lunch break
13:00 - 14:00 NAISS-SENS
14:00 - 15:00 Module system (continued) and Slurm
15:00 - 16:00 Software installation without internet connection
## Invitation text
Are you working with your sensitive data in your research? If yes, welcome to a full day introduction to handling sensitive data on the UPPMAX cluster, Bianca. We will tell you about NAISS-SENS, how to login to Bianca, transfer files via wharf, Slurm and the module system, and how to work with Conda packages.
You do not need to be a member of a NAISS-SENS project in order to join the workshop. A SUPR course project will be available to all participants. The workshop will consist of both lectures and exercise sessions.
When: March 15, 2023.
Where: online via Zoom.
For more information and registration please visit: https://www.uppmax.uu.se/support/courses-and-workshops/bianca-workshop-2023.
## Registration form
https://forms.gle/QzQrdcg3yn3TXPJ38
## Staff
- Planning content
- [name=Björn]
- [name=Diana]
-
- Teachers
- [name=]
## Meetings
### Meeting 13 March
- Status
- HackMD
#### Status
- 9-9.10 Syllabus and intro
- fixes
- 9.10-10 login ssh/ThinLinc, 2FA [name=Diana]
- fixes or more work?
- interactive 8 h -p core -n 2
- 10-10:30 Command-line intro specific Bianca (30 min) [name=Marcus]
- more work
- 10:30 - 11:00 Module system 30m [name=Douglas]
- more work
- 11-12 Transfering files to/from Bianca [name=Björn C], [name=Pavlin]
- fixes or more work?
- 40
- 13-14 NAISS-SENS [name=Lars & Marcus]
- ?
- 14 Intro to SLURM
- fixes
- 14.15 Intro to installing
- fixes
- 14.30 Q/A and extra material
- slurm
- more work
- conda
- fixes (formatting)
- pip
- more work
- R
- more work
- julia
- fixes
- containers
- fixes (images)
### Meeting 10 March
- the timings
- time for exercises
- afterwards, we can if we have time prepare,so finished by end of Wednesday
- focus on demos
### Next meeting w8, Tuesday, after the staff meeting
TO-DO:
- [x] decide who teaches what
- [x] which format should we use for the material
- Studium/Canvas
- PPT/PDF
- GH-pages
- MKdocs, like Python course or User guide by Matias
- material from introday of UPPMAX intro course can be re-used and easily ported to MkDocs
- Björn C votes for this
- Sphinx
- Read the docs style
- material from introday of UPPMAX intro course can be re-used
source: https://github.com/UPPMAX/bianca_workshop
page: https://uppmax.github.io/bianca_workshop/
## Content (to be discussed)
Names correspond to person responsible for the material. Teachers to be decided next time.
- 9-9.10 Syllabus and intro
- 9.10-10 login ssh/ThinLinc, 2FA [name=Diana]
- differences in terms of operating system/...
- exercise session included
- working from abroad / outside SUNET
- GUI and its limitations on Bianca
- pros and cons ssh vs ThinLinc
- connect ssh to the part "Transfering files"
- 10-10:30 Command-line intro specific Bianca (30 min) [name=Marcus]
- prompt, mv, cp, pwd, ls, mkdir
- navigate through the file system
- https://uppmax.github.io/uppmax_intro/linux.html
https://uppmax.github.io/uppmax_intro/linux_basics.html
- 10:30 - 11:00 Module system [name=Douglas]
- interactive
- how to interact with R packages / R studio
- do demos in Thinlinc
- gatk, vep, snipf, conda (loading only)
- jupyter
- 11-12 Transfering files to/from Bianca [name=Björn C], [name=Pavlin]
- wharf, transit server, ...
- pros/cons of different solutions
- stress throughoout the day that is it important to keep the entire chain of transfering the data secure
- rsync, scp/sftp
- grus/delivery (refer to documentation) 2min
- 12 - 13 Lunch break
- 13-14 NAISS-SENS [name=Lars & Marcus] material+teacher
- extended from intro course
- what is sensitive data?
- puba & timelines
- how to apply for projects
- project management
- LINK to extra material
- contact persons
- 14 Just intro to below sessions [name=Björn] + intro teacher
- 14-15 Module system (continued) and Slurm [name=Diana] (material)
- interactive vs. non-interactive sessions
- why is interactive needed at all?
- e.g. nextflow - main node should be on interactive
- short (assume people have images)
- 15-16 Software installation without internet connection (flera subdokument)
- intro to subtopics (at 14.00) [name=Björn]
- subtopics given (if asked for) by the responsible (individual documents)
- Conda packages: installing/using [name=Jonas]
- alternative: installing with pip
- R package installation not included in R packages [name=Marcus]
- Julia [name=Björn C]
- running jupyter notebooks [name=Lars]
- ruta i Douglas moduler
- containers [name=Björn V]
- Q&A??
Participants are asked what they'd like to learn in the registration form, so we may tweak the content accordingly.
## Earlier material
https://uppmax.github.io/uppmax_intro/linux.html
https://uppmax.github.io/uppmax_intro/linux_basics.html
## Prereqs
Details to be sent out before:
- SUPR/course project
- set up 2FA
- need to have access to univ network or set up VPN
- /access via Rackham
- mention that getting an acoount and setting up 2FA may take several days
- detailed schedule - emphasize that it is OK to take parts of the course only
- 1st day of the intro course, 1/2-1 day at your own pace, beneficial to do before the workshop
## Material type
- make use of present material as long as possible
- Choose one or several
- GH pages
- sphinx
- **MkDocs** !!!
- PPT/PDF
- Studium
- Exercises...
## Overview
```mermaid
graph TB
Node1 -- interactive --> SubGraph2Flow
Node1 -- sbatch --> SubGraph2Flow
subgraph "Snowy"
SubGraph2Flow(calculation nodes)
end
thinlinc -- usr-sensXXX + 2FA + VPN? ----> SubGraph1Flow
terminal -- usr --> Node1
terminal -- usr-sensXXX + 2FA + VPN? ----> SubGraph1Flow
Node1 -- usr-sensXXX + 2FA + no VPN ----> SubGraph1Flow
subgraph "Bianca"
SubGraph1Flow(Bianca login) -- usr+passwd --> private(private cluster)
private -- interactive --> calcB(calculation nodes)
private -- sbatch --> calcB
end
subgraph "Rackham"
Node1[Login] -- interactive --> Node2[calculation nodes]
Node1 -- sbatch --> Node2
end
```
```mermaid
graph TB
terminal -- usr --> Node1
terminal -- usr-sensXXX + 2FA + VPN? ----> SubGraph1Flow
Node1 -- usr-sensXXX + 2FA + no VPN ----> SubGraph1Flow
subgraph "Bianca"
SubGraph1Flow(Bianca login) -- usr+passwd --> private(private cluster)
end
subgraph "Rackham"
Node1[Login] -- interactive --> Node2[calculation nodes]
Node1 -- sbatch --> Node2
end
subgraph "Local"
Node1[Login] -- interactive --> Node2[calculation nodes]
Node1 -- sbatch --> Node2
end
```