---

tags: Meetup

---

# Nordic RSE Online Meetup

This document: https://hackmd.io/@nordic-rse/online-meetup/edit
Schedule: https://nordic-rse.org/events/2020-online-get-together/
Code of conduct: https://nordic-rse.org/events/2020-online-get-together/code-of-conduct/
Nordic RSE Membership Form: https://forms.gle/zUu9CgEF9nW8mPjM8

We can use this pad for collaborative notes, practical announcements, and also for questions/comments.

## Welcome and Introduction (Wednesday)

* Agenda overview
* more suggestions welcome for last session (unconference session)
* Please take notes from all sessions (https://hackmd.io/@nordic-rse/meetup_notes), also late additions from day2 are very welcome
* Code of conduct https://nordic-rse.org/events/2020-online-get-together/code-of-conduct/
* Feedback on the meetup is welcome in any format (one way: section of this HackMD)

## Panel session

* this session will be recorded by Peter for the podcast ‘Code for Thought’
* we recommend to turn video off during the panel and to use "hide non-video participants" (where is that toggle?) so that you can see all panelists on one screen
* Short HackMD intro (Richard)
1. Panelist introduction round

If you would like to ask a question to the panelists, write it below (current bottom of this document) and we will try to merge it into the discussion:

- How can we find a better/common strategy to make the leaders (professors, etc.) understand the need to allocated resources for RSE work?

---

## Discussions

Numbers correspond to breakout rooms (all times in CETEO)

Website: https://nordic-rse.org/events/2020-online-get-together/

* **Break until xx:30**. You can continue discussing here or in breakout rooms, if you would like.

* **Wednesday 10:30 CET** Talks in main room
* Ericsson Research and Distributed Applications in the Cloud (15 min) (Andrew Williams)
* R <-> Python transpiler (15 min) (Dmitrii Borisevich)
* **Wednesday 11:00 CET** Discussions based on talks. You can join breakout rooms yourself. If you can't, ask in chat to be put into a room.
1. Discussion about Ericsson Research talk
2. Discussion about R <-> Python transpiler
3. (Insert Room topic here) <- Everyone can do so!
4. ...
5. ...

* **Wednesday 12:30 - 13:30 CET** Talks in breakout rooms.
1. European Environment for Scientific Software Installations (EESSI)
2. Is it possible to make code quality more important than the number of published papers in academia?
3. Lessons learned from procuring a fairly large HPC system
4. [ResearchSoftwareHour](https://researchsoftwarehour.github.io/) live from Nordic RSE: Rust for science
- can only be joined via Twitch: https://www.twitch.tv/rshour
5. ...

:::success
* **Wednesday 14:00 - 15:00 CET** Breakout rooms:
1. How to explain containers are friends of scientists? Are they?
2. < Add your topic here >
3. < Add your topic here >
4. Problematic demographics within research computing and ways to rectify the problem
5. Collecting ideas/suggestions for Nordic RSE conference in May ([name=Radovan] I haven't prepared this and maybe it was discussed already somewhere)
6. Anyone interested in starting a Nordic openMP user group?
7. < Add your topic here >
:::

---

## General questions, day 3:

### Panel discussion questions
* Ask a question like this
* You can answer/comment on questions like this
* Please try to use HackMD, Zoom chat questions are watched but will get lost soon.
* [name=rkdarst] you can include names like this, this will be posted but all names will be removed.
* Upvote questions by adding +1. Increment the number to add to votes
* Please leave your video off (and if you "Hide non-video participants" then you see just the panelists)

Questions:
* ~~Please introduce yourself, and tell about your relationship to research, software, or engineering.~~
* How do you find out about RSE? +2
* ~~What points of resistance in your RSE career have you noticed? +1~~
* What do you know now that you wish someone had told you when you began as a researcher. +6
* ~~How can universities promote a greater diversity of career paths? +4~~
* ~~What trends in academia will lead to more opportunities for RSEs? +1~~
* ~~Advice for aspiring RSEs? +6~~
* ~~What can other RSEs learn from Bioinformaticians (who received much more recognistion over the last years)? +2~~
* ~~Do you see a role for industry in the RSE community? If so, what would you like to see the relationship being?~~
* ~~How could a career as RSE look like? What "levels" are there? +3~~
* ~~What defines an RSE in your opinion? Who is an RSE and who isn't? +1~~

* ~~What are the trade offs between having embedded RSEs in research groups and having a centralized RSE group. Is it possible to provide the needed long term software development when working as software consultants out of a central RSE group? +2~~

* How much value would an RSE add to their unit by being able to secure funding (this is time that could have been spent developing software instead)?

* Comment: In Sweden within the UPPMAX hpc facility (I dont know how it is handled at other centers within the SNIC org) and NBIS organisations Application experts and RSE (devlopers) are now mainly hired fulltime and not temporary while there are still people hired with shorter contracts as exception. It has been recognized that its very hard to retain talent if you only hire shortterm/temporary contracts.

* Comment: really liked the point about having and getting time for "pet project". not only to have something to show later on CV but also to stay current and motivated and to learn "for free".+5
* I started one early this year and told my boss about it a few months later. He enjoyed the initiative, then told me a heartwarming story about investing in projects that _just might_ become a big thing one day. Plus, I also think that this sort of initiative can bring in ideas for an RSE to apply for funding, which is always welcome.
* yes! a number of established projects started as side projects and these can often open up funding opportunities and cross-discipline collaborations.
* There is a good model at my work: 70+20+10. spend 70% on regular work, 20% learning something new that will make your work better, 10% learning something new that is good for you but does not have to be directly work/project related.+1

* Comment: perhaps another way to make scientific software as "important" as research papers is to always try to publish it in a popular repository (CRAN, PyPI, etc.). Not sure how applicable that would be for software that are not R or Python packages, though.
* yes, also connecting to yesterday's talk. this can really help using standard practices. publishing packages always felt/sounded difficult/scary until I tried it.

* Comment : Excellent point Patric about the standards to which we need to hold resarch software to be able to keep it at the same level as experimental setups and methodologies +3
* May we get more details about such standards so we can register them on this document?
* **Recognizing methodology development:** I'm thinking for example of my colleagues building or working on experimental setups and detectors (e.g. at CERN, GSI and other institutes). They spend the majority of their PhD/Postdoc building setups and developing methodology, similar to how others might spend a lot of their time developing and building software. They have several journals where they can publish their work and get recognized for their contribution, which they base their PhD thesis/project upon. There are of course many journals where software and research software can be published, but at least in my experience, I could not build my PhD thesis or project on this work, since there is a complete focus on the research. The attitude is generally that code is just methodology, "it should just work" and it is not important how it works. I think that this viewpoint is wrong. If you do significant contributions, maybe even breakthroughs, this should be recognized and valued.
* **Testing:** Nobody would trust experimental equipment that has not been calibrated or tested. Building an experimental setup, it is therefore assumed that you will have to spend time on calibrating and testing. Sadly, the same cannot in general be said about research codes (again, this is in my experience, but I've heard others commenting the same in this conference). As an example, I took over a large code that was lacking testing and proper documentation. As I started adding unit tests, regression/physics tests, the senior PI was starting to get impatient because they wanted to see research. Even though several important bugs had been discovered that put the validity of results in jeopardy, I was told "enough tests already". An issue is that PIs might be more focused on research and know very little about writing sustainable codes; they just want them to work. I have a hard time believing that the same would happen in the building of an experimental setup (sure, there are horror stories from experiments, but this is far from the norm).
* **Reproducibility:** Reproducibility is another issue. It is often completely acceptable to neglect mentioning implementation/code etc in methods sections, as long as you describe e.g. the physics theory you are using. Even if the code is mentioned, it is rarely made open access, and when it is made open access, it is often hard to use it to reproduce the results. This goes against the core tenet that research should be reproducible. There needs to be more acceptance for the time it takes to make a code re-usable. Of course, everyone cannot re-build a detector for the Large Hadron Collider at home, but at least a great deal of time and effort in spent in describing how said detector works and can be built. +1
* **TL;DR:** So these are some of the main points (recognition of code/methodology development, testing, reproducibility) when I make the comparison between experiments and "numerical experiments", and when I say that they should be held to the same standards. If we make this analogy, it might be easier to promote RSE-related work as not just important but crucial, and to have a greater acceptance of time spent doing these tasks for e.g. PhDs and postdocs.+1
- Excellent point to compare software development with experimental method development. How can something lead to good data if bugs are in the code and nobody verifies them? Reproducibility is also the key to prevent wrong conclusions. I like to mention PyFAI. It is an anlysis tool, it is published on github, and is developed by a core team, but users, scientist can contribute to its development. It was published also in traditional publications.

* Comment: great point about mentoring groups and connecting to others. +1

### Ericsson Research and Distributed Applications in the Cloud

Breakout room 1

- [name=Radovan] the speaker sent me the slides and I will put them on the event website ASAP.
- working on it ...

Questions/comments:

- [name=West] There are already long-standing software stacks to do distributing computing. How are you interacting with these existing projects? I do hate to continual pound this drum but [HTCondor](https://htcondor.readthedocs.io/en/latest/overview/exceptional-features.html) written by the folks in [Wisconsin](https://chtc.cs.wisc.edu/) have been doing this sort of platform agnostic distributed computing for a long time now.
- Any plans for running on HPC and how that would work?
- A: Currently looking for stakeholders write now for research projects. Under discussion and debating whether the software should be open sourced.
- In the era of GDPR and sensitive data how are you seeing the ocean of computing working with localisation requirements that these regulations impose?
- A: Not working with the infrastructure but specifying locations where stuff should run.
- Comment [name=Zdenek]: We have quite currious application at particle accelerators: the guys making "particle re-energiteze" devices for the particle accelerator started using networks/clouds of *smart-devices*. So instead of well-defined dedicated networks the control system is running over wifi.
- Where does Ericsson see potential benefits in building links with the RSE community? Would this, for example, ultimately be as a route to build users of specific codes, would it be as a route to get/share technical expertise and input from the RSE community, etc? Are there other drivers?
- This has to be a two-way partnership, giving back. Trying to get people to better communicate both internally as well as across stake-holder groups. The Met-Office showed how training is important for getting researchers up to speed on making production code and the unique requirements this entails. An on-boarding program for getting new hires acclimatized to the standards of the projects is very important. All code should be group-code. Beware of personalized ownership.
- Code ownership is a particularly tricky point, both in Sweden and elsewhere.
- This is a very interesting topic that is getting more attention recently. Who owns the intellectual property for e.g. the code developed to do research at a university? Universities have a policy that can seem quite aggressive in the way they claim ownership. It was discussed that teachers can fall under a special clause, but that researchers and students do not. I'm not sure if the latter is true, students (from bachelor up to and including postdocs) also have a special clause, and the tools they develop to "conduct their studies" (including thesis work and postdoc work) are their own. At least this seems to be the case at my university. For researchers and other staff, the situation looks different. Please comment if you have any experience or encountered a different policy.
- Patents & Papers as measures for success, but are these actually good measures for progress and in particular the value of collaborative work?

### R <-> Python transpiler

##### Join to collab at https://github.com/borisevichdi/rtopython-mapping (also text the speaker at GitHub or better at Twitter - https://twitter.com/siberianpython)

- The demo:
- https://rtopython-pa7qfqmwhq-ew.a.run.app/

Breakout room 2

Questions/comments:

- comment: great point about tools possibly dividing communities
- Links for related software
- The OneLang demo: https://ide.onelang.io/ (https://github.com/onelang/OneLang)
- Transfer data frames between python and R: https://blog.rstudio.com/2016/03/29/feather/
- An Rstudio supported projects for using python code in R with R syntax: https://cran.r-project.org/web/packages/reticulate/vignettes/calling_python.html
- A non-profit that has been supporting software development in both the PyData stack and R for the last decade: https://numfocus.org/
- CBioVikings - Danish Comp Bio community - https://twitter.com/CBioVikings, https://www.instagram.com/cbiovikings/

- Q: optimal R programming would likely involve making use of its vectorization capabilities, for example by using apply functions instead of loops. Is this something that rtopython-mapping plans to implement?
- numpy provides these opportunities, so it is possible, hopefully feasible :-)
- Q: Why the comments cannot be translated? Just currious.
- this can be done, the PoC uses ASTs (abstract syntax trees) for the conversion, and comments are not a part of AST; but this could have been added on top
- Q: how it works under the hood?
- AST converter from R to python for the grammar + huge map, implenting the functions from R as python AST structure

### Lessons learned from procuring a fairly large HPC system

Slides: https://hackmd.io/@KTW/HPC-procurement-lessons#/

1. Are you monitoring your cluster usage? What tools?
2. Do you regularly run regression tests? Which tools?
3. How did you select application benchmarks?
4. How did you design the scoring system?
5. Was there anything that surprised you during the procurement process?
6. What was particularly challenging?
7. Do you think the procurement could have been more successful if you had done anything differently? If so, what?
8. Should we score the quality of benchmark reports?
9. Is it worth all the work to use real application benchmarks? Or use only synthetic or kernels--benchmarks?
10. How should we estimate the "real" power usage of the system?
11. Do we need to run a job mix to evaluate how different users affect each other?
12. Is benchmarks a good way to evaluate the "competence" of the vendor?

* Q: How did your requirments gathering procedure work, how did you implement riskassesment to requiremnt process?

* Q: Is the benchmark list exhaustive for the application benchmarks? Can you elaborate on why you chose such narrow scope of the benchmarks?
- 7 benchmarks are probably at the upper end and is approching painful for vendors.
- including more benchmarks leads to overall worse results
- one often ends up benchmarking the vendor benchmark teams
- is this a good thing because we will need software support later?
- benchmark teams (might) not the same group that does support

- how much modification of benchmark codes should be allowed?
- major refactoring not representative of typical future use, so might want to keep it minimal

- performance for individual proxy benchmarks and application benchmarks can point in different directions, but often becomes robust after averaging
- to eliminate small/less competent vendors, can require certain minimum placement on top500 or alternatively have requirements on vendor stability or financial aspects

* Q: did you have a dynamic rebalancing of scores for phase 1 and phase 2 depending on the bids?
- we had minimum performance on each phase
- turned out that it paid off for vendors to focus efforts on phase 2
- we decided to have phase 2 with accelerators because that's where HPC is heading, even if all users were not in favor

### European Environment for Scientific Software Installations (EESSI)

- Question: Is it possible to test the whole stack, please add links?
- Yes, see https://eessi.github.io/docs/pilot/
- To get help, join the EESSI Slack, see https://www.eessi-hpc.org/join/
* Question: Will you also support AMD Rocm and AMD ecosystem overall?
- Yes, eventually. Right now there already are optimized installations for AMD Zen2 (Rome).
OpenMPI is included and is installed on top of UCX & libfabric, so *should* properly support AMD Rocm interconnect, but this is currently untested.
* Comment: I like this idea as for us it is important people can use it in their laptops.
Personally I not much loosing time in setting up sw at my laptop but I see for the users it is important
to have an option to install/use it also in their lab. They like it more.
- Yes, this could allow people to *literally* write a job script that just works on the HPC cluster. Same modules, same software.
(and no need to build containers, or copy them over, etc.)
* Question: This builds on existing projects so it has some content from the begining.
- Thanks to EasyBuild we can easily provide 1000s of installations.
Right now we limit what we provide, so we can focus on solving the problems we're hitting first.
* Question: Why European in the name?
- Because it started with European sites.
We're already thinking about changing the first E to "Easy" :)
"EESSI is the Easy Environment..."
* Question: Question: what are the possibilities to add “own dirty module”, is it like same as e.g. with EasyBuild itself?
- You can easily install additional software on top, for example in your home directory on in /tmp, just like you can with any other software stack built with EasyBuild).
* Question: Sensitivity of central Stratum-0 component, in terms of resilience?
- The CernVM-FS design is very robust. If the Stratum-0 dies, the only impact is that you can't add new software to the repositories.
As long as *one* Stratum-1 server is still alive, the software remains available (all Stratum-1 servers have a full copy of the provided software).
So it comes down to having enough Stratum-1 servers, spread across the world, in different sites and cloud providers.
- W.r.t adding software: we plan to fully automate the workflow of adding software to the EESSI repository, such that adding software comes down to opening a pull request on GitHub. When the PR is approved by a reviewer, the software gets built automatically on all supported CPU architectures, and added to Stratum-0, fully automatically. Ideally we also have (small) test cases to verify that the installations are functional before deploying them.
* Question: You mentioned that CernVM-FS only relies on HTTP connections. Shouldn't that be HTTPS for security reasons?
- No, switching to HTTPS has no added value in terms of security, we've discussed that with the CernVM-FS developers.
CernVM-FS has built in security checks between server and clients, so HTTPS doesn't provide any additional security (I think, should be checked in CernVM-FS documentation).
* How would this work for large jobs across multiple nodes, can a lot of network traffic to pull in the software be avoided?
- Yes, you can set up a shared CernVM-FS cache on a shared filesystem.
If there's no internet access on the cluster workernodes, you can use a squid proxy in the cluster network (on a login node for example).
This setup has been tested with the EESSI pilot stack at the Jülich Supercomputing Centre, worked really well!
* Comment: Detection of CPU architecture is a very nice feature. This is a big issue with containers where generic binaries are often, which can have a big impact on performance.
- Yes, indeed! Containers are also very rigid: what if you want to add additional software?
The EESSI environment is way more dynamic, easy to add software on top of it (without paying for it in terms of performance), etc.
* Comment: This would also work really well in heterogenous environments with a mix of old/new CPUs, thanks to the auto-detection mechanism.
- Yes, very correct, this is an interesting use case!

### How to explain containers are friends of scientists? Are they?

(please copy it where it fits better)

- Mentioning where containers are used and what are the the alternative and competing tools. Conda was mentioned many times.
- A story of advertising Containers at an institution to scientists, in particular by organising a semminar giving overview of containers and brief intro into them, including documentation where and how can be used at the institute. But even such activities has not resulted in a serious adaption from the scientists side. On the other side it found other paplication, e.g. where isolation is needed for dividing conservative stable software for controlling scientific instruments and up-to-data data analysis environment. Mentioned that in early times, containers were presented in way expressing only limitted view and applications of them.
- Mentioned that application in the instrument control, requiring isolation, are interesting.
- There are different type of containers (docker, singularity, proto???mark) that are appropriate for different situations: service deployment, HPC, isolation etc. A proper one should be choosen depending on the application case.
- There are many possible options how the given container technology can be adapted/used. One should think it is a versatile technology.
- Other particular cases where containers were used: a) glibc issues, b) dot.net on CentOS
- Expressing a thought that if containers (or other tools) are not widely used/adopted in the given community (scientists) it can die out.
- View from BioInformatics: Conda was used a lot (Bioconda), nowadys many computing intensive pipelines are packed into Singularity

## Wrap Up Talk
* Thanks to all contributors, organizers and keynote speakers
* Especially Samantha, Richard, Thor, Jeremy, Radovan, Naoe for technical setup
- Highlights from the notes
* Great intro to the international RSE movement and to RSEs on Monday, thanks Alys Brett, Richard and Samantha
- Good introductions to groups and projects
- several expanding RSE groups
- Several technical tools and topics
- The further we go, the more we can focus on sharing experiences and tools
- Great to see interest in these discussions!
- We will try to make all slides/contributions findable and accessible
- authors: please send us the DOI or the pdf version
- Interesting and inspiring panels session.
- What the Nordic RSE should do?
- Build an identity. Create a network with local hubs.
- About local hubs: multiple things need to happen on university level
- Build a network, connect RSEs who are currently only connected to researchers in a field
- Have a more specific definitions of RSE
- Job board (separate from CR chat?)
- Give feedback to national and Nordic organisations (funders, for example)
- Collection of resources on the website (to hands of scicomp, citation file format …)
- It is easy to see the problems. We should implement solutions.
- Make good, well designed tools that also professors will use.
- Use and advocate for best practices
- Ask for feedback
- Invite everyone to
- Biweekly meetings
- Coffee breaks for more freeform chats

## Feedback about the meetup

One thing you enjoyed, one thing that we should improve for next time:

Good:
- Beside many intersting topics and presentations, even online over Zoom it works! I like many discussions and open topics, as well as several well prepared demos and panel sessions.
- Many interesting topics and good discussions
- easy to schedule time for due to being online
- A very very good idea to "professionalize" the community around RSEs
- Very well spaced out sessions and great breakouts.
- Great opportinuty to meet fellow RSEs
- I liked the topics, meeting a new community.

To improve:
- break-out rooms/sessions concurrency scheduling, a year ago in Helsinki I think they were organized more dynamically
- minor issues with zoom/issues with joining breakout rooms
- HackMD quickly becomes hard to follow, a lot of text
- Make more modular by linking out of a main HackMD
- Setting a little stronger directions for the different sessions. For instance make some guidelines for the discussion sessions so the discussion stays on track
- Linkout in the markdown to improve readability.(Accessability adaptation)
- I found it really odd that there were holes in the schedule. I also found it odd that there was a block just with discussions, at least some intros there would have been nice followed by the discussions.

## Feedback after the meeting:

- Nordic-RSE organization on LinkedIn to increase visibility?

---
*Always write at the very bottom, above this line*