# 2020-01-21-colchester-workshop https://bit.ly/codimaEssex Notes for https://ccp-codima.github.io/2020-01-21-colchester-workshop/ In this notepad, we invite you to share organisational information, and details about what you want to work on/have worked on during the workshop. Please contribute to the agenda of the workshop to make the most of it! We are going to finalise the agenda in January, depending on the audience skills and priorities, so everything in this document is subject to discussion - suggestions welcome. We will also use this or newly create notepads to keep notes from the workshop, and share linkis and code snippets during the workshop. # Workshop concept This workshop will be useful to you if you are a mathematician or computer scientist interested in using GAP and/or SageMath in your research. You do not necessarily intend to contribute to the development of these systems or to submit your code to be included in them in a form of packages - what you need is to write your own code to work with some examples and conduct computational experiments. To do that efficiently, you not only need to know the relevant functionality in GAP and/or SageMath, but also know how to manage the whole process to make it as automated and reproducible as possible. You don't need to have any previous knowledge of the tools that will be presented at the workshop. For collaborative practical sessions, we will aim at organising participants in groups, so that novices will be paired up with someone more experiences. We will start with basics in order to boost up novices' skills and prepare everyone to more advanced topics. For the practical sessions, you are encouraged to bring your own code from experiment that you run with GAP or SageMath in the past, or have a particular problem that you would like to tackle with their help. Please bring a laptop with a Mac, Linux, or Windows operating system (not a tablet, Chromebook, etc.) on which you have administrative privileges. A working installation of GAP and SageMath is desirable. If you have problems with installing them, we can try to resolve them in the beginning of the workshop, if time permits. Installation instructions: - **GAP**: https://carpentries-incubator.github.io/gap-lesson/setup/ - **SageMath**: - Linux and macOS: http://www.sagemath.org/download.html - Windows: https://github.com/sagemath/sage-windows/releases - **Git**: we suggest to use the standard Software Carpentry setup, see instructions e.g. at https://ccp-codima.github.io/2020-02-20-lancaster/#setup The main format of workshop sessions will be live coding: instead of using slides, the instructor will type in the code and/or commands, run them, and explain their acions. The participants are strongly encouraged to repeat the same steps with the instructor. We call this "participatory live coding" or "coding along". This approach is used by the Carpentries. Howeber, this is not a full-scale official Software Carpentry workshop: we will use Carpentries pedagogical approaches a lot, but will not have clearly identified lessons on Unix shell and Git (which would take a whole day by themselves). Instead, we will interleave elements of these into the topics and mini-projects outlined below, making pauses when necessary to explain concepts, and embedding practice of relevant skills into collaborative exercises. Attendees of previous CCP-CoDiMa training events will be very welcome to act as helpers for topics which they already know, and help others to work through things like installation problems, error messages and unexpected output. You don’t need to be an expert in the tools we’re covering, just to have used one or more of them and be comfortable helping people to troubleshoot. # Participants If you are coming to the workshop, please briefly intoduce youself below (keep the order alphabetic). Please indicate briefly if you have any prior experience of using GAP and/or SageMath. - Erik (Madison) Bray (https://iguananaut.net): Research software engineer at CNRS / Université Paris-Sud. I'm one of the core maintainers of SageMath, and developer of the Windows port of Sage. Also have contributed a little bit to GAP development. I am not a researcher and do not often *use* the very software I develop on, but I do know many of the basics of Sage, and can help with all manner of technical issues. I am also an expert at Python, and am a regular Software Carpentry instructor. - Chris Jefferson (https://caj.host.cs.st-andrews.ac.uk): I am a reader at the University of St Andrews. My background is in A.I. and Computational Group Theory. I am one of the developers of the internals of GAP, including debugging and profiling code. My main (group theory related) research interestes are computationally hard problems on permutation groups, such as intersection and normaliser. - Olexandr Konovalov (https://olexandr-konovalov.github.io/year-archive/): I am a lecturer at the School of Computer Science at the University of St Andrews (and an organiser for this workshop). Mathematician by training, I started to use GAP in my research in the theory of group rings, and eventually became an active contributor to GAP. I am an instructor and an instructor trainer for the Carpentries, and the author of the Software Carpentry-style lesson "Programming with GAP" (https://carpentries-incubator.github.io/gap-lesson/) which we plan to teach at this workshop. - Alastair Litterick (https://www.essex.ac.uk/people/litte00606/alastair-litterick) I am a lecturer in the Department of Mathematical Sciences at the University of Essex, and a local organiser for this workshop. My primary research interests are in algebraic groups and finite simple groups, broadly understood, and this has included quite a lot of computational algebra, but primarily not in GAP (and I have no SageMath experience at all). I am increasingly interested in algebraic geometry, but have not yet attempted to apply computational methods to this. - Jesus Martinez-Garcia (https://jesusmartinezgarcia.net): I am a lecturer in mathematics at the University of Essex (and a local organiser for this workshop) and I have a background in computer engineering. I have applied some programming techniques to geometric problems, from rudimentary calculations in linear algebra and symbolic methods to, more recently, creating ad-hoc libraries that solve probelms in Geometric Invariant Theory (AKA GIT). I created a library to describe the GIT quotient of hypersurfaces in projective space in Python and more recently, with collaborators Gallardo, Moon and Swinarski, I have completed a prototype of GIT library in SageMath which describes the GIT quotient of a Projective space by any non-exceptional simple Lie group. I belive this library can be useful to many researchers in moduli spaces. My personal goal for this workshop is to understand what I need to do to make our library be part of the SageMath distribution. - Emilio Pierro (https://www.researchgate.net/profile/Emilio_Pierro): I am a Guest Teacher at the London School of Economics. My research is in finite simple group theory and I've been using GAP (and Magma) for many years to study various properties of finite groups. - Michael Torpey (https://mtorpey.github.io): I am an associate lecturer in the School of Computer Science at the University of St Andrews. My background in computational algebra led to a natural interest in GAP, and I am the author of GAP's package manager and the Memoisation package, and a co-author of the Semigroups and Digraphs packages. I have a particular interest in teaching these sorts of technologies to others, and so I'll be delighted to be participating in another Software Carpentry--style workshop! - Madeleine Whybrow (https://madeleinewhybrow.wordpress.com/): I am a postdoc at the University of Primorska in Slovenia. I use GAP and other computer algebra systems extensively in my research. I will be talking about my work developing algorithms and software for studying axial and Majorana algebras and will be around throughout the workshop to help out and answer any questions you might have. - James Williams (https://seis.bristol.ac.uk/~qd19176/) I am a postdoc at the University of Bristol. I use GAP to gain insight in my research of p-groups. # Topics This is the initial list of topics, listed in some kind of logical dependencies between them. Please add a topic which you would like to learn, or suggest something what you would like to teach yourself. Some of the topics may be merged, or incorporated into the exercise sessions as means to achieve other tasks. We will be making the final programme in mid-January, taking into account these suggestions. - **How to install our systems and packages** (Olexandr, Erik): this may be a short slot in the beginning to check that everyone can run GAP and SageMath, and help if necessary. Some problems may be identified and dealt with later, if they can't be resolved immediately. - **GAP Software Carpentry lesson** (Olexandr, Chris, Madeleine, Michael): this is a Software Carpentry-style introduction to GAP (https://carpentries-incubator.github.io/gap-lesson/). In average, it could be taught over 4.5 hours; depending on the audience, that could be anything between 3 and 6 hours. The suggestion is to spend Day 1 morning on GAP and switch to SageMath after lunch, and then continue GAP training on Day 2. At Day 1, an absolute necessity is to cover episodes 1 and 3 of the GAP lesson from the link above. - **Introduction to SageMath** (Erik): Run the notebooks online: https://mybinder.org/v2/gh/ccp-codima/2020-01-21-colchester-workshop/master?filepath=notebooks%2Fintroduction_to_sage%2F Git repository: https://github.com/ccp-codima/2020-01-21-colchester-workshop Nice Sage notebooks: https://sagemanifolds.obspm.fr/examples.html - **Working with files and directories in the terminal** (Olexandr, Erik, Michael and others): this topic is usually covered at Software Carpentry workshops in more detail. We will not have a separate session on this topic here due to time constraints, but will be using it a lot during live coding sessions, making stops, when necessary, for more detailed explanations. Mastering the command line will save you a lot of time, so please ask questions, if you will need us to repeat something or explain some tricks. Thus, these skills will be reinforced via the transfer of the _tacit knowledge_, by watching _how_ the instructors will work and repeating their actions on your computer. We expect to achieve this by the end of Day 1. - **Running tests to check that code works correctly** (Olexandr, Erik): for GAP, this will be introduced in the GAP Software Carpentry lesson in the Episode 4, which will be the start the GAP lesson on Day 2. It will be then revisited several times in later sessions, e.g. in the one on publishing reproducible experiments, and another about debugging and profiling in GAP. For SageMath, FIXME. - **Version control with Git and GitHub**: again, we will not run a full half-day course on this topic, like we usually do at Carpentries, but will start to introduce it from Day 2. For this, on Day 2 we should ensure that everyone has installed Git and created a free GitHub account. Then before Episode 4 on GAP, we will establish the repository for our project with the initial version of the lesson's `avgord.g` file, and then will keep its future versions and a test file under version control. This will provide an opportunity to learn most common Git commands, and practise a typical workflow for making changes. When we are fluent at working with our local repository, we will put the code on GitHub and practise making further changes with pushing and pulling to/from the repository. - **Continuous integration with Travis CI**: after finishing GAP Episode 4, we will have code and tests, but they will not be performed automatically. Now we can activate Travis CI integration, and add our standard setup for regression tests to the repository. We will then continue with Episode 5, checking how our changes are tested on Travis CI after being pushed to GitHub. - we can start with a general introduction to Travis before diving into the GAP specifics; an input from SageMath developers at that point would be great. At the moment I think there is not much state of the art on this, though I think a few projects are using Sage in Travis; I will look into it (EB) - **Using Git and GitHub to publish code and collaborate with others** (while teaching GAP Episodes 5 and 6): - collabrative exercises in pairs to work on the same repository where both people have write access - submitting pull requests to other repositories where you do not have write access (e.g. adding your details to the list of workshop participants) using GitHub web interface - submitting pull requests to test changes and perform code review *before* they are getting merged - **Tips and pitfalls for using GAP** (Olexandr): this is a lecture given using a Jupyter notebook hosted on Binder. Participants will be able to open it and run it in their browser. This way, I will combine the lecture about GAP with introducing Jupyter notebooks. Since it will be running remotely, there's no need for local installation. - **Publishing reproducible experiments using Jupyter notebooks running on Binder** (Olexandr can cover both GAP and Sage): we will now add Jupyter notebook and Binder setups to our `avgord.g` repository - Olexandr is now creating a template for publishing reproducible GAP experiments in Jupyter notebooks runnable on Binder: https://github.com/rse-standrewscs/gap-binder-template. It is possible to incorporate it into episodes 4-5 of the GAP Carpentries style lesson, or have a separate session on it, depending on time constraints and the progress achieved on Day 1. - **Recommendations for reproducible research practices using Jupyter** (Olexandr): how to avoid going from spaghetti code to spaghetti notebooks. - If there is an interest in using GAP and SageMath in Jupyter locally: **walkthrough of the installation process** (GAP: Olexandr; SageMath: Erik) - **Automate tasks with UNIX shell scripts**: again, this is something which we will not teach expliciltly, but may come across explaining Travis CI setup or starting GAP or SageMath using a shell script on a university cluster. - **Debugging and profiling in GAP** (Chris): GAP has many ways to help you understand why your code isn't working, or is performing slowly. I will give an overview of various techniques, including the 'profiling' package. - **Run distributed computations with GAP and SageMath** - who: - Olexandr (GAP, based on https://github.com/olexandr-konovalov/scscp-demo) - this is based on the example from the Software Carpentry GAP lesson, so will naturally be its continuation - ??? (Sage) (what do we mean here by distributed? - distributed as in shared memory parallelising vs distributed memory parallelising - AFAIK, there is no multithreading in Sage) - JMG: I'm actually very interested in this. I have asked to get Sage installed into our parallel computation centre and my code is potentially parallelisable. It was my understanding that this was possible, see [here](http://doc.sagemath.org/html/en/reference/parallel/index.html) - **Success stories**: talks by people who have applied GAP or SageMath in their own research - who: Madeleine - who: James - I would be interested in giving a short talk (20 mins?) on how GAP helped to drive my research in my PhD. - who: Chris - Both developing new algorithms for permutation groups, and using these algorithms in A.I. - who: Jesus (SageMath for moduli spaces), happy to talk about this for 10' on whiteboard. - **GAP package manager** (Michael): an optional tool for installing and updating GAP packages from within GAP, automating a lot of the more complicated parts. Particularly useful for those needing custom packages or their latest versions ahead of including in GAP release. This might come up in other parts of the workshop, but if there is interest, Michael can present a short demo. - **Contributing to our systems** (overview, sources of information, communication channels, etc.) - GAP and its packages: Olexandr, Michael - SageMath: Erik - **"Code is Science"** (Olexandr: wrap-up talk about research software): Scientific code needs to be treated as a primary research output. For this reason, we have created the Code Is Science Manifesto, which calls for making genuine efforts to make research code open, correct, sustainable, and citable. I will discuss the main principles of the Manifesto and provide some practical tips on their implementation. The talk is intended for anyone who deals with research software, including researchers, research software engineers, and administrators. - also where and how to publish software and/or software papers in computational algebra # Mini-projects These could be mini-projects to work on during practical sessions e.g. - collaboration using GitHub (pull requests, code review) - for example, submit a PR to the list of participants to specify your website and/or GitHub username - set up regression tests and continuous integration for your code - bring your code to archive it on Zenodo and publish your reproducible experiment on Binder - reviewing and further developing SageMath Software Carpentry lesson # Draft schedule ## Tuesday January 21st 09:30 Registration on the NTC (North Teaching Centre) foyer (location [here](https://findyourway.essex.ac.uk/?cmsid=4653&project_id=1)). - Since it can be tricky to get oriented around campus, Alastair Litterick will be at the Silberrad Student Centre (AKA The Welcome Centre, location [here](https://findyourway.essex.ac.uk:8080/share/e17705ecee7c57736c23625451e8dfa8)) at 09:20 to lead a party to the NTC. ### Morning [[NTC 1.03](https://findyourway.essex.ac.uk:8080/share/e5ef2d3e735446ac37a28720f02cfa55)] 10:00 Start of the workshop - Welcome, housekeeping and introductions - Checking and helping with GAP and SageMath installations (at least up to the state sufficient to start the GAP training session; the rest of the problems can be identified and dealt with later) - Programming with GAP (Olexandr Konovalov): Software Carpentry-style lesson at https://carpentries-incubator.github.io/gap-lesson/ 11:15 Break 11:45 Programming with GAP (continued) Troubleshooting: - `GapWriteIniFile` what to do if it tries to write a file on drive M: which is not mounted - GAP error message `Error, file <filename> must exist and be readable at path/to/gap/installation/lib/files.gi etc.` is very confusing to a beginner - it looks like GAP is trying to find a file in the directory of GAP installation This is the content of `avgorg.g` which by the end of the morning GAP session should appear in your home directory: ``` AvgOrd:=function ( G ) local sum, g; sum := 0; for g in G do sum := sum + Order( g ); od; return sum / Size( G ); end; ``` Q: how to enumerate "first" 100000 elements of a free group of rank 2? A: using iterators ``` G:=FreeGroup("x","y"); i:=0; for g in G do Print(i, " : ", g,"\n"); i:=i+1; if i>100000 then break; fi; od; ``` 13:00 Lunch [[NTC 1.02](https://findyourway.essex.ac.uk:8080/share/162130ffcb2242caf8cd8e778170e5a4)] ### Afternoon [[STEM 4.1](https://findyourway.essex.ac.uk:8080/share/78ca3e3060b208db87fdc3919a603499)] 14:00 Programming with SageMath (Erik Bray) Troubleshooting Q&A: Q: How to start SageMath in Jupyter A: use `sage -n jupyter` Q: Where are my ipynb files in Windows? A: In your home directory in `SageMath-X.Y\runtime\home\sage` 15:15 Break 15:45 Programming with SageMath (continued) 16:45 Wrapping up - Reminder: everyone needs to install Git and create a free GitHub account for tomorrow's exercises ### Evening 18:30 Workshop dinner - We will be having a workshop dinner on the 21st at 6.30pm in [The Old Siege House Bar and Brasserie](https://www.theoldsiegehousebarandbrasserie.co.uk/) at 75 East Street, Colchester, Essex CO1 2TS. The 3-course menu is paid for by the organisers. Drinks can be purchased by attendees themselves. Due to the large number of participants we need to provide our dinner choice from the menu in advance -- please fill in the form at https://forms.gle/s3rzVn6GbStb9W5c8 ## Wednesday January 22nd ### Morning [[Lakeview Room](https://findyourway.essex.ac.uk:8080/share/49bb55d7d612d7a475bce22139d11de0)] 10:00 Overview of the agenda for the day Announcements: - Software Carpentry workshop: Unix shell, Git and SageMath (Lancaster, Feb 20-21): https://ccp-codima.github.io/2020-02-20-lancaster/ - Install Git: https://ccp-codima.github.io/2020-02-20-lancaster/#setup - Create a free GitHub account: https://github.com/ - Expenses claims: https://www.st-andrews.ac.uk/policy/staff-pay-and-benefits-expenses/staff-expenses.pdf To those staying in the Wivenhoe House Hotel: we provisionally booked a table for 8 people at 8pm tonight. If you're joining us, please add your name to the list, and we will update the reservation: - Olexandr - Madeleine - Michael - Erik - Dean - Awss - Diego - Chris Jefferson - Rhys 10:15 GAP (Olexandr Konovalov) Links: - GAP Carpentries-style lesson: https://github.com/carpentries-incubator/gap-lesson - Template for reproducible GAP experiment: https://github.com/rse-standrewscs/gap-binder-template/ - Example of a reproducible GAP experiment: https://github.com/sukru-yalcinkaya/unipoly - OpenDreamKit Report describing the unipoly example (see Appendix C, page 133): https://github.com/OpenDreamKit/OpenDreamKit/raw/master/WP5/D5.15/report-final.pdf - OpenDreamKit demos of GAP Jupyter notebooks: https://github.com/OpenDreamKit/gap-demos ### Break (11:15-11:45) 11:45 James Williams: Powerfully Nilpotent Groups - Insights due to GAP Chris Jefferson: Debugging and Profiling in GAP - NoMethodFound errors - LoadPackage errors (wrong name/uncompiled package) - Undocumented and ALLCAPS functions - Common syntax error message - Using "profiling" package: https://gap-packages.github.io/profiling/ ### Lunch (13:00-14:00) [[NTC 3.02](https://findyourway.essex.ac.uk:8080/share/04b5b644ac3bf0098c8acb6a66de200b)] ### Afternoon [[NTC 3.02](https://findyourway.essex.ac.uk:8080/share/04b5b644ac3bf0098c8acb6a66de200b)] 14:00 **Binderfest** Bring your GAP or SageMath code Useful links: - Software Carpentry lesson on UNIX shell: https://swcarpentry.github.io/shell-novice/ - Software Carpentry lesson on Git: https://swcarpentry.github.io/git-novice/ Most useful shell commands: - pwd - cd - ls - mkdir - cp - mv - rm Most useful Git comands - non-destructive (for inspection, call as many times as you wish) - status - diff - log - commands that change the state of the repository - add - commit - push - pull - setting up: see https://swcarpentry.github.io/git-novice/02-setup/index.html - to use "nano" for commit messages (once on your computer) - `git config --global core.editor "nano -w"` - to clone a repisitory (once for each clone) - `git clone` Most useful Git commands: `git help` and `git help <command>` (e.g. `git help commit`) ### Break (15:15-15:45) 15:45 Sage Q & A 16:45 Wrapping up (maybe staying a bit longer, if the room remains available) 20:00 Dinner Some of us staying in the Wivenhoe House Hotel go to the dinner in the hotel's Brasserie. Please add your name to the list (line 298) if you're coming. What do we want to do tomorrow? Definitely this: - Chris: partitions backtrack - Madeleine: Majorana algebras (slides: https://madeleinewhybrow.files.wordpress.com/2020/01/whybrow-slides-website.pdf) - Michael: Package manager (see abstract above) - Olexandr: Say "NO" to spaghetti notebooks (based on https://github.com/olexandr-konovalov/repro-jupyter) Optionally this (please put an exclamation mark after the item if you are interested) - Olexandr, Michael: walk through the rest of the GAP lesson! - Olexandr: Code is Science (wrap-up, see abstract above)!!! - Olexandr: Tips and pitfalls for using GAP!! - Olexandr: SCSCP (see abstract above)! - Help with installation (e.g. of GAP Jupyter notebooks) - Time for exercises, questions - More practice in using version control/GAP/SageMath (maybe in small groups) - Mastering Git & GitHub: pull requests!! - if you wish to try, you may submit a pull request to update your details in https://github.com/ccp-codima/2020-01-21-colchester-workshop/blob/master/_data/participants.yml - More experience with publishing notebooks on Binder - Anything else? Please suggest ## Thursday January 23rd ### Morning [[STEM 4.1](https://findyourway.essex.ac.uk:8080/share/78ca3e3060b208db87fdc3919a603499)] 10:00 (with a break at 11:15-11:45): - Using GAP Effectively: Some Tips and Pitfalls (Olexandr): https://github.com/OpenDreamKit/gap-demos/blob/master/notebooks/TipsAndPitfalls.ipynb - Say "NO" to spaghetti notebooks (Olexandr): https://github.com/olexandr-konovalov/repro-jupyter - Majorana Algebras (Madeleine): https://madeleinewhybrow.files.wordpress.com/2020/01/whybrow-slides-website.pdf - Partition backtrack (Chris) ### Lunch (13:00-14:00) [[6.300](https://findyourway.essex.ac.uk:8080/share/dd0b7ec2d064d29a02bdf426743092a0)] ### Afternoon [[STEM 4.1](https://findyourway.essex.ac.uk:8080/share/78ca3e3060b208db87fdc3919a603499)] 14:00 (with a break at 15:15-15:45) - Code Is Science: Research Software is a Primary Research Output (Olexandr) ######################################################################## Links for Olexandr's talk: * Software Sustainability Institute (SSI): https://software.ac.uk * UK Society of Research Software Engineering: https://society-rse.org/ * The Carpentries: https://carpentries.org/ * Code Is Science Manifesto: https://codeisscience.github.io/manifesto/ * Software Citation Principles: https://doi.org/10.7717/peerj-cs.86 * Code4REF: https://code4ref.github.io/ What does “release” mean? - at least should have a version - GitHub releases: https://help.github.com/en/articles/creating-releases - ideally, should also have a DOI - Zenodo (https://zenodo.org/) - Figshare (https://figshare.com/) - institutional repository - ... - getting a DOI can be automated using GitHub-Zenodo integration: https://guides.github.com/activities/citable-code/ - Have an open source license (helpful: https://choosealicense.com/) Sharing reproducible experiments with Jupyter on Binder and Azure Notebooks: - GAP demo/template at https://github.com/rse-standrewscs/gap-binder-template/ - Python demo at https://github.com/olexandr-konovalov/repro-jupyter - R demo at https://github.com/olexandr-konovalov/repro-r Code Deserves Credit - Software citation principles: A.M. Smith, D.S. Katz, K.E. Niemeyer, FORCE11 Software Citation Working Group. (2016) Software Citation Principles. PeerJ Computer Science 2:e86. https://doi.org/10.7717/peerj-cs.86 - As a developer: - Advise your users how to cite it - Automate sample citation generation - Provide a CITATION.md file - Provide a CFF file: https://citation-file-format.github.io/ -As a user: - Follow recommendations by developers, if they are provided - Cite accurately. It helps both developers and readers. Where to publish software papers: - JOSS (Journal of Open Source Software): https://joss.theoj.org/ - JORS (The Journal of Open Research Software): https://openresearchsoftware.metajnl.com/ - ReScience C: http://rescience.github.io/ - More journals accepting papers that primarily about the software - https://www.software.ac.uk/which-journals-should-i-publish-my-software - JSAG (Journal of Software for Algebra and Geometry): https://msp.org/jsag/ ######################################################################## - Package manager (Michael): https://gap-packages.github.io/PackageManager/ - SCSCP demo (Olexandr): https://github.com/olexandr-konovalov/scscp-demo - Walk through the rest of the GAP lesson (Michael) - Mastering Git & GitHub: pull requests (Michael) ### Wrapping up We anticipate finishing the workshop between 4pm and 5pm. Olexandr: I plan to leave at 4.30pm by taxi to the Colchester train station for a 5.30pm train. You can join me if you are interested in a lift.