---
# System prepended metadata

title: UoD Power outage 2020-10-10
tags: [SA]

---

---
tags: SA
---

# UoD Power outage 2020-10-10

## References

- [image.sc announcement](https://forum.image.sc/t/ome-resources-down-due-to-uod-outage/43957)
- [UoD services spreadsheet](https://docs.google.com/spreadsheets/d/1P6F9s9DS0bp372VQUPxCLDYKdPoREg7_KFdNzEd-C-8/edit#gid=0)

## 2020-10-15

Attending: Seb, Jason, Josh, Simon, J-M, Will, Dom, Petr, June, Frances

### Status
- Code references (Josh/Chris)
  - upgrade checks (known)
  - `omero import` --> download hangs
    - CloudFlare, S3, etc.
  - Where on the priority list?
- Shim aka new-registry (Josh/Chris)
  - Lambdas: GS: Haven't paid a dollar (UoD: dollars a month)
  - Storage: digital ocean postgres (multi-master) no effort
    - 120-160 USD / month
    - 120 GB (1 standby, auto-failover)
    - Note: flatted IP table
  - Steps
    - Update https://github.com/ome/qa-shim
    - Create database
    - Spin up via terraform
    - Change DNS
- **Hardware**
    - Simon: was working with JD. All production VMs are back online esp. high priority. Some networking issues.
    - Engineer coming this morning to fix remaining GPFS boxes. No redundancy
    - Should have no downtime
- www.openmicroscopy.org
    - deployed on GitHub pages
    - several components missing (QA, forums, site redirects, Schemas). Causes PRs to fail validation
    - Options: switch back to ome-www or keep working with GH pages (need redirect work)
        - Simon: in-progress work on redirects
        - J-M: GH pages seems to be the way
        - Jason: main concern is that spending time now means splitting resources
        - J-M: if 1-2 days away, worth investigating
    - Test website and add the schemas by EOB
        - List all broken 
- gate.openmicroscopy.org
    - Working with ports 22 + 443
- downloads.openmicroscopy.org/docs.openmicroscopy.org
    - Seem to be working. Tested via several builds
    - J-M: looking at it. Everything is working
    - Petr: presentations **ok**
- artifacts.openmicroscopy.org
    - currently redirected to GS artifactory
    - some issues with artifacts
    - question of whether we switch back to our own artifactory
        - J-M: working on Maven Central/Sonatype
        - Focusing on Bio-Formats first
    - Seb: wait until we need to push artifacts to switch ?
    - How many days of investigation before deciding to switch to the old workflow?
        - J-M: need to sort out various issues (shared used, GPG)
        - David: 1 week necessary to test various things
- learning.openmicroscopy.org
    - Jason to email Paul and ask to test. Minimal login works
    - Josh: only potential issue would be corruption when PSQL would shut down
        - No reason to think otherwise?
        - J-M: did some testing on outreach/workshop
    - Josh: assuming initial outage only looked like power outage
        - Simon: a priori yes
- outreach.openmicroscopy.org/workshop.openmicroscopy.org
   - Petr: fully working
- demo.openmicroscopy.org/pub-omero.openmicroscopy.org
    - Petr: fully tested and working (**yes** per spreadsheet)
    - Seb: looking at pub-omero today
- nightshade.openmicroscopy.org
    - Simon: starting the omero-server should be fine
        - Josh: assuming no storage bump. Just access problem?
    - Jason: nightshade mailing list?
        - Simon: wait for GPFS to be back
        - Josh: we're still in maintenance window
        - Petr: to start drafting
- idr-redmine.openmicroscopy.org
    - Frances: confirmed it's working. updating issues
- merge-ci.openmicroscopy.org/latest-ci.openmicroscopy.org
    - Seb: could not ssh into the boxes
    - Simon to check the DRAC
- ci.openmicroscopy.org
    - running
    - J-M: test these jobs if we are not switching the release workflow
- image.sc
    - Seb to draft

## 2020-10-14

Attending: Will, Seb, PEtr, Dom, Josh, Simon, Frances, J-M, Jason, David, 

### Status

- Hardware
    - Simon: 2 arrays down. One out-of-support, one under warranty. Out-of-support one was fixed with no charge. Dell engineer or UoD IT person need to go to Data Centre (need to ask who)
    - Getting VMWare back online requires UoD IT technician
    - Potentially getting VMWare back except nightshade, devspaces
    - Impact on current services of priority 4-5
        - gate back
        - downloads/docs still unavailable
        - other web-prod would work
        - website woudl come back
        - nightshade requires GPFS fix
        - demo/learning could come back
        - ome-lochy (Redmine, monitoring requires GPFS for persistent storage but could potentially bring back redmine without attachments)
- Artifactory ("running")
  - Josh: DNS updated from UoD to GS.
    - Mirrors scijava.
    - Note that actions can timeout while mirroring is taking place.
    - There may be JARs not there. Caching fun
    - Some changes on GS side at code level to be reviewed (bioformats2raw)
    - Are we rolling back or switching to GS artifactory long-term plan
        - Jason: much harder discussion. Problems we are having also affect NPSC, proteomics, MRC...
        - Seb: artifactory is one of the first to come up
        - Simon: need resiliency
        - Seb: have 3 of them effectively (Scijava + OME + GS) containing a portion of the artifacts
        - Simon: encode them in the builds to start relying on them
        - JRS: need to make this decision about what is the ideal.
        - JMB: pushing to maven central will also help (may be easier to push these days). More resiliency.
    - JMB: can't yet build. i.e. not "fixed"
- Investigations
  - See artifactory discussion above
  - Website (Simon):
    - website is in 2 parts. static content straighforward to move to GH pages
    - get our static content on GH pages?
    - Long-term could look at CDN 
    - JRS: minimal viable? Seb: static pages and then need things to add. Most critical is **/Schemas**
    - Simon: basically same as https://snoopycrimecop.github.io/www.openmicroscopy.org/
    - JRS: use one as the backup? Simon: could keep a list of DNS changes if there's another outage.
  - Docs (J-m): looked at various hosting options (e.g. GitHub pages). javadoc should be automatically available via javadoc.io. 
      - David available to discuss Bio-Formats deployments?
      - Josh: capture the impossibility to build first. J-M: to start with omero-model
  - Petr: snoopy + www

### Next steps

- Simon: redirect WWW to GitHub? Seb to change the DNS.
  - Simon working on redirects
  - Seb working on the Schemas
- Josh: propose **registry shim** with help from Glencoe
  - Simon: GDPR ok? Will check.
- J-M + David: look into artifact hosting
    - Potentially include Dom for omero-* artifacts
- As soon as VMWare is up, test learning + demo + outreaches
    - Simon: expecting no need to rerun the playbooks for Ansible managed systems 
- Python: Josh push omero-web to readthedocs
    - Seb: omero-py?
 - Misc
   - List of External Resources we're focusing on
     - PyPI
     - ReadTheDocs
     - javadocs.io
     - Maven Central
     - GitHub Pages
     - GitHub Releases
 - Presentations (Petr)
   - Possible to have resiliency?
   - Seb: GitHub pages?
   - Simon: hitting limits?
   - e.g. https://ome.github.io/presentations 