# Flock 2024 Infrastructure and Release Engineering hackfest

## schedule

<https://cfp.fedoraproject.org/flock-2024/talk/P9AV9Z/>

friday aug 9th starting at 9am and running until 1pm.

## attendees (for however long)

*
* remote: abompard (ping me in the Infra channel on Matrix)

## proposed items to work on:

- ✓Come up with some proposal for standards for openshift apps
- ✓Discuss infra-sig packages maint: Propose list to add/remove
- ✓Discuss releng packages (fedora-packager/fedora-release,etc)
- ✓Discuss proxy network: move to nginx? change things? or keep?
- ✓Discuss making aws more ansiblized/managed, or not?
- ✓Discuss onboarding, what we can do to make it better
- ✓short openshift intro for new folks (communishift/stg/prod)
- ✓Look ahead: gitforge, bugzilla, matrix server
- ✓Retire wiki pages / migrate to docs
- ✓Comm ops. wants to get information from datagrepper
- ✓ARA (ansible running ansible) in fedora infrastructure?
- ✓moving AWX forward
- ✓zabbix checkin/testing/planning
- Onboarding new contributors on infra applications development -- @abompard
- Among the people in the room, would there be some that are interested in coding on the Fedora Infra apps?
- If so, which apps?
- What would you need to start contributing? What's currently holding you back?
- Aurélien would be happy to mentor a few folks on our apps (well, those he knows, so not all apps) and help people get up to speed with the dev environment, the code layout, the tests, etc.
- Recent examples of apps that could use help/work/love: Badges, MirrorManager, FMN, Noggin, Datanommer, Bodhi, etc.
- It's mostly Python with the exception of FMN that also has TypeScript. Frameworks are Flask & FastAPI
- Ipsilon replacement by Keycloak + IPA-Tuura -- @abompard
- Anyone interested in looking into that? I heard about IPA-Tuura at FOSDEM but haven't found the time to prototype with it yet.
- talk to me (@abbra). nirik promised to send a list of technical reqs for Keycloak to be able to replace Ipsilon (OpenID need, etc). ipa-tuura has konflux-built images already available so can be tried but we haven't yet merged Keycloak's plugin to support ipa-tuura into Keycloak itself (planned by end of the year/2025q1).
- GEOip for Fedora, AlmaLinux flock talk
- Until the beginning of the week, we had 6 years old GeoIP databases stored on batcave and available for infra apps, and the update script was broken.
- @abompard fixed the script, it requires a Maxmind login now, he created that and the databases are now being updated
- ...but it's the free databases, they are somewhat inaccurate. We've had a report of a mirror in South Korea being misplaced on MirrorManager's map (in the capital instead of in the south of the country, hundreds of km away)
- We could switch to ipinfo.io, it's also a Freemium service, but it's based on HTTP API calls instead of downloading databases. The free plan is 50k queries/month. People say it's more accurate. We could also ask them if they want to sponsor Fedora with a free premium account, they do that for Alma
- help attendees with infra / releng problems, questions, concerns
- Find repositories we can archive on pagure.io or github.com
- Can sort repos. by activity ... that should be a good first start.

## notes:

- general
- For all our images/playbooks ... have a simple README pointing to the main links for the project, Eg. current main source repo. Next problem is making sure the README is correct 18 months after someone touched it.
- containers
- need to rework all the things that use deployment config.
- Need to deploy/use ACS (redhat product) that looks inside containers and tells you what's in it and what the security issues are.
- Need best practice kind of document for building containers for internal openshift deployment
- FAS ids of people interested in collaborating on the openshift apps best practices/standards: humaton, zlopez, smiller, dkirwan, abompard, lachmanfrantisek, lsm5, mohanboddu
- infra sig group and the packages in it
- First step: culling the packages, Eg. stuff that isn't built for el9.
- Only kevin updates it
- lots of things can be removed
- maybe remove some people from the group?
- python sig has taken some of the packages, maybe take more?
- run our own pypi mirror instead of making rpms? cert for govt. less impressed by this idea
- Packit folks (lachmanfrantisek) are willing to help with the (mass) onboarding for the suitable packages (just [provide a list of packages](https://github.com/packit/packit-service/issues/new?assignees=&labels=onboarding&projects=&template=onboarding.yml&title=Onboard+fedora-infra+packages))
- releng packages
- move to be owned by infra-sig group (not the same as above repo)
- proxies
- Apache-httpd doesn't have HTTP/3 support.
- do we just give all the content to a CDN
- Ask someone/group in RH what we should do
- Just do nothing? :)
- AWS more managable
- Our acct. is used by us, centos, openQA, etc.
- All of the setup is done manually ... but after setup can be ansibled. Ansible is capable of doing the AWS setup.
- Terraform maybe helps a lot
- onboarding
- Kevin has done some work done, but not committed/pushed
- Less community involvement/pipeline for new sysadmins than we used to have
- Market who the current infra. people are, and how you can speak to them
- People want to help a little bit, but don't want to touch the servers when they'll be doing it once or twice a year.
- Problem between Kevin just doing easy fixes, and they aren't there for new people, or them not being done for a month.
- Hello days (Eg. after every release)
- openshift apps. intro
- Three clusters, communishift (anything goes, AWS); staging (deploy by ansible, VLAN); production
- refactor playbooks, don't start builds move to deployment objects
- gitforge
- Should change the processes to use gitforge features (on push do X, on branch do Y, etc.)
- retire wiki
- user documentation is on wiki and doesn't fit with current docs
- get a list of categories for wiki pages, then can migrate some of those categories
- gitforge might help
- datagrepper in Communishift
- dump data weekly into an amazon postgres instance?
- Robbert is going to use the public dumps and stand up a new instance somewhere.
- ARA
- Just use AWX instead? Might be much easier to deploy though.
- moving AWX forward
- problems with current split of ansible repos. for public/private info.
- migrate to ansible Vault or something else that is more AWX compatible?
- Does reporting, so don't need ARA?
- Using Vault means we can have a dummy set of variables to do CI on changes.
- Zabbix
- Setup 2nd matrix channel so that zabbix can complain without spamming the main channel
- Still problems with network connectivity, some VPN issues, potential RH firewall ports need opening
- Porting custom nagios alerts is still undecided.
- Stop warning for CPU on builders etc.
- More levels than nagios, just message on the big ones?