# Flock 2024 Infrastructure and Release Engineering hackfest ## schedule <https://cfp.fedoraproject.org/flock-2024/talk/P9AV9Z/> friday aug 9th starting at 9am and running until 1pm. ## attendees (for however long) * * remote: abompard (ping me in the Infra channel on Matrix) ## proposed items to work on: - ✓Come up with some proposal for standards for openshift apps - ✓Discuss infra-sig packages maint: Propose list to add/remove - ✓Discuss releng packages (fedora-packager/fedora-release,etc) - ✓Discuss proxy network: move to nginx? change things? or keep? - ✓Discuss making aws more ansiblized/managed, or not? - ✓Discuss onboarding, what we can do to make it better - ✓short openshift intro for new folks (communishift/stg/prod) - ✓Look ahead: gitforge, bugzilla, matrix server - ✓Retire wiki pages / migrate to docs - ✓Comm ops. wants to get information from datagrepper - ✓ARA (ansible running ansible) in fedora infrastructure? - ✓moving AWX forward - ✓zabbix checkin/testing/planning - Onboarding new contributors on infra applications development -- @abompard - Among the people in the room, would there be some that are interested in coding on the Fedora Infra apps? - If so, which apps? - What would you need to start contributing? What's currently holding you back? - Aurélien would be happy to mentor a few folks on our apps (well, those he knows, so not all apps) and help people get up to speed with the dev environment, the code layout, the tests, etc. - Recent examples of apps that could use help/work/love: Badges, MirrorManager, FMN, Noggin, Datanommer, Bodhi, etc. - It's mostly Python with the exception of FMN that also has TypeScript. Frameworks are Flask & FastAPI - Ipsilon replacement by Keycloak + IPA-Tuura -- @abompard - Anyone interested in looking into that? I heard about IPA-Tuura at FOSDEM but haven't found the time to prototype with it yet. - talk to me (@abbra). nirik promised to send a list of technical reqs for Keycloak to be able to replace Ipsilon (OpenID need, etc). ipa-tuura has konflux-built images already available so can be tried but we haven't yet merged Keycloak's plugin to support ipa-tuura into Keycloak itself (planned by end of the year/2025q1). - GEOip for Fedora, AlmaLinux flock talk - Until the beginning of the week, we had 6 years old GeoIP databases stored on batcave and available for infra apps, and the update script was broken. - @abompard fixed the script, it requires a Maxmind login now, he created that and the databases are now being updated - ...but it's the free databases, they are somewhat inaccurate. We've had a report of a mirror in South Korea being misplaced on MirrorManager's map (in the capital instead of in the south of the country, hundreds of km away) - We could switch to ipinfo.io, it's also a Freemium service, but it's based on HTTP API calls instead of downloading databases. The free plan is 50k queries/month. People say it's more accurate. We could also ask them if they want to sponsor Fedora with a free premium account, they do that for Alma - help attendees with infra / releng problems, questions, concerns - Find repositories we can archive on pagure.io or github.com - Can sort repos. by activity ... that should be a good first start. ## notes: - general - For all our images/playbooks ... have a simple README pointing to the main links for the project, Eg. current main source repo. Next problem is making sure the README is correct 18 months after someone touched it. - containers - need to rework all the things that use deployment config. - Need to deploy/use ACS (redhat product) that looks inside containers and tells you what's in it and what the security issues are. - Need best practice kind of document for building containers for internal openshift deployment - FAS ids of people interested in collaborating on the openshift apps best practices/standards: humaton, zlopez, smiller, dkirwan, abompard, lachmanfrantisek, lsm5, mohanboddu - infra sig group and the packages in it - First step: culling the packages, Eg. stuff that isn't built for el9. - Only kevin updates it - lots of things can be removed - maybe remove some people from the group? - python sig has taken some of the packages, maybe take more? - run our own pypi mirror instead of making rpms? cert for govt. less impressed by this idea - Packit folks (lachmanfrantisek) are willing to help with the (mass) onboarding for the suitable packages (just [provide a list of packages](https://github.com/packit/packit-service/issues/new?assignees=&labels=onboarding&projects=&template=onboarding.yml&title=Onboard+fedora-infra+packages)) - releng packages - move to be owned by infra-sig group (not the same as above repo) - proxies - Apache-httpd doesn't have HTTP/3 support. - do we just give all the content to a CDN - Ask someone/group in RH what we should do - Just do nothing? :) - AWS more managable - Our acct. is used by us, centos, openQA, etc. - All of the setup is done manually ... but after setup can be ansibled. Ansible is capable of doing the AWS setup. - Terraform maybe helps a lot - onboarding - Kevin has done some work done, but not committed/pushed - Less community involvement/pipeline for new sysadmins than we used to have - Market who the current infra. people are, and how you can speak to them - People want to help a little bit, but don't want to touch the servers when they'll be doing it once or twice a year. - Problem between Kevin just doing easy fixes, and they aren't there for new people, or them not being done for a month. - Hello days (Eg. after every release) - openshift apps. intro - Three clusters, communishift (anything goes, AWS); staging (deploy by ansible, VLAN); production - refactor playbooks, don't start builds move to deployment objects - gitforge - Should change the processes to use gitforge features (on push do X, on branch do Y, etc.) - retire wiki - user documentation is on wiki and doesn't fit with current docs - get a list of categories for wiki pages, then can migrate some of those categories - gitforge might help - datagrepper in Communishift - dump data weekly into an amazon postgres instance? - Robbert is going to use the public dumps and stand up a new instance somewhere. - ARA - Just use AWX instead? Might be much easier to deploy though. - moving AWX forward - problems with current split of ansible repos. for public/private info. - migrate to ansible Vault or something else that is more AWX compatible? - Does reporting, so don't need ARA? - Using Vault means we can have a dummy set of variables to do CI on changes. - Zabbix - Setup 2nd matrix channel so that zabbix can complain without spamming the main channel - Still problems with network connectivity, some VPN issues, potential RH firewall ports need opening - Porting custom nagios alerts is still undecided. - Stop warning for CPU on builders etc. - More levels than nagios, just message on the big ones?