✓Discuss proxy network: move to nginx? change things? or keep?
✓Discuss making aws more ansiblized/managed, or not?
✓Discuss onboarding, what we can do to make it better
✓short openshift intro for new folks (communishift/stg/prod)
✓Look ahead: gitforge, bugzilla, matrix server
✓Retire wiki pages / migrate to docs
✓Comm ops. wants to get information from datagrepper
✓ARA (ansible running ansible) in fedora infrastructure?
✓moving AWX forward
✓zabbix checkin/testing/planning
Onboarding new contributors on infra applications development – @abompard
Among the people in the room, would there be some that are interested in coding on the Fedora Infra apps?
If so, which apps?
What would you need to start contributing? What's currently holding you back?
Aurélien would be happy to mentor a few folks on our apps (well, those he knows, so not all apps) and help people get up to speed with the dev environment, the code layout, the tests, etc.
Recent examples of apps that could use help/work/love: Badges, MirrorManager, FMN, Noggin, Datanommer, Bodhi, etc.
It's mostly Python with the exception of FMN that also has TypeScript. Frameworks are Flask & FastAPI
Ipsilon replacement by Keycloak + IPA-Tuura – @abompard
Anyone interested in looking into that? I heard about IPA-Tuura at FOSDEM but haven't found the time to prototype with it yet.
talk to me (@abbra). nirik promised to send a list of technical reqs for Keycloak to be able to replace Ipsilon (OpenID need, etc). ipa-tuura has konflux-built images already available so can be tried but we haven't yet merged Keycloak's plugin to support ipa-tuura into Keycloak itself (planned by end of the year/2025q1).
GEOip for Fedora, AlmaLinux flock talk
Until the beginning of the week, we had 6 years old GeoIP databases stored on batcave and available for infra apps, and the update script was broken.
@abompard fixed the script, it requires a Maxmind login now, he created that and the databases are now being updated
…but it's the free databases, they are somewhat inaccurate. We've had a report of a mirror in South Korea being misplaced on MirrorManager's map (in the capital instead of in the south of the country, hundreds of km away)
We could switch to ipinfo.io, it's also a Freemium service, but it's based on HTTP API calls instead of downloading databases. The free plan is 50k queries/month. People say it's more accurate. We could also ask them if they want to sponsor Fedora with a free premium account, they do that for Alma
help attendees with infra / releng problems, questions, concerns
Can sort repos. by activity … that should be a good first start.
notes:
general
For all our images/playbooks … have a simple README pointing to the main links for the project, Eg. current main source repo. Next problem is making sure the README is correct 18 months after someone touched it.
containers
need to rework all the things that use deployment config.
Need to deploy/use ACS (redhat product) that looks inside containers and tells you what's in it and what the security issues are.
Need best practice kind of document for building containers for internal openshift deployment
FAS ids of people interested in collaborating on the openshift apps best practices/standards: humaton, zlopez, smiller, dkirwan, abompard, lachmanfrantisek, lsm5, mohanboddu
infra sig group and the packages in it
First step: culling the packages, Eg. stuff that isn't built for el9.
Only kevin updates it
lots of things can be removed
maybe remove some people from the group?
python sig has taken some of the packages, maybe take more?
run our own pypi mirror instead of making rpms? cert for govt. less impressed by this idea
Packit folks (lachmanfrantisek) are willing to help with the (mass) onboarding for the suitable packages (just provide a list of packages)
releng packages
move to be owned by infra-sig group (not the same as above repo)
proxies
Apache-httpd doesn't have HTTP/3 support.
do we just give all the content to a CDN
Ask someone/group in RH what we should do
Just do nothing? :)
AWS more managable
Our acct. is used by us, centos, openQA, etc.
All of the setup is done manually … but after setup can be ansibled. Ansible is capable of doing the AWS setup.
Terraform maybe helps a lot
onboarding
Kevin has done some work done, but not committed/pushed
Less community involvement/pipeline for new sysadmins than we used to have
Market who the current infra. people are, and how you can speak to them
People want to help a little bit, but don't want to touch the servers when they'll be doing it once or twice a year.
Problem between Kevin just doing easy fixes, and they aren't there for new people, or them not being done for a month.
Hello days (Eg. after every release)
openshift apps. intro
Three clusters, communishift (anything goes, AWS); staging (deploy by ansible, VLAN); production
refactor playbooks, don't start builds move to deployment objects
gitforge
Should change the processes to use gitforge features (on push do X, on branch do Y, etc.)
retire wiki
user documentation is on wiki and doesn't fit with current docs
get a list of categories for wiki pages, then can migrate some of those categories
gitforge might help
datagrepper in Communishift
dump data weekly into an amazon postgres instance?
Robbert is going to use the public dumps and stand up a new instance somewhere.
ARA
Just use AWX instead? Might be much easier to deploy though.
moving AWX forward
problems with current split of ansible repos. for public/private info.
migrate to ansible Vault or something else that is more AWX compatible?
Does reporting, so don't need ARA?
Using Vault means we can have a dummy set of variables to do CI on changes.
Zabbix
Setup 2nd matrix channel so that zabbix can complain without spamming the main channel
Still problems with network connectivity, some VPN issues, potential RH firewall ports need opening
Porting custom nagios alerts is still undecided.
Stop warning for CPU on builders etc.
More levels than nagios, just message on the big ones?
-
Any changes
Be notified of any changes
-
Mention me
Be notified of mention me
-
Unsubscribe
Subscribe