= Introduction

Currently Fedora riscv efforts are using a external koji hub/builders at http://fedora.riscv.rocks/koji

As the next step toward being a full arch in fedora/eln/centos stream the next step is to move to a koji hub managed the same way the primary one is in Fedora Infrastructure. We still cannot use local builders as the currently available hardware isn't up to enterprise hosting/use, but it should be before moving to primary

= secondary arch scope

Ideally we get everything 100% like it is on primary. This will make it more likely/practical to merge to primary. We don't have to do everything all at once however, we can start out with laxer rules and work toward primary.

> [name=davidlt] One example would be koji hub and builder default configuration (maxjobs, timeout values, repo lifetime, etc.) Simply put we need more time as we are slowest of all.
>> [name=kevin] Yeah, we can conditionalize the config for this using ansible templates. ie, primary does A and if riscv do B

> [name=davidlt] We might want to have different (or additional) channels. Our builder have different core-memory ration, different core count, different ST performance. Today we are manually assigned builds to specific machines to save tons of time, avoid timeouts (doesn't require spec change), or just to avoid running out of memory. Koji as-is has now smart scheduling based on what resources historically were needed to build a package. "Capacity" and "Weight" doesn't help especially and even hurts us with default 1.5 value for a new builds. We probably should have channels for board types (Unmatched, VF2, etc.) and maybe additional ones for high-memory, low-memory, core-memory ratio, etc. I do want to avoid sending the largest builds to smallest board situation. Today that's done manually.
>> [name=kevin] Yeah, I completely understand that. We could continue to manually do things I guess, but it would be nice to try and figure channels, etc.
>>> [name=davidlt] Adding channels is an easy part, but we would need to look at Koji CLI tools to allow specifying a channel.
>>>> [name=kevin=] well, what channel is chosen is a hub policy thing. It could be done by package name or submitter name or tag or whatever. So perhaps we could come up with a method that works by using all those things.

We are already having to use external builders, possibly a git overlay, and possibly other things, but those should reduce/go away with time/effort.

= timeline

2024-09 - kevin installed bvmhost-x86-riscv01 host in a seperate vlan in iad2
2024-10-09 ish - kevin to setup hub / db / builders

= policy questions

== git overlay - Allowing divergence from primary fedora source. We ideally want to get rid of this, but perhaps we need it longer?

> [name=davidlt] Very likely we cannot make it in time before the switch. We could use the upstream dist-git if we are allowed to have our own branches similar to what we do in dist-git overlay (`fXY-riscv64`). We could have a git hook to ensure that once the main branch is modified it's auto-merged into riscv64 branch, or riscv64 branch gets rebased. Notification (bot or/and email) is generated to riscv64 group if it fails (required manual human intervention). I am suggesting this because I noticed that folks tend to have private branches in dist-git too. Might require a policy change (I didn't check the current rules).
>> [name=kevin] it's possible, but messy. You cannot remove branches ever that were used for a build. I think if we wanted to do this it would start with a discussion on devel list. Some maintainers really won't like it. If we use forks on src.fp.o, it's a bit more seperated, but less like mainline
>>> [name=davidlt] Forks work, it's a bit annoying as there will be no single place to reference those. One would have to query Koji for build information to finda proper fork. Maybe have a user like Packit? https://src.fedoraproject.org/user/packit If we have a group assigned, could multipe folks access/modify/fork under it?
>>> [name=kevin] sadly, I don't think so. groups only give people ability to commit to packages the group has permissions on. It would indeed be messy to have those spread around, let me think on it.

> [name=davidlt] Another issue is bumping NVR if we need to rebuild because we aren't building in the same order as upstream. We usually don't need large scale rebuilds, but targetted ones. When SOVERSION bumps happen in side tags maintainer will not wait for us (and we don't jump working on those ASAP too). Also sometimes side-tags get deleted extremely quickly before I get to export their build history. Some (if not all) of that is recoverable checking Bodhi updates and Koji (quering history of specific folks that were working on that Bodhi update). Majority of `fXY-riscv64` branches are for NVR bumps where we add `.rvreX` before `%{?dist}`. We don't have a seperate database/file to track riscv64 specific things. We place that information in NVR (unique, not used by upstream Koji).
>> [name=kevin] This is not great, but I understand the constraint. I wonder if we could use the new koji 'draft builds' for this. ie, do a draft build, make sure it's built against the right things and promote it. If it's not, do another draft with the soname bump or whatever, etc.
>>> [name=davidlt] I am not sure it helps. Today we go wide & parallel to achieve high number of builds. SOVERION bumps are announced (but not always) on the mailing list. In these cases I typically slow down, drain the queue and handle non-overlapping SOVERSION bumps on the main tag without any side tags. Once in a while there are still mistakes that require NVR bumping. Maybe we should use the existing method for now? Fork, bump, rebuild without asking to bump upstream NVR.

> [name=davidlt] The both items above (merges/rebases and NVR bumps (commits)) have their own issues if `rpmautospec` is used. Sometimes I am forced to rebase as `rpmautospec` fails because there is a merge. `rpmautospec` naturally will bump `Release:`. For example, upstream `firefox-131.0.2-1.fc40` and if rebuilt by us it would become `firefox-131.0.2-2.rvre0.fc40` IIRC. That's because `rpmautospec` is looking at commit count for `Release:`. Not a major issue, but could confuse some folks. It would be better if we could trigger a rebuild of a package in general, but I assume not everyone could like it. We typically ask folks if they could bump NVR and most do that without much discussion.

== builders - since they are external we need to make sure they are run by trustworthy folks.

> [name=davidlt] We don't have Koji Shadow, and we need to figure how we progess on how things are built.
>> [name=kevin] we might look at https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/distrobuildsync which is used by centos and eln

> [name=davidlt] Composes should we disabled initially, or should be only could images + containers (would be nice to have them in the registry) as those use Kiwi-NG. It's very fast, works on riscv64. I doubt we can deliver images like Workstation beased on Image Builder. I did some work last year, got some ISOs, but never finished full Pungi pipeline. See: http://fedora.riscv.rocks/compose/rawhide/latest-Fedora-Rawhide/compose/Server/riscv64/iso/ I didn't upstream it (change to Pugi configuration, productmd, anaconda, etc.). It also required some hacks. The problem is that Pungi needs to boot VM, run anoconda, and then mount image with libguestfs and execute RPM to dump installed packages. Funny things that this information is available in anaconda logs. This fails becuase `binfmt_misc` just didn't work via libguestfs IIRC. Without KVM this is extremely expensive natively on riscv64, and it triggers way too many issues and timeouts in too many places to ever get somewhere useful. I hacked `kojid` to force it go on `x86_64` as Image Builder uses `libvirt` to spin the VM as it doesn't need to be natively on riscv64. This is how I managed to get those ISOs generates, but again next steps are annoying too.
>> [name=kevin] Yeah, we should focus on kiwi. primary is going to be moving more to it anyhow.

== importing into secondary koji - what do we import and were do we focus?

> [name=kevin] I think we should just import rawhide/f42 and try and keep that up to date, but perhaps f41 is also worth importing?

> [name=davidlt] We have to import a stable branch too that will be used by the builders. If we deliver working F41 images soonish we can ignore F40 branch. We haven't started F42 efforts yet, but we can catch up before the import.

== Infrastructure logistics

> [name=kevin] I plan to make a sysadmin-riscv fas group. This group would allow ssh access/sudo on the koji hub/composers/buildvm's/db as well as ability to run playbooks to change configuration (after pr's are reviewed/merged)

>[name=kevin] I hope to use kerberos auth/keytabs for builders. Hopefully with one of the above machines being able to pull them locally so they can be securely distributed to builders owners. Or alternately we may be able to allow specific users to get those keytabs from ipa and we just need to setup the host/service in ipa and ask them to download. There will be a keytab per host/service.