HackMD
  • Prime
    Prime  Full-text search on all paid plans
    Search anywhere and reach everything in a Workspace with Prime plan.
    Got it
      • Create new note
      • Create a note from template
    • Prime  Full-text search on all paid plans
      Prime  Full-text search on all paid plans
      Search anywhere and reach everything in a Workspace with Prime plan.
      Got it
      • Sharing Link copied
      • /edit
      • View mode
        • Edit mode
        • View mode
        • Book mode
        • Slide mode
        Edit mode View mode Book mode Slide mode
      • Note Permission
      • Read
        • Only me
        • Signed-in users
        • Everyone
        Only me Signed-in users Everyone
      • Write
        • Only me
        • Signed-in users
        • Everyone
        Only me Signed-in users Everyone
      • More (Comment, Invitee)
      • Publishing
        Everyone on the web can find and read all notes of this public team.
        After the note is published, everyone on the web can find and read this note.
        See all published notes on profile page.
      • Commenting Enable
        Disabled Forbidden Owners Signed-in users Everyone
      • Permission
        • Forbidden
        • Owners
        • Signed-in users
        • Everyone
      • Invitee
      • No invitee
      • Options
      • Versions and GitHub Sync
      • Transfer ownership
      • Delete this note
      • Template
      • Save as template
      • Insert from template
      • Export
      • Dropbox
      • Google Drive
      • Gist
      • Import
      • Dropbox
      • Google Drive
      • Gist
      • Clipboard
      • Download
      • Markdown
      • HTML
      • Raw HTML
    Menu Sharing Create Help
    Create Create new note Create a note from template
    Menu
    Options
    Versions and GitHub Sync Transfer ownership Delete this note
    Export
    Dropbox Google Drive Gist
    Import
    Dropbox Google Drive Gist Clipboard
    Download
    Markdown HTML Raw HTML
    Back
    Sharing
    Sharing Link copied
    /edit
    View mode
    • Edit mode
    • View mode
    • Book mode
    • Slide mode
    Edit mode View mode Book mode Slide mode
    Note Permission
    Read
    Only me
    • Only me
    • Signed-in users
    • Everyone
    Only me Signed-in users Everyone
    Write
    Only me
    • Only me
    • Signed-in users
    • Everyone
    Only me Signed-in users Everyone
    More (Comment, Invitee)
    Publishing
    Everyone on the web can find and read all notes of this public team.
    After the note is published, everyone on the web can find and read this note.
    See all published notes on profile page.
    More (Comment, Invitee)
    Commenting Enable
    Disabled Forbidden Owners Signed-in users Everyone
    Permission
    Owners
    • Forbidden
    • Owners
    • Signed-in users
    • Everyone
    Invitee
    No invitee
       owned this note    owned this note      
    Published Linked with GitHub
    Like BookmarkBookmarked
    Subscribed
    • Any changes
      Be notified of any changes
    • Mention me
      Be notified of mention me
    • Unsubscribe
    Subscribe
    # EPEL-8 Production Layout ## TL; DR: 1. EPEL-8 will have a multi-phase rollout into production. 2. EPEL-8.0 will build using existing grobisplitter in order to use a 'flattened' build system without modules. 3. EPEL-8.1 will start in staging without grobisplitter and using default modules via mock. 4. The staging work will allow for continual development changes in koji, 'ursa-prime', and MBS functionality to work without breaking Fedora 31 or initial EPEL-8.0 builds. 5. EPEL-8.1 will look to be ready by November 2019 after Fedora 31 around the time that RHEL-8.1 may release (if it uses a 6 month cadence.) ## Multi-phase rollout [As documented elsewhere](https://smoogespace.blogspot.com/2019/06/epel-8.html), EPEL-8 has been slowly rolling out due to the many changes in RHEL and in the Fedora build system since EPEL-7 was initiated in 2014. Trying to roll out an EPEL-8 which was 'final' and thus the way it always will be was too prone to failure as we find we have to constantly change plans to match reality. We will be rolling out EPEL-8 in a multi-phase release cycle. Each cycle will allow for hopefully greater functionality for developers and consumers. On the flip side, we will find that we have to change expectations of what can and can not be delivered inside of EPEL over that time. Phases: 1. 8.0 will be a 'minimal viability'. Due to unshipped development libraries and the lack of building replacement modules, not all packages will be able to build. Instead only non-modular RPMs which can rely on only 'default' modules will work. Packages must also only rely on what is shipped in RHEL-8.0 BaseOS/AppStream/CodeReadyBuilder channels versus any 'unshipped -devel' packages. 2. 8.1 will add on 'minimal modularity'. Instead of using a flattened build system, we will look at updating koji to have basic knowledge of modularity, use a tool to tag in packages from modules as needed, and possibly add in the Module Build System (MBS) in order to ship modules. 3. 8.2 will finish adding in the Module Build System and will enable gating and CI into the workflow so that packages can tested faster. Due to the fact that the phases will change how EPEL is produced, there may be need to be mass rebuilds between each one. There will also be changes in policies about what packages are allowed to be in EPEL and how they would be allowed. ## Problems with koji, modules and mock If you are wanting to build packages in mock, you can set up a lot of controls in ``/etc/mock/foo.cfg`` which will turn on and off modules as needed so that you can enable the ``javapackages-tools`` or ``virt-devel`` module so that packages like ``libssh2-devel`` or ``javapackages-local`` are available. However koji does not allow this control per channel because it is meant to completely control what packages are brought into a buildroot. Every build records what packages were used to build an artifact and koji will create a special mock config file to pull in those items. This allows for a high level of auditability and confirmation that the package stored is the package built, and that what was built used certain things. For building an operating system like Fedora or Red Hat Enterprise Linux (RHEL), this works great because you can show how things were done 2-3 years later when trying to debug something else. However when koji does not 'own' the lifecycle of packages this becomes problematic. In building EPEL, the RHEL packages are given to the buildroot via external repositories. This means that koji does not fully know the lifecycle of the packages it 'pulls' in to the buildroot. In a basic mode it will choose packages it has built/knows about first, then packages from the buildroot, and if there is a conflict from external packages will try to choose the one with the highest ``epoch-version-release-timestamp`` so that only the newest version is in. (If the timestamp is the same, it tends to refuse to use both packages). An improvement to this was adding code to ``mergerepo`` which allows for dnf to make a choice on which packages to use between repositories. This allows for mock's dnf to pull in modules without the repositories having been mangled or 'flattened' as with grobisplitter. However, it is not a complete story. For DNF to know which modules to pull in it needs to set an environment variable for the platform (for fedora releases it is something like f30 and for RHEL it is el8). Koji doesn't know how to do this so the solution would be to set it in the build systems ``/etc/mock/site-defaults.cfg`` but that would affect all builds and would cause problems for building Fedora on the same build system. ## Grobisplitter A second initiative to deal with building with modules was to try and take modules out of the equation completely. Since a module is a virtual repository embedded in a real one, you should be able to pull them apart and make new ones. [Grobisplitter](https://pagure.io/puiterwijk/grobisplitter) was designed to do this to help get CentOS-8 ready and also allow for EPEL to bootstrap using a minimal buildset. While working on this, we found that we needed also parts of the '--bare' koji work because certain python packages have the same src.rpm name-version but different releases which koji would kick out. Currently grobisplitter does not put in any information about the module it 'spat' out. This will affect building when dnf starts seeing metadata in individual rpms which says 'this is part of a module and needs to be installed as such'. ## Production plans We are trying to determine which tool will work better long term in order to make EPEL-8.0 and EPEL-8.1 work. ### EPEL-8.0 | Start Date | End Date | Work Planned | Party Involved | | ---------- | ---------- | ------------ | -------------- | | 2019-07-01 | 2019-07-05 | Lessons Learned | Smoogen, Mohan | | 2019-07-01 | 2019-07-05 | Documentation | Smoogen | | 2019-07-08 | 2019-07-12 | Release Build work | Mohan, Fenzi | | 2019-07-08 | 2019-07-12 | Call for packages | Smoogen | | 2019-07-15 | 2019-07-19 | Initial branching | Mohan, Dawson | | 2019-07-22 | 2019-07-31 | First branch/test | Dawson, et al | | 2019-08-01 | 2019-08-01 | EPEL-8.0 GA | EPEL Steering Committee | | 2019-08-01 | 2019-08-08 | Lessons Learned | Smoogen, et al | | 2019-08-01 | 2019-08-08 | Revise documentation | Smoogen, et al | | 2019-09-01 | 2019-09-01 | Bodhi gating turned on | Mohan | #### EPEL-8.0 Production Breakout 1. **Lessons Learned.** Document the steps and lessons learned from the previous time frame. Because previous EPEL spin-ups have been done multiple years apart, what was known is forgotten and has to be relearned. By capturing it, we hope that EPEL-9 does not take as long. 2. **Documentation.** Write documents on what was done to set up the environment and what is expected in the next section (how to branch to EPEL-8, how to build with EPEL-8, dealing with unshipped packages, updated FAQ) 3. **Call for Packages** This will be going over the steps that packagers need to follow to get packages branched to EPEL-8. 4. **Release Build Work.** This is setting up the builders and environment in production. Most of the steps should be repeats of what was done in staging with additional work done in bodhi to have signing and composes work 5. **Initial Branching.** This where the first set of packages are needed to be branched and built for EPEL-8: epel-release, epel-rpm-macros, fedpkg-minimal, fedpkg (and all the things needed for it). 6. **First Branch** Going over the various tickets for EPEL-8 packages, a reasonable sample will be branched. Work will be done with the packagers on problems they find. This will continue as needed. 7. **EPEL-8.0 GA** Branching can follow normal processes to get done. 8. **Lessons Learned.** Go over problems and feed into other groups backlogs. 9. **Documentation** Update previous documents and add any that were found to be needed. ### EPEL-8.1 | Start Date | End Date | Work Planned | Party Involved | | ---------- | ---------- | ------------ | -------------- | | 2019-07-01 | 2019-07-05 | Lessons Learned | Fenzi, Contyk, et al | | 2019-07 | ???? | Groom Koji changes needed | ??? | | 2019-07 | ???? | Write/Test Koji changes needed | ??? | | 2019-07 | ???? | Non-modular RPM in staging | ??? | | 2019-07 | ???? | MBS in staging | ??? | | 2019-08? | ???? | Implement Koji changes? | ??? | | 2019-08? | ???? | Implement bodhi compose in staging? | ??? | | 2019-09? | ???? | Close off 8.1 beta | ??? | | 2019-09? | ???? | Lessons learned | ??? | | 2019-09? | ???? | Begin changes in prod? | ??? | | 2019-10? | ???? | Open module builds in EPEL | ??? | | 2019-11? | ???? | EPEL-8.1 GA | EPEL Steering Committee | | 2019-11? | ???? | Lessons Learned | ??? | | 2019-11? | ???? | Revise documentation | ??? | #### EPEL-8.1 Production Breakout This follows the staging and production of the 8.0 with additional work in order to make working with modules work in builds. Most of these dates and layers need to be filled out in future meetings. The main work will be adding in allowing a program code-named 'Ursa-Prime' to help build non-modular rpms using modules as dependencies. This will allow for grobisplitter to be replaced with a program that has long term maintenance.

    Import from clipboard

    Advanced permission required

    Your current role can only read. Ask the system administrator to acquire write and comment permission.

    This team is disabled

    Sorry, this team is disabled. You can't edit this note.

    This note is locked

    Sorry, only owner can edit this note.

    Reach the limit

    Sorry, you've reached the max length this note can be.
    Please reduce the content or divide it to more notes, thank you!

    Import from Gist

    Import from Snippet

    or

    Export to Snippet

    Are you sure?

    Do you really want to delete this note?
    All users will lost their connection.

    Create a note from template

    Create a note from template

    Oops...
    This template is not available.


    Upgrade

    All
    • All
    • Team
    No template found.

    Create custom template


    Upgrade

    Delete template

    Do you really want to delete this template?

    This page need refresh

    You have an incompatible client version.
    Refresh to update.
    New version available!
    See releases notes here
    Refresh to enjoy new features.
    Your user state has changed.
    Refresh to load new user state.

    Sign in

    Forgot password

    or

    By clicking below, you agree to our terms of service.

    Sign in via Facebook Sign in via Twitter Sign in via GitHub Sign in via Dropbox

    New to HackMD? Sign up

    Help

    • English
    • 中文
    • Français
    • Deutsch
    • 日本語
    • Español
    • Català
    • Ελληνικά
    • Português
    • italiano
    • Türkçe
    • Русский
    • Nederlands
    • hrvatski jezik
    • język polski
    • Українська
    • हिन्दी
    • svenska
    • Esperanto
    • dansk

    Documents

    Tutorials

    Book Mode Tutorial

    Slide Mode Tutorial

    YAML Metadata

    Contacts

    Facebook

    Twitter

    Feedback

    Send us email

    Resources

    Releases

    Pricing

    Blog

    Policy

    Terms

    Privacy

    Cheatsheet

    Syntax Example Reference
    # Header Header 基本排版
    - Unordered List
    • Unordered List
    1. Ordered List
    1. Ordered List
    - [ ] Todo List
    • Todo List
    > Blockquote
    Blockquote
    **Bold font** Bold font
    *Italics font* Italics font
    ~~Strikethrough~~ Strikethrough
    19^th^ 19th
    H~2~O H2O
    ++Inserted text++ Inserted text
    ==Marked text== Marked text
    [link text](https:// "title") Link
    ![image alt](https:// "title") Image
    `Code` Code 在筆記中貼入程式碼
    ```javascript
    var i = 0;
    ```
    var i = 0;
    :smile: :smile: Emoji list
    {%youtube youtube_id %} Externals
    $L^aT_eX$ LaTeX
    :::info
    This is a alert area.
    :::

    This is a alert area.

    Versions

    Versions and GitHub Sync

    Sign in to link this note to GitHub Learn more
    This note is not linked with GitHub Learn more
     
    Add badge Pull Push GitHub Link Settings
    Upgrade now

    Version named by    

    More Less
    • Edit
    • Delete

    Note content is identical to the latest version.
    Compare with
      Choose a version
      No search result
      Version not found

    Feedback

    Submission failed, please try again

    Thanks for your support.

    On a scale of 0-10, how likely is it that you would recommend HackMD to your friends, family or business associates?

    Please give us some advice and help us improve HackMD.

     

    Thanks for your feedback

    Remove version name

    Do you want to remove this version name and description?

    Transfer ownership

    Transfer to
      Warning: is a public team. If you transfer note to this team, everyone on the web can find and read this note.

        Link with GitHub

        Please authorize HackMD on GitHub

        Please sign in to GitHub and install the HackMD app on your GitHub repo. Learn more

         Sign in to GitHub

        HackMD links with GitHub through a GitHub App. You can choose which repo to install our App.

        Push the note to GitHub Push to GitHub Pull a file from GitHub

          Authorize again
         

        Choose which file to push to

        Select repo
        Refresh Authorize more repos
        Select branch
        Select file
        Select branch
        Choose version(s) to push
        • Save a new version and push
        • Choose from existing versions
        Available push count

        Upgrade

        Pull from GitHub

         
        File from GitHub
        File from HackMD

        GitHub Link Settings

        File linked

        Linked by
        File path
        Last synced branch
        Available push count

        Upgrade

        Danger Zone

        Unlink
        You will no longer receive notification when GitHub file changes after unlink.

        Syncing

        Push failed

        Push successfully