owned this note
owned this note
Published
Linked with GitHub
# Deployers Team Meeting
Triage Query:
https://github.com/pulp/pulp-oci-images/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-Needed
https://github.com/pulp/pulp-operator/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-Needed
## Pending actions items
[mikedep333] to look into long startup time
[dkliban] to file docs issue about using systemd to manage your containers
## March 21, 2023
* Still working on wrapping operator in clowder
* old versions aren't publishing, looking into it
## March 14, 2023
* Status of getting pulp on clowder
* Determined we cannot use multi-process container
* Did a sample "hello" app
* going to verify a route
* Haven't decided whether to pursue the operator or "mutliple containers without operator" yet
* whether we can create something like routes to replace webserver container is tbd
* Migration to golang operator
* backup/restore PR is open, awaits AAP review
* support AAP CEE
* need to figure out priority for https://issues.redhat.com/browse/AAP-5094
* status emails for staging environment
* they *should* (IETF definition) be brief
## March 7, 2023
* pulp-operator support
* k8s? openshift?
* what do we claim and what we want to claim
* Current CI we have?
* Pre-merge: regualr 8ks
* nightly schedule: openshift org CI (for pre-merge, use crc on your laptop)
* What openshift-specific features do we have?
* routes and the elimination of the webserver - solves some big problems
* scc (openshift 4.12) - you cannot run a pod in openshift without an scc, we do not want to run containers as root. regular k8s has a "pod secuirty context", which is like a limited version of SCC.
* OLM - catalog to install the operator. OLM can be installed on regular k8s, but does not come by default.
* In the future, openshift monitoring / logging
* Agreed: In the medium-term, try to CI test against ephemeral / staging
* platform team will provide jenkins
* Agreed: In the short-term, we manually deploy against ephemeral / staging
* 3.22 images getting pushed by both 3.22 branch and latest branch
* [PR](https://github.com/pulp/pulp-oci-images/pull/456) will stop building 3.22 branch
* We should rebase the 3.22 branch after 3.23 release
* Decko mentioned that he needs to customize the s6 service files
* Perhaps I should look into refactoring the s6 service files to make this less burdensome?
* Clowder deployment
* https://github.com/pulp/pulp-clowder-deployments/issues/1
## February 28, 2023
* pulp-oci-images CI code deduplication is merged
* minor and micro tags for all images now
* periodic-ci-pulp-pulp-operator-main-deploy-pulp-on-openshift failures in our slack channel
* have not been updated since moving to golang operator
* CI is not building old branches (properly)
* working on updating multi-process container section of website
* How much should I mention about other container solutions?
* Researching more into getting HCaaS to staging
* multiple stakeholders desire to start testing soon
* Auto-scaling info sharing
* opentelemetry-instrument command?
* bmbouter advised me on the correct pip packages to install. That fixed the command error.
## February 21, 2023
* I do not have notes from the meeting that included discussing the staging approach (clowder vs extending-our-operator)
* clowder does use deployment templates though
* I believe Pete Savage was the main person I talked to
* auto-scaling will be a required feature very early, probably before we go into production
* Partially because on the days of product releases, Red Hat traffic will increase greatly
* Let's add documentation about https://www.redhat.com/sysadmin/quadlet-podman
## February 14, 2023
* Want to discuss the design of solving [this](https://github.com/pulp/pulp-oci-images/issues/361#issuecomment-1423240451)
* Pulp Operator scope of responsibilities
* openshift-specific routes needed to fix issues / improve performance by eliminating the webserver
* AH requires multiple openshift-specific features
* It makes sense for downstream to make the operator cluster-scoped rather than namespace-scoped
* HCaaS will require a pod auto-scaler, this is openshift-specific
* nightly building/publishing multi-process images [is failing](https://github.com/pulp/pulp-oci-images/actions/runs/4169508576/jobs/7217545076#step:10:1393) with a curl SSL error during tests, but those tests succeed for PR building
* Should I just fix this during the CI files merge?
* Yes
## February 7, 2023
* released pulp_installer 3.21.5 and 3.22.1 with the token auth key fix
* [mikedep333] Proposed agenda for pulp_installer is:
* pin ansible-lint on 3.21 & 3.22
* fix & re-enable EL9 CI
* Help galaxy fix their [dependency issue](https://issues.redhat.com/browse/AAH-2137) that is breaking release-galaxy
* [mikedep333] Figured out how to make docker-compose forward ports to scaled containers
* What ranges to use on the host? 24816 and 24817 are adjacent to eachother.
* pulp-content on 24716-24816 and pulp-api on 24817-24917
* The 1st pulp-content container will be 24716 rather than 24816, but good enough
* [mikedep333] "CEE enablement notes" was on my to-do list from last week
* follow-up with Tanya (after configmgmtcamp)
* [bmbouter] want to try pulp_ostree in ephemeral environment
* hcaas ephemeral environment instructions: https://gist.github.com/mikedep333/7ace9690bba6a70875e1bfd5af608234
* pulp_ostree is not in the GA image, it's in the nightly image so use that
* talked with Christian about the migration and tests
* let him aware that conversion webhook has a limitation of "installMode: AllNamespaces", so he agreed on not using it
* explained about the plan B (manually implement the conversions and run it in a patch release) and opened a draft for it
* we talked with jlanda and rcarrillocruz to hear their opinion on this and they said that they will check it this week
* started to work on plan C
* do not do any modifications in ansible version
* adapt golang version to allow the ugprade from ansible, but considering it a "breaking" change
## Feb 1, 2023 Goals and priorities discussion
* Goals of this group
* enable users to deploy pulp in an easy way
* be compatible with a broad set of user environments
* address stakeholder needs
* growing upstream community
* visibility
* Stakeholders
* Community
* HCaaS
* AAP
* pulp_installer (maintenance only) (on-prem)
* pulp-operator (on-prem and cloud)
* Katello CI (build team)
* Internal PyPI (possibly inactive)
* Known needs/ongoing efforts
* [to discuss] Sunsetting pulp_installer
* supporting AAP till AAP 2.4 is out
* [Mike P1] pulp-oci-images priorities
* pulp-oci-images has technical debt
* pulp-oci-images is lacking some basic functionality
* pulp-oci-images has a backlog of bugs
* operator has a backlog of bugs
* pulp-oci-image version/tag consistency between pulpcore version and operator version
* impacts CI
* e.g., operator pulls "latest" pulpcore image, but "latest" could be old
* documentation
* Need to make it easier to get started with runnig Pulp
* [Mike/Humberto P2] HCaaS will need a c.rh.c compatible operator (hard deadline - July, preferably April-May?)
* either add functionality to pulp-operator, or develop 2nd operator based on clowder framework
* auto-scaling based on availabity of contentapp, tasking needed
* pulpcore domain compatibility (we will need to investigate the modifications needed on the operator side - for example, should the operator allow to configure different types of storage or multiple object storages? is it possible to configure wildcard path in routes and ingresses? a change in pulp-web/nginx configmap will also be needed?)
* [Humberto P1] make pulp-operator upgrade possible before April (probably) - release of aap 2.4
* in progress
* upgrade = going from AAP operator based on ansible to AAP operator based on go
* compose lacks many features and features require refactoring (low priority, it's more of an example file than a broadly compatible deployment method)
* AAP HA knowledge sharing with productization team
* Questions
* use pulp-operator ourselves?
* maybe CI
* Capacity
* Mike & Humberto - 100%
* Dennis - light to no participation
## February 2, 2023
* Top priorities for pulp-oci-images are:
* combine CI files and [fix the image tagging](https://github.com/pulp/pulp-oci-images/issues/323) so all images have a 3.y and 3.y.z tags
* [Document building with containerfiles](https://github.com/pulp/pulp-oci-images/issues/269)
* Traceback bugs
## January 31, 2023
* pulp_installer 3.22.0 released
* katello CI will use pulp_installer 3.22 for older pulpcore releases, no need to maintain 3.18 to 3.21 branches
* [mikedep333] working on pulp_installer token auth issue
* more tests and adjustments for the operator migration/upgrade
* investigation and fix of operator CI issues
## January 24, 2023
* [mikedep333] Finished fixing several issues in pulp_installer, 3.22.0 release PR is ready
* 3.22 images are released
* [dkliban] will follow up with x9c4 about branch protection rules on the pulp-oci-images repository
## January 17, 2023
* [mikedep333] Fixed/worked around 2 issues in pulp_installer CI, working on 3rd
* [dkliban] status of latest image builds? I saw x9c4 posted https://github.com/pulp/pulp-oci-images/actions/runs/3932927873/jobs/6735199102
* we have an issue filed about long start up time https://github.com/pulp/pulp-oci-images/issues/318
* [dkliban] operator permissions problems?
* need to use Operator Lifecycle manager to install the operator
* Operator needs to discover the hostname which is a cluster level permission. If we remove this users will need to provide the hostname manually.
* Cluster level permissions are needed to setup ingres which allows access to Pulp from outside the cluster
* Some items from AAP
* https://issues.redhat.com/browse/AAP-7795
* https://issues.redhat.com/browse/AAP-8344
* https://github.com/pulp/pulp-oci-images/issues/402
* [ttereshc] goals and priorities (can be discussed last if we have any time left)
## January 10, 2023
* image publishing failures https://github.com/pulp/pulp-oci-images/actions
* issue with `dnf install` on centos 8
* dkliban to write it up
* Is a manual image build/publish needed when a new version of pulpcore released?
* https://docs.pulpproject.org/pulp_oci_images/developer-instructions/
* Working on reproducing https://github.com/pulp/pulpcore/issues/3492
* Seems like a permissions issue
* About to start on fixing the issue to copy the token auth key https://issues.redhat.com/browse/AAP-7795?
* https://github.com/pulp/pulpcore/issues/3492
* Mike / Humberto to create meeting to explain OCP templates
## Dec 20 Agenda
* Status of HCaaS ephemeral environment
* A lot of work was done on permissions for the operator
* Using the OCP templates right now instead
* Only plugin needed currently is RPM
* Looks like every content plugin, not just some, will require minio. Because there are multiple pods.
* Looking into whether we can (organizationally) use the openshift "projects"
## Dec 13 Agenda
* Plan for HCaaS ephemeral environment:
* jsherrill will be given cluster admin rights, then deploy the pulp-operator CRD
* Each dev will run `bonfire` to create an ephemeral namespace
* Each dev will run `oc apply` to deploy each individual pulp-operator deployment
* mikedep333 to make any needed changes to pulp-operator for it to be applied successfully
* Persistent volumes will likely need additional rights. Mike to test them after CRD is deployed.
* NOTE: We cannot use object storage until production, and an ephemeral "emptyDir" is incompatible with pulp-operator's current design of having multiple pods.
## Dec 6 Agenda
* [PR up](https://github.com/pulp/pulp-oci-images/pull/393) with migration instructions (many different scenarios)
* Need someone to work on pulp-oci-images while Mike works on the ephemeral environment for crhc
* Humberto will, availability permitting
# Actions Items
* [dkliban] write up issue about migration (pulp_installer -> pulp_oci_images) docs
* steps predicted:
* chown/chgrp to new uid/gid
* database upgrade (if not already postgres 13)
## Nov 29 Agenda
* Unimplemented features/docs of single container vs pulp_installer
* docs for signing service scripts (issue filed)
* docs for keys/certs (issue filed)
* variables like gunicorn workers (issue filed)
* agreed: do these as soon as migration docs are done
* Continue providing Apache snippets now that pulp_installer is EOL?
* None of our deployers use Apache.
* Foreman uses Apache. Do they use our snippets?
* Because of containers, every user is going to have several plugins in their database schema, and we are stuck maintaining those plugins forever
* agreed: Make a post about this on discourse
* Separate image names for the complete list of plugins?
* x9c4 had some thoughts on this.
* agreed: Let's get more feedback
* We should decrease our image sprawl by implementing https as a variable (rather than a tag) 1st
## Nov 23 Agenda
* need to file an issue for upgrade docs on going from pulp_installer to container based install
* need to write a post on discourse announcing the stop of development of pulp_installer post 3.22
* pulp-operator triage at future meetings
## Nov 16 Agenda
* Inconsistency between container images https://github.com/pulp/pulp-oci-images/pull/336/files
* [mikedep333] filed [an issue](https://github.com/pulp/pulp-oci-images/issues/378) to move this to the base image
* postgres upgraded in single container to 13. We need to make sure that neither pulp-operator nor single-container upgrade it without the other doing so.
* New image names for docs "single-process" vs "multi-process"
* we are going to do a pulp_installer 3.22
## pulp_installer fate meeting
* we have agreement that upstream we can stop support for pulp_installer
* We need to work with all of the stakeholders to find a path forward with them.
* AAP Installer wraps around pulp_installer
* Katello uses it in their build pipeline
* RHUI customizes it and uses it as their installer
* Mike will work on the upgrade path
* Dennis will reach out to Sat build team
* CI seems to take longer since multi-arch
* Don't do multi-arch builds in PR CI?
## Nov 2 Agenda
* status of compose development
* How to announce breaking changes, or potentially breaking changes?
* Put in development section of discourse?
* Agreed: Ask in open floor
## Oct 26 Agenda
* pulp-operator PR for the new images: https://github.com/pulp/pulp-operator/pull/698
* fao89's comment about not testing s6 images with the go operator
* agreed: Only test the s6 images with the ansible operator. Since the Go operator will be a breaking change, and we do not want to create unnceccessary CI tests.
* galaxy / galaxy-minimal image vs pulp-galaxy-ng image
* ansible core CI (ansible/ansible on github) uses this image, but under the ansible namespace instead of pulp
* Agreed: consolidate pulp-galaxy-ng to galaxy
## Oct 19 Agenda
* End of lifing the installer
* Need to include more people in the decision-making process
* provide an upgrade path to containers
* https://github.com/pulp/pulp-oci-images/pulls
## Oct 12 Agenda
* [mikedep333] Been spending almost all my time fixing the CI, including for various repos
* Includes some python test code
* Need to check out the remainder of the repos
* Could not reproduce the "double service" issue in oci-env
* click env issue in single container is cluttering up CI logs
## Sept 28 Agenda
* Single container "production ready" coming along steadily
* Made the decision to put the pulp process on a limited UID
* ansible-lint keeps on getting stricter
* RHEL support the installer (with repos to be enabled) was broken for months
## Sept 14 Agenda
* Just found out about [Azure Pipelines](https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml)
* Has Mac support
* oci_env needs mac CI
* GHA has Mac support also
* https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
* https://github.com/pulp/oci_env/pull/41
* pulp_installer 3.21.0 released
* About to do a 3.21.1 with fixes for AH
* /etc/pulp/certs permissions fix
* galaxy-ng UI not loading
* working on the pulp-container repair command PR
* Mitogen to speed up ansible
* strategy=free to speed up ansible
* Strongly leaning against it
* Output looks like: https://github.com/pulp/pulp_installer/actions/runs/3010481546/jobs/4836624106#step:14:69
## Sep 7 Agenda
* CentOS 9 and Postgres 13 upgrade in single container
* includes upgrading existing database
* No CI test for this though
* Make discourse post
* [single container issues](https://github.com/pulp/pulp-oci-images/issues) are the "production ready" task list
* Some are marked prio-list. These are those being currently / soon worked on.
* prio-list should be limited to 3 items at all times. Drop existing ones to add new ones.
* Create new mkdocs site for single container
* Includes changelog
* Website docs page will link to it
* Website single container page will have quick start + link
* https://github.com/pulp/pulp-oci-images/issues/241
* Container signing service landed
* Includes cleaning up accidental logic from ansible signing service
* file a bug, galaxy_ng's example script is broken
## Aug 31 Agenda
* Ubuntu 22.04 support landed. Hurray!
* New config format for S6
* Service commands broke, but quickly addressed
* We can now make the oneshot services (formerly initializaiton scripts) depend on the postgres service, instead of having a postgres-down oneshot.
* https://github.com/pulp/pulp-oci-images/issues/235
## Aug 24 Agenda
* Figured out single container's EL7 rootless compatibility issue
* Figured out how to debug the initialization scripts for the single container
* Doing lots of reading S6
* Still having trouble with upgrade to v3, not sure whether to continue doing it or not
* S6 is poorly documented and complex
* "not simple at all"
* Agreed: Use systemd instead of s6 if it doesn't require any container permissions, and if it doesn't slow down container startup
## Aug 17 Agenda
* Merged rocky linux support
* Starting on container docs now, running into possible compatibility issues with old (EL8) podman
* https://pulpproject.org/pulp-in-one-container/
* Should we switch to mkdocs?
* If we need more than 1 page
## Aug 10 Agenda
* Multiple big docs updates
* Finally followed & updated the manual (pypi) [install instructions.](https://github.com/pulp/pulpcore/pull/3058)
* Another uses wants RockyLinux
* Proposal: Only add it to short-running CI tests
## Aug 3 Agenda
* Catching up on issue backlog for pulp_installer
* Some were user support
* One of them improves usability. The prior need to set postgresql_hba_entries.
* pulp-operator being possibly rewritten in golang?
* https://github.com/pulp/pulp-operator/pull/505
* https://docs.google.com/document/d/1j6eanLgVz4HboEslzhjhOGKK4HF-jSSuC2wSC34eVpA/edit
## Jul 27 Agenda
* I finished the special variable cleanup
* In the middle of docs updates
* Worry that the table will be too wide
* In the middle of fixing 3.9 CI
* For last 3.9 release, CI did not run. It seems to not run at all occassionally.
* How to fix the [ulimit bug](https://issues.redhat.com/browse/AAP-4343) on the operator
* ulimit cannot be changed in the container
* cgroups does not prescribe file limits (or other ulimits)
## Jul 20 Agenda
* Making progress on special variable cleanup
* Installer mimics the default behavior of pulpcore settings.py
## Jul 13 Agenda
* How to implement https://issues.redhat.com/browse/AAP-4668
* Agreed: We do not want to implement. Reach out to Yanis to disucss.
* single container documentation improvements
* need similar docs to https://docs.gitlab.com/ee/install/docker.html
* agreed: Focus on single container as primary installation mechanism from here on out
* agreed: Focus on docs updates for single container & pulp_installer, including overhauling single container docks.
* Note: I created https://discourse.pulpproject.org/t/proposal-to-focus-on-single-container-and-improve-docs-for-installers/540 to discuss this further
* [Review usability changes](https://hackmd.io/RXKyTOecQCax128qbuJlXg?edit) before posting to community
* Not entirely reviewed
* Agreed: Do not do the "pip freeze" or "default branch of plugins" because engineering effort should be focused on container.
## Jun 29 Agenda
* [mikedep333] Focusing on the dev env for AH
* Not blocking anyone's work on the usability epic, am I?
* vagrant for the old pulp2/pulp3 combo is broken
* probably won't need it soon because migration plugin will be EOL
* look into whether hostmanager plugin breaks regular VMs
* planning for the usability epic
* desire for the installer to provide a "distribution" of pulp + plugins at specific versions
* start a thread on discourse to discuss the installer providing default z-stream releases for plugins
* updated centos 8stream vagrant images still not available
* Consider https://app.vagrantup.com/generic/boxes/centos8
* Also try reaching out to CentOS via IRC/google chat
## Jun 22 Agenda
* Need to answer these questions: https://issues.redhat.com/browse/AAP-4342
* Only need help answering the question about what bug it will cause
* Ask Ina
* Status of fixing [#1196](https://github.com/pulp/pulp_installer/issues/1196) re-labeling mid-install
* Can't use ansible_facts.mounts because the options are excessively verbose, need to parse /etc/fstab instead
* Status of fix for dev environments
* Merged the fix https://github.com/pulp/pulp_installer/pull/1188
* Fix only affects new provisions, broken users need to run `sudo bash /home/vagrant/.bashrc.d/venv.bashrc`
## Jun 15 Agenda
* Research into other ansible installers:
* ansible-freeipa
* Has 1 role (e.g. "ipaclient") per machine-type, but also has 1 hostgroup (e.g, "ipaclients") per machine type.
* Uses lots of custom python modules, and plugins
* Custom python modules put the server application in the correct state after packages lay down files
* installs RPM/deb packages
* Has an uninstaller (`state|default('present')``)
* But wraps around a binary uninstall command
* Specifies role short name rather than FQCN
* foreman_basic_installer
* Has 1 role (e.g. "foreman_smart_proxy") per machine-type, but also has 1 hostgroup (e.g, "foreman_smart_proxy") per machine type.
* has a group_vars/all.yml.sample, with a long list of variables, that users can copy & modify
* Has an inventory.sample file, which shows the hostgroups
* Calls foreman-installer commands
* Not idempotent, always restarts the webserver service for example.
* AAP
* Each webserver/pulp-host sets the content_origin to its own individual ansible_fqdn.
* We could default it to the 1st host in pulp_webservers
* brightcomputing/installer
* Uses a table to describe variables.
* specifies role's FQCN
* Team agreements:
* Adopt hostgroups for the usability epic
* lint failing on 3.18 branch
* It's as if GHA is like "I'm going to ignore existing lint errors, but not ones that are fixed on the other branch."
* Agreed: Just fix the lint errors
## Jun 8 Agenda
* In the LDAP PR, I managed to make the extras variable usable even when users provide a string rather than a list.
* In the LDAP PR, I did not do a case-insensitive search, even though extras are case-insensitive.
## Jun 1 Agenda
* Addressing another EL7 old RPM-provided jinja2 compatiblity issue
* About to start on the usability epic
* Need links to videos from pulpcon 2021
* https://www.youtube.com/playlist?list=PLwm8_O6oKSS3GEl1NHgtY4G-Tq2byivpa
## May 26 Agenda
* Will discuss with AH whether they would use the webserver role automatically determining the api/content hosts to talk to.
* docs refactoring and updates
* "deployment scenarios" -> clustering"
* "Workflows for versioning" -> "Upgrades and Installing Exact Versions"
* Fedora 36 weirdly failing on idempotence:
https://github.com/pulp/pulp_installer/runs/6607348348?check_suite_focus=true#step:14:31
but it doesn't fail on geerling's repo: https://github.com/geerlingguy/ansible-role-postgresql/pull/215
## May 17 Agenda
* Mike's desire to make vagrant installs no longer build & install the collection
* https://github.com/pulp/pulp_installer/pull/1099
* This conflicts with molecule, which does not build and install the collection. But the installed collection takes precedence over the local repo.
* I have repeatedly run molecule commands, 2 only to have to repeat them a 3rd time after deleting the collection.
* Making this change would require all vagrant users to run `rm -rf ~/.ansible/collections/ansible_collections/pulp/`
* agreed: Make this change, and communicate it well. Devs often use vagrant envs for months.
## May 11 Agenda
* Remaining CentOS 9 work
* vagrant box
* upgrade images
* Additional complexity in implementing webserver support for multiple api/content hosts
* This is basically load balancing
* Load balancing parameters per-host
* Global load balancing parameters
* Proposed design
* pulp_webserver_api_balancing_params:
foo: bar
foo2: bar2
* pulp_webserver_api_servers:
- url: pulp-api-1:24817
parameters:
foo: bar
foo2: bar2
- url: pulp-api-2:24817
parameters:
foo: bar
foo2: bar2
* pulp_webserver_content_balancing_params:
foo: bar
foo2: bar2
* pulp_webserver_content_servers:
- url: pulp-content-1:24816
parameters:
foo: bar
foo2: bar2
- url: pulp-content-2:24816
parameters:
foo: bar
foo2: bar2
* Lets triage open issues
## May 9 Agenda
* Status of 2 big cluster support PRs:
* Cluster CI
* pulp_webserver independence
* dependent on the epel7 PR
* Status of el7 support in packages?
* el7 packages will still be built for pulpcore 3.18 RPMs.
* docs examples not showing up properly https://docs.pulpproject.org/pulp_installer/customizing/
* They're showing up properly now
* Itemized lists are not showing up properly though
* Mike to look into this
* https://github.com/pulp/pulp-operator/pull/387
* https://github.com/pulp/pulp_installer/pull/1042
* Suggestions on renaming / moving: https://docs.pulpproject.org/pulp_installer/customizing/#deployment-scenarios
* How about move to a new page called "cluster examples"?
* Not technically accurate because 1 example is an external postgres/redis, but they could be postgres/redis clusters.
* Suggestions on renaming / moving: https://docs.pulpproject.org/pulp_installer/customizing/#recommended-workflows-for-pulpcore-plugin-versioning
* How about "specifying plugin versions" or "Installing specific plugin versions"
* Running into an issue with my easy-approach-to-settings
* Desire: settings like content_origin get set to "the 1 host that will run pulp-webserver"
* Problem example: content_origin needs to be set for the pulp-api host, but pulp webserver gets deployed afterwards. I cannot determine "the 1 host that will run pulp-webserver", only "the 1 host that has already run pulp-webserver".
* Possible solution: Special group names like pulp_webservers? A host could be in multiple groups. Users would still need to apply the correct roles list to each host.
* agreed: follow up with pavel
## Apr 27 Agenda
* Updated the db fields encryption keys PR to support replacing other hosts' keys with 1 host's key
## Apr 20 Agenda
* Status of database fields key PR
* Satoe messaged me privately with an error, which leads to a design question.
* Should we continue to wait for the database fields key PR to release pulp_installer 3.19.0?
* agreed: Release it beforehand if someone complains.
* 3.18 RPMs
* started being developed, not published on yum.theforeman.org/pulpcore yet
## Apr 13 Agenda
* Removing the firewall https://discourse.pulpproject.org/t/removing-the-firewall-feature-from-pulp-installer/419
* Openshift installer does configure it
* Agreed: Remove it
## Apr 6 Agenda
* Welcome Humberto!
* Figured out how to set the most global of variables
* access once set with hostvars['localhost']['var_name']
* as opposed to the normal way "var_name" which can be also done as: hostvars[inventory_hostname]['var_name']
* set with "set_fact:" and "delegate_facts: True" "delegate_to: localhost" "run_once: True"
## Mar 30 Agenda
* Inconsistency in default config for postgres/redis
* Currently:
* postgres binds to 0.0.0.0
* postgres only permits connections from 127.0.0.1
* redis binds to 127.0.0.1
* What should we default to?
* Accept connections / bind to 0.0.0.0
* Can we configure postgres to allow the other hosts by IP?
* we cannot guess correctly enough which is the correct IP address
* Refactoring pulp_webserver to use the __pulp_database_config_real_sole_host instead of installing pulp-common
* Current status of fixing non-identical database fields key
* Will not do anything about non-identical existing keys but to throw a proper error message in the installer
* Lots of effort involved in picking the correct host to run pulp_database_config.
## Mar 23 Agenda
* Most of the way done on making db encryption keys the same
* Use of run_once / register / debug module to desgignate the primary host
* How to handle non-identical keys already on the cluster?
* Worker nodes need the key too, right?
* Working partial implementation of a cluster for release-dynamic
* Resolved multiple accidental dependencies of pulp_common on pulp_database
* Need to do a release still for the AH signing service
* SELinux updates just merged
* galaxy-importer support in SELinux
* Need test steps
* agreed: reach out to AH
* settings.local.py
* agreed: Put header at the top of settings.py saying to modify settings.local.py instead
* [Settings pulp_user_home should set the entirety of /var/lib/pulp](https://github.com/pulp/pulp_installer/issues/934)
* There is a mismatch between certain variables and certain sub-variables of pulp_settings
* We should look for ways to merge these variables into 1.
* ppicka will address if he has time
## Mar 16 Agenda
* Is it possible to have 2 dynaconf settings files, one generated by installer, and a user-override?
* Looks like it: https://www.dynaconf.com/settings_files/#local-settings-files
* AI: pavel
* Status of AH signing service
* 1 remaining bug affecting release installs on EL7 only
## Mar 10 Agenda
* onboarding new team members
* read pulp 3 docs, particularly architecture
* familiarity with task tracking
* Status of galaxy signing service
* going fairly well
* wrapping around gpg commands is awkward but doable
* this might help for some gpg commands: https://galaxy.ansible.com/mistermiles/gpg_key_import
* PR for repos role usage
* merging of certs fix
## Mar 3 Agenda
* [Galaxy Signing Service](https://issues.redhat.com/browse/AAP-1874)
* Looks like it can be done in a few days to a week
* HA
* Need to refactor the dynamic molecule tests into multi-node tests
* Need some adjustments on keys (encryption, token auth, webserver, ...)
## Feb 24 Agenda
* Remaining CI performance improvement: reduce the time to run source-static
* Proposal: Only run the devel role in nightly CI or for PRS when the devel role gets modified. For most PRs, just install from source.
* Sub-proposal: Don't create a new test, just run `sed` to not include pulp_devel for the regular PR CI test
* Note: This still tests installing the latest, master branch, of pulp.
* pulp_repos & redis role: https://github.com/pulp/pulp_installer/pull/901
* Where does redis come from on EL7? [EPEL7](https://centos.pkgs.org/7/epel-x86_64/redis-3.2.12-2.el7.x86_64.rpm.html), for both centos and RHEL
* Wrong dependency on pulp_common
## Feb 16 Agenda
* We really do not need the "source-dynamic" test.
* Nobody uses this exact combination. (devel role + dynamic include)
* I cannot see why someone would need to use it.
* It's realistically tested by release-dynamic and source-static
* Already done a day ago: Moved to nightly CI only
* "packages-dynamic" or "release-dynamic" should be moved to nightly CI
* release-dynamic always tests the latest version of pulpcore
* packages-dynamic is quicker
* Already moved a day ago
* Status of CI refactoring (& performance improvements)
* Made the 2 above changes, only on nightly CI now
* PR CI tests dropped from 11 to 9
* tox removed, python versions and everything now determined by GHA logic
* ansible-core tests merged into our other test matrix and GHA logic
* PR CI tests dropped from 9 to 7
* Added more recent versions of ansible to the test matrix
* We can see how long each molecule phase takes
* Performance still not satisfactory.
* Will look into dropping idempotency from the longest running test or 2.
* Will look into more general performance improvements
* Noticed that packages tests take like 10 fewer minutes than release tests
* I'm trying to use (with molecule) an ansible plugin that reports the longest running tasks.
* fao89: Try comparing only uses CentOS 7 and CentOS 8 in the release tests
* packages-static takes 8 more minutes than packages-dynamic?
* Dynamic should take longer, it runs pulp_common repeatedly.
* maybe ansible interpreter version? Python version?
* yes, 2.10 vs latest, 3.7 vs 3.9
* fao89: See variables
* differences are unix sockets and different webserver.
## Feb 9 Agenda
* Created pulp-installer channel
* I found another way to do different variables per distro:
* https://github.com/mullholland/ansible-role-repository_epel/blob/main/tasks/epel.yml#L16
* https://github.com/mullholland/ansible-role-repository_epel/blob/main/defaults/main.yml#L5
* Still need to announce vagrant environments changes
* pulp_repos design
* include_role with "public" option
* "disable all repos" single variable
* Ansible core main branch still broken in CI
* Ping Yanis
## Feb 4 Agenda
* What a week full of CI and installer breakages!
* CentOS 8 discontinued was kind of foreseeable and avoidable
* pip-tools being broken by pip 22 is a routine breakage
* pulp_devel idempotency breakage is a routine brekage
* release CI breakage (and nightly CI breakage) - not sure if we can test for this better. I just run yamllint.
* Rocky Linux 8 support?
* Ask on discourse
* There's also alma - can be supported via generic check for "RedHat" family and major version 8.
* Release announcement for 3.17.2?
* agreed: Do an announcement
* GitHub generated release notes?
* agreed: Just continue to link to https://docs.pulpproject.org/pulp_installer/CHANGES/
* Reviewed https://github.com/pulp/pulp_installer/pull/870
## Jan 26 Agenda
* Is waiting for a long time to merge now https://github.com/pulp/pulp_installer/pull/382
* [mikedep333] I will re-review
* Finished triaging https://pulp.plan.io/issues/9211 by linking to it in 2 GH issues
* Is this the appropriate way to migrate to GH issues?
* Is [tox](https://github.com/pulp/pulp_installer/blob/main/tox.ini) actually overall beneficial for us?
* It causes silly/laborious integration between CI and tox
* How often do we test locally with all the python versions?
* How often do we test locally with any different python version that then venv's?
* It is preventing me from calling molecule steps individually in the CI (which helps identify performance regressions.)
* agreed: Remove tox
* Drop idempotence from the longest running test, source-dynamic?
* +1 for PRs, keep on nightly [fao]
* +1 [mikedep333]
* Design of https://github.com/pulp/pulp_installer/pull/830
* Agreed: Use the cache_enabled variable to determine whether or not the pulp_redis role actually does anything. Wrap the tasks in a when condition. No more need for a separate role.
## Jan 19 Agenda
* Fixed 3 CI issues / actual bugs in pulp_installer, now encountering 2 more
* Force merged since it makes the CI overall less buggy, and ppicka's PR will be better fixed with it.
* Noticed in Ansible docs that `ansible-galaxy collection install` can operate on a source directory. Builds and installs in 1 step.
* CI issue with pulp_rpm migrations not being idempotent
* This was fixed by pulp_rpm.
* ["pulp_repos" role improvement / bugfixes:](https://pulp.plan.io/issues/8701)
* Eliminates need for workaround in example-use playbook.
* Fixes actually adding EPEL on RHEL7 for the pulp_database/pulp_redis roles
* Assigned to ppicka.
* [postgresql role issue](https://github.com/geerlingguy/ansible-role-postgresql/pull/203)
## Jan 12 Agenda
* CI issue with pulp_rpm migrations not being idempotent
* CI issue with locale_gen module missing from ansible-core 2.12 from geerlingguy role
* ppicka looking into this
* Will drop support for EL7 pip mode while doing performance improvements over next week
## Dec 10 Agenda
* 3.15 RPMs support landed
* I did code fixes / workarounds for all the upgrades issue.
* 3.16 RPMs support
* Running into another upgrade issue from 3.3
* Decided to actually drop support for upgrading from pre-3.8 this time
* Need to announce on discourse.
* Cluster support
* Clarify what changes are still needed?
* Identical token auth keys
* galaxy_ng users probably need this, not just regular pulp_container users.
* Even if their QA hasn't caught it.
* Molecule test for multiple containers
* Fixing any bugs along the way
* Prototyping with a load balancer or something
* Documentation updates.
* Work with someone from Ansible QE to test it out. With their HA environment.
* Focus before cluster support or usability will be molecule performance
## Dec 1 Agenda
* 3.15 RPMs support
* [Figured out EL8 upgrade](https://github.com/pulp/pulp_installer/pull/793/commits/21bb6b64b406a28771881cbfb8d903b41c9e146c) at last
* Figured out EL7. Now idempotency is failing, but that is resolvable
* Just realized I could easily create a 3.8 container with the old installer. Because no pip dependency issues to worry about.
* [Issue with affinity](https://github.com/pulp/pulp-operator/issues/289#issuecomment-982702761)
* Moving pulp_pkg_repo URLs from the molecule (CI) variables to the installer itself
* [mikedep333] In a convo several months ago, I recall the Foreman build team saying this is OK to do now.
* agreed: Put the URLs in pulp_installer, along with docs warnings about being unsupported, and have the Foreman build team review.
* Making redis optional in the installer
* Agreed: If a user does not want redis, tell them to use a roles list [pulp_database, pulp_services, pulp_health_check, pulp_webserver]
## Nov 17 Agenda
* track sprint[s] on github issues? && how?
* We can use labels
* agreed: This subteam never followed sprints closely, so no need now.
* agreed: We will implement if we feel the need for it.
* [mikedep333] Still trying to make time to work on pulpcore 3.15 RPM packages support
* [mikedep333] Still need to rename master branches to main
* [mikedep333] Just renamed them. fao89 and I agreed we would just fix any breakages quickly, since it's easy to do.
* pulp_installer master branch checks out pulpcore-selinux master. This needs to be updated ASAP to fix CI.
* Gave 2 talks and colected excellent feedback at pulpcon.
* Still ongoing discussions about the feedback and engineering plans based on it. Such as:
* pulp-pip-installer
* Changing the docs or separating pulp_installer into "installing pulp" (`pulp_services`) vs "orchestrating the entire pulp node" (`pulp_all_services`)
## Nov 03 Agenda
* Fedora 35 - https://github.com/pulp/pulp_installer/pull/797
* Risky file permissions
* ansible-lint warning
* dkliban to file issue, ppicka to work it
* https://github.com/pulp/pulp_installer/issues/800
* 3.15 release (Satoe request)
* https://github.com/pulp/pulp_installer/pull/793
* Mike to hopefully finish by end of day. Will follow up then.
* https://github.com/pulp/pulp_installer/pull/796
* Plugins
* https://github.com/pulp/pulp_installer/pull/382
* Working on pulpcore 3.15 RPM packages support
* Debugging an issue, just FYI
* I know it is blocking the 3.15.2-3 release
## Oct 20 Agenda
* Moving to github issues
* check our backlog to close stale issues
* https://blogs.harvard.edu/rprasad/2014/07/10/moving-from-redmine-to-github-issues/
* Agreed: File new issues in GitHub.
* Agreed: Move over existing issues when discussed/updated.
* Agreed: Just rename the Redmine "category" "Installer" to "Installer - move to GitHub". Most users will not even use Redmine, or get the hint.
* https://pulp.plan.io/issues/9450
* Didn't we ask about the squid setup? In matrix/irc?
* Squid proxy was run on same machine as pulp.
* We do not understand why they would do that.
* Agreed: Dkliban will talk to Yanis about it.
* How to run entire GHA workflow locally, not just molecule locally?
## Oct 13 Agenda
* SELinux
* error when pulpcore-worker tries to do an outgoing connection to ephemeral_port_t
* Probably pre-worker-change pulpcore-worker trying to connect to redis
* https://pulp.plan.io/issues/9450
* We'll ask for clarification on setup. Like is squid (caching proxy) on the same machine or a different?
* Pulp/Django itself does caching.
## Oct 6 Agenda
* Finally merging the big minor-release PR!
* Ironically, this was Mike's original design, but talked out of it in order to more quickly implement a version-scheme.
* SELinux support for migration plugin finally delivered!
* Mike had a lot of trouble figuring out why 3.14 source-upgrade CI specifically was failing.
* debian-10 could not access the repo on opensuse OBS (in pulp_devel), but error was misleading
* Actually because of a TLS certificate issue, debian-10 was unpatched
* Tips for debugging & development
* Command versions of what molecule does. Like "add-apt-repository" vs ansible's apt_repository module.
* ansible-playbook --start-at-task
* Hitting ctrl-c while molecule is running, like a pause in a debugger
* Running ansible-playbook or ansible with a local docker connection (against a molecule container)
* Writing a one-off playbook with a task in it
* molecule --destroy=never
* `molecule converge` rather than `molecule test` (preceeded by `molecule destroy`, `molecule create`)
* `molecule login` rather than `docker exec`
## Sept 29 Agenda
* SELinux BZ 1991030 / [9468](https://pulp.plan.io/issues/9468)
* Code seems complete and it builds on EL7.
* Working on testing it.
* running into [error](https://pastebin.com/SSm356M5) with test commands.
* swadeley asking Katello devs to look into the error
* Had to outline the overall process for developing, testing, etc
* TBD how much back and forth on testing / debugging and PR review.
* Probably 1 day.
* https://github.com/pulp/pulp_installer/pull/762/
* Discussion on docs updates, what is needed exactly
* Discussed adding 4th workflow in docs like "stay on minor release, get latest micro"
## Sept 22 Agenda
* Lint warnings during PR reviews
* Indentation
* Lack of file permissions (when using copy module, etc)
* The single meta service.
* [mikedep333] Very happy about this!
* Code paths for old versions
* Yes, we can drop 3.13. Since packages are on 3.14 now.
* Need to release 3.15.2-2
* issues with 3.15.2+1
* Should minor release land on 3.16.0 or on 3.15.x?
* Yes, branch 3.15 right before we merge
* Versions of bugfix installer releases
* [agreed] z-stream not tied to pulp_installer z-stream release. Like pulp_installer 3.16.1 installs pulpcore 3.16.z.
* We need to communicate this somehow?
* Blog post?
* Announcement on discourse?
* [agreed] One or the other, possibly both.
* multi-container molecule test case
* Looks like molecule "platforms" ties to [ansible hosts / groups](https://molecule.readthedocs.io/en/latest/configuration.html#molecule.platforms.Platforms), which is for nodes running a particular service.
* Possible CI/molecule performance improvements
* Publish a pulp_common image. When we modify a role other than pulp_common, we just pull this image.
## Sept 15 Agenda
* Minor release support
* https://github.com/pulp/pulp_installer/pull/762
* Packages scenarios in molecule now require that a `pulpcore_version` be set for the scenario
* Upgrade scenario needs to do more
* [Proposals for multiple updates in 1 molecule test](https://hackmd.io/m71OYcvBRuSbAsaocj6uyw?edit)
* High Availability
* what we need?
* Create multi-container molecule test case
* Replaces dynamic
* installer support for identical token auth keys
* address other issues discovered along the way
* document integration with load balancers, etc.
* Possible CI/molecule performance improvements
* Look into why simple tasks run as root do not go extremely fast compared to other ansible uses (overhead per task)
* Review settings in ansible.cfg
* Look into performance optimizations for docker connection
* Eliminate unnecessary operations like lint from being run multiple times
* Remove the idempotency check from the longest running test
* Review last pulpcon notes - https://hackmd.io/cQf2zUKlR2G4_Vqb3rwP8g
* done: merge pulplift into pulp_installer
* not done:
* Docker based vagrant box
* partially done on migration box
* Ubuntu support
* we may need to explain the *not done* during pulpcon 2021 panel
## Sep 9 Agenda
* PulpCon - Any Installer topic?
* https://discourse.pulpproject.org/t/pulpcon-2021-call-for-proposals/134
* Demonstration of, and ideas, to improve the user experience?
* dependency checker (separate tool, plugin is a PoC)
* Alternative ways of explaining variables
* past prototype to make the installer a pip/rpm package using Foreman tooling (solves "installing the installer")
* Let's Encrypt support (with cloud instance demonstration)
* Cluster support (demonstration, explanation of 3rd party resources)
* Panel discussion (we are the panel) of users come talking to us about their experiences with the installer
* Mike working on SELinux bugs by end of month.
* Decided: Pulp 3 will access pulp 2 content on same system
* Most likely: A boolean for Pulp 3 to access pulp 2 content (httpd labels)
* Most likely not at this time: Access to NFS, gluster, etc mounts due to workaround of mounting entire filesystem with a label.
* https://bugzilla.redhat.com/show_bug.cgi?id=1991030 is a giant list of selinux alerts.
* I can open up separate issues in plan.io.
* Still need to remove the vagrant tests from CI
* Dkliban working on minor version support
## Sep 1 Agenda
* Difficulty fixing CI for vagrant "powertools"/"PowerTools" issue
* difficulty uploading the package to PPA from Fedora
* I can probably upload it easily from an existing ubuntu VM I have, that's what I did in the past
* Considering dropping the CI for vagrant
* fao89 & mikedep333 agreed: Let's finally drop the CI for vagrant
* Division of responsibilities due to large amount of pulp_installer work:
* fao89 primarily responds to user requests, mike secondary
* mikedep333 handles stakeholder requests
* rationale: stakeholder requests are generally far more complex
* Mike to focus on remaining SELinux issues, [Satellite QE reports](https://github.com/pulp/pulp_installer/pull/748) they are blocked by them
* Mike to also reach out to swadeley
## Aug 25 Agenda
* pulpcore-selinux update
* CI has issue with "powertools"/"PowerTools" repo name with centos8-fips box
* Look into vagrant vagrant-sshfs plugin version on CI
* CI has been more reliable in general though!
## Aug 18 Agenda
* upgrade with different webservers
* from nginx to apache
* from apache to nginx
* not a use case, maybe document it
* error with friendly message?
* Disable the other service?
* FIPS errors
* Leave the tests enabled on the nightly CI.
* Team will investigate them when noticed.
* Who will routinely looking at them and filing issues?
* Bring up in a month, of Mike doesn't do it.
* Looking into speeding up CI?
* Mike to look into settings and tweaks
* ansible.cfg
* why isn't molecule so fast if it is running as root, not using become?
* disable encryption for vagrant.
* gunicorn issue
* We are doing upstream & downstream kind of. Like with galaxy_ng role.
* Mike: We have releases for them on branches though.
* Agreed: Discuss with Yanis when he's back.
## Aug 11 Agenda
* db key fixes
* Almost complete set of requirements, to be written up a bug / epic: https://hackmd.io/0BcyTTltQ3qXZal9mDws2w
* Mike will write an epic for the auth token key (main issue currently: different on each pulp-api node)
* fao89 testing the slurp module, useful for both
* post-release issue on galaxy
* https://pypi.org/project/semantic-version/
* de-coupling pulp_installer z-stream from pulpcore z-stream would also fix it
* Agreed: We will land the de-coupling within the 3.15.z release stream, rather than at 3.16.0.
* Mike will work on this (new convention, and getting release out the door) if build team / AH team are not OK with waiting for the de-coupling.
```python=
In [1]: import semantic_version
In [2]: semantic_version.Version("3.14.0") > semantic_version.Version("3.14.0-1"
...: )
Out[2]: True
In [3]: semantic_version.Version("3.14.0") < semantic_version.Version("3.14.0-1")
Out[3]: False
In [4]: semantic_version.Version("3.14.0") > semantic_version.Version("3.14.0-post")
Out[4]: True
In [5]: semantic_version.Version("3.14.0") > semantic_version.Version("3.14.0.post")
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-6-24cbe611d9e3> in <module>
----> 1 semantic_version.Version("3.14.0") > semantic_version.Version("3.14.0.post")
~/.pyenv/versions/3.9.1/envs/venv/lib/python3.9/site-packages/semantic_version/base.py in __init__(self, version_string, major, minor, patch, prerelease, build, partial)
103
104 if has_text:
--> 105 major, minor, patch, prerelease, build = self.parse(version_string, partial)
106 else:
107 # Convenience: allow to omit prerelease/build.
~/.pyenv/versions/3.9.1/envs/venv/lib/python3.9/site-packages/semantic_version/base.py in parse(cls, version_string, partial, coerce)
309 match = version_re.match(version_string)
310 if not match:
--> 311 raise ValueError('Invalid version string: %r' % version_string)
312
313 major, minor, patch, prerelease, build = match.groups()
ValueError: Invalid version string: '3.14.0.post'
```
## Aug 4 Agenda
* The SELinux Epic
* https://pulp.plan.io/issues/97
* I keep on running into situations that would be easier to resolve if this were implemented. https://pulp.plan.io/issues/8701
* Team busy, discuss next week
* db key fix needed by 3.15 - https://pulp.plan.io/issues/9200
* david - I'm happy to test this out if it can be done by thurs/friday
* fao89 assigned
* minor branch model
* How to handle upgrades?
* Current model for plugins: `upgrade` or `version`. They are mutually exclusive.
* `upgrade`: `pip install --upgrade plugin`
* `version`: `pip install plugin==version`
* `update` could be mutually exclusive with `version`, but it could also be combined with it to be like `~=version`
* `update`: "determine current-version", then `pip install plugin~=current-version`
* `version`: `pip install plugin==version`
* `update` + `version`: `pip install --upgrade plugin~=version` (but this may not be feasible due to the Ansible' `pip` module's limitations.)
* How to handle upgrades?
* Option 1: Make users specify the new x.y.z `version`, within the new y-stream (currently implemented)
* Option 2: `upgrade` option. (currently implemented)
* Option 3: `update` + `version`
* These options are not mutually exclusive. Users could have multiple ways. But Option 2 is insufficient.
## Jul 28 Agenda
* Should we drop RTD?
* https://docs.pulpproject.org/pulp_operator/
* https://docs.pulpproject.org/pulp_installer/
* Mike working on fixing the packages CI failures
* Started occurring about 5 days ago on the master branch
* Seems to be an issue with an undeclared dependency not in the pulpcore repo, but in the chain. Like an EPEL package.
* Mike still hasn't gotten to work on the SELinux issue
* Python 3.8 distros
* We need to create a 3.14 branch. Python 3.8 distros should only be for 3.15 (although it may work with 3.14, we do not want to **upgrade** users python when they **update** pulpcore / pulp_installer)
## Jul 21 Agenda
* Problem on CentOS 8 with disabling the Python 3.6 module
* Agreed: Just use a shell command
* How to prevent ansible from breaking when we "uninstall" Python 3.6 on CentOS 8?
* Mike: Use ansible_python_intepreter=/usr/bin/unversioned-python
* Can we create a PR checklist?
* Coding standards
* ansible_facts syntax
* issue in commit message
* Dkliban: Yes we can
* Mike will look for it in GitHub settings pages
## Jul 14 Agenda
* https://pulp.plan.io/issues/8846 - use Y release of pulpcore
* subtask 8848 - Mike updated/clarified the description
* 2 user requests for offline installation
* My IRC explanation
* Moved to subtask I wrote up under the pulp_repos role refactoring story: https://pulp.plan.io/issues/9087
* Also, we are not going to try to make pulp_devel work offline
* Agreed: Do not document this until the "pulp repos" role refactoring is done
* 3.14.2 release
* done! https://galaxy.ansible.com/api/v2/collections/pulp/pulp_installer/versions/
* Fixes the "needs become" issue with checking for the db encryption key
## Jul 07 Agenda
* [Optimizing the db fields key generation](https://github.com/pulp/community/discussions/47)
* I think I should stop spending time on this just because it is optimizing code, and I've spent like 1.5 days on it.
* Mike to follow up with Dennis
* Mike still debugging the pulplift Mac issues
* Mike unable to reproducing DavidN's virtualbox performance issue, which suggests something else on his laptop is amiss
## Jun 30 Agenda
* tons of backporting to 3.11
* lots of little CI breakage (actually ability to install breakage often)
* [pulplift on Mac issues](https://hackmd.io/IsVY2y8RQn6bPZjWOQTERQ)
* Dennis working on docker kfor the sake of the galaxy_ng devs
## Jun 16 Agenda
* Fixed CI with inspec
* Still need to schedule a scope reduction meeting
* Mike to survey what features / support are adding lots of code complexity.
* Like causing really complex jinja2 in certain tasks.
## Jun 10 Agenda
* People not updating code in multiple places
* I want to apologize if it sounded like I was scolding in the 5/27 meeting.
* Bruno Rocha was affected by this recently (added cloning to pulp_common, but cloning was in molecule prepare.yml)
* Do people have any suggestions on this other than encouraging devs to grep?
* [ppicka] Dropping complexity will reduce code & minimize this happening.
* I noticed inconsistencies between how we import keys:
* pulp webserver TLS has 3 vars:
* `pulp_webserver_tls_key`: Relative or absolute path to the TLS (SSL) key
one wants to import.
* `pulp_webserver_tls_custom_ca_cert` A custom CA certificate to import on the server.
* `pulp_webserver_tls_files_remote`: Whether or not `pulp_webserver_tls_cert`,
`pulp_webserver_tls_key` & `pulp_webserver_tls_custom_ca_cert` are on the webserver (`true`)
or on the ansible management node (`false`). Defaults to `false`.
* pulp token has 1 var:
* `pulp_token_auth_key`: Location of the openssl private key (in pem format) to use for token
authentication. If not specified, a new key wil be generated.
* Should we add the latter 2 options for token auth?
* Agreed: Ask (& write up) if anyone will need the cert import. They both make sense though.
* Scope reduction proposals
* Agreed: We will cover this in a separate meeting. Advertise to users?
* Problem: Too much work to maintain pulp_installer
* Proposed:
* 1. Drop support for upgrades from early Pulp 3.x releases.
* Eliminates need for this portion of [this epic]() "We no longer need system-wide packages, so we should remove support for it, and migrate user installs off of it, as safely as possible."
* 2. Drop EL7 support
* However, [adding Python 3.8 support was easy after all](https://github.com/pulp/pulp_installer/pull/650/checks)
* [Awaiting confirmation that we can do this](https://github.com/pulp/community/discussions/3#discussioncomment-841815)
* Pavel & mikedep333 agree that maintaining support for Ubuntu would be much easier than maintaining support for EL7.
* Drop Python2 support from the managed node, but we can just pre-require Python3 (assuming SELinux bindings exist.)
* 3. Drop Python2 support from the managmenet node.
* 4. Drop FIPS (& future-implemented SELinux) CI tests
* Another reason is that on some distros the source/devel CI takes 1:40, others more like 4:00.
* We will encounter breakage at `vagrant up` time though.
* 5. What else? Othercustomization options like custom install paths that bloat our ansible tasks with lots of jinja2?
* Latest CI failure
* Error downloading debian key: https://github.com/pulp/pulp_installer/runs/2789307801?check_suite_focus=true#step:7:6348
* https://github.com/pulp/pulp_installer/pull/651
* Note: This is good that CI is catching unreliable 3rd party repos, users would occassionally experience this.
## Jun 03 Agenda
* [mikedep333] I don't think we should specify the pulpcore-selinux version in the release script
* The installer never installs the master branch anyway.
* We'll need the fixes of the latest pulpcore-selinux version in the pulp master branch.
* It's good to have the fixes in the changelog when we bump it via a pulp_installer PR.
* Alternatives could include having it track a branch for devel installs, and specify the version at release time.
* We'll need separate selinux policies for pulpcore stable release branches eventually.
## May 27 Agenda
* Is fao89 the primary pulp_installer maintainer still, or is mikedep333 it again?
* [agreed] mikedep333 will step up
* ansible 4.0.0 is out!
* read more about the changes later
* continue to support the last 2, or possibly last 3, versions of ansible-base
* [RPM signing service script PR from months ago](https://github.com/pulp/pulp_installer/pull/371)
* Agreed: mike to rebase. If it doesn't work, either fix it, or hand it off.
* Also, pulpcore-manager command can install it. (rather than the shell)
* Operator nodeports
* https://github.com/operate-first/support/issues/176#issuecomment-848976108
* https://pulp.plan.io/issues/8833
* Not doing a pulp_installer release for each micro release
* This is similar to the original plan for the installer to revolve around minor release of pulpcore & of plugins.
* Original discussion:
* [mikedep333's proposal](https://github.com/pulp/pulp_installer/pull/203#issue-361269733)
* [bmbouter's couter-proposal to do micro-versioned releases](https://github.com/pulp/pulp_installer/pull/203#issuecomment-577903411)
* [mikedep333's agreement/details for micro-versioned releases](https://github.com/pulp/pulp_installer/pull/203#issuecomment-579450153)
* Lack of CI on branches (or at least I don't see it.)
* This is partially mitigated currently by doing releases.
* Common breakage we catch is that python package needs to be pinned for pulp to actually install (e.g., pip)
* Implementation would include:
* `pulpcore_update` variable (false for idempotency)
* I'm noticing lots of instances where we update code/docs in some places, but not in the other places. (molecule tests, GHA workflows, especially)
* FIPS runners disabled from PR due to GHA runners being blocked, still running on nightly
* https://github.com/pulp/pulp_installer/pull/632
* agreed: This was part of the plan all along. (If the FIPS tests take up too much time, we'll disable them at PR time.)
## May 19 Agenda
* https://pulp.plan.io/issues/8679
* Mike to create a "triaged but unassigned" query, rather than just listing this in the meeting outline
* vagrant-sshfs 1.3.6 (rpm in fedora testing) enables CentoS 8.3 / 8-stream "powertools" but not 8.2 "PowerTools"
* Updates images?
* regular:
* 8.3 available: https://cloud.centos.org/centos/8/x86_64/images/
* or switch to vagrant's "generic" project images
* FIPS: We'll rebuild https://github.com/mikedep333/fips-box-builder
* Stream is working anyway, so maybe this PR isn't necessary?
* Probably isn't breaking anything, since the old version works with CentOS Stream
* Agreed: Mike to update images at his leisure, but do not worry about breakage
## May 12 Agenda
* community survey about installers
* what do we want to know from users?
* Encryption key support
* Thinking of putting this non-cert under /etc/pulp/certs/
* Give users the ability to import, like with certs
* Clarification about repo workaround in the example playbook
* Issue isn't actually fixed yet
## May 5 Agenda
* https://pulp.plan.io/issues/8701
* I just thought of a good variables design for it
* Example playbooks (in folders and in docs) contains workarounds
* Workaround merged https://pulp-installer.readthedocs.io/en/latest/quickstart/#example-playbook-for-installing-plugins
* Mike to clean up while working on other docs
* example-use playbook has S3 checks
* Mike to move them to the pulp_common role https://pulp.plan.io/issues/8702
## April 28 Agenda
* vendoring collection dependencies
* https://github.com/pulp/pulp_installer/pull/562
* Turn POC into real work
* apache x nginx:
* https://github.com/pulp/pulp_installer/pull/594
* decided to go forward with this
## April 23 Agenda
* Installing with proxy server
* https://listman.redhat.com/archives/pulp-list/2021-April/msg00039.html
* We do not want to switch the URL from https to http by default to workaround curl's inability to talk to a proxy server.
* curl's / libcurl's inability to talk to a proxy server is a big issue that the user should fix.
* But I will make sure the URL is configurable.
* We will probably end up writing a howto on using pulp_installer with a proxy server
* SELinux issue: https://pulp.plan.io/issues/8620
* The comment on the handler "name: Restore SELinux contexts on Pulp dirs that must exist" will help mikedep333 figure out how to fix this. (When to flush the handlers)
* We've run into multiple bugs lately due to the lack of [a pulp_repos dependency role.](https://pulp.plan.io/issues/7773)
* Create another issue for it, assign to pavel.
* We've run into multiple bugs lately that would be caught by doing a CI with 1 service per container
* How to handle testing multiple distro at once? 2 sets of group variables?
* Mike and fao89 to meet and spin off (separate) 1 service into a separate molecule container on 1 OS, then fao89 to do the rest.
*
## April 14 Agenda
* pulplift on mac (virtualbox)
* centos-**fips** images are libvirt only
* [mikedep333] Please use this repo to create & upload any images you need: https://github.com/mikedep333/fips-box-builder
* Discussion of virtualbox vs libvirt on Mac: libvirt on Mac was very early in development a year ago when ppicka last tried it, and we want to support VirtualBox-using contributors anyway.
* https://pulp.plan.io/issues/8550
* Agreed: Remove the duplicate docs
* https://pulp.plan.io/issues/8552
* Agreed: Also clarify pulp_settings vs other vars. And that users using the installer should only set pulp_settings via the installer.
## April 7 Agenda
* https://pulp.plan.io/issues/8490
* Assigned to ppicka
* https://pulp.plan.io/issues/8477
* Agreed: Mike/mcorr to create sub-tasks for individual issues.
* Mike to work on the variables section, with a new page like "customizing your install", moving the "Recommended Workflows for Pulpcore & Plugin Versioning" section to it. It will list the most commonly used vars, with some examples, but references the roles for more info.
* https://pulp.plan.io/issues/8519
* [mikedep333] Modifying the dynamic tests to operate on individual containers would have caught this
* [mikedep333] I'll bring it up at the next quarterly planning. (won't take too long to implement though)
## March 31 Agenda
* Install & document storage support: https://pulp.plan.io/issues/8446
* https://github.com/pulp/pulp_installer/pull/571
* should we move to github issues?
* pulp_installer
* pulp-oci-images
* pulp-operator
* FIPS test
* unix socket improves the CI
* the faster it goes, less failures we see
* Agreed: Move it to a schedule. Put it back in PR in the PR that fixes it.
* I think this CI failure was just due to galaxy being down: https://github.com/pulp/pulp_installer/runs/2189226913?check_suite_focus=true#step:7:15
* https://github.com/theforeman/forklift/pull/1325
* Dependencies
* https://github.com/pulp/pulp_installer/pull/567#discussion_r600735562
* AI: fao to file an issue
* Should we keep supporting python 2.7?
* [mikedep333] EPEL7 Ansible 2.9 is on Python 2
* [fao89] molecule & other ansible tooling is dropping python 2 support
* Agreed: drop python 2 support next time CI breaks
## March 24 Agenda
* pulp ansible module: https://github.com/ansible-collections/community.general/blob/main/plugins/modules/packaging/os/pulp_repo.py
* https://docs.ansible.com/ansible/latest/collections/community/general/pulp_repo_module.html
* [mdellweg] reach out to the author to ideally include a statement this is for pulp2
* collection dependencies
* community.general
* https://github.com/pulp/pulp_installer/pull/562
* pulp-cli should be treated as plugin?
* AI: use a new variable `install_pulp_cli`
## March 17 Agenda
* Ansible 3.1 is out now
* Still ansible-base 2.10
* More on why we should support Ansible 2.9, 2.10, 3.0, 3.1
* [Ansible 2.10, 3.0 & 3.1 all use ansible-base 2.10.] See ["What is the correlation between Ansible 3.0.0 and ansible-base 2.10.x?"](https://www.ansible.com/blog/ansible-3.0.0-qa)
* Has anyone found a feature list for Ansible 3.0 / 3.1?
* [bullhorn newsletter](https://github.com/ansible/community/issues/546) is the closest
* Is it possible to replace "ini_file"?
* avoid community.general
* [mikedep333] Removed dependency from my FIPS box pair PR, but it still is using community.general.postgresql_user
* [mikedep333] We should enumerate all our community.general modules somehow
* [mikedep333] Let's talk to Yanis about this
## March 10 Agenda
* ansible 3
* POC - https://github.com/pulp/pulp_installer/pull/545
* CI requiring a removed scenario
* support from 2.9 to 3.0? Or keep with the latest 2 releases (2.10, 3)?
* [mikedep333] Definitely drop 2.8, added comment to PR on how to do that.
* [mikedep333] Let's keep 2.9 for now, due to confusion over new version scheme, irregular pip upgrade process to 2.10, and to distros often having 2.9.
* runtime metadata
* https://github.com/pulp/pulp_installer/pull/540
* currently not enforced, but it helps as a hint
* [mikedep333] Awesome!
* https://pulp.plan.io/issues/8186
* ppicka cannot reproduce it on a CentOS 7 VM
* [mikedep333] Such a simple fix, so let's just add `become:true` and add a comment that on some environments, it is necessary.
* Warnings about GHA upgrading ubuntu-latest to Ubuntu 20.04 soon (plugin-template and pulp_installer)
* [mikedep333] IIRC, on 1/20 we agreed that because our repos are mostly compatible with 20.04, we will cross that bridge when we get to it.
* fao89 just confirmed that pulp_installer is compatible, as well as all but 1 of the rest of his forks. (His forks are on 20.04, GHA is doing a rolling release.) The sole incompatible repo is pulp-operator.
* [mikedep333] We should upgrade pulp_installer, easier than plugin-template, with no plugin scripts.
## March 3rd Agenda
* Status of pulp 2 + pulp 3 FIPS vagrant boxes
* Implementing as a docker box on top of a libvirt box. (for performance largely)
* Currently dealing with issues specifically with docker on top of libvirt
* May be able to work around by not trying to build the image, which means additional shares to the VM.
* pulp-operator status:
* Chris Hambridge is doing an awesome job!
## February 24 Agenda
* pulp-operator status:
* Chris Hambridge is doing an awesome job!
* Mike to start working on it hopefully in a week, after finishing vagrant box pair for fips pulp2 + pulp3
* Mike would like to reassign https://pulp.plan.io/issues/3800 to fao89
* ansible 3.0
* should we support it?
* supporting it means dropping support for 2.9? (due to CI load)
* we can support 2.9, 2.10 and 3.0, maybe 2 jobs for each
* According to our existing policies, we should only support 2.10 and 3.0.
* However, we do suspect that users will use 2.9, the last monolithic release, for a while. `pip install --upgrade` from 2.9 does not work.
* https://docs.ansible.com/ansible/latest/user_guide/collections_using.html#simplifying-module-names-with-the-collections-keyword
* With the collections keyword, we can presumably be able to support 2.9 and 3.0
* AI: fao89 to start a PoC
## February 17 Agenda
* Gerrod recently deployed pulp on a database server + a pulp server
* Mike will offer to collaborate with him on updating [the instructions](https://pulpproject.org/2020/07/09/pulp-3.5-installer-roles/) & move them from blog to installer docs?
* Task to copy SELinux packages is not idempotent, triggers slow SELinux Relabel
* I forget who was mentioning about how slow the SELinux relabelling is
* Carry over to next meeting
* Triage
* Zarro Boogs Found
## February 10 Agenda
* Status update on CI's FIPS / Vagrant / Qemu emulation.
* It's merged at last!
* https://pulp.plan.io/issues/8228
* Mike to update issue to clarify that if the 1st request times out, all subsequent ones will too.If the 1st request succeeds, all later ones will be much faster.
* Seems like the best approach to the problem.
* We will advice plugin-template users to configure the FIPS tests as optional to merge. So they can merge before they finish running after 2.5+ hours (inadvertently too, if they fail.)
* https://pulp.plan.io/issues/3800#note-8
## January 27 Agenda
* Cannot figure out why [pulp health check via vagrant on GHA times out](https://github.com/pulp/pulp_installer/runs/1772162654?check_suite_focus=true#step:4:841), but running the role individually via ansible-playbook works
* change in MEDIA_ROOT - https://github.com/pulp/pulp_installer/pull/510/
* FYI about multiple CI failures
## January 20 Agenda
* Warnings about GHA upgrading ubuntu-latest to Ubuntu 20.04 soon
* nginx x apache scheme inconsistency:
* pulp-api
* nginx: does not include http:// - https://github.com/pulp/pulp_installer/blob/master/roles/pulp_webserver/templates/nginx.conf.j2#L29
* apache: includes http:// - https://github.com/pulp/pulp_installer/blob/master/roles/pulp_webserver/vars/main.yml
* current code breaks plugin snippets (they don't expect http in it): https://github.com/pulp/pulp_container/blob/master/pulp_container/app/webserver_snippets/apache.conf
* https://github.com/ansible/galaxy_ng/pull/616/commits/a4bf60fb017fd95e82c39698c4de4f60a250ea5f
* should we update nginx and all plugin snippets?
* fao: yes (only installer should know the scheme)
* Issue: https://pulp.plan.io/issues/8130
## January 13 Agenda
* What was the previously proposed layer for the operator to take RWO storage and convert to RWM or S3?
* [dkliban] Rook
* Need someone to join pulp-operator / galaxy_ng meeting next week (chouseknecht's)
* fao89 is already joining.
* fao89 will be mentored by Mike on containers / k8s / operator concepts as he makes changes
* See ["Introduction to pulp-operator"](http://people.redhat.com/mdepaulo/presentations/Introduction%20to%20pulp-operator.pdf) slides from pulpcon 2019
* Particularly note the ground-up explanation of layers: containers -> k8s -> operator
* pulp-operator on github actions
* works when running at master, but fails to deploy as images are built with podman and CI tries to push with docker - https://github.com/pulp/pulp-operator/pull/60
* approved
* insta demo fails on PRs (failing to get remote/branch from upstream) https://github.com/pulp/pulp-operator/pull/59/commits/0b6e5210f055731e5f64ecfb69c8e5de9861c8b2
* finish review post-meeting
* Performance issue [#8055](https://pulp.plan.io/issues/8055)
* We could selectively rework the handlers into smaller ones, and trigger the smaller handlers, and only when needed.
* For example The task "Create configuration directory for Pulp" need not trigger the entire big handlers "Restore SELinux contexts on Pulp dirs that must exist"
* We could do the Pulp 2 approach: Version pulp_installer with the SELinux policies, and run selective relabel tasks depending on what we're upgrading to and from
* This would involve submodules or putting it in the pulp_installer repo, ties into [#7575](https://pulp.plan.io/issues/7575)
## January 6 Agenda
* does the installer use the latest version of pip?
* No - https://github.com/pulp/pulp_installer/blob/master/roles/pulp_common/tasks/install_pip.yml#L86-L93
* [dkliban] open task to start using latest version of pip in the installer
* we may need to start working on pulp-operator
* CI for pull requests is currently failing
## December 16 Agenda
* need for a new release now that https://github.com/pulp/pulp_installer/pull/495 is merged, so the Tower installer can pull it from galaxy
## December 9 Agenda
* Older issues still at modified. No milestone set.
* https://pulp.plan.io/issues/7804
* https://pulp.plan.io/issues/7234
* https://pulp.plan.io/issues/7780
* https://pulp.plan.io/issues/7746
* https://pulp.plan.io/issues/7107
* https://pulp.plan.io/issues/7155
* https://pulp.plan.io/issues/7798
* https://pulp.plan.io/issues/6753
* Releasing doc doesn't mention updating redmine
* https://github.com/pulp/pulp_installer/blob/master/RELEASING.md
* agreed
* Release process is somewhat painful
* e.g. having to manually close out issues and assign them to the correct milestone
* Maybe worth automating?
* https://pulp.plan.io/issues/7961
* pulpcore version x pulp_installer version: use template docs/index.md.j2?
* Continuous Release "with every commit"?
* Versions like 3.8.1-1-dev-20201209123456
* pretty similar to nightly.yml from pulp plugins
* [spredzy] to make a writeup for pulp-dev ML
* FIPS/SELinux CI Approaches
* [Qemu Emulation on GHA](https://github.com/pulp/pulp_installer/pull/503)
* 100 min runtime per job
* Given a matrix of centos7 vs centos8, master branch of pulpcore/plugins vs releases, and fips vs selinux
* we could do 2 of those tests, with the opposite set of options
* we could also do FIPS & SELinux in the same test, most users with FIPS also have SELinux enabled
* No security implications
* No management overhead
* Agreed: This approach. Which tests to run at PR time is TBD, but definitely for cron.
* persistent server from QE
* Possibly only 1 job at a time
* High security implications
* Moderate management overhead
* Agreed: Tell them we'll follow up in a week as we pursue Qemu emulation
* using a cloud provider w/ ephemeral instances (from GHA runner w/ a cloud provider vagrant plugin)
* Moderate security implications
* Little management overhead
* https://github.com/pulp/pulp_installer/pull/495
* can we safely rename variables we never advertised to exist?
* 1 or 2 users (from the ML, told to use them with a warning) will probably be broken
## December 2 Agenda
* [CentOS CI denied us](https://pagure.io/centos-infra/issue/158#comment-703994) at this time, due to too few hardware resources
* Agreed: follow up with rchan about any available hardware resources.
* Agreed: Pursue AWS instances called from GHA.
* https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories
* we won't use self hosted runners, we will be using GHA runners, that call out to AWS instances.
* We still need to be mindful of security for our to-be-created AWS credentials being used to create the instances.
## November 25 Agenda
* Vagrant tests waiting on [CentOS CI onboarding ticket](https://pagure.io/centos-infra/issue/158)
* I seem to be following [the process](https://docs.fedoraproject.org/en-US/cpe/day_to_day_centos/) correctly
* Agreed: I'm planning on just starting to write our expected Jenkinsfile in the meantime. [Very different syntax](https://www.jenkins.io/doc/book/pipeline/syntax/) though.
* Satellite QE used to use Jenkins for some tests, [look at theirs](https://github.com/pulp/pulp-ci/tree/master/ci/jjb), Bruno used to maintain it
* Look at the forklift Jenkinsfile
* Agreed: FIPS work: Mike cannot use Travis minutes, but should resume investigating locally w/ Vagrant
* Submit PR at end, because david disabled *all* travis automatic jobs
## November 19 Agenda
* Migrating pulplift off of Travis
* Investigating how other projects test FIPS/SELinux on Travis or GHA - no luck
* Backup plan of vagrant cloud instances from GHA - we may use [IBM cloud](https://github.com/IBM/deploy-ibm-cloud-private/blob/master/docs/deploy-vagrant.md) instead of AWS.
* Yes, IBM cloud does have VMs/virtual servers.
* Yes, IBM cloud does have a Vagrant plugin
* Travis CI failing on geerlingguy role in all 4 FIPS tests
* Mike to investigate
## November 11 Agenda
* Releasing 3.6.5 seems to be broken
* https://github.com/pulp/pulp_installer/pull/491/checks
* Are we really able to install old versions of pulp with the installer?
* [mikedep333] Set the pulp-rpm version, and modify the debian-10 host_vars files
* should we have a rpm installed box in "pulplift"?
* x9c4 try in a spare minute and document it
* not going to make a big effort here
* [mikedep333] Just add [the vars](https://pulp-installer.readthedocs.io/en/latest/roles/pulp_common/#role-variables-if-installing-from-rpms) to install from rpm here, commented out: https://github.com/pulp/pulp_installer/blob/master/example.user-config.yml
* Can we stop releasing a new version of the installer with every bugfix release of pulpcore?
* It should always be ok to upgrade to the latest 3.8.z of pulpcore
* If it were tightly coupled, why are core and installer separate repositories?
* from pulp_version: "3.8.1" to pulp_version: "<3.9" ?
* [mikedep333] Reasoning: https://github.com/pulp/pulp_installer/pull/203#issuecomment-579450153
* agreed: read/research this more for next meeting
* pros and cons list would be helpful
* Pros:
* Installer pins pulpcore but no plugin which seems inconsistent
* You can version pin all the components already and that is documented
* Cons:
## November 4 Agenda
* https://pulp.plan.io/issues/4968
* Good ramp up task for Pavel
* Triaging process - up to #6747
* We spent most of the meeting going through open pulp_installer PRs
## October 28 Agenda
* https://pulp.plan.io/issues/7750 - an installation docs issue, need assignee
* How to update triage query? "assignee none"?
* Agreed: Handled during backlog session (triaged but unassigned)
* How to announce pulpcore-selinux releases?
* [mikedep333] Agreed: Issues against pulp_installer, included in pulp_installer changelog when __pulp_selinux_version is bumped.
* I did a trivial merge commit so I could merge a PR, but now it breaks CI: https://github.com/pulp/pulp_installer/pull/446
* Info on how we have the permissions for owners to modify branches of PRs. This explains why commit on his branch in his repo.
* Agreed: Up to Mike's discretion.
* Redis SCL switch
* build team really wants it
* We have a bug anyway, no more EPEL getting added in pulp_redis
* impact to pulp is migrating existing installs
* This looks problematic based on [different /var path.](https://centos.pkgs.org/7/centos-sclo-rh-x86_64/rh-redis5-redis-5.0.5-1.el7.x86_64.rpm.html)
* Include Brian & Ina on convo, since they are redesigning tasking system. Maybe it won't even use Redis?
* spredzy requested some backports to 3.7.2
* https://github.com/pulp/pulp_installer/commit/505ab087b23864a057e4d0226e63071fb66c88cc
* agreed: Mike to do the 3.7.2-1 or 3.7.3 release
## October 21 Agenda
* Meeting keeps needing to be rescheduled due to company meeting
* Agreed: Reschedule for 9:00 / 15:00 if schedules are free.
* Backup: Reschedule for 8:30 / 14:30 or so. Americans will wake up earlier :)
* [bmbouter] [pulpcore 3.7.0 retro] installer requires pulp-file and pulp-rpm to be released, which prevents pulpcore from fully releasing
* installer needs to drop pulp_rpm tests for y-releases
* pulp-rpm needs to be tested due to https://pulp-installer.readthedocs.io/en/latest/prereq_roles/pulp_rpm_prerequisites/
* Agreed: this was addressed by the deprecation policy implemented in the 3.8.0 development cycle.
* [mdellweg] pulplift merge has been ready for some while now and i want to get it merged
* https://github.com/pulp/pulp_installer/pull/440
* Mike to review it, now that he is back at work.
* redis SCL switch from 10/14 Agenda?
* Agreed: write a ticket, then discussion with AH. Mike shouldn't do lots of investigation 1st due to workload.
## October 14 Agenda
* Documenting RPM installation - https://www.mail-archive.com/pulp-list@redhat.com/msg05886.html
* unix sockets:
* [#7524 Apache fails to use unix sockets to talk to pulp-api](https://pulp.plan.io/issues/7524)
* [Docs PR with recommended pathss.](https://github.com/pulp/pulp_installer/pull/454)
* Non-list email thread with ehelms about unix socket binds:
* 1) Does y'alls default deployment, putting them in this directory structure, provide access to the 'apache' user for use with Apache as the reverse proxy?
* 2) Does the selinux policy support these paths and unix socket bind already?
* 3) Are you u* pulp_redis role EPEL issues raised by katello/spredzy
* Katello / galaxy_ng issues with pulp_redis role
* undeclared dep on EPEL
* Katello will use the SCL instead
* galaxy_ng has an organizational requirement to use the SCL rather than EPEL
* Does galaxy_ng need us to remove EPEL entirely from pulp_installer?
* Next time new folderpaths should be discussed with Katello team
* They added /var/lib/pulp/docroot, we added /var/lib/pulp/pulpcore_static to do the same.
* How to make sure this happens? Awareness?
## October 7 Agenda
* After much experimenting, I discovered we can check if we are running on a podman container with a very different variable. ansible_env.container == 'podman'
* A bunch of easy next tasks for Pavel. Not urgentnow, but may be needed soon: https://pulp.plan.io/issues/7638
## September 30 Agenda
* Test ansible 2.10 and drop 2.8 - https://github.com/pulp/pulp_installer/pull/448
* [mikedep333] to review the PR
* CI failing - https://github.com/pulp/pulp_installer/runs/1185310416#step:7:2174 - already addressed https://github.com/pulp/pulpcore/commit/45827312f87c5d5fb88e01f206d27e5d813a85fd
* The majority of the issues at NEW are from the installer team, and most of them have been carried from multiple sprints ago - https://pulp.plan.io/agile/board?query_id=151
* [dkliban] schedule a meeting for next week to go through the backlog of issues - https://pulp.plan.io/projects/pulp/issues?query_id=162 (this query is only for the installer, container and operator not included)
* molecule has dropped python2 support since March, and we are using an ancient version in our CI that lacks support for collections on many levels.
* Can we drop python2 support in the installer? (The software we are installing doesn't have it.)
* https://github.com/ansible-community/molecule/issues/2629
* https://github.com/ansible-community/molecule/pull/
* Mike: This is about the control node (management node), not the managed node.
* Mike: Many control nodes will have Ansible installed via Python2.
* If an EL7 control (management) node user just does "sudo yum install ansible", [as the Ansible 2.10 install docs say to do](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible-on-rhel-centos-or-fedora), they'll have python2 ansible.
* Many orgs have large, 3rd-party, Python 2 stacks. e.g., installed under /opt or an NFS share. I've seen this in the ML, experienced this IRL.
* If a Mac user runs `pip`, [they get python2, pip3 is a stub.](https://apple.stackexchange.com/a/376081) If a Mac user follows [the Ansible 1st party install instructions](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible-on-macos), they use Python2.
* If an Ubuntu user follows their Ubuntu install instructions (install from PPA), they get python2 also. (Tested on 18.04)
* If a Mac / EL7 / Ubuntu 18.04 user follows [the Ansible pip install instructions](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible-with-pip), they use Python2.
* All good arguments to bring to the molecule team
* https://github.com/ansible-community/molecule/issues/2764
* Mike: Can we just not test Python2 anymore with Molecule?
* What we don't test, we don't support. +1
* Can we drop to use molecule and test with vagrant only?
* We can use py3 with molecule and install one vagrant box with py2
this is blocked by https://github.com/pulp/pulp_installer/pull/440
* Can molecule on py3 use ansible on py2?
* Travis resources will be overwhelmed with running the vagrant box tests (approximately 9 out of 17) with every pulp_installer PR.
* New org only for installers repos?
## September 9th Agenda
* [mikedep333] I want to say that the rest of the subteam was right about something from ~3 months ago. It was best not to create a branch supporting CI for pulp_installer. The effort would not have paid off.
* We cannot run CentOS 8 molecule containers on our Fedora 32 systems right now. service/systemd module fails:
* https://bugzilla.redhat.com/show_bug.cgi?id=1853407
* https://bugzilla.redhat.com/show_bug.cgi?id=1853736#c8
* switching to systemd module does not help
* Can we ask RHEL8 for a fix?
* agreed: Just run molecule on a VM
* CI is red on source-upgrade: https://pulp.plan.io/issues/7479
* try pipdeptree
* try https://github.com/pulp/pulp-oci-images/commit/755fa0f816bfac40fbf97f2dc2ca8fa5d6a6c873#diff-ef37552d3bc0dec828902c8331d481c7R10
* What is needed from the installer to support smooth roll out of https://github.com/pulp/pulpcore/pull/799 ?
* Created installer subtask.
* how docker changes can affect us? (Free plan – anonymous users: 100 pulls per 6 hours ) - https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/
* agreed: This will probably break many projects, so they might revert. Travis/GHA may also have a cache. So let's address this later. We can consider Quay if it is more reliable then.
* How Mike responded to the user email about RPMs install
* Team thinks it was good to spend time investigating, for the user's sake, for the sake of other users, and to look into the underlying issues (we want to find out about them early in CI, not when users report.)
## September 2nd Agenda
* Ewoud's layout proposal is awaiting feedback https://github.com/pulp/pulpcore/pull/799?
* Agreed: Installer team approves of this design
* FIPS dev environment
* pulplift VM ready! * https://pulp.plan.io/issues/6983
* 3.6.1 has been released
* Thank you fao89!
* Added 3-month plan summary (who's assigned) to top of this document
* Agreed: FIPS dev env line item should be marked complete. Installer team has nothing more to do. CI team does though.
## August 26th Agenda
* https://pulp.plan.io/issues/7155#note-2 Discuss implementation scope
* We have 3.5 GB to 4GB RAM free (out of 7GB) with 4 Pulp containers running in CI:
https://github.com/pulp/pulp_installer/pull/401#issuecomment-678363192
* Apache/Nginx config
* whitenoise issue: https://github.com/pulp/pulp_installer/pull/402#discussion_r474194225
* stakeholders may have other configs? katello? awx?
## August 19th Agenda
* archive https://github.com/pulp/pulp_rpm_prerequisites
* done on galaxy
* Brian to archive on gh
* done: https://github.com/pulp/pulp_rpm_prerequisites
* look into https://github.com/ansible-collections/overview/blob/master/collection_requirements.rst
* [fao89] this is nice, thank you!
* We need to restart fresh-install CentOS 7 testing to CI badly.
* working on it: https://github.com/pulp/pulp_installer/pull/395
* AI: file an issue about selecting OS on molecule
* anyone interested in a tour through tower CI pipeline?
* AI [fao89]: will talk with Elyezer to set it, and try to record it
* operator-sdk 1.0.0 is released
* https://github.com/operator-framework/operator-sdk/releases/tag/v1.0.0
* https://sdk.operatorframework.io/docs/building-operators/ansible/migration/
* https://sdk.operatorframework.io/docs/upgrading-sdk-version/v1.0.0/
* [mikedep333] Is it really a big rewrite?
* rename and move to pulp org?
* https://github.com/fao89/pypdc/runs/1003319080?check_suite_focus=true#step:5:6
* This bug is important to fix
* https://pulp.plan.io/issues/7328
* Assigned to mikedep33 to complete at end of today. If not, re-assign to x9c4.
* These bugs needed to fix to unblock @ipanova
* https://pulp.plan.io/issues/7330
* https://pulp.plan.io/issues/7334
* https://pulp.plan.io/issues/7335
* All assigned fao89, but mikedep333 will follow up later this week after finishing DevConf.US presentation.
## August 12th Agenda
* FYI: Ansible 2.10 release date is 8/13.
* FYI: It's a good thing Our CI tests Python 2 on the Ansible mgmt node. Found a 2nd jinja2 limitation [in this PR.](https://github.com/pulp/pulp_installer/pull/371)
* Limit the scope of the installer (collection x bunch of roles)
- Limit use to collection and not source.
- Will update https://pulp.plan.io/issues/7281 with this limitation
## August 5th Agenda
* plugin for pre-flight?
* https://github.com/pulp/pulp_installer/pull/372
* Agreed: Pursue this, and using pip 20.2 / 20.3's new dependency solver. Assume we will not refactor the installer to actually install based on pip-tools (the preflight).
* Testing the pulp_installer as a collection not a bunch of roles
* collection testing is not well supported by molecule
* but testing it as a collection is testing it the way it should be used
* helps us shipping ansible plugins with the collection
* https://github.com/pulp/pulp_installer/pull/382
* Agreed: Good idea to test. We should accept the limitation of "Ansible 2.8 + collection + dynamic include does not work."
* Support for doing multiple installations at once?
* https://github.com/pulp/pulp_installer/pull/377
* No - document it https://pulp.plan.io/issues/7281
* Lets Encrypt support in time for 3.6
* Unlikely to be finished
* Assigned to mikedep333
* mikedep333 offloaded the remaining work on pulp_installer tests and the pip issue to fao89
* Agreed: dkliban to meet with mikedep333 on Wed, and work on it on Friday. x9c4 is backup, definite reviewer.
## July 31th Agenda
* Matthias commit bit
* +1 [fao89] +1 [bmbouter] +1[dkliban] +1[mikedep333]
* tower issue (hitting /status)
* https://pulp.plan.io/issues/6586 - Agreed to just drop the /tmp isolation to solve quickly
* [Our tests (not run currently)](https://github.com/pulp/pulp_installer/blob/master/molecule/scenario_resources/tests/test_default.rb) are currently in Ruby and part of the molecule "verify" phase, not the Ansible run itself. Need to be quickly ported.
* Put verification in a new role? Part of pulp_all_services, but not pulp_services?
* lots of changes in operator-sdk (1.0.0 is coming)
* https://github.com/operator-framework/operator-sdk/releases/tag/v1.0.0-alpha.1
* Noticed some docs issues - mkdocs markdown specific
* https://pulp-installer.readthedocs.io/en/latest/roles/pulp_common/
* maybe we need to add a plugin? (For the links next to the section title)
* [indentation error for ENV in example](https://pulp-installer.readthedocs.io/en/latest/quickstart/)
* Reminder to read and follow [the "oasis" ansible role standards](https://github.com/oasis-roles/meta_standards/blob/master/README.md)
* https://pulp.plan.io/issues/6846 - Letsencrypt integration for the installer
###### tags: `Minutes`
