pulp_installer usability changes

• Stop using special variables in the installer, use pulp_settings.variables.
  • Solves Florian "Ansible variables hide away the Variables in Pulp itself"
• Advertise the 2 different modes: pulp_services (& pulp_webserver) vs pulp_all_services
  • Solves Florian "More of an orchestration, not as an installation."
  • Solves Florian "It's not debian anymore, it's now orchestration."
• Find way to read settings.local.py, or use "lineinfile" and read settings.py
  • Solves Florian 'Installer blows away variables in /etc/pulp/settings.py. (we do not do "lineinfile")''
  • May not need to read settings.local.py if the settings do not concern the installer behavior.
• Each plugin will have a default minor branch. The compatible minor branches are maintained on pulp_installer minor releases.
  • Solves Florian's partially "Pulp itself no longer has a single version. It's a distribution."
  • Solves Florian's "What versions are tested with what version of pulpcore."
  • Solves Florian's "Needs some agreement on distribution wide values."
  • Solves Florian's partially "Re-running the installer against new blank systems installs different versions."
  • Partially Solve's Florian's "Everytime he runs the installer, in his pipeline, he has to check what versions actually got installed. That does not make sense for an "installer". What gets installed should be according to certain properties."
• Create a "pip-freeze" mode, which installs the entire set of dependencies for all plugins, but not the actual plugins unless specified. Installs user-specified plugins at specific versions.
  • Solves Florian's partially "Pulp itself no longer has a single version. It's a distribution."
  • Solves Florian's "Re-running the installer against new blank systems installs different versions."
• Document supported versions of redis, postgres & webservers
  • Solves Florian's "supported versions of redis, postgres, etc."
• Advertise local settings as an option, users can customize later, but clarify that some settings need to be set by the installer
  • Perhaps a table of "settings that must be set by the installer"
  • Solves Florian's "Provides and initial configration"
  • Solve's Florian's "a distribution-wide agreement with all devs is neccesary to establish what configuration values are neccesary to stand up the service."
  • Solves Florian's "an installer usually has the right to refuse to be concerned with other configuration values than mandatory or integration."
• Systemd overrides rather than templated unit file
  • Solve's Florian's "Provides and initial configration"
  • Solve's Douglas's desire for static systemd unit file
  • Also document it on pulpcore docs
• Provide webserver config files either on disk, or in pulpcore docs
• Create an architectural overview, or add diagrams to pages
  • Solves Douglas's "Docs need an architectural overview."

Unsolved

• Florian's "Pulp does an endless starting loop because stuff isn't available. (postgres, redis)"
  • We purposely made the systemd unit always restart a while back. "Restart=always" Because a temporarily unavailable postgres/redis should not cause the service to fail.

Already Solved

• "localhost" should be the default value for connecting to postgres & redis

Orchestration vs installation

What pulp_common does

• Runs checks for variables
• Fails if FIPS is not satisfied
• Enables repos via role
• Enables python module
• Installs prerequisites
• Gets SCL-specific info like LD_LIBRARY_PATH
• Creates pulp group
• Creates pulp user
• Adds developer user to pulp group
• Adds pulp user to redis group
• Directory management ("creates" includes setting permissions)
  • Creates /var/lib/pulp
  • Creates pulp_media_root
  • Migrates old artifact dir
  • Creates artifact dir
  • Creates cache dir
  • Creates scripts dir
  • Creates cert dir
  • Sets pulp 2 dir permissions
  • pip mode: Creates pulp_install_dir
  • Creates configuration directory
• pip mode:
  • Installs dependencies to build SELinux policy
  • Installs SELinux policy
  • Upgrades/prepare the venv
  • preflight check
  • prereq roles
  • prereq RPM/deb dependencies
  • Installs prereq_pip_packages
  • clones/checks out pulpcore/plugins via git
  • Installs pulpcore/plugins
  • Sets SELinux labels on pulpcore binaries
  • Pathces md5usedforsecurity if fips & devel install
  • Upgrades dynaconf/django-storages
  • Creates contaraints file to lock django & pulpcore versions
  • Installs gunicorn
• packages mode:
  • Creates temporary dnf.conf for best=false for pulp_pkg_upgrade_all
  • Installs pulp_pkg_undeclared_deps
  • Installs pulpcore & plugins
  • Installs SELinux policies
• Creates configuration file for pulp from pulp_settings
• Creates (empty) local configuration file for Pulp
• Creates pulpcore-manager wrapper
• Configures sudo for pulpcore-manager
• Creates galaxy signing service gpg key
• Creates galaxy signing service script
• galaxy_ng post_install role
• modifies /etc/fstab and remounts /var/lib/pulp for SELinux label
  • Only needed because we do runtime-like activities
• Creates meta service
• Starts pulp meta service
• Enables pulp meta service
• Makes variables available for the pulp_webserver role
• Restores SELinux contexts
• Collects static content
  • Runs pupcore-manager
  • Requires pulp running?
  • Requires database?
  • Requires redis?
• Restarts pulp services

What pulp_rpm_prerequisites does

• Installs zchunk-devel on EL9 from koji
• installs RPM dependencies
• installs certain pip dependencies

What galaxy_post_install does

• Installs "tar" package
• Creates /etc/galaxy-importer
• Creates /etc/galaxy-importer/galaxy-importer.cfg
• Creates /var/log/galaxy_api_access.log
• Creates /etc/default/pulpcore-api with GALAXY_ENABLE_API_ACCESS_LOG
• Imports the collection signing service gpg key file
  • Runs gpg commands for pulp user
• Imports the collection signing service script
• Creates /etc/default/pulpcore-worker with PULP_SIGNING_KEY_FINGERPRINT
• Restarts pulpcore-api.service

What pulp_database_config does

• Picks the correct host to run the remainder of the role, based on the database fields encryption key
• Checks if requires variables are set
• Imports or generates database fields encryption key
• Installs openssl command
• Sets permissions on database fields encryption key
• Runs database migrations
  • Runs pupcore-manager
  • Requires database - yes
  • Requires pulp running - no
  • Requires redis - no
• Sets Pulp admin user's password
  • Runs pupcore-manager
  • Requires database - yes
  • Requires pulp running - no
  • Requires redis - no
• Adds the galaxy signing service to the Pulp application
  • Runs pupcore-manager
  • Requires database - yes
  • Requires pulp running - no
  • Requires redis - no
• Restarts all Pulp services

What pulp_api does

• Installs python-cryptography package
• Restarts pulpcore-api.service
• Generates or imports token auth key
• Applies the SELinux type fo the TCP port
• Creates pulpcore-api service
• Starts pulpcore-api service
• Enables pulpcore-api service

What pulp_content does

• Copies and installs database fields encryption key from pulp_database_config node
• Applies the SELinux type fo the TCP port
• Creates pulpcore-content service
• Starts pulpcore-content service
• Enables pulpcore-content service
• Restarts pulpcore-content service

What pulp_workers does

• Copies and installs database fields encryption key from pulp_database_config node
• Creates pulpcore-worker template service
• Disables and removes pulpcore-resource-manager service
• Determines existing workers
• Scales down no-longer-used workers
• Enables and starts workers
• Restarts pulpcore workers services