Pulp3 RPM Advanced-Copy depsolving issue(s)

Related issues

• 1965942 - warnings RE missing dependencies
  • 9293 (backport, CLOSED-CURRENT)
• 2003764 - depsolving once-per-rpm is nonperformant
  • 9387, PR 2123
  • 9388 Backport request
  • symptoms of this problem - closed-DUP of '3764/9387/9388
    • 1934545 - assert during depsolving/logging
      • 9336, 9381 Backport request
    • 1965936 - memory use bz
      • 9335
• 1995232 - postgres-cpu/explain bz
  • 9331, 9378 Backport request

Executive Summary

There are two underlying problems. One is the query-performance raised in 1995232. The other is that Pulp3 is computing repoclosure on every single rpm added to a destinatoin during a copy, 2003764

Fixing '3764 addresses the majority of the depsolving-problem.

Issue/BZ shenanigans

Final state desired

• 6.10 GA

  • Fix 1965942 - warnings RE missing dependencies [DONE]
  • Fix 2003764 - depsolving once-per-rpm is nonperformant
    • Backport '3764 to pulp-rpm 3.14
    • Close various BZs/issues/backport-requests around the symptoms as DUPS

• Post-6.10

  • Address 1995232 - postgres-cpu/explain bz

Observations

Pulp issues

• Logging: "can't install" warnings aren't relevant to the repo-depsolve-closure case. PR submitted to remediate.
• assert/OOM signal handing can't happen, the code that needs to set a singal-handler isn't running in python main-thread
• postgres query performance is…terrible.
• assert-cause still unclear

Katello issue

• Filter-API doesn't do what the user expects
  • errata-exclude-before - pulls in ALL 32K RPMS
  • results in 4 copy-tasks
  • each task takes progressively longer, as dest fills
• Doing advanced-copy 'well' (or even 'acceptably') probably wants a new feature for next-year-katello, "advanced-copy". that sends what the user is actually asking for ("repo as of date-X plus security fixes not including any of the following RPMs")

Investigation

• Satellite-CV-creation actually fails with a cancelled task, caused by this error message:

pulpcore-worker-4[1160]: python3: ../src/rules.c:261: solver_addrule: Assertion `!p2 && d > 0' failed.
pulpcore-worker-4[1160]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Cleaning up and canceling Task b4d6e7ec-bcff-4663-9ef5-599bd3bce24b

• This error is FATAL - but rather than failing the task with a traceback, it CANCELS the task with no explanation
• need to open an issue for this, it's…Rude
• Have not recreated the fatal error above yet
• see 1934545
• On standalone Pulp, can recreate the warnings by doing an advanced-copy, of RHEL7, of "all RPMs that are not associated with Advisories" (see recipes, below)
  • biggest thing Not Found: libc.so.6 (!!), 705 (!!!) times

Investigating on Pulp3-upstream shows that, when adding base-RPMs to advanced-copy, RHEL7-x86_64 and RHEL8-x86_64 tested, all of the RPMs that couldn't find their files, were i686 (ie 32-bit) RPMs. Multi-arch strikes again?

RE assert and logging - we're going to need a SIGABRT handler instantiated prior to solver.solve(). The handler can log the problem more thoroughly; not sure how much access we have "inside" such a thing? Investigation needed.

• SQL performance during advanced copy is terrible. Here is an example of a single running postgres task - note the current elapsed time:

  5032 | 01:28:04.176268 | pulp     | SELECT "core_content"."pulp_id", "core_content"."pulp_created", "core_content"."pulp_last_updated", "core_content
"."pulp_type", "core_content"."upstream_id", "core_content"."timestamp_of_interest" FROM "core_content" LEFT OUTER JOIN "core_repositorycontent" ON ("c
ore_content"."pulp_id" = "core_repositorycontent"."content_id") WHERE ("core_repositorycontent"."pulp_id" IN (SELECT U0."pulp_id" FROM "core_repository
content" U0 INNER JOIN "core_repositoryversion" U2 ON (U0."version_added_id" = U2."pulp_id") LEFT OUTER JOIN "core_repositoryversion" U3 ON (U0."versio
n_removed_id" = U3."pulp_id") WHERE (U0."repository_id" = '17e556a3-9e77-4ada-9b61-f5fa0f31b112'::uuid AND U2."number" <= 1 AND NOT (U3."number" <= 1 A
ND U3."number" IS NOT NULL))) AND "core_content"."pulp_id" IN ('577501a8-5c99-4980-9295-eb8ce1d8bbec'::uuid, '4a9a4042-484d-4f8b-a19e-612c848dab99'::uu
id, 'cfbefb76-5587-4404-99ba-2024e96f67b3'::uuid, '13a83bb6-8c05-4624-95ed-7b8f79cd8ee8'::uuid, 'ea76d66d-948a-4080-94e7-5b4126561bda'::uuid, 'be0f23dc
-872

Useful Sat6 recipes

basic functions

• Canceling a task:
  ​​​​curl -X PATCH -d state=canceled \
  ​​​​  --cert /etc/pki/katello/certs/pulp-client.crt \
  ​​​​  --key /etc/pki/katello/private/pulp-client.key \
  ​​​​  https://$(hostname -f)/pulp/api/v3/tasks/UUID/
  

• Accessing postgres:
  ​​​​[root@sat-r220-09 ~]# sudo su - postgres
  ​​​​-bash-4.2$ psql pulpcore
  

Preparing an advanced-copy call "by hand"

• Find all advisories with an updated-date prior to a specified date

http :/pulp/api/v3/content/rpm/advisories/?repository_version=/pulp/api/v3/repositories/rpm/rpm/a534c985-4add-46f3-8ef1-49de3bbcff8b/versions/1/\&fields='pulp_href,updated_date'\&limit=5000 \
  | jq '.results[] \
  | select(.updated_date < "2016-03-17")' \
  | jq .pulp_href > errata_config

• Find packages that aren't part of an UpdateRecord (note: works when there's only one repo sync'd. Needs work to find just "…for a specific repository")
  ​​​​-bash-4.2$ psql pulpcore
  ​​​​pulpcore=# select count(p.name) from rpm_package p where not exists (select 1 from rpm_updatecollectionpackage ucp where ucp.name = p.name and ucp.epoch = p.epoch and ucp.version = p.version and ucp.release = p.release and ucp.arch = p.arch);
  

• Find content-ptr-id for all packages not in UpdateRecords:
  ​​​​psql -U pulp -d pulp --host 127.0.0.1 \
  ​​​​  -c "select content_ptr_id from rpm_package p where not exists (select 1 from rpm_updatecollectionpackage ucp where ucp.name = p.name and ucp.epoch = p.epoch and ucp.version = p.version and ucp.release = p.release and ucp.arch = p.arch)" \
  ​​​​   > base_packages
  

• Prepare base_packages to hand off to /copy/:
  • Delete header/footer
  • Replace header with:
  ​​​​[
  ​​​​{
  ​​​​"source_repo_version": "/pulp/api/v3/repositories/rpm/rpm/e9e67b6c-5a50-4a48-907b-5d527169c633/versions/1/", 
  ​​​​"dest_repo": "/pulp/api/v3/repositories/rpm/rpm/d0422d2b-e232-4f5c-9ec2-fdf2d737c5bf/",
  ​​​​"content": [
  

  • Replace footer with:
  ​​​​        ]
  ​​​​}
  ​​​​]
  

  • Replace UUIDs with "/pulp/api/v3/content/rpm/packages/UUID/",
• EXAMPLE:
  ​​​​[
  ​​​​{
  ​​​​"source_repo_version": "/pulp/api/v3/repositories/rpm/rpm/e9e67b6c-5a50-4a48-907b-5d527169c633/versions/1/", 
  ​​​​"dest_repo": "/pulp/api/v3/repositories/rpm/rpm/d0422d2b-e232-4f5c-9ec2-fdf2d737c5bf/",
  ​​​​"content": [
  ​​​​    "/pulp/api/v3/content/rpm/packages/ceb32b0a-11cf-41fc-ac06-96ed4b8a9cec/",
  ​​​​    "/pulp/api/v3/content/rpm/packages/6b78b4fd-5cb3-48cf-a3d1-a448a529fed4/",
  ​​​​....
  ​​​​"/pulp/api/v3/content/rpm/packages/e18707da-dae6-4309-b523-be0393f72a65/"
  ​​​​  ]
  ​​​​}   
  ​​​​]
  

• Issue the /copy/ command:
  ​​​​http POST :/pulp/api/v3/rpm/copy/ \
  ​​​​  dependency_solving=True \
  ​​​​  config:=@./base_packages
  

Pulling useful info from journalctl

• Find the things missing-from depsolve warnings:
  ​​​​journalctl | \
  ​​​​  grep "WARNING: Encountered problems solving dependencies, copy may be incomplete: package" | \
  ​​​​  awk -F"copy may be incomplete:" '{print $2}' | \
  ​​​​  awk -F" " '{print $4}' | 
  ​​​​  sort | uniq -c | sort -n
  

• Find the packages-looking-for-missing:
  ​​​​journalctl | \
  ​​​​  grep "WARNING: Encountered problems solving dependencies, copy may be incomplete: package" | \
  ​​​​  awk -F"copy may be incomplete:" '{print $2}' | \
  ​​​​  awk -F" " '{print $2}' | \
  ​​​​  sort | uniq -c | sort -n