owned this note changed 3 years ago
Published Linked with GitHub

Pulp3, FIPS, and MD5 repositories

Sync md5 fixture on_demand and look at rpm_package.checksum_type and .pkgId. The checksum_type is "md5" (because that's what's in primary.xml) and the pkgId is the matching md5 checksum from that file.

Especially in the on_demand case, if the repo only has md5 checksums, you either use them here, or you have to refuse to sync - we have no other data for identifying the package/pkgId

Questions:

  • Why did this even work?!?
    • A: Wasn't running on a FIPS machine, so found md5 to validate download
  • Declarative Artifact - why did it allow MD5?
    • Why didn't init-and-validate catch this? - not involved
    • BulkCreate is not calling pre_save? - pre_save not called on Bulk

At this location:

if the Artifact specifies a checksum that is not in DIGEST_FIELDS, we end up with an EMPTY expected_digests. After that, the Artifact ends up NOT VALIDATING ANY CHECKSUM (because it can't). We need a check/exception thrown here?

  • What to do about on-demand?
    • Should prevent users from syncing content in this scenario
    • How?

In pulp_rpm, currently this is the first place we can notice a problem: https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/tasks/synchronizing.py#L601

In order to address this problem for All The Things, we need to fix/address this here (ie, verify that DeclaritiveArtifacts coming thru the pipeline are declaring valid checksum-types) (also means wee do NOT need anything in synchronizing - we'll never get that far):

https://github.com/pulp/pulpcore/blob/b94abd64d76ea4554e6750ff38ce458eaa888cc8/pulpcore/plugin/stages/artifact_stages.py#L48

Conclusions:

  • two fixes (repository 302 (7853) and artifact_stages 48 (7854)
    • needs two issues
  • need a test that attempts to sync the md5-fixture AND FAILS, regardless of FIPS-box or not
    • required in one of the above issues (7854)
  • pre_save() needs to stay
  • pulp_rpm - needs to limit CHECKSUM-CHOICES
    • do this in serializer
    • needs new issue (7855)
    • needs to not have a duplicate list in contants.py, but rather generate from Artifact
  • add doc to plugin-authors-guide about ALLOWED_CHECKSUMS and how they should pay attention to it
    • needs an issue (7856)
  • testing MUST happen on a FIPS-enabled box

Testing

Here is a script to sync/distribute/publish a repo using the md5-only fixture:

#!/bin/bash

# Poll a Pulp task until it is finished.
wait_until_task_finished() {
    echo "Polling the task until it has reached a final state."
    local task_url=$1
    while true
    do
        local response=$(http $task_url)
        local state=$(jq -r .state <<< ${response})
        case ${state} in
            failed|canceled)
                echo "Task in final state: ${state}"
                exit 1
                ;;
            completed)
                echo "$task_url complete."
                break
                ;;
            *)
                echo -n "."
                sleep 1
                ;;
        esac
    done
    echo ""
}

ZOO_URL="https://fixtures.pulpproject.org/rpm-with-md5/"
ZOO_NAME="zoo"
# create repo
ZOO_HREF=$(http POST :    :/pulp/api/v3/repositories/rpm/rpm/ name=$ZOO_NAME | jq -r '.pulp_href')
echo "repo_href : " $ZOO_HREF
if [ -z "$ZOO_HREF" ]; then exit; fi
# add remote
http POST :/pulp/api/v3/remotes/rpm/rpm/ name=$ZOO_NAME url=$ZOO_URL  policy='immediate'
# find remote's href
REMOTE_HREF=$(http :/pulp/api/v3/remotes/rpm/rpm/ | jq -r ".results[] | select(.name == \"${ZOO_NAME}\") | .pulp_href")
echo "remote_href : " $REMOTE_HREF
if [ -z "$REMOTE_HREF" ]; then exit; fi
# sync
TASK_URL=$(http POST :$ZOO_HREF'sync/' remote=$REMOTE_HREF | jq -r '.task')
echo "Task url : " $TASK_URL
if [ -z "$TASK_URL" ]; then exit; fi
# wait for task
wait_until_task_finished :$TASK_URL
# find repo-version
REPOVERSION_HREF=$(http :$TASK_URL| jq -r '.created_resources | first')
echo "repoversion_href : " $REPOVERSION_HREF
if [ -z "$REPOVERSION_HREF" ]; then exit; fi
# publish
TASK_URL=$(http POST :/pulp/api/v3/publications/rpm/rpm/ repository=$ZOO_HREF | jq -r '.task')
echo "Task url : " $TASK_URL
if [ -z "$TASK_URL" ]; then exit; fi
wait_until_task_finished :$TASK_URL
# find latest publication
PUBLICATION_HREF=$(http :$TASK_URL| jq -r '.created_resources | first')
echo "publication_href : " $PUBLICATION_HREF
if [ -z "$PUBLICATION_HREF" ]; then exit; fi
# show it
http :$PUBLICATION_HREF
# Distribute it
TASK_URL=$(http POST :/pulp/api/v3/distributions/rpm/rpm/ name=$ZOO_NAME base_path=$ZOO_NAME publication=$PUBLICATION_HREF | jq -r '.task')
echo $TASK_URL
if [ -z "$TASK_URL" ]; then exit; fi
# wait for task
wait_until_task_finished :$TASK_URL
# find latest distribution
DISTRIBUTION_HREF=$(http :$TASK_URL | jq -r '.created_resources | first')
echo "distribution href : " $DISTRIBUTION_HREF
if [ -z "$DISTRIBUTION_HREF" ]; then exit; fi
# show it
http :$DISTRIBUTION_HREF
tags: FIPS
Select a repo