Related Issues: Incremental-FS-export Clean up import/export CLI Add fs-exporter to CLI Executive Summary There are two kinds-of Exporters in use, PulpExporter and FilesystemExporter. There is currently just one import-type, PulpImporter and PulpImport.
1/24/2023Introduction/problem statement There is a use-case for extracting repository-versions out of a running instance of Pulp into a file that can be transferred to another Pulp instance and imported. This is not the Pulp-to-Pulp sync case; the assumption is that the receiving Pulp instance is network-isolated. This design aims to describe the requirements, limitations, and a possible implementation for this functionality. Additional Documents low level design document
1/24/2023Problem statement Large datasets are serialized in memory before written to disk. json.dump cannot be tricked to loop over itertators (in any sane way) django_import/export export() returns a list in memory instead of a lazy iterator. This is a real problem "in the wild" with actual Red Hat data and users - we need a short-term solution, ASAP Proposed solutions Export in chunks Write "[" to a tempfile Call export(...).json.encode(...) on chunks of our queryset (which will be a string with json-list starting/ending with "[]")
1/24/2023Executive Summary Pulp Import/Export cannot rely on pulp_id to differentiate objects at import-time, due to pulp_id being 'owned' by the downstream instance, not the upstream. We propose adding a field to Content for the use of Pulp Import/Export, export_id, to give us a way to be able to use an upstream-pulp-id to identify entities created downstream, in order to be able to rebuild relationships. Introduction The PulpImport/Export (PIE) feature is aimed at allowing the export of repository content and artifacts from one Pulp instance ("upstream") in a way that allows them to be transported to another Pulp instance ("downstream"), such that "downstream" ends up with a Repository/Repositories with RepositoryVersions that contain the same Content as the exported "upstream" had. PIE accomplishes this in two ways.
1/24/2023