# PulpImport/Export Memory Investigation

## Problem statement
* Large datasets are serialized in memory before written to disk.
* `json.dump` cannot be tricked to loop over itertators (in any sane way)
* django_import/export `export()` returns a list in memory instead of a lazy iterator.
* **This is a real problem "in the wild"** with actual Red Hat data and users - we need a short-term solution, ASAP

## Proposed solutions

### Export in chunks
1. Write `"["` to a tempfile
2. Call `export(...).json.encode(...)` on chunks of our queryset (which will be a string with json-list starting/ending with `"[]"`)
3. remove first and last character from the exported string
4. write the string to the tempfile
5. if chunks are left write `","`
6. repeat from 2.
7. Write `"]"` to the tempfile
8. add the tempfile to the tar

#### Advantages:
* contained in one place (`pulpcore.app.importexport._write_export()`)
* we own "all" the code involved
* doesn't try to "take advantage of" current implementation details of lower level libs
* never more than one resource-worth of tempfile at a time on disk
* never more than one "chunk" in memory at a time ("how much" depends on what batch-size we use to work w/ the queryset as we export)
* this change doesn't change the export-file-format - so it doesn't require 'import' to know that we've done anything
* We continue to call the libraries as intended
#### Disadvantages
* post_export() called per-batch, instead of once-per-entire-export-set
* this may not be a problem
* if it is - we already have it in low-memory situations
* we have tests!
* this is UGLY - much commentary needed in the code to explain why we're doing this
* We do string operations on the output

### Create our own export_to_file in QueryModelResource
* This is not necessary for the fix, but a possible way to refactor the code towards the ultimate goal below.
* Would use the above approach, but encapsulate inside of `pulpcore.plugin.importexport.QueryModelResource`, which all PIE model-resources subclass from.
* Sets us up for making it possible to do stream-to-file in the future, by changing this one method (instead of inline somewhere else)
* `export_to_file` would take a `query_set`, a `file_stream`, the `format` and maybe a `batch_size` parameter.

## Ultimate Goal
* show django-import-export what we have to do to work through this problem
* work w/ them to add new d-i-e API to make it possible
* accept new release of d-i-e, and remove all this ugly code all at once

###### tags: `import/export`