# Document Service Scaling

## File Upload / Download

It seems like we have scaling problems with Document Service and high failure rate (~90%) with 1k concurrent requests.

There are several issues which popped up but they all related to file handling:
* File upload is done via JSON (base64 encoded string in `file` prop)
* JSON is parsed and whole file is loaded in memory to be saved in AWS S3
* Template file is being downloaded from S3 in to memory to be copied as Document file to S3 when `template_id` is being used
* When file is being requested it is downloaded from S3 to memory and being sent all at once

Current file limit is 25mb so if we wanna have 1k requests we need 21Gb of memory.
10Mb files = 10Gb mem
5Mb file = 5Gb.

OK, lets discuss different approaches:

**On slightly unrelated note:**
We need to enable XML body parser for `webhook` only.
It makes no sense to have parser enabled application wide.

this.router.post(
  ‘/v1/document/webhook/:id’,
  xmlParser(),
  this.validate({ params: idParameterSchema }),
  async (req, res, next) => {}
)

**On memory:**
So maybe we should stick to 5Mb (Also Pulsar default) for now and increase memory limit on pods to 3Gb each pod (we have 2 in prod).

> File upload is done via JSON (base64 encoded string in `file` prop) JSON is parsed and whole file is loaded to memory to be saved in AWS S3

**On file handling:**

Here we can do several things.

*Option A*
Separate file upload and document creation.

First document should be created via API endpoint using JSON blob and then file should be attached to that document (similar to signing definition) using `multipart/form-data` or `application/octet-stream`.
This way we can pipe request to stream and write file to FS and then save it on S3.

    // pipe upload to temporary file
    await new Promise((resolve, reject) => {
      const writeStream = fs.createWriteStream(tmpFile);
      req.pipe(writeStream);
      req.on('error', reject);
      writeStream.on('close', resolve);
      writeStream.on('error', reject);
    });

*Option B* (**preferable**)

We need to disable `bodyParser` for all routes where we have file upload.
Maybe for that we need to move them to a separate router and enable `bodyParser` for all routers except this one - [Using Express middleware](https://expressjs.com/en/guide/using-middleware.html#middleware.router)

For routes where we deal with files we need to use stream based JSON parsers like [stream-json](https://www.npmjs.com/package/stream-json) or [JSONStream](https://www.npmjs.com/package/JSONStream)
*First one looks more promising, imo.*

To do that we need to read `request` as stream. Node.js HTTP request extends streams and we can do `req.pipe()`.

This way we can try to exclude file from JSON body and do validations of document creation.

As far is the file concerned we need to find a way how we can pick file property and get it as stream. `stream-json` might be suitable for that.
That stream then could be piped to S3 sdk for upload. We don’t need to create buffer from stream. S3 `upload` supports streams.

Alternative option is pipe request to file and use out-of-process tools to deal with it.
So whole request body will be saved as file containing JSON and then we can use shell commands (to run outside of event loop) to extract base64 encoded file in to separate file. Afterwards request body (document props) could be read from FS and parsed as JSON for validation and persistence and file could be read as stream and piped to S3 sdk for upload.
Maybe we would need to use some sort of scheduler or task queue here. Internal IPC or [Worker Threads](https://nodejs.org/docs/latest-v12.x/api/worker_threads.html) could do it as well.

> Template file is being downloaded from S3 in to memory to be copied as Document file to S3 when `template_id` is being used

To avoid that we can use AWS S3 sdk `copyObject` method - [Class: AWS.S3 — AWS SDK for JavaScript](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#copyObject-property)

> When file is being requested it is downloaded from S3 to memory and being sent all at once

We actually need to stream data back form S3 instead of reading it all at once.
For that we need to read data in chunks `getObject` with `Range` specified ([Class: AWS.S3 — AWS SDK for JavaScript](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property)) and we need to know file size.
We can get file size from S3 by sending `HEAD` request and reading `Content-Length` - `headObject` [Class: AWS.S3 — AWS SDK for JavaScript](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject) and [res.ContentLength](https://stackoverflow.com/questions/3910071/check-file-size-on-s3-without-downloading)

Then we can use streams to send data back by piping stream to response via `res.pipe()`.

We need to test will node.js use `Transfer-Encoding: chunked` in this case or do we need to set this header ourselves to enable "http streaming".

**On file storage:**

Currently we store reference to the file in db and we do lookup there. I think it is legit approach atm but if db becomes bottle neck in the future we could test s3 scan vs db query and see what performs better.

We need to start using `prefix`es for sharding.

We could use something like `Murmur32Hash(document_uuid) % 10` to calculate the shard. It will produce sigle digit between `0` and `9`.
Or we can use just `Murmur32Hash('bc52232a-d894-407f-8b6f-04a35acbde74')` which will produce `1221424980` and it will let S3 to autoscale shards starting from `1/` and then going to `12/` and so on.
Similar objects will be sorted together in one bucket, i.e. `1221424980` and `1262090988`, but when load increases they can go to different shards i.e. `122/` and `126/`.

Full path could look like `<prefix>/<country>/<customer_id>/<document_id>.<ext>`, i.e. `1221424980/us/CUST-1337/bc52232a-d894-407f-8b6f-04a35acbde74.pdf`

More info could be found [AmazonS3](https://docs.aws.amazon.com/AmazonS3/latest/dev/optimizing-performance.html).

**On file compression:**

We should consider enabling `gzip` compression for ingress and egress, since we use `base64` encoding for the files.

> Thus, the actual length of MIME-compliant Base64-encoded binary data is usually about 137% of the original data length, though for very short messages the overhead can be much higher due to the overhead of the headers. Very roughly, the final size of Base64-encoded binary data is equal to 1.37 times the original data size + 814 bytes (for headers).

[Source - Wikipedia](https://en.wikipedia.org/wiki/Base64)
[Additional information](https://lemire.me/blog/2019/01/30/what-is-the-space-overhead-of-base64-encoding/)

I did the test with 20 paragraphs of Lorem ipsum (~14Kb) encoded with `base64` (~19Kb) and then compressed with `gzip`. End result was approximately 8.2Kb which is slightly more than half of original content size.
Of cource it worked that well with text and won't show such results with images or documents but still it will be improvement comparing to `base64` encoding overhead.

## Data queryability
In order to keep querying capabilities under control (especially if we will add querying by multiple values, i.e. list of `customer_id`s, list of `types` and so on) we need to have either `BATCH`ed responses, i.e. pagination and/or using `CURSOR`s.

Currently I am leaning towards `BATCH`ed responses to stay consistent with the rest of our services.

Also for the future we need to think about how to shard data (probably by region) and accessibility frequency, i.e. reads vs writes.
How to setup a cluster and replication to reduce RTT.

###### tags: `audibene` `work/audibene/tech`