---
# System prepended metadata

title: Store ChRIS Logs in Object Storage

---

# Store ChRIS Logs in Object Storage

```text
---
title: Store ChRIS Logs in Object Storage
authors:
  - "@PridhiArora"
reviewers:
  - 
  
creation-date: 2022-10-14
last-updated: 2022-10-14
status: implementable
```

## Table of Contents

- [Store ChRIS Logs in Object Storage](#store-chris-logs-in-object-storage)
  - [Table of Contents](#table-of-contents)
  - [Glossary](#glossary)
  - [Summary](#summary)
  - [Motivation](#motivation)
    - [Goals](#goals)
  - [Proposal](#proposal)
    - [User Stories](#user-stories)
      - [Story 1 -Decreasing the latency in ChRIS backend.](#story-1---decreasing-the-latency-in-chris-backend)
    - [Requirements](#requirements)
      - [Functional Requirements](#functional-requirements)
      - [Non-Functional Requirements](#non-functional-requirements)
    - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)
      - [Proposed Changes](#proposed-changes)
    - [Test Plan](#test-plan)
      - [Subtasks](#subtasks)
    - [Graduation Criteria](#graduation-criteria)
  - [Implementation History](#implementation-history)

## Glossary

- S3 - Amazon S3 is an object storage service that stores data as objects within buckets
- Minio - Minio is an open-source distributed object storage server written in Go, designed for Private Cloud infrastructure providing S3 storage functionality

## Summary

ChRIS project currently stores log messages as strings in Postgres DB, and this causes Postgres DB to be populated with relatively large strings. The large size causes the backend to slow down.
One quick fix is to truncate the log string that is already implemented and use only a limited string. Right now, a substring until 3000 words is populated into the database, and the rest is truncated. But this devoids us of a lot of information. The decoration of log strings is done to make messages aesthetically pleasing. Removing unnecessary characters, such as emojis and Chinese characters, reduces the string size from 4000B to 2000B. Both the above fixes are still not enough as log size increases daily, and it is essential to find a better solution, and this is where object storage comes into the picture.


## Motivation

The absence of a proper storage system for logs is slowing the backend, causing many users to be frustrated. The idea is to have an appropriate design so that ChRIS system is seamless. 



### Goals

1. To allow implementation of object storage for storage of logs.
2. To increase performance as well by using asynchronous data streams instead of buffering full responses.

## Proposal

### User Stories

#### Story - Decreasing the latency in ChRIS backend.

Stacy works at an IT department of an organization where ChRIS backend is used; she intends to use it regularly and needs logs to refer to. Storing logs in object storage would give her more readable logs.

### Requirements

#### Functional Requirements

- FR1: Run Docker container with Minio
- FR2: Connect Docker container to ChRIS' backend.
- FR3: Storge logs inside of Minio

#### Non-Functional Requirements

- NFR1: Unit tests MUST exist for the support.


### Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints

#### Proposed Changes

Currently, the summary gets saved in Postgres. 

When object storage will be implemented, the following lines would need to be changed.

```python
 self.c_plugin_inst.status = 'started'
self.c_plugin_inst.summary = self.get_job_status_summary()  # initial status
self.c_plugin_inst.raw = json_zip2str(d_resp)
self.c_plugin_inst.save()
```

Following is a code snippet taken from the official [Minio Docs](https://min.io/docs/minio/linux/developers/python/minio-py.html) that can be used to implement the functionality.

```python
from minio import Minio
from minio.error import S3Error


def main():
    # Create a client with the MinIO server playground, its access key
    # and secret key.
    client = Minio(
        "play.min.io",
        access_key="Q3AM3UQ867SPQQA43P2F",
        secret_key="zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
    )

    # Make 'asiatrip' bucket if not exist.
    found = client.bucket_exists("asiatrip")
    if not found:
        client.make_bucket("asiatrip")
    else:
        print("Bucket 'asiatrip' already exists")

    # Upload '/home/user/Photos/asiaphotos.zip' as object name
    # 'asiaphotos-2015.zip' to bucket 'asiatrip'.
    client.fput_object(
        "asiatrip", "asiaphotos-2015.zip", "/home/user/Photos/asiaphotos.zip",
    )
    print(
        "'/home/user/Photos/asiaphotos.zip' is successfully uploaded as "
        "object 'asiaphotos-2015.zip' to bucket 'asiatrip'."
    )


if __name__ == "__main__":
    try:
        main()
    except S3Error as exc:
        print("error occurred.", exc)
Run File Uploader
$ python file_uploader.py
'/home/user/Photos/asiaphotos.zip' is successfully uploaded as object 'asiaphotos-2015.zip' to bucket 'asiatrip'.

```

### Test Plan

- Unit tests.



### Graduation Criteria

- Docker-supported Minio
- Integration of ChRIS with Minio
- Unit tests to test the functionality

