# Refactoring mall-data configs

## Context
There is significant level of redundancy in config files. The goal is to reduce that while maintaining a clean config structure.

## Suggested config updates:
#### Structure:

> mall-data/confs

    > common_configs
        > scraped_collections.yaml
            # will contain all the scraped feeds' raw db names and urls.

    > locale_configs
        > en_ca_configs # will contain configs for all en_ca locale feeds.
             > feed_configs.yaml
             > affiliate_stores.yaml # will have map: {"cjcom": {child_store_name: details of the store, child_store_name: details of the store}, "adept_portal": {child_store_name: details of the store, child_store_name: details of the store}, ...}
        > en_us_configs # will contain configs for all en_us locale feeds.
             > feed_configs.yaml
             > affiliate_stores.yaml # will have map: {"cjcom": {child_store_name: details of the store, child_store_name: details of the store}, "adept_portal": {child_store_name: details of the store, child_store_name: details of the store}, ...}
        > fr_ca_configs # will contain configs for all fr_ca locale feeds.
             > feed_configs.yaml
             > affiliate_stores.yaml # will have map: {"cjcom": {child_store_name: details of the store, child_store_name: details of the store}, "adept_portal": {child_store_name: details of the store, child_store_name: details of the store}, ...}
        > fr_us_configs # will contain configs for all fr_us locale feeds.
             > feed_configs.yaml
             > affiliate_stores.yaml # will have map: {"cjcom": {child_store_name: details of the store, child_store_name: details of the store}, "adept_portal": {child_store_name: details of the store, child_store_name: details of the store}, ...}
        > fr_fr_configs # will contain configs for all fr_fr locale feeds.
             > feed_configs.yaml
             > affiliate_stores.yaml # will have map: {"cjcom": {child_store_name: details of the store, child_store_name: details of the store}, "adept_portal": {child_store_name: details of the store, child_store_name: details of the store}, ...}
        > pl_pl_configs # will contain configs for all pl_pl locale feeds.
             > feed_configs.yaml
             > affiliate_stores.yaml # will have map: {"cjcom": {child_store_name: details of the store, child_store_name: details of the store}, "adept_portal": {child_store_name: details of the store, child_store_name: details of the store}, ...}
        > es_mx_configs # will contain configs for all es_mx locale feeds.
             > feed_configs.yaml
             > affiliate_stores.yaml # will have map: {"cjcom": {child_store_name: details of the store, child_store_name: details of the store}, "adept_portal": {child_store_name: details of the store, child_store_name: details of the store}, ...}
        > ...

    > client_configs
        > atrium
            > attrium_pipeline.yaml
            > feed_configs.yaml
                # will contain the configs only if client has certain details which are different from locale level e.g. brand filtering, memory requirements etc. If not specified, locale level configs will be picked.
            > affiliate_stores.yaml
                # Will contain all the affiliate feeds of the client with their child affiliate stores. The details provided here will override the locale level store details.
            > feed_schedule.yaml
        > carbonleo
            > carbonleo_pipeline.yaml
            > feed_configs.yaml
            > affiliate_stores.yaml
            > feed_schedule.yaml
        > ...

#### Reasoning:
- Feeds for a particular locale have almost the same configurations across clients hence they have been clubbed together.
- Scrape urls can be common across clients hence they have been moved separately.

## Advantages of the new structure:
- Minimum code changes will be needed to integrate an already integrated feed for some other client as only name need to be added in pipeline.yaml file. It will speed up our integration process and will reduce dependency over integration scripts.
- '**One fix for all**' approach will work here. Any bug fixed(related to configurations) will fix that bug for all the clients for that region eg. big_store change.
- It will speed up our integration process. Will reduce dependency over integration scripts.
- Feed removal will also be simplified.

## Code changes - efforts and estimation:
- Will explore it post design is finalaized after discussion.

## Demo repo:
- Repo url: https://github.com/deepak-bhardwaj-adeptmind/refactor-configs
- output of `generate_configs.py`

Client configs:
{
    "shared": {"db_host": "db_host", "region": "ca"},
    "feeds": {
        "flexoffers": {
            "english": {
                "feed_info": {"API_KEY": "overriding_api_key"},
                "db_name": "flexoffers_en_ca",
            }
        },
        "cjcom": {
            "english": {
                "feed_info": {
                    "authorization": "bearer_token",
                    "website-id": "website-id",
                },
                "db_name": "cjcom_en_ca",
            }
        },
        "boathouse": {
            "english": {
                "feed_info": {"is_brand": True, "raw_product_db": "raw_boathouse"},
                "db_name": "final_boathouse_en_ca",
            }
        },
        "claires": {
            "english": {
                "feed_info": {
                    "raw_product_db": "raw_claires",
                    "include_brands": ["Clairs"],
                },
                "db_name": "final_claires_en_ca",
            }
        },
    },
}



Affiliate stores:
{
    "cjcom": {
        "Fossil": {
            "STORE": "FOSSIL",
            "store_name": "Fossil",
            "advertiser_ids": "xxxxxxFossil",
        }
    },
    "flexoffers": {
        "Club Monaco": {
            "store_name": "Club Monaco",
            "cid": "xxxxxxClub",
            "product_store_id": "clubmonaco",
            "is_brand": False,
        },
        "Eddie Bauer": {
            "store_name": "Eddie Bauer",
            "cid": "xxxxxxEddie",
            "product_store_id": "eddiebauer",
        },
    },
}