Date: 05/07/2023 **Incident** During the integration of the Ethereum indexer with the Terminal, Ethereum environment variable was added directly to the Terminal production environment. During the addition of the Ethereum variable, Polygon environment variable got misconfigured due to which the terminal couldn’t reach the Polygon GraphQL server, and this lead to undefined data on the accounts with Polygon. The env file was manually modified, and one extra Polygon environment variable was added by mistake: ``` POLYGON_SERVER_DEV=http://87.98.139.116/polygon/graphql POLYGON_SERVER_DEV=http://87.98.139.116/ethereum/graphql ETHEREUM_SERVER_DEV=http://87.98.139.116/ethereum/graphql ``` **Process to be be followed to avoid such instances** * SDE's should create a list of features/tasks which is getting released in Production and get explicit approvals from PMs. Always have these done on common Slack groups with Anuj and all relevant developers CCed * The branch (to be deployed in Prod) must be E2E tested in staging and only this branch must be pushed to Production. In case of any changes in the branch, it must go through E2E testing in staging again. This must be done by both PMs & SDEs * Inform CT & Business team of the deployment date and estimated time window for which the website/terminal will be down. Do this atleast 24 hours in advance so that they have enough time to inform the stakeholders * Call will be setup with the PMs and the developers at a pre-agreed date & time when the deployment will happen. Post deployment testing of the features released & sanity testing of the entire platform will be done on the call itself. For sanity testing, we ll have basic test cases written which must be tested out (Product team is responsible for coming up with these test cases) * SDEs must make sure that all deployments follow proper conventions ( DEV -> STAGING -> PROD ). Ensure all test cases provided are tested in in each of the stages. Always ask for test cases from PMs * Code modifications must be peer reviewed THOROUGHLY to ensure proper deployment. * All prod deployments should be dome at a fixed time slot of 12AM to 1AM IST(To be finalized with Anuj on basis of client's geography)