# Cloud automation self-learning project : Cron-send-receipt-spotify

[](https://hackmd.io/DlraECW4Q_imvwXGCAM1Ag)

A Python job executed by Cloud Run and scheduled by Cloud Scheduler at the end of every month) which retrieves from
Gmail the latest receipt of Spotify Family and notifies all family members within a Facebook Messenger group.

## Prerequisites

- Services:
- Gmail + Facebook account
- GCP account
- Authentification:
- Enabled GCP APIs:
- Cloud Run Admin API
- Cloud Scheduler API
- Secret Manager API
- Compute Engine API
- Gmail API
- Cloud Logging API
- Eventarc API
- Cloud Resource Manager API
- Identity and Access Management (IAM) API
- Cloud Billing API
- OAuth 2.0 Client IDs generated
- Tools:
- Poetry >= 1.1.10
- Python >= 3.7.2
- Docker latest
- Terraform 1.0.11

## Installation

### 1. Clone the repository

With Terminal:

  git clone [e-mail]:doanhat/cron-send-receipt-spotify.git
  cd cron-send-receipt-spotify/

With Pycharm:

> Open `Pycharm` -> `GET FROM VCS` -> Copy and paste the git url in the field `URL`

### 2. Install dependencies:

Open Terminal (system or Pycharm), **within the repository folder**:

  poetry install # install virtual environment and python packages
  poetry export -f requirements.txt --output requirements.txt --without-hashes # generate requirements.txt

:warning: In Pycharm, remember to config interpreter to choose the virtual environment generated by Poetry

:warning: If some dependecy problems occur, delete the `poetry.lock` file and retry the above command

### 3. Set up the cron job !!

1. Set up authentification:
1. Google API credentials for Gmail readonly access:
- Create a new project in `Google Cloud Console`, for example `send-facture-spotify`
- In `APIs and Services`, enable Gmail API
- In `APIs and Services/OAuth Consent Screen`, enter application name and add your email to `Test users`
- In `APIs and Services/Credentials`, create a `OAuth 2.0 Client ID` and download the JSON file, save it
to `Google Secret Manager` as `gmail_credentials`
2. Set up GCP service account:
- https://cloud.google.com/docs/authentication/getting-started
- Grant roles for the service account: `Cloud Run Admin`, `Cloud Scheduler Admin`, `Service Account user`
, `Secret Manager Accessor and Version Adder`
3. Set up remote credentials, perform these additional steps:
1. In `Google Cloud Console/APIs and Services/Credentials`, add a `Service Account` for the project if not yet
created
2. In `Google Cloud Console/Security/Secret Manager`, add this secret :
1. name: `facebook_login`, value:

                   {
                      "FB_USER_EMAIL_ADDRESS":<fb_email>,
                      "FB_USER_PASSWORD":<fb_password>
                   }

2. At this moment, two secrets have been created, which are `facebook_login` and `gmail_credentials`, after
the first run of Cloud Run, there will be also `facebook_session` and `gmail_token` generated, which will be reused for the next run

2. Build image Docker:
- In Terminal (system or Pycharm), **within the repository folder**:

  docker build -t cron-send-receipt-spotify:<version> -f docker/gcp_cloud_run/Dockerfile .
  docker tag cron-send-receipt-spotify:<version> <gcp_region>/<gcp_project_name>/cron-send-receipt-spotify
  docker push <gcp_region>/<gcp_project_name>/cron-send-receipt-spotify 

3. Deploy GCP resources with terraform:
- Define environment variables in `terraform/main.tf`, in `locals`:

   locals {
      service_account = "[e-mail]"
      thread_id = "<messenger_group_id>" # messenger group id
      project_id = "send-facture-spotify" 
      cre_secret_id = "gmail_credentials"
      tok_secret_id = "gmail_token"
      ses_secret_id = "facebook_session"
      log_secret_id = "facebook_login"
      gcp_region = "europe-west1"
      imap_url = "imap.gmail.com"
  }

> :information_source: `<messenger_group_id>` example: `6947468915279299` is the id of https://www.facebook.com/messages/t/6947468915279299

- Verify the job `schedule` :

   resource "google_cloud_scheduler_job" "scheduler-send-receipt-spotify" {
      name             = "monthly-report-spotify-receipt"
      description      = "It will send monthly receipt to messenger group"
      schedule         = "59 10 28-31 * *"
      time_zone        = "CET"
      http_target {
         http_method = "GET"
         uri = google_cloud_run_service.run-send-receipt-spotify.status[0].url
         oidc_token {
            service_account_email = local.service_account
        }
     }
  }

     which means: At 10:59 on every day-of-month from 28 through 31, run the script

:information_source: Check out [crontab.guru](https://crontab.guru/#59_10_28-31_*_*) for more cron schedule expressions

- In Terminal (system or Pycharm), **within the repository folder**:
- ⚠️ Replace `PROJECT_NUMBER` with current gcp project number

  make terraform-init
  make terraform-plan
  make terraform-apply

### 5. Voilà, all set !!

### 6. Improvement :

There's always enough rooms for improvement, we can take advantage of this project to learn :

- Using Airflow to schedule more complicated tasks
- Using CI/CD for more reliable deployment

## Author

- [@doanhat](https://github.com/doanhat)

## Links

[](https://www.linkedin.com/in/minhdoan272/)
[](https://www.facebook.com/dnminhhhhh/)
[]([e-mail])

## License

[](https://choosealicense.com/licenses/mit/)

## Reference:

[Read Emails from Gmail using Gmail API](https://www.geeksforgeeks.org/how-to-read-emails-from-gmail-using-gmail-api-in-python/)

[Gmail API - Quick Started](https://developers.google.com/gmail/api/quickstart/python)

[Fbchat documentation](https://fbchat.readthedocs.io/en/stable/)