# API Integration
## Description
This document contains how to integrate API from many sources to the metrics. Eg Stripe, Google API, Facebook API and etc.
## Folder structure
Here's a folder structure for allmetrics:
```
allmetrics/ # Root directory.
|- apps/ # Folder used to store apps files.
|- - connects / # Folder used to store API integration from many sources
|- - - urls.py # which uses to set URLconf for the sources and authorization
|- - - utils.py # collection of functions for general data-processing or numerical. widely used by other modules
|- - - views.py # which uses to store authorize and callback function for oauth2
|- config
│- - settings
│- - - __init__.py
│- - - base.py # which uses to store base and core settings
│- - - local.py # which uses to store local settings, we will put our API settings in here
│- - - production.py # which uses to store production settings, we will put our API settings in here
│- - urls.py # which uses to set main URLconf
│- - wsgi.py # which uses for wsgi setting
|- .env # which uses to store our credentials key and other configuration parameters.
```
## Creating Stripe API Integration
Let's start to create the API integration, We will use Stripe API as the example. Please note that every source has a different way to get the credential keys. At first, we need to set up our stripe connect so we can let users granted our application to their stripe account, so we can access their data using the access token.
### Set API KEY
1. To get the our application Credentials from stripe we need to go to stripe [Dashboard Live](https://dashboard.stripe.com/account/apikeys) or [Dashboard Test](https://dashboard.stripe.com/test/apikeys)
2. Point to **API keys** -> **Standard keys** -> Copy the credentials **Publishable key** and **Secret key** and put it on **.env** with variables **STRIPE_API_SECRET_KEY** and **STRIPE_API_PUBLISHABLE_KEY**
```
cd allmetrics
open .env
```
```
STRIPE_API_SECRET_KEY=
STRIPE_API_PUBLISHABLE_KEY=
```
> Note: Stripe uses test data with test credentials so we can test our API at their sandbox without using our live data.
### Set Application Authorize and request token URL
Go to https://dashboard.stripe.com/test/settings/applications and get the application URL by clicking on **Test OAuth integration** -> **Standard** and add to .env
```
STRIPE_REQUEST_TOKEN_URL=https://connect.stripe.com/oauth/token
STRIPE_AUTHORIZE_URL=https://connect.stripe.com/oauth/authorize
```
### Set Application CLIENT_ID
Bellow **Test OAuth integration** there is **Test mode client ID**, copy and add to .env
```
STRIPE_CLIENT_ID=
```
### Set Redirect URI
The last thing is to set up the redirect URI. Bellow **Test mode client ID** there is **Redirects**, set the redirect URI to `http://localhost:8000/connect/oauth/callback/stripe/` and add to .env
```
STRIPE_REDIRECT_URI=/connect/oauth/callback/stripe/
```
So the whole settings will look like this
```
STRIPE_API_SECRET_KEY=
STRIPE_API_PUBLISHABLE_KEY=
STRIPE_REQUEST_TOKEN_URL=https://connect.stripe.com/oauth/token
STRIPE_AUTHORIZE_URL=https://connect.stripe.com/oauth/authorize
STRIPE_CLIENT_ID=
STRIPE_REDIRECT_URI=/connect/oauth/callback/stripe/
```
### Add the settings to locals.py and production.py
```
# STRIPE ACCOUNT
STRIPE_API_SECRET_KEY = env("STRIPE_API_SECRET_KEY", default='')
STRIPE_API_PUBLISHABLE_KEY = env("STRIPE_API_PUBLISHABLE_KEY", default='')
STRIPE_REQUEST_TOKEN_URL = env("STRIPE_REQUEST_TOKEN_URL", default='')
STRIPE_AUTHORIZE_URL = env("STRIPE_AUTHORIZE_URL", default='')
STRIPE_CLIENT_ID = env("STRIPE_CLIENT_ID", default='')
STRIPE_REDIRECT_URI = env("STRIPE_REDIRECT_URI", default='')
```
> Note: Set the default to empty so it will leave blank instead otherwise the .env variable is will produce error because key not found.
## Connect to Oauth2
We put all oauth2 logic on `apps/connect/views.py` which contains an **authorize** and **callback** function where it can be used by any sources that using oauth2.
### Create Data Source
Point to http://localhost:8000/admin/sources/datasource/ and add **Stripe** as the name of data source and make sure the slug is lower **stripe**. It will look like this
**Why we need to create a data source?** Because we will load it to user templates especially on sources dashboard, e.g
`apps/templates/sources/index.html`
```
<div class="modal-body">
<div class="container">
<div class="row">
{% for i in data_source %}
<div class="col">
<a href="/connect/authorize/{{i.slug}}/">{{i.name}}</as>
</div>
{% endfor %}
</div>
</div>
</div>
```
Users will see what sources available in our system, and they can just connect their sources to our app by clicking one of them.
This approach will be efficient because we can control what sources that available in our system. Let's say stripe has changed its API object structure, we can just disable the stripe source, fixing it, and then enable it again. and if we want to integrate/add more source we don't need to change the code on the front end like adding new data source because it will be added automatically.
### Define API_CREDENTIALS
1. Open `apps/connect/views.py` and define **API_CREDENTIALS** as a dictionary so it can be easy to define other sources credentials in the future.
```
SITE_URL = settings.SITE_URL
API_CREDENTIALS = {
"stripe": {
"base_url": settings.STRIPE_AUTHORIZE_URL,
"client_id": settings.STRIPE_CLIENT_ID,
"redirect_id": SITE_URL + settings.STRIPE_REDIRECT_URI,
"client_secret": settings.STRIPE_API_SECRET_KEY,
"token_url": settings.STRIPE_REQUEST_TOKEN_URL,
"user_key": "stripe_user_id",
"access_token_key": "access_token"
}
}
```
If we want to add more sources in the future we can just update the dictionary.
```
API_CREDENTIALS = {
"stripe": {
"base_url": settings.STRIPE_AUTHORIZE_URL,
"client_id": settings.STRIPE_CLIENT_ID,
"redirect_id": SITE_URL + settings.STRIPE_REDIRECT_URI,
"client_secret": settings.STRIPE_API_SECRET_KEY,
"token_url": settings.STRIPE_REQUEST_TOKEN_URL,
"user_key": "stripe_user_id",
"access_token_key": "access_token"
}, "google": {
"base_url": settings.GOOGLE_AUTHORIZE_URL,
"client_id": settings.GOOGE_CLIENT_ID,
"redirect_id": SITE_URL + settings.GOOG_REDIRECT_URI,
"client_secret": settings.GOOGE_API_SECRET_KEY,
"token_url": settings.GOOG_REQUEST_TOKEN_URL,
"user_key": "key_user_id",
"access_token_key": "key_access_token"
}, "facebok": {
...
}
}
```
Most sources have different objects result when we request their authorization. That's why we need to know what objects key to get **user key** and **access token key**. Which in this case stripe using **stripe_user_id** and **access_token_key** https://stripe.com/docs/connect/oauth-reference#post-token-request
### Define authorize and callback
1. We have connect_authorize_view with url config /connect/authorize/<slug>/ and connect_authorize_callback
```
def connect_authorize_view(request, slug):
if not request.user.is_authenticated:
return HttpResponseRedirect(reverse('login'))
# We will match the slug with our key on API_CREDENTIAL dictionary, from there we can get stored values and keys.
credentials = API_CREDENTIALS.get(slug)
if not credentials:
return HttpResponse("Credentials not found")
url = credentials.get('base_url')
client_id = credentials.get('client_id')
redirect_id = credentials.get('redirect_id')
params = {
'response_type': 'code',
'scope': 'read_write',
'client_id': client_id,
'redirect_uri': redirect_id
}
url = f'{url}?{urllib.parse.urlencode(params)}'
return redirect(url)
```
2. The **callback** is for receiving an access tokens when the user successfully authorized by the platforms and we store the access token information into our database.
```
def connect_authorize_callback(request, slug):
code = request.GET.get('code')
if code:
# slug could be: stripe, facebook, or google we define it on DatasSource
credentials = API_CREDENTIALS.get(slug)
client_secret = credentials.get("client_secret")
client_id = credentials.get("client_id")
token_url = credentials.get("token_url")
user_key = credentials.get("user_key")
access_token_key = credentials.get("access_token_key")
data = {
'client_secret': client_secret,
'grant_type': 'authorization_code',
'client_id': client_id,
'code': code
}
response = requests.post(token_url, params=data)
result = response.json()
user_id = result.get(user_key)
access_token = result.get(access_token_key)
#Look the data source first so we can query to Userauthsource
data_source = DataSource.objects.get(slug=slug)
#UserAuthSources where we store user credentials
Obj, create = UserAuthSources.objects.update_or_create(
data_source=data_source, user=request.user)
'''
we store all of data on objects_json in case is something happen in future
so we can check all of result to see if the key result is change.
'''
Obj.objects_json = result
Obj.access_id = user_id
Obj.access_token = access_token
Obj.owner = request.user
Obj.save()
redirect_url = reverse('sources:list_view')
response = redirect(redirect_url)
return response
```
> Note: connect_authorize_view and connect_authorize_callback is reusable function where we can use for other source, so we just need to update API_CREDENTIALS, that's it.
Let's test the oauth2 integration, please go to your browser and open http://localhost:8000/connect/authorize/stripe/
The page will direct you to oauth2 authentication and just click at the top to skip the step and allmetrics applications will be granted with your accounts.
Congratulations, we just successfully implement stripe for oauth2 connect!
## Request data using user access token
We will try to request object data from stripe API using user access tokens that we stored before.
1. We need to create a directory called **stripe_api**, if you are about integrate other source eg. **google** you can just follow the name **google_api**.
```
cd allmetrics/
cd apps/
cd connects/
mkdir stripe_api/
cd stripe_api/
touch __init__.py
touch views.py
```
The folder structure will look like this
```
allmetrics/ # Root directory.
|- apps/ # Folder used to store apps files.
|- - connects / # Folder used to store API integration from many sources
|- - - stripe_api/ # by default we had stripe API integrated
|- - - - __init__.py # tell python to look for submodules inside
|- - - - views.py # which uses to request object from stripe API and process the results
|- - - urls.py # which uses to set URLconf for the sources and authorization
|- - - utils.py # collection of functions for general data-processing or numerical. widely used by other sources
|- - - views.py # which uses to store authorize and callback function for oauth2
```
2. Let's open **views.py** on `apps/connect/views.py` and define the `CONNECT_HANDLER` which store functions of sources. In stripe we will called it `stripe_objecs_request`.
```
CONNECT_HANDLER = {
"stripe": {
"objects_request": stripe_objects_request #we will create this function later
}
}
```
3. Take a look on `connect_graph_query` function. This function works like a routing which handle source requests and assign to the specific function.
```
def connect_graph_query(request, slug):
# slug could be: stripe, facebook, or google we define it on DatasSource
connect_handler = CONNECT_HANDLER.get(slug)
if not connect_handler:
return HttpResponse("no handler")
results = connect_handler.get('objects_request')(request.user, request.GET)
return HttpResponse(json.dumps(results), content_type='application/json')
```
After we define the `connect_handler = CONNECT_HANDLER.get(slug)`
connect_handler is now equivalent to stripe_objects_request and then we only need to put the function parameters to pass the argument `(request.user, request.GET)`. It's more simple than define `if slug == "stripe" elif slug == "facebook` and etc.
4. We need to import the utils from **apps.connects.utils**, The **utils** here are a set of functions that have been made to provide generate data, aggregate data, and object request data. and we can just use it for any sources.
```
from apps.connects.utils import (
get_connect_objects, #which uses to get stored access_token
get_periodicals, #which uses to get a periodical date
aggregate_data, #which uses to aggregate result data from API
generate_stacked_graph, #which uses to generate the graph
get_date_range #which uses to generate date range from periodical date
)
# This is the x field where we set it default and stripe key for created is `created`.
X_FIELD = "created"
```
5 Ok, let's make function `stripe_objects_request` on apps/connects/stripe_api/views.py, This function will called when we get data from `/dashboars` `/home` and `/metrics/post`. This is where we proccess data to aggregate data and draw the graph.
```
def stripe_objects_request(user, request):
```
As we remember that the request here is coming from `connect_handler.get('objects_request')(request.user, request.GET)`
We don't just passing all of the request methods because it will be useless. So we only need to have request.user and request.GET.
```
response_dict = {}
stripe = get_connect_objects(user, 'stripe')
```
6. `get_connect_objects` uses to get a stored access token so we can perform query to stripe API. In future if we integrate other source we can just reuse this function `get_connect_objects(user, 'facebook')`
### Define data range
We don't want to request all data, because it cost our resources. So we need to get data from a period of time. We are going to use the `get_periodicals` function where we pass parameters monthly and 30 (which mean 30 days) as default and then on the dashboard page there will be some buttons where the user can jump back to the previous or ahead of 7 days and 30 days.
```
period = request.get("period")
total_days = request.get("total_days")
if not period:
period_date, period_timestamp = get_periodicals('monthly', 30)
else:
period_date, period_timestamp = get_periodicals(period, int(total_days))
```
### Request data to stripe API.
```
object_name = request.get("object_name")
operator = request.get("operator")
field_name = request.get("field_name")
if not field_name:
field_name = "object"
get_list = getattr(stripe, object_name).list(created=period_timestamp)
```
1. We have 3 data from metric form which is **object_name, operator, field_name**. Where **object_name** representing to Stripe Objects, **Operator** as the operator (Count, Sum, Average) and **field_name** representing to stripe data objects. For an example, let's say we have this request from metrics form.
```
<QueryDict: {'object_name': ['Customer'], 'operator': ['count'], 'field_name': ['currency']}>
```
This mean that we need to get **Customer objects**, and group the data object by **currency** and **count** the results. https://stripe.com/docs/api/customers
2. Instead of using this original syntax from stripe documentation
```
if object_name == "Customer":
stripe.Customer.list(created=period_timestamp)
elif object_name == "Invoice":
stripe.Invoice.list(created=period_timestamp)
```
Its always better if we can make it more dynamic by using **getattr**. https://www.geeksforgeeks.org/python-getattr-method/
```
get_list = getattr(stripe, object_name).list(created=period_timestamp)
```
### Aggregate result and draw graph
1. We need get the **object data keys**. This key will define as **field_name** so it can be easily recognize by users.
```
if get_list.get("data"):
get_keys = list(get_list["data"][0].keys())
```
2. `aggregate_date` has parameter where we pass the query result from **stripe, operator, x field and field_name**. This will return the result of processed data which is **labelX, labeY and labelZ**. And after that we pass the data to `generate_stacked_graph` to get the data for our chart.
```
labelX, labelY, labelZ = aggregate_data(get_list['data'], operator, X_FIELD, field_name)
newX, newY, newZ = generate_stacked_graph(period_date, labelX, labelY, labelZ)
response_dict = {
"keys": get_keys,
"labelX": newX,
"labelY": newY,
"labelZ": newZ
}
```
Let's see how the data looks like before we pass it to aggregate_data.
```
print (get_list['data'], operator, X_FIELD, field_name)
...
}, <Customer customer id=cus_IkUKlAWXiSC2Vk at 0x1181da360> JSON: {
"address": null,
"balance": 0,
"created": 1610505870,
"currency": "usd",
"default_source": null,
"delinquent": false,
"description": null,
"discount": null,
"email": "rahmat.ramadhanirianto@gmail.com",
"id": "cus_IkUKlAWXiSC2Vk",
"invoice_prefix": "7167AB22",
"invoice_settings": {
"custom_fields": null,
"default_payment_method": null,
"footer": null
},
"livemode": false,
"metadata": {},
"name": null,
"next_invoice_sequence": 2,
"object": "customer",
"phone": null,
"preferred_locales": [],
"shipping": null,
"tax_exempt": "none"
}] count created currency
```
> Note: Remember a format is a dictionary of data, count (operator), created (X_FIELD), and currency (field_name). Next time if we integrate other sources, we should follow this structure.
3. Let's see what the data looks like after we aggregate the data.
```
print (labelX, labelY, labelZ)
['01-13-2021'] [10] ['usd']
print (newX, newY, newZ)
['12-21-2020', '12-22-2020', '12-23-2020', '12-24-2020', '12-25-2020', '12-26-2020', '12-27-2020', '12-28-2020', '12-29-2020', '12-30-2020', '12-31-2020', '01-01-2021', '01-02-2021', '01-03-2021', '01-04-2021', '01-05-2021', '01-06-2021', '01-07-2021', '01-08-2021', '01-09-2021', '01-10-2021', '01-11-2021', '01-12-2021', '01-13-2021', '01-14-2021', '01-15-2021', '01-16-2021', '01-17-2021', '01-18-2021', '01-19-2021', '01-20-2021'] [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0] {'usd': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0]}
```
> Note: that we need to generate the data, and a list of numbers to follow the graph structure from chart.js. In the future, if we need to upgrade to another lib chart we can just custom it from `generate_stacked_graph`.
4. Leave the dictionary empty if there was no data found on the stripe API results. So the graph will be shown as an empty graph otherwise it will produce errors.
```
else:
response_dict = {
"keys": [],
"labelX": get_date_range(period_date),
"labelY": [],
"labelZ": []
}
return response_dict
```
Note: remember that we must provide **keys, labelX, labelY and labelZ** as the return of data. Where **keys** are a set of objects field name uses to group the data, **labelX** as list range of date, and **labelY** is the value of the data, and **labelz** is the set of the dictionary where we put currency and the value. It will be used as a stacked chart.
Full of source code
```
from apps.connects.utils import (
get_connect_objects,
get_periodicals,
aggregate_data,
generate_stacked_graph,
get_date_range
)
X_FIELD = "created"
def stripe_objects_request(user, request):
response_dict = {}
stripe = get_connect_objects(user, 'stripe')
period = request.get("period")
total_days = request.get("total_days")
if not period:
period_date, period_timestamp = get_periodicals('monthly', 30)
else:
period_date, period_timestamp = get_periodicals(period, int(total_days))
object_name = request.get("object_name")
operator = request.get("operator")
field_name = request.get("field_name")
if not field_name:
field_name = "object"
get_list = getattr(stripe, object_name).list(created=period_timestamp)
if get_list.get("data"):
get_keys = list(get_list["data"][0].keys())
labelX, labelY, labelZ = aggregate_data(get_list['data'], operator, X_FIELD, field_name)
newX, newY, newZ = generate_stacked_graph(period_date, labelX, labelY, labelZ)
response_dict = {
"keys": get_keys,
"labelX": newX,
"labelY": newY,
"labelZ": newZ
}
else:
response_dict = {
"keys": [],
"labelX": get_date_range(period_date),
"labelY": [],
"labelZ": []
}
return response_dict
```
## Creating Routing Source in Admin
The routing source contains an **object's name** of the source, where we will show this in the form of the metric and also help us to make our code easier to modify without touching the code.
For example, we can see the stripe API object name on this documentation page https://stripe.com/docs/api/, you can go to **core resource** menu and you will see **Balance, Charges, Customer, Disputes and etc**. That's what we called **object names.**
In this picture we add Customers to our routing source, so we can access a list of customers from Stripe API. If we want to add more routing sources we just copy and use the name as the routing source name.
After that we can test the code by add source in http://localhost:8000/sources/ and then add metric in http://localhost:8000/metrics/
Congratulations, the stripe API has been integrated to our system!
The next step we will learn about the javascript (front end code). But we will just review the code, nothing to add on javascript when we want to integrate a source since the code itself design to be dynamic and reusable.
Sign in with Wallet
Connect another wallet