<style>
body {
background-color: #10455B;
/* color: #F9F3ED !important; */
}
h1, h2, h3, h4 {
color: #FFAF20 !important;
}
code {
font-size: 12px;
line-height: 1.15;
}
</style>
<!-- Situation, Task, Action, Result -->
# Metasearch API
Note: Metasearch API is the engine that collects all the data from all relevant Limehome sources and structures it for omni-channels usage.
---
## For what?
To integrate Limehome's properties with a third-party search providers and aggregators
Note: The main goal of the Metasearch API | is to provide | and keep up to date Limehome's properties data for a third-party search providers and aggregators like Google Hotels, Trivago, Expedia, and so on.
Alone with the properties data Metasearch API provides the availability and room rates.
---
## Providers' criteria
1. Relevant data
2. Response delay
Note: There are two crucial criteria - up-to-date data and response delay. Many of providers have timeout limit (for instance, Google Hotels has the timeout of 4000ms). Whithin time range we have to perform many operations to provide the data, such a make request to Property API to get the properties data, and same time we have to fetch all offers for multiple properties that provider requested.
Best case scenario if we have only one property in the city, worst case scenario is, for instance, Berlin location where we have, if I remember correctly, more than 15 properties is live and many properties are ready to go.
In order to get an offers' data we have to make request to Properties API, then Properties API has to make request to `apaleo`, which doesn't support requests with multiple ids. This is actually is bottle neck that we have to expand.
---
## The acceptance criteria
<ul style="line-height: .75;">
<li><!-- .element: class="fragment" data-fragment-index="1" -->Reuse existing Limehome resources</li><br>
<li><!-- .element: class="fragment" data-fragment-index="2" -->Do not touch the existing core code</li><br>
<li><!-- .element: class="fragment" data-fragment-index="3" -->Support high load traffic</li><br>
<li><!-- .element: class="fragment" data-fragment-index="4" -->Flexibe</li><br>
<li><!-- .element: class="fragment" data-fragment-index="5" -->and Scalable application.</li><br>
</ul>
Note: The acceptance criteria we defined are the following: (show criteria)
So, we came up with the following architecture:
---
### Metasearch API Architecture
Note: The first integration that we made is for Trivago. This application is done as a serverless solution with couple of routes with Lambda functions that connected to the Lambda authorizer, and one State Machine that performs a few operations Lambda functions not necessary for.
---
### Hotel data
Note: So, I've put the pink badges to mark the flow of the first endpoint which is `hotel_data`
This endpoint returns a list of all properties.
Let's have a look what do we have. The `hotel_data` route performs the following operations:
<ol>
<li>It receives a request from Trivago</li>
<li>Authorizes the request with the pair of username and password</li>
<li>Invokes the Lambda</li>
<li>Lambda itself makes request to the Properties API</li>
<li>Properties API respond with the data from the data base</li>
<li>And finally, Lambda responds with this data</li>
</ol>
</ul>
**So, what is the Lambda function looks like?**
----
#### Hotel data Lambda function
```typescript=
// hotel_data.ts
// static value only for demo
const URL = 'https://api.limehome.com/properties/v1/public/properties';
let dataString: string = '';
const result: PropertiesResponse = await new Promise((resolve, reject) => {
const req = https.get(URL, function (res) {
res.on('data', (chunk) => { dataString += chunk; });
res.on('end', () => { resolve(JSON.parse(dataString)); });
});
req.on('error', (e) => {
reject({ statusCode: 500, body: e.message });
});
});
Log.debug({ result });
const response = {
api_version: 4,
lang: 'en_GB',
payload: result.payload,
};
return response;
```
Note: This Lambda does simple request to the Proiperties API and respond with the data. Nothing more!
But! How do we prepare the response that validates against the JSON schema Trivago expects?
**Mapping template**
----
#### Response mapping template
```c=
#set($body = $input.path('$'))
#define( $loop )
[
#foreach($hotel in $map)
#if($hotel.location.lat == 'n/a')
#set($hotel.location.lat = 0)
#end
#if($hotel.location.lng == 'n/a')
#set($hotel.location.lng = 0)
#end
{
"partner_reference": "$util.escapeJavaScript($hotel.external_id)",
"name": "$util.escapeJavaScript($hotel.name)",
"street": "$util.escapeJavaScript($hotel.location.addressLine1)",
"city": "$util.escapeJavaScript($hotel.location.city)",
"postal_code": "$util.escapeJavaScript($hotel.location.postalCode)",
"country": "$util.escapeJavaScript($hotel.location.country)",
"latitude": $hotel.location.lat,
"longitude": $hotel.location.lng
}
#if( $foreach.hasNext ),#end
#end
]
#end
{
"api_version": $body.api_version,
"lang": "$body.lang",
#set($map = $body.payload)
"hotels": $loop
}
```
---
### Hotel availability route
Note: So, the second route a bit complex them previous one and contains the State Machine along with the Lambda function.
Actually, we can connect API Gateway directly to the State machine, but we need to perform some business logic to create a Step Functions input object which is quite complex to do it on the API Gateway side with a request mapping templates.
So, the operations that this route performs are following:
<ol>
<li>Trivago makes request to the specific route</li>
<li>API Gateway authorizes the request with Lambda authorizer</li>
<li>Invokes the Lambda function</li>
<li>Lambda function invokes the Step Functions</li>
<li>From the step functions we do call two Property API's endpoints: 1. Fetch specific property by its id, 2. Fetch an offers' list for the property with the same property id.</li>
<li>The first request goes to the Properties API DB</li>
<li>The second one goes through the Properties API to the apaleo api and returns the offers' data</li>
<li>After generating a response we return it to Trivago</li>
</ol>
----
#### Execute Step Functions
```typescript
result = await stepfunctions.startSyncExecution(params).promise();
```
Note: Inside of Lambda we do synchronous invocations of the State Machine (express type) in order to get the result of this operation and then generate the response.
**One more application we made is Currency rate app:**
----
### Currency rates
<span>`cron(30 0 ? * TUE-SAT *)`</span>
Note: The architecture of currency rate handler is quite simple:<ol>
<li>We've created the CloudWatch rule that triggering the State Machine at the midnight</li>
<li>Then, state Machine performs request to the currency data provider via API Gateway proxy and receives the data</li>
<li>The last step is to put the data to the DynamoDB table</li>
</ol>
---
### The Currency rates State Machine
Note: All requests from Trivago assume to provide a rate plans in various currency. So, we have created the simple State Machine that updates these currency rates and stores it in the DynamoDB.
The State Machine contains no Lambdas and do requests through the predefined API Gateway proxy routes. Also, for soring the data the native Step Functions state type do it for us.
The State Machine is triggered by schedule which is defined as cron job string. The invokation would be triggered once a day at 00:30 from Tuersday till Saturday, the days when currency rates get updated.
----
### State Machine
Note: Let's have a look at the State Machine. Here we do three operations:
<ol>
<li>Retrieve the currency rates from the DynamoDB table</li>
<li>Since we have a list of ids we simply perform parallel requests to get a list of properties</li>
<li>and list of offers</li>
</ol>
The type of the state that contains two others is `Parallel`. Inside of it we have two `Map`-type states that do requests to a specific endpoint with concurrency of 3. It means the same time three requests would be performed for each `Map` state. In this way we simply drive down the latency of response.
Each of `Map` states performs request to the predefined API Gateway proxy route.
We use API Gateway proxy due to Step Functions can be integrated with only AWS resources natively. We can not do request to a third-party endpoint without from a state without Lambda.
Also, this state machine contains no Lambda that allows us to avoid cold starts and all integrations the State Machine with AWS resources are done natively.
---
## Achievements
<ul>
<li>Flexible</li>
<li>Scalable</li>
<li>Sustainable</li>
<li>Resilient</li>
</ul>
---
## TODO
<ol>
<li><!-- .element: class="fragment" data-fragment-index="1" -->Add an option to select a properties that should be visible for specific providers</li>
<li><!-- .element: class="fragment" data-fragment-index="2" -->Define what are cancellation policies should be provided</li>
<li><!-- .element: class="fragment" data-fragment-index="3" -->Apply custom domain settings</li>
</ol>
---
## Q 'n' A
{"metaMigratedAt":"2023-06-15T19:01:07.802Z","metaMigratedFrom":"YAML","title":"Metasearch API (MVP)","breaks":true,"slideOptions":"{\"spotlight\":{\"enabled\":true},\"fragments\":true,\"transition\":\"fade\"}","contributors":"[{\"id\":\"95b62735-55ed-4f45-b782-d554f867000a\",\"add\":12912,\"del\":3086}]"}