## Example request body for POST `projects/{project_id}/batch/source`
Here's how to set up a request body when making a POST request to `projects/{project_id}/batch/source`.
- [Postgres](#Postgres)
- [MongoDB](#MongoDB)
- [File (CSV, JSON, Excel, Feather, Parquet)](#File-CSV-JSON-Excel-Feather-Parquet)
### Postgres
This example provides the request body for a Postgres database:
```json
{
"name": "Postgres",
"sourceDefinitionId": "decd338e-5647-4c0b-adf4-da0e75f5a750",
"connectionConfiguration": {
"port": 5432,
"schemas": [
"public"
],
"ssl_mode": {
"mode": "disable"
},
"tunnel_method": {
"tunnel_method": "NO_TUNNEL"
},
"replication_method": {
"method": "Standard"
},
"host": "hh-pgsql-public.ebi.ac.uk",
"username": "reader",
"password": "NWDMCE5xdipIjRrp",
"database": "pfmegrnargs"
}
}
```
### MongoDB
This example provides the request body for a MongoDB:
```json
{
"name": "MongoDB",
"sourceDefinitionId": "b2e713cd-cc36-4c0a-b5bd-b47cb8a0561e",
"connectionConfiguration": {
"auth_source": "admin",
"instance_type": {
"instance": "atlas",
"cluster_url": "ac-wlxczet-shard-00-02.7qjtkcw.mongodb.net"
},
"database": "sample_airbnb",
"user": "fathom",
"password": "xpRO2PwEJVZVefIG"
}
}
```
### File (CSV, JSON, Excel, Feather, Parquet)
This example provides the request body for various file types including CSV, JSON, Excel, Feather, and Parquet:
```json
{
"name": "File (CSV, JSON, Excel, Feather, Parquet)",
"sourceDefinitionId": "778daa7c-feaf-4db6-96f3-70fd645acc77",
"connectionConfiguration": {
"format": "json",
"provider": {
"storage": "HTTPS",
"user_agent": false
},
"dataset_name": "sample_file",
"url": "https://drive.google.com/uc?export=download&id=1r-FaKcRT6eUzumAu1EufR_9FtHTKuYzP"
}
}
```
## Example request body for POST `/projects/{project_id}/batch/data_feeds`
Here's how to set up a request body when making a POST request to `/projects/{project_id}/batch/data_feeds`.
- [Postgres](#Postgres1)
- [MongoDB](#MongoDB1)
- [File (CSV, JSON, Excel, Feather, Parquet)](#File-CSV-JSON-Excel-Feather-Parquet1)
#### Request Body Explanation
The following delineates a sample request body structure, illustrating both the required and optional fields:
```json
{
"sourceId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"parentPath": "string",
"schedule": "string",
"scheduleType": "string",
"scheduleData": "string",
"syncCatalog": "string"
}
```
Mandatory Fields:
1. `sourceId`: This identifier is essential as it specifies the batch source that is targeted for synchronization.
2. `parentPath`: This element is integral in pinpointing a distinct location within the delta table storage hierarchy
and functions as a prefix in the S3 bucket structure.
Optional Fields:
- `schedule`, `scheduleType`, `scheduleData`, and `syncCatalog` are additional, non-mandatory fields enhancing the
functionality and specificity of the request.
Detailed Insight:
- The `sourceId` is a unique identifier assigned to the batch source intended for synchronization. It is a critical
element that ensures the accurate mapping and management of data.
- The `parentPath` is another pivotal component. It is instrumental in locating a specific segment within the delta
table storage hierarchy, acting as a prefix in the S3 bucket configuration. It is analogous to the 'database id' in
the structured path: `s3://{bucket}/{organization id}/{project id}/{database id}/{table id}`, a format that
underscores the systematic approach adopted for optimal data organization and access.
This explicit structure facilitates a streamlined, efficient, and error-free data synchronization process, ensuring that
data integrity and accessibility are maintained at optimal levels.
> Notes:
> - It is imperative to substitute `sourceId` and `parentPath` with their appropriate values, as elucidated in the
_Request Body Explanation_ section, to ensure accuracy and functionality.
> - The example provided below is applicable
exclusively under the condition that the batch source, as detailed in this
document, has been previously established and is operational.
### Postgres
> WIP
### MongoDB
```json
{
"sourceId": "paste here sourceId returned by GET /projects/{project_id}/batch/sources",
"parentPath": "paste here parentPath of your choice",
"scheduleType": "manual",
"syncCatalog": {
"streams": [
{
"config": {
"syncMode": "incremental",
"cursorField": [
"_id"
],
"destinationSyncMode": "append",
"primaryKey": [
],
"aliasName": "listingsAndReviews",
"selected": true,
"suggested": true
},
"stream": {
"name": "listingsAndReviews",
"jsonSchema": {
"type": "object",
"properties": {
"amenities": {
"type": "array"
},
"TESTCOLUMN": {
"type": "string"
},
"notes": {
"type": "string"
},
"access": {
"type": "string"
},
"house_rules": {
"type": "string"
},
"description": {
"type": "string"
},
"first_review": {
"type": "string"
},
"calendar_last_scraped": {
"type": "string"
},
"neighborhood_overview": {
"type": "string"
},
"availability": {
"type": "object",
"properties": {
"availability_365": {
"type": "number"
},
"availability_30": {
"type": "number"
},
"availability_60": {
"type": "number"
},
"availability_90": {
"type": "number"
}
}
},
"number_of_reviews": {
"type": "number"
},
"space": {
"type": "string"
},
"review_scores": {
"type": "object",
"properties": {
"review_scores_checkin": {
"type": "number"
},
"review_scores_communication": {
"type": "number"
},
"review_scores_rating": {
"type": "number"
},
"review_scores_accuracy": {
"type": "number"
},
"review_scores_location": {
"type": "number"
},
"review_scores_value": {
"type": "number"
},
"review_scores_cleanliness": {
"type": "number"
}
}
},
"cleaning_fee": {
"type": "number"
},
"reviews": {
"type": "array"
},
"price": {
"type": "number"
},
"reviews_per_month": {
"type": "number"
},
"host": {
"type": "object",
"properties": {
"host_verifications": {
"type": "array"
},
"host_url": {
"type": "string"
},
"host_response_time": {
"type": "string"
},
"host_about": {
"type": "string"
},
"host_picture_url": {
"type": "string"
},
"host_has_profile_pic": {
"type": "boolean"
},
"host_id": {
"type": "string"
},
"host_listings_count": {
"type": "number"
},
"host_total_listings_count": {
"type": "number"
},
"host_location": {
"type": "string"
},
"host_is_superhost": {
"type": "boolean"
},
"host_neighbourhood": {
"type": "string"
},
"host_thumbnail_url": {
"type": "string"
},
"host_response_rate": {
"type": "number"
},
"host_name": {
"type": "string"
},
"host_identity_verified": {
"type": "boolean"
}
}
},
"property_type": {
"type": "string"
},
"summary": {
"type": "string"
},
"monthly_price": {
"type": "number"
},
"images": {
"type": "object",
"properties": {
"picture_url": {
"type": "string"
},
"xl_picture_url": {
"type": "string"
},
"medium_url": {
"type": "string"
},
"thumbnail_url": {
"type": "string"
}
}
},
"security_deposit": {
"type": "number"
},
"address": {
"type": "object",
"properties": {
"market": {
"type": "string"
},
"country": {
"type": "string"
},
"country_code": {
"type": "string"
},
"street": {
"type": "string"
},
"suburb": {
"type": "string"
},
"location": {
"type": "object",
"properties": {
"coordinates": {
"type": "array"
},
"type": {
"type": "string"
},
"is_location_exact": {
"type": "boolean"
}
}
},
"government_area": {
"type": "string"
}
}
},
"weekly_price": {
"type": "number"
},
"bed_type": {
"type": "string"
},
"listing_url": {
"type": "string"
},
"guests_included": {
"type": "number"
},
"maximum_nights": {
"type": "string"
},
"bathrooms": {
"type": "number"
},
"bedrooms": {
"type": "number"
},
"extra_people": {
"type": "number"
},
"minimum_nights": {
"type": "string"
},
"last_review": {
"type": "string"
},
"transit": {
"type": "string"
},
"accommodates": {
"type": "number"
},
"name": {
"type": "string"
},
"interaction": {
"type": "string"
},
"cancellation_policy": {
"type": "string"
},
"_id": {
"type": "string"
},
"beds": {
"type": "number"
},
"last_scraped": {
"type": "string"
},
"room_type": {
"type": "string"
}
}
},
"supportedSyncModes": [
"full_refresh",
"incremental"
],
"defaultCursorField": [
],
"sourceDefinedPrimaryKey": [
],
"namespace": "sample_airbnb"
}
}
]
}
}
```
### File (CSV, JSON, Excel, Feather, Parquet)
```json
{
"sourceId": "paste here sourceId returned by GET /projects/{project_id}/batch/sources",
"parentPath": "paste here parentPath of your choice",
"scheduleType": "manual",
"syncCatalog": {
"streams": [
{
"config": {
"syncMode": "full_refresh",
"cursorField": [],
"destinationSyncMode": "append",
"primaryKey": [],
"aliasName": "sample_file",
"selected": true,
"suggested": true
},
"stream": {
"name": "sample_file",
"jsonSchema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": [
"children",
"network"
],
"properties": {
"children": {
"type": "object",
"required": [
"pipelines",
"trunkLine"
],
"properties": {
"pipelines": {
"type": "array",
"items": {
"type": "object",
"required": [
"children",
"pipeline"
],
"properties": {
"pipeline": {
"type": "object",
"required": [
"code",
"codeOfDesign",
"coordinates",
"designPressure",
"designTemperature",
"elevations",
"flowRate",
"id",
"impactUrgencyAndPriorityLevel",
"isPiggable",
"location",
"maximumOperatingTemperature",
"moap",
"mop",
"name",
"operationStatus",
"parentId",
"pipelineSystemType",
"pipelineType",
"productType",
"yearOfConstruction",
"yearOfOperation"
],
"properties": {
"moap": {
"type": "number"
},
"operationStatus": {
"type": "string"
},
"code": {
"type": "string"
},
"designTemperature": {
"type": "number"
},
"maximumOperatingTemperature": {
"type": "number"
},
"coordinates": {
"type": "null"
},
"designPressure": {
"type": "number"
},
"yearOfOperation": {
"type": "string"
},
"pipelineSystemType": {
"type": "string"
},
"yearOfConstruction": {
"type": "string"
},
"parentId": {
"type": "string"
},
"mop": {
"type": "number"
},
"codeOfDesign": {
"type": "string"
},
"name": {
"type": "string"
},
"flowRate": {
"type": "number"
},
"isPiggable": {
"type": "boolean"
},
"location": {
"type": "string"
},
"id": {
"type": "string"
},
"pipelineType": {
"type": "string"
},
"elevations": {
"type": "null"
},
"productType": {
"type": "string"
},
"impactUrgencyAndPriorityLevel": {
"type": "string"
}
}
},
"children": {
"type": "object",
"required": [
"brachConnects",
"casings",
"clossure",
"corrosionCouponRetrievalPoints",
"elbows",
"flanges",
"luncher",
"meteringPoints",
"pipeBends",
"receiver",
"reducers",
"repairDevices",
"segments",
"valves",
"vents"
],
"properties": {
"receiver": {
"type": "null"
},
"valves": {
"type": "array"
},
"meteringPoints": {
"type": "array"
},
"elbows": {
"type": "array"
},
"repairDevices": {
"type": "array"
},
"clossure": {
"type": "array"
},
"brachConnects": {
"type": "array"
},
"segments": {
"type": "array",
"items": {
"type": "object",
"required": [
"coatingType",
"designFactor",
"domain",
"id",
"length",
"manufacturer",
"materialGrade",
"nominalDiameter",
"pipeJointManufactureType",
"segmentNo",
"smys",
"startingPoint",
"suts",
"wallThickness"
],
"properties": {
"designFactor": {
"type": "number"
},
"smys": {
"type": "number"
},
"length": {
"type": "number"
},
"coatingType": {
"type": "string"
},
"segmentNo": {
"type": "integer"
},
"nominalDiameter": {
"type": "number"
},
"manufacturer": {
"type": "string"
},
"startingPoint": {
"type": "number"
},
"suts": {
"type": "number"
},
"domain": {
"type": "string"
},
"wallThickness": {
"type": "number"
},
"pipeJointManufactureType": {
"type": "string"
},
"id": {
"type": "string"
},
"materialGrade": {
"type": "string"
}
}
}
},
"corrosionCouponRetrievalPoints": {
"type": "array"
},
"vents": {
"type": "array"
},
"casings": {
"type": "array"
},
"flanges": {
"type": "array"
},
"luncher": {
"type": "null"
},
"reducers": {
"type": "array"
},
"pipeBends": {
"type": "array"
}
}
}
}
}
},
"trunkLine": {
"type": "array",
"items": {
"type": "object",
"required": [
"children",
"trunkLine"
],
"properties": {
"children": {
"type": "array"
},
"trunkLine": {
"type": "object",
"required": [
"id",
"name",
"pipelineCount"
],
"properties": {
"pipelineCount": {
"type": "integer"
},
"name": {
"type": "string"
},
"id": {
"type": "string"
}
}
}
}
}
}
}
},
"network": {
"type": "object",
"required": [
"id",
"organizationId"
],
"properties": {
"organizationId": {
"type": "string"
},
"id": {
"type": "string"
}
}
}
}
},
"supportedSyncModes": [
"full_refresh"
],
"defaultCursorField": [],
"sourceDefinedPrimaryKey": []
}
}
]
}
}
```
Sign in with Wallet
Connect another wallet