---
# System prepended metadata

title: DBS 2021 challenge table 135

---

# DBS 2021 challenge table 135

## testingfile.parquet
tran_amount	            14.495358	
tran_code	            775
tran_direction	        D
mcc	                    3082
online_ind	            N	
prescored_category      Shopping
card_number            	4453333811552830
tran_date               2021-04-10	
prod_sys                A

## vendors-v3.csv
id                 V00001
name               Hyatt - Ondricka
isActive           True

## wallets-v3.csv
id                W00000001
customerId        C00000001
cashBalance       7268
rewardPoints      742


## brands-v3.csv
id                B00000001
name              Iynn
isActive          True

## customers-v3.csv
id                C00000001
name              Clara Keebler
isActive          True

## items-v3.csv
id             I00000001
brandId        B00000004
vendorId       V00095
name           Fantastic Rubber Tuna
price          5868
category       Foods
specialOffer   4


## Target
- 1. Develop Data Ingestion & Compute services to Ingest, Transform & Aggregate the data using AWS native services.
- 2. Develop scalable data persistence solution to store the raw transactions and computed aggregates.
- 3. Develop Data Quality & Error handing services to capture erroneous data.
    -Category have empty string.The dataset has 50,000,000 rows, the empty category value has 5,000,000 rows. Therefore, we decide to keep the value and change the empty string to 'unknown'.
    -People pay online or not almost have the same transaction amount,and who paid online is less than paid online. 
- 4. Develop Data Services to handle changes in the datasets (Delta and Full Feeds).
- 5. For the records which have the spend category missing, deploy a prebuilt ML model to classify the spend category