# Reflaunt Data Scraping

## What's wanted

* Scrape data from second hand marketplace websites for specific brands, category and condition.
* In total 8 sites to scrape but pilot with just one
* In total 1003 brands but pilot with just 20
* In total 322 categories but pilot with with 20
* 4 conditions - New, Like New, Gently Used and Slight Damage
* Only scrape products that has estimated original price listed in the website
* Expected 10 products per cluster of brand, category and condition
* The scraper should run daily. Initial test with daily output but will later be daily. The scraper should scale to scrape 4.8 millions products daily in the long run.

### Required fields

* high priority
* Which website the item is scraped from
* Product category assigned by scraping algorithm
* Lowest level of category as listed in website
* Path of category in the category tree (crumb)
* Brand of the product
* Condition of the product
* Currency of the listed prices
* Estimated original price in the website
* Listed prices in website
* Percentage discount as listed in product page
* average priority
* Value based discount as listed in product page
* Name of product
* Description of product
* Url of the page the product was scraped from
* Image url for main image of the product
* Date when the product was last scraped
* Date when the product was scraped for first time
* Is the item listed as sold in the website
* Is an item that was listed as sold now delisted from the website
* Date when the item is found as sold
* Date when the item listed as sold is removed
* Seller name
* Does the product come with original tag attached
* Some additional low and very low priority fields also present in RFP

## What's timeline

* **Pilot** : 1 Website\*20 brands\*20 categories\*4 conditions\*10 products
* Rough work estimation
* Scrapping
* Categorising
* Other (Like automating scrapper for daily running etc.,)

***Deadline as in RFP March 23rd***

* **Launch**: 1 website\*(1000 brands\*0.3)\*20 categories\*4 conditions\*10 products = 240,000 products
* Rough work estimation
* Scrapping
* Categorising
* Other

***Deadline as in RFP April 6th***

* **Long term**: (8 websites\*1000 brands\*100 categories\*4 conditions\*5 products)\*0.3 = 4.8M products

***Deadline as in RFP - 1 new website every 2 weeks from April 6th until July 13th***

## Research

### Scrapping website: https://www.therealreal.com/

#### Possible complications (Red Flags)

* The site asks for login in the begining and also have frequent pop-ups which may complicate the scrapping.
* Could not find one average priority field (Seller Name) - and few low priority fields
* Few fields will require to check previously scraped and stored data, for eg
* Date when the product was last scraped
* Date when the product was scraped for first time
* Is an item that was listed as sold now delisted from the website
* Date when the item is found as sold
* Date when the item listed as sold is removed
* The condition as listed in the website might not the the 4 condition category wanted by reflaunt so we will need to document and decide which category in the website will map to which category by reflaunt.
* The site has a bot checker in place - see screenshot.
* We should be able to scrape the site for our elements though accoring to Robots.txt - https://www.therealreal.com/robots.txt

#### High priority fields (and the relevant div elements)

* Which website the item is scraped from
* Product category assigned by scraping algorithm
* Lowest level of category as listed in website
* Path of category in the category tree (crumb)
The screenshot covers both lowest level category and category tree
* Brand of the product
* Condition of the product
* Currency of the listed prices
* Estimated original price in the website
* Listed prices in website
* Percentage discount as listed in product page

#### Average priority fields (and the relevant div elements)

* Value based discount as listed in product page
* Name of product
* Description of product
* Url of the page the product was scraped from
* Image url for main image of the product
* Date when the product was last scraped
* Date when the product was scraped for first time
* Is the item listed as sold in the website
* Is an item that was listed as sold now delisted from the website
* Date when the item is found as sold
* Date when the item listed as sold is removed
* Seller name
* Looked at 10 different production but didn't find seller name
* Does the product come with original tag attached

#### Low priority fields
* is_with_receipt
* Not found
* is_with_serial_number
*May be...*
* is_with_original_box
* Not found
* is_with_dust_bag
* Not found
* size
* material
* weight
* style

#### Very low priority fields (ideas)
* is_real_gold
* is_real_precious_stone
* is_limited_edition

#### Other information

* SiteMap: http://trr-sitemap-production.s3.amazonaws.com/sitemaps/sitemap.xml.gz