---
# System prepended metadata

title: Reflaunt Data Scraping

---

# Reflaunt Data Scraping

## What's wanted

* Scrape data from second hand marketplace websites for specific brands, category and condition.
  * In total 8 sites to scrape but pilot with just one
  * In total 1003 brands but pilot with just 20
  * In total 322 categories but pilot with with 20
  * 4 conditions - New, Like New, Gently Used and Slight Damage
  * Only scrape products that has estimated original price listed in the website
  * Expected 10 products per cluster of brand, category and condition
* The scraper should run daily. Initial test with daily output but will later be daily. The scraper should scale to scrape 4.8 millions products daily in the long run.

### Required fields

* high priority 
  * Which website the item is scraped from
  * Product category assigned by scraping algorithm
  * Lowest level of category as listed in website
  * Path of category in the category tree (crumb)
  * Brand of the product
  * Condition of the product
  * Currency of the listed prices
  * Estimated original price in the website
  * Listed prices in website
  * Percentage discount as listed in product page
* average priority
  * Value based discount as listed in product page
  * Name of product
  * Description of product
  * Url of the page the product was scraped from
  * Image url for main image of the product
  * Date when the product was last scraped
  * Date when the product was scraped for first time
  * Is the item listed as sold in the website
  * Is an item that was listed as sold now delisted from the website
  * Date when the item is found as sold
  * Date when the item listed as sold is removed
  * Seller name
  * Does the product come with original tag attached
* Some additional low and very low priority fields also present in RFP 


## What's timeline

* **Pilot** : 1 Website\*20 brands\*20 categories\*4 conditions\*10 products
  * Rough work estimation
    * Scrapping
    * Categorising
    * Other (Like automating scrapper for daily running etc.,)

***Deadline as in RFP March 23rd***

* **Launch**: 1 website\*(1000 brands\*0.3)\*20 categories\*4 conditions\*10 products = 240,000 products
  * Rough work estimation
    * Scrapping
    * Categorising
    * Other

***Deadline as in RFP April 6th***

* **Long term**: (8 websites\*1000 brands\*100 categories\*4 conditions\*5 products)\*0.3 = 4.8M products

***Deadline as in RFP - 1 new website every 2 weeks from April 6th until July 13th***

## Research

### Scrapping website: https://www.therealreal.com/

#### Possible complications (Red Flags)
    
* The site asks for login in the begining and also have frequent pop-ups which may complicate the scrapping.
* Could not find one average priority field (Seller Name) -  and few low priority fields
* Few fields will require to check previously scraped and stored data, for eg
  * Date when the product was last scraped
  * Date when the product was scraped for first time
  * Is an item that was listed as sold now delisted from the website
  * Date when the item is found as sold
  * Date when the item listed as sold is removed
* The condition as listed in the website might not the the 4 condition category wanted by reflaunt so we will need to document and decide which category in the website will map to which category by reflaunt.
* The site has a bot checker in place - see screenshot.
  * We should be able to scrape the site for our elements though accoring to Robots.txt - https://www.therealreal.com/robots.txt 
![](https://i.imgur.com/BO8v8cT.png)

#### High priority fields (and the relevant div elements)

* Which website the item is scraped from
* Product category assigned by scraping algorithm
* Lowest level of category as listed in website
* Path of category in the category tree (crumb)
![](https://i.imgur.com/XhpL8zD.png)
  The screenshot covers both lowest level category and category tree
* Brand of the product
![](https://i.imgur.com/Q4TNWx1.png)
* Condition of the product
![](https://i.imgur.com/EtW3zF4.png)
* Currency of the listed prices
![](https://i.imgur.com/A8F3JCK.png)
* Estimated original price in the website
![](https://i.imgur.com/4kte0Mj.png)
* Listed prices in website
![](https://i.imgur.com/0rVgOU6.png)
* Percentage discount as listed in product page
![](https://i.imgur.com/9KcT37r.png)

#### Average priority fields (and the relevant div elements)

* Value based discount as listed in product page
![](https://i.imgur.com/wAloAUV.png)
* Name of product
![](https://i.imgur.com/zUTUW2V.png)
* Description of product
![](https://i.imgur.com/3YqaNXt.png)
* Url of the page the product was scraped from
* Image url for main image of the product
![](https://i.imgur.com/F77PxAX.png)
* Date when the product was last scraped
* Date when the product was scraped for first time
* Is the item listed as sold in the website
![](https://i.imgur.com/syTgtkC.png)
![](https://i.imgur.com/x0Dkoz2.png)
* Is an item that was listed as sold now delisted from the website
* Date when the item is found as sold
* Date when the item listed as sold is removed
* Seller name
    * Looked at 10 different production but didn't find seller name
* Does the product come with original tag attached
![](https://i.imgur.com/bt2ZSxv.png)
![](https://i.imgur.com/2T9YWqk.png)

#### Low priority fields
* is_with_receipt
    * Not found
* is_with_serial_number
  *May be...*
  ![](https://i.imgur.com/iMbwoxw.png)
* is_with_original_box
    * Not found
* is_with_dust_bag
    * Not found
* size
  ![](https://i.imgur.com/Lz1ri7b.png)
* material
  ![](https://i.imgur.com/x82JXrr.png)
* weight
  ![](https://i.imgur.com/kHzmftD.png)
* style
  ![](https://i.imgur.com/PKl3rPp.png)

#### Very low priority fields (ideas)
* is_real_gold
![](https://i.imgur.com/5H04NSu.png)
* is_real_precious_stone
![](https://i.imgur.com/QMsELEE.png)
* is_limited_edition

#### Other information

* SiteMap: http://trr-sitemap-production.s3.amazonaws.com/sitemaps/sitemap.xml.gz
