Design a twitter crawler

This document is used to record the development process.

1. Use cases and constraints

Gather requirements and scope the problem.

Use cases

We'll scope the problem to handle the following use cases

• Monitor :
  • Monitor multiple twitter accounts activities (tweet, retweet, delete)
• Crawler :
  • Collect the first 5 tweets (retweets), and their replies and likes count given a user account.
• User app :
  • Add/delete multiple user accounts to the system.
  • Access/download all data has been collected.

Constraints and assumptions

State assumptions

• Support only anonymous users
• The web crawler should not get stuck in an infinite loop
  • We get stuck in an infinite loop if the graph contains a cycle
• 500 users to monitor
• Monitor a user every 15 min

Calculate usage

• 2 MB of stored content per day
  • One account posts 1 tweet per day, while we need to get 5 tweets
  • 7 Kb per tweet
  • 100 replies per tweet
  • 4 kb per reply
  • 5 * 100 * 4 kb + 7 kb = 2 MB

2: Create a high level design

Outline a high level design with all important components.

3: Development environment

• Python 3.7
• MongoDB
• MySQL
• Javascript
• Tweepy, twarc
• Ubuntu 14.04

4: Hardware requirements

• 2 CPUs
• 8 GB RAM
• 40+ GB hard-drive space

5: Next step

The next step is to develop each core componets