---
# System prepended metadata

title: README

---

# README #


### Group 13 - Localhost4555

* Aishwarya Wesanekar (awes96@seas.upenn.edu)
* Blien Habtu (habtub@seas.upenn.edu)
* Yi Qin (qinyi@seas.upenn.edu
* Yike Li (yikel@seas.upenn.edu)
 

### Features

* Distributed crawling with Amazon SQS
* Distributed indexer using EMR Hadoop MapReduce
* Full-pagerank using Stormlite - The overall approach is based on the full-pagerank algorithm discussed in the class, based on the number of inbound links i.e. links redirecting to the website and the number of outbound links i.e. links redirecting from the website. Stormlite has been used to implement mapreduce functionality. There are two topologies - init and process. The process topology is iterated over for convergence.
* Search engine webserver which handles multiple concurrent requests 



### Extra Credit
* Search Suggestions: If a word has an idf of less than 0.3, we search from a list of index terms and suggest a word in its place with minimum edit distance. For example, the word "crow" would be replaced by the more common word in the corpus, "eagle". (Listed as extra credit under Section 2.4 in the project description)
* Bolding Keywords: If there's a description for the webpage that includes a query term, we return a snippit of it with the term bolded. (Listed as extra credit under Section 2.2 in the project description)


### Source File
```
project
│   README.md
│   file001.txt    
│
└───cloud
│   │   DDBStore.java
│   │   DDBStoreIndexer.java
│   │   Document.java
│   │   IdfEntity.java
│   │   InvertedIndexEntity.java
│   │   S3Store.java
│   │   TestEntity.java
│   │   TfEntity.java
│   │   Utils.java
└───crawler
│   │   Crawler.java
│   │   CrawlerQueue.java
│   │   CrawlerWithSQS.java
│   │   CrawlerWithSQSWorker.java
│   │   DocumentFetcher.java
│   │   Filter.java
│   │   OutputLinks.java
│   │   Recorder.java
│   │   URLInfo.java
│   │   URLQueue.java
│   │   Utils.java
└───indexer
│   │   Driver.java
│   │   Indexer.java
└───pagerank
    └───v1
│   │   │PageRank.java
│   │   │Page.java
│   │   │FileToObj.java
        
    └───v2
        └───init
│   │   │   │PRInitBolt.java
│   │   │   │PRInitSpout.java
│   │   │   │PRStormInit.java
        └───process
│   │   │   │PRProcessBolt.java
│   │   │   │PRProcessSpout.java
│   │   │   │PRStormProcess.java
        └───cleanse
│   │   │   │PRCleanseBolt.java
│   │   │   │PRCleanseSpout.java
│   │   │   │PRStormCleanse.java
        └───output
│   │   │   │PROutputBolt.java
│   │   │   │PROutputSpout.java
│   │   │   │PRStormOutput.java
│   │   │   │PROutput.java
│   │   │FlattenedLink.java
│   │   │LinkMapFileWriter.java
│   │   │PageRankStormlite.java
│   │   │Pair.java
│   │
└───searchengine 
│   │   RankingResults.java
│   │   SearchEngine.java
│   │   SearchHandler.java
│   │   WordFilter.java
└────
```


### Installation 


### EC2 set up
```

sudo yum update -y
sudo yum install git -y
sudo wget https://www-eu.apache.org/dist/maven/maven-3/3.8.5/binaries/apache-maven-3.8.5-bin.tar.gz
sudo tar xf apache-maven-*.tar.gz -C /opt
sudo ln -s /opt/apache-maven-3.8.5 /opt/maven
sudo amazon-linux-extras install java-openjdk11 -y
sudo nano /etc/profile.d/maven.sh

# export JAVA_HOME=/usr/lib/jvm/jre-11-openjdk
# export M2_HOME=/opt/maven
# export MAVEN_HOME=/opt/maven
# export PATH=${M2_HOME}/bin:${PATH}
sudo chmod +x /etc/profile.d/maven.sh
source /etc/profile.d/maven.sh
git clone https://yikeyike@bitbucket.org/awesanekar/555-finalproject.git
cd 555-finalproject


```


### Run Crawler

```
mvn clean install exec:java@sqscrawl
```




### Run Indexer
* Pull Driver.java in indexer package into an independent Java project
* Create a .jar file from the above project
* Upload the .jar file into AWS S3
* Create an EMR cluster. Add a step and set the jar file reference to the one uploaded in previous step
* Provide 4 command-line arguments: the input path, the output path for inverted index/idf, the output path for tf, and the total number of documents
* Run the step. After it finishes, results of indexer would be stored in the two output paths, respectively.


### Run PageRank
```
mvn clean install exec:java@pagerank
```

### Run Search Engine
```
mvn clean install exec:java@searchengine
```
Note that you will need to wait for the "Ready!" confirmation to be printed to the terminal before being able to run the server on its EC2 instance. The current configuration runs on http://54.81.29.236:45555/search.
