# Hadoop筆記(2)
*5/23
## Fully Distributed
[上課講義](https://hackmd.io/@yillkid/r1f7HrFHc/https%3A%2F%2Fhackmd.io%2F%40yillkid%2FrJANg-pBq)
* Master與Slaves的主機要先用ssh確定彼此都能連進彼此主機後才能開始進行fully distributed的設定
* 關閉防火牆

* 綁定主機名稱，之後就用主機名稱取代ip，設定好後要重新登入

#Master
$ hostnamectl set-hostname master

#Slave1
$ hostnamectl set-hostname slave1

#Slave2
$ hostnamectl set-hostname slave2

* 修改Hosts 設定檔，Master與所有的Slave都要設定。

#路徑 /etc/hosts
$ sudo vi /etc/hosts

<MASTER_IP_ADDRESS> master
<SLAVE1_IP_ADDRESS> slave1
<SLAVE2_IP_ADDRESS> slave2

![](https://hackmd.io/_uploads/By5kY45S2.png)

* 修改四個設定檔

路徑: $ cd /usr/local/hadoop/etc/hadoop
$ sudo vim core-site.xml
$ sudo vim hdfs-site.xml
$ sudo vim mapred-site.xml
$ sudo vim yarn-site.xml

#### core-site.xml：
* fs.defaultFS是NameNode位置；fs.tmp.dir是NameNode暫存資料夾

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
<description></description>
</property>
<property>
<name>fs.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
<description></description>
</property>

</configuration>

#### hdfs-site.xml
* dfs.replication 代表一台 Node 要儲存幾份檔案，不能超過機器數量
* dfs.namenode.name.dir 是FsImage 鏡像的目錄，存放 hadoop 的 namenode 的 metadata
* dfs.namenode.data.dir 是HDFS 文件目錄

<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
<description></description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop/hdfs/name</value>
<description></description>
</property>
<property>
<name>dfs.namenode.data.dir</name>
<value>/usr/local/hadoop/hdfs/data</value>
<description></description>
</property>
</configuration>

#### yarn-site.xml
* 設定RM的hostname為master
* 告知 NodeManager 要額外啟用 Shuffle 服務
* 設定RM上的各項服務，port號可以先jps查看服務的pid，然後用指令sudo netstat -ntlp | grep pid找接近的埠號

<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
<description></description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description></description>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>master:44357</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>

#### mapred-site.xml
* 只留下配置 MapReduce 框架屬性

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description></description>
</property>
</configuration>

* 若跑map-reduce出此設定檔error，此設定當可改回:

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description></description>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
</configuration>

#### 各項角色的定義
* master 節點與slave/worker 節點
* 到路徑/usr/local/hadoop/etc/hadoop/下新增works、slaves、master三個檔案(沒有副檔名)

/usr/local/hadoop/etc/hadoop/works:
slave1
slave2

/usr/local/hadoop/etc/hadoop/slaves:
slave1
slave2

/usr/local/hadoop/etc/hadoop/master:
master

#### 確定無密碼登入
* master重新產生一組公私鑰，把公鑰複製到master自己與slaves的authorized_keys，可以用【ssh 帳號@slaves】確定是否可以無密碼登入

$ ssh-keygen
$ cd ~/.ssh/
$ cat id_rsa.pub >> authorized_keys
#然後再把內容複製到slaves主機的 ~/.ssh/authorized_keys

### 把四個設定檔複製到slaves

scp /usr/local/hadoop/etc/hadoop/* <slave_username>@<slave_IP>:/usr/local/hadoop/etc/hadoop/

### Restart Master
#### Stop

$ cd /usr/local/hadoop/sbin/
$ ./stop-dfs.sh ; ./stop-yarn.sh

#### 刪暫存

$ sudo rm -rf /usr/local/hadoop/hdfs/name/current
$ pkill -9 java
$ sudo rm /usr/local/hadoop/tmp/*
$ rm -rf /tmp/hadoop*

#### JAVA環境變數

$ export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/

#### 格式化NameNode

$ /usr/local/hadoop/bin/hdfs namenode -format

#### 啟動Hadoop
* master 啟動後，會自動帶起 slave 節點

$ cd /usr/local/hadoop/sbin/
$ ./start-dfs.sh ; ./start-yarn.sh

### 完成啟動服務
* 網址可打http://master:9870/看概覽
* 在HDFS開資料夾或是推資料

$ /usr/local/hadoop/bin/hdfs dfs -mkdir /test

![](https://hackmd.io/_uploads/HynvprcBn.png)

* 到Slave檢查是否可以看到資料夾/資料

/usr/local/hadoop/bin/hdfs dfs -ls /

![](https://hackmd.io/_uploads/B1YyEUqrh.png)

* Slave可以試著推檔案上去HDFS

$ touch a.txt
$ /usr/local/hadoop/bin/hdfs dfs -put a.txt /test/

* Master確認是否有檔案新增

/usr/local/hadoop/bin/hdfs dfs -ls /test/

![](https://hackmd.io/_uploads/Skgl7aNU2.png)

* 可上傳檔案後執行map-reduce

/usr/local/hadoop/bin/hadoop jar '/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-3.3.5.jar' \
-mapper 'python3 mapper.py' \
-file /home/samantha/mapper.py \
-reducer 'python3 reducer.py' \
-file /home/samantha/reducer.py \
-input hdfs:/workshop/input.txt \
-output hdfs:/workshop/output

* 監控 application:

$ /usr/local/hadoop/bin/yarn application -list

* 監聽 jps:

$ watch jps

* 若master的node manager起不來，可以修改works與workers內容加上master。

$ sudo vi /usr/local/hadoop/etc/hadoop/works
$ sudo vi /usr/local/hadoop/etc/hadoop/workers

![](https://hackmd.io/_uploads/S11LwbPL3.png)

* Slave的yarn-site.html內要改成自己的，slave1:xxxxx
![](https://hackmd.io/_uploads/rJQX_-wI2.png)

* 開始跑map-reducde後，會在master/slave1/slave2的watch jps看到服務會不斷分散在三者之間，包含YarnChild、MRAppMaster等
![](https://hackmd.io/_uploads/S1D8K-DIh.jpg)
![](https://hackmd.io/_uploads/SyikcZPI2.png)

* 執行完成後可以到網頁打8088確認RM與App的相關細節，以及每個任務是哪個節點執行的。
![](https://hackmd.io/_uploads/rkuStfPLn.jpg)
![](https://hackmd.io/_uploads/BkZPKMwIn.jpg)


* Workshop:
https://hackmd.io/@uf57VA2KROuAfxAa6dLehQ/HyLpeF4In


###### tags: `hadoop` `fully distributed`

補充
https://hackmd.io/@WVuRUYslRZm9PNDSsjjDiA/HJXpBevSh
https://hackmd.io/@WVuRUYslRZm9PNDSsjjDiA/H1lu4gqHn