Try   HackMD
tags: Hadoop Linux Python 網路爬蟲

Hadoop HA + Line 綜合運用

一般而言機房不適合管理人員長時間待著，惟監控叢集是必須的，但是如果能藉由 Line 即時通知管理人員或是相關主管目前叢集就顯得格外方便。

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
網路爬蟲應用非常廣泛，還能持續延伸…

  1. 準備事項:
    1. 要有LINE帳號，註冊方式請參考官網
    2. LINE加入 Line Notify 官方帳號好友
    3. 註冊 Line Notify 服務
    4. Hadoop 為高可用性 HA(high availability)，架設步驟請參閱
    5. bdseHadoopNotify等相關程式，GitHub請參閱

加入 Line Notify 官方帳號好友

  1. 搜尋好友輸入 @LineNotify 並加入官方帳號
    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
  2. 建立可以接收訊息的群組並加入 Line Notify 官方帳號進入群內
    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

註冊 Line Notify 服務

  1. 登入官方Line notify網站

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  2. 登入Line帳號

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  3. 點選右上方帳號名稱，並選擇管理登錄服務

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  4. 選擇登錄服務

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    如果都沒有使用過相關服務，上面呈現空白很正常

  5. 電子信箱請使用可以收認證信為主，其餘資料依個人喜好輸入

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  6. 確認無誤後按下登錄

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  7. 至信箱收認證信後，點即前往服務一覽

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  8. 屆時將出現Line Notify服務，並點選個人頁面

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  9. 點擊發行權杖

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    如果都沒有使用過相關服務，上面呈現空白很正常

  10. 依個人喜好設定機器人名稱，並選擇稍早建立好的接收訊息的群組

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    請確保Line Notify 官方帳號於選擇的群組內

  11. 複製權杖

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    請確保權杖複製起來並儲存，不然會找不到該權杖

  12. 將權杖貼製bdse-conf.txt內

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    請確保貼上為第一行，並無任何註解、空白或是換行元素

  13. 啟動bdseHadoopNotify

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  14. 停止NameNode或是Resourcemanager

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  15. 顯示運行狀態

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

  16. 與Line連動，於Line同步顯示運行狀態

  17. 停止程式運行

    1. Ctrl-Z 將程式丟入背景
    2. 正常模式停止程序
    ​​​​kill -15 %1  # %後面帶程序工作號碼(數字)

    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    -9參數 為強制關閉程序

    1. 檢查程序是否關閉
    

    ​​​​jobs
​​​​jobs

bdseHadoopNotify相關程式

程式連結~~

檔案計有:

  1. bdseHadoopNotify #主要腳本程式
  2. bdse_linenotify.py #呼叫line的python程式
  3. bdse-conf.txt #放置Token(權杖)

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
使用前請先看README.txt

參考資料

  1. LINE Notify API 官方文件請參閱
  2. Hadoop HA 官方文件請參閱
Last changed by 
 
Jeff
0
510
Published on HackMD