Python 基礎爬蟲

--- title: 'Python 基礎爬蟲' --- Python 基礎爬蟲 === ###### tags: `SIRLA` `python` `爬蟲` >目次 >[TOC] > >Reference >1.[Python爬蟲實戰](https://www.slideshare.net/tw_dsconf/python-78691041) >2.[Python爬蟲筆記](https://medium.com/@gordonfang_85054/python-%E7%88%AC%E8%9F%B2%E7%AD%86%E8%A8%98-1-15fdec38393c) >3.[Python 使用 Beautiful Soup 抓取與解析網頁資料，開發網路爬蟲教學 ](https://blog.gtwang.org/programming/python-beautiful-soup-module-scrape-web-pages-tutorial/) ## **認識爬蟲** 網路爬蟲，英文:Web Crawler 爬蟲是一種機器人，會自動瀏覽網路網頁把目標的資訊擷取下來。 Google就是一個爬蟲的例子，爬遍世界的網站，建立索引來成立搜尋引擎。應用：比價、分析評價、趨勢、…… ### **爬蟲程式步驟** 1. 向目標網址發送HTTP請求封包，取得HTML資料 2. 解析資料 3. 取得目標資訊 ### **爬蟲套件** requests：用來對目標網頁的server發出request BeautifulSoup：用來解析HTML ## **爬蟲實戰** ### **取得網頁內容** * GET vs. POST * GET * 直接透過URL傳參數 * POST * 回傳參數資料到Sever * 安裝套件：Requests ``` pip install requests ``` * 第一次測試取得網頁內容 ```gherkin= import requests response = requests.get('https://www.railway.gov.tw/tra-tip-web/tip/tip001/tip112/querybytime') print(response.text) ``` * 台鐵時刻表與票價 ```gherkin= import requests res = requests.post('https://www.railway.gov.tw/tra-tip-web/tip/tip001/tip112/querybytime') print(res.text) ``` ```gherkin= data1 = {'_csrf': '56538d1c-2a43-41cf-a65c-d0ed7cec7c8f', 'trainTypeList': 'ALL', 'transfer': 'ONE', 'startStation': '3360-彰化', 'endStation': '3470-斗六', 'rideDate': '2019/12/19', 'startOrEndTime': 'true', 'startTime': '00:00', 'endTime': '23:59'} res = requests.post('https://www.railway.gov.tw/tra-tip-web/tip/tip001/tip112/querybytime', data = data1) ``` * 模擬用戶代理 * 股票網站: https://www.stockdog.com.tw/stockdog/index.php?m=overview&sid=1101 * 一般直接請求取得資料 ```gherkin= import requests res = requests.get('https://www.stockdog.com.tw/stockdog/index.php?m=overview&sid=1101') print(res.text) ``` * 加上user-agent ```gherkin= user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'} res = requests.get('https://www.stockdog.com.tw/stockdog/index.php?m=overview&sid=1101',headers = user_agent) ``` * encoding : 遇到亂碼 ```gherkin= import requests res = requests.get('http://isbn.ncl.edu.tw/NEW_ISBNNet/main_DisplayRecord_Popup.php?Pact=view&Pkey=1080117*0046') print(res.text) ``` * 加上 encoding ```gherkin= res.encoding = 'utf-8' ``` ### **擷取網頁內容** * 安裝套件：BeautifulSoup ``` pip install BeautifulSoup4 ``` * 範本 ```gherkin= from bs4 import BeautifulSoup # 原始 HTML 程式碼 html_doc = '<html> \ <body> \ <h1 id="title">Hello World</h1> \ <a href="#" class="link">This is link1</a> \ <a href="# link2" class="link">This is link2</a> \ </body> \ <html> ' # 以 Beautiful Soup 解析 HTML 程式碼 soup = BeautifulSoup(html_doc, 'html.parser') ``` * 顯示結果 ```gherkin= print(soup.text) print(soup.contents) #ALL print(soup.select('html')) #TAG print(soup.select('h1')) #TAG print(soup.select('a')) #TAG print(soup.select('#title')) #ID print(soup.select('.link')) #CLASS ``` * 使用 find ```gherkin= print(soup.find(id='title')) print(soup.find(href='# link2')) print(soup.find_all('a')) ``` ## **Lab1** * 爬蟲 : 圖書資訊 * URL : https://www.kingstone.com.tw/new/basic/2010230006305?zone=book&lid=search&actid=WISE * Output ``` 資訊組織作者：張慧銖、陳淑燕、邱子恒、陳出版社：華藝數位出版日：2017/6/21 ISBN：9789864371310 適讀年齡：全齡適讀 ``` * 參考解法 ```gherkin= import requests from bs4 import BeautifulSoup user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'} res = requests.get('https://www.kingstone.com.tw/new/basic/2010230006305?zone=book&lid=search&actid=WISE',headers = user_agent) soup = BeautifulSoup(res.text, 'html.parser') a = soup.select('.pdname_basic')[0] print(a.text) b = soup.select('.basiccol')[0] result = '' for i in range(1,6): c = b.select('.basicunit')[i] if i < 3: d= c.select('.title_basic')[0] result += d.text e = c.select('a')[0] result += e.text else: result += ' '.join(c.text.split()) if i < 5: result += '\n' print(result) ``` ## **Lab2** * 爬蟲 : 電影資訊 * URL : https://movies.yahoo.com.tw/movieinfo_main/%E5%B0%8F%E5%B0%8F%E5%A4%9C%E6%9B%B2-little-nights-little-love-10213 * Output ``` 小小夜曲上映日期：2019-11-22 片　　長：02時00分發行公司：天馬行空 ``` * 參考解法 ```gherkin= import requests from bs4 import BeautifulSoup res = requests.get('https://movies.yahoo.com.tw/movieinfo_main/%E5%B0%8F%E5%B0%8F%E5%A4%9C%E6%9B%B2-little-nights-little-love-10213') soup = BeautifulSoup(res.text,'html.parser') a = soup.select('.movie_intro_info_r')[0] b = a.select('h1')[0] print(b.text) for i in range(0,3): c = a.select('span')[i] print(c.text) ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.