---
# System prepended metadata

title: Use python requests to crawl website
tags: [爬蟲]

---

# Table of contents
[TOC]
# Sample code
```python
import requests
from bs4 import BeautifulSoup

def main():
    url = 'target_url'

    try:
        # Send an HTTP GET request to the URL
        response = requests.get(url)
        response.raise_for_status()  # Check for errors in the HTTP response
        response.encoding = 'utf-8'

        # Parse the HTML content of the page
        soup = BeautifulSoup(response.text, "html.parser")

        tag = soup.find('tag_name', attrs={'attribute': 'attribute_value'})

    except requests.exceptions.RequestException as e:
        print(f"An error occurred: {e}")
if __name__ == "__main__":
    main()
```
# common command
## soup.find()
```python
a_soup = soup.find('tag_name', attrs={"attribute": "value", ...})
```
- return an tag object
    - 
## soup.find_all()
```python
b_soup = soup.find_all('tag_name')
```
- reutrn an array. Can be indexed
    - ex. `b_soup[i]`
# References
- [Beautiful Soup — Python 的網路爬蟲套件](https://medium.com/@yuhsienyeh/beautiful-soup-python-%E7%9A%84%E7%B6%B2%E8%B7%AF%E7%88%AC%E8%9F%B2%E5%A5%97%E4%BB%B6-be09be3d1a21)
- [Beautiful Soup 函式庫](https://steam.oxxostudio.tw/category/python/spider/beautiful-soup.html)