Scratch and Python 2018 - Python Lecture 4
===

## File I/O
+ `open`
+ Read text file: `FILE = open(filename,'r')`
+ Read whole file as a string: `FILE.read()`
+ Read while file as lines: `FILE.readlines()`
+ Read a single line: `FILE.readline()`
+ Is iterable: `for line in FILE:`
+ Write text file: `FILE = open(filename,'w')`
+ Print to file: `print(things,file=FILE)`
+ Write a string: `FILE.write(string)`
+ `close`
+ `FILE.close()`
+ `with` block
+ Prevent you forget to close the file.
+ ```python3
with open(filename,'r') as FILE:
lines = FILE.readlines()

## JSON
+ [Reference](https://docs.python.org/3/library/json.html#module-json)
+ `import json`
+ `dumps`
    + `json.dumps({'name':'MZ','height':177.5,'weight':115})`
+ `dump`
    + ```python3
      with open(filename,'w') as FILE:
          json.dump({'name':'MZ','height':177.5,'weight':115},FILE)

+ `loads`
+ `lst = json.loads('[1,2,3,4,5]')`
+ `load`
+ ```python3
with open(filename,'r') as FILE:
obj = json.load(FILE)

## CSV
+ [Reference](https://docs.python.org/3/library/csv.html)
+ `import csv`
+ `csv.reader`
    + ```python3
      import csv
      with open(csvfile,'r') as FILE:
          rd = csv.reader(FILE)
          rows = [row for row in rd]
          
      print(rows)

+ `csv.writer`
+ ```python3
import csv
with open(csvfile,'w') as FILE:
wt = csv.writer(FILE)
wt.writerow(['Name','Height','Weight'])
wt.writerow(['MZ',177.5,115])

+ `csv.DictReader`
    + ```python3
      import csv
      with open(csvfile,'r') as FILE:
          rd = csv.DictReader(FILE)
          rows = [row for row in rd]
          
      print(rows)

+ `csv.DictWriter`
+ ```python3
import csv
with open(csvfile,'w') as FILE:
fns = ['Name','Height','Weight']
wt = csv.DictWriter(FILE,fieldnames=fns)
wt.writerow({'Name':'MZ','Height':177.5,'Weight':115})

      
## `requests`: a browser (?)

+   `import requests`: don't forget `s`
+   URL: Uniform Resource Locator
    +   這術語好長我們叫他網址好了
+   `result = requests.get('http://www.nctu.edu.tw/')`
    +   這網址好長請試著複製貼上吧
+   `result.text`
    +   這內容好字串：`type(result.text)`
    +   String processing
        +   `str`
        +   `import re`: regular expression
+   `result.raise_for_status()`
    +   連網頁做了好多事情，要是網址有錯瀏覽器會直接當掉給你看嗎？
    +   It is OK to raise an exception, always using `try-except` blocks is tedious.
        +   Java is tedious: almost always `try` or `throws` (`raise` in Python)
    +   `raise_for_status()`: If there is any exception, raise it now.
+   Save the content
    +   Open a file to save it: `the_file = open('a_name.html', 'wb')`
        +   Filename: `a_name.html`
        +   Mode: `wb` means "write binary"
            +   `the_file.write(chunk)`: write a chunk of bytes
        +   Mode: `wt` means "write text"
            +   `print(some_str,file=the_file)`: write a string
    +   `for chunk in result.iter_content(102400):` to iterate 102400-byte chunks of `result`
    +   Remember to close the file: `the_file.close()`
    +   Sample code

import requests

url = input('Input URL: ')
result = requests.get(url)
result.raise_for_status()

name = input('input filename: ')
FILE = open(name,'wb')
for chunk in result.iter_content(102400):
FILE.write(chunk)

FILE.close()

    +   Sample code 2: `with`-block (Probably, every body sometimes forgets to close the file.)

import requests

url = input('Input URL: ')
result = requests.get(url)
result.raise_for_status()

name = input('input filename: ')
with open(name,'wb') as FILE:
for chunk in result.iter_content(102400):
FILE.write(chunk)

    +   Try to download some images with the sample codes.

import requests

def url_to_file(url,filename):
result = requests.get(url)
result.raise_for_status()
with open(filename,'wb') as FILE:
for chunk in result.iter_content(102400):
FILE.write(chunk)

## Open Data

+ [政府開放資料平台](https://data.gov.tw/)
+ [紫外線即時監測資料](https://data.gov.tw/dataset/6076)
    + Task: 請撰寫一個程式，透過紫外線即時監測資料，找出 UVI 前三高的地方。
        + Hint: `requests.get('http://opendata.epa.gov.tw/ws/Data/UV/?$format=json')`
        + Sample Code

import requests, json

res = requests.get('http://opendata.epa.gov.tw/ws/Data/UV/?$format=json')
data = json.loads(res.text)
uvi_place = [(float(d['UVI']),d['SiteName']) for d in data if d['UVI']!='']
uvi_place.sort(reverse=True)
print(uvi_place[:3])

+ [全國電子發票B2C開立資料集](https://data.gov.tw/dataset/36846)
    + Task: 請撰寫一個程式，統計 2014 年至 2017 年，各行業平均客單價。輸出為 CSV 格式的檔案。
        + Hint: `requests.get('http://sip.einvoice.nat.gov.tw/ods-main/ODS308E/download/3886F055-EB77-4DF9-98E2-F3F49A7D3434/1/845E38D0-76D4-4B49-922A-96F41705F175/0/?fileType=csv')` 
        + Sample Code

import requests, csv

def url_to_file(url,filename):
result = requests.get(url)
result.raise_for_status()
with open(filename,'wb') as FILE:
for chunk in result.iter_content(102400):
FILE.write(chunk)

url = 'http://sip.einvoice.nat.gov.tw/ods-main/ODS308E/download/3886F055-EB77-4DF9-98E2-F3F49A7D3434/1/845E38D0-76D4-4B49-922A-96F41705F175/0/?fileType=csv'
filename = 'C:\\Users\\user\\Desktop\\task2.csv'
url_to_file(url, filename)

with open(filename,'r',encoding='utf8') as FILE:
rd = csv.DictReader(FILE)
rows = [row for row in rd]

avg = {}
for row in rows:
if row['\ufeff發票年月'].startswith('2018'): continue
avg.setdefault(row['行業名稱'],[]).append(float(row['平均客單價']))

for k, v in avg.items():
print(k,sum(v)/len(v))