---
# System prepended metadata

title: readme

---

---
title: 'readme'
disqus: hackmd
---

大數據與商業分析（Group 5）
===
HackMD好讀版：https://hackmd.io/@BZX5xmFVQ6-Y4RO6QBIJ_A/SkFmRHSyB
### Requirement
1. Python 3.6
2. Jupyter Notebook

### Required Packages
1. pandas
2. numpy
3. json
4. time, datetime
5. graphviz *建議使用brew install graphviz*
6. pydotplus, Image
7. pickle
8. sklearn


## 廣告效益分析(1)
### File Name
Advertisement analysis 1.ipynb
### Preprocessing
#### Behavior Data
移除behavior data中不需要的欄位：
```python
BD = BD.drop(columns=['ProductId', 'SearchKeyWord','CategoryId','TransactionNum','TransactionRevenue',
                      'VisitorId', 'ProductQuantity'])
```
behavior data只保留Facebook、Instagram、Line、LineShopping四個廣告管道，以及ViewSalePage、Cart、Purchase三種行為，並移除含有Na值得資料:
```python
BD = BD[BD['TrafficSourceCategory'] != 'Direct']
BD = BD[BD['TrafficSourceCategory'] != 'GoogleOrganic']
BD = BD[BD['TrafficSourceCategory'] != 'Email']
BD = BD[BD['TrafficSourceCategory'] != 'Others']
BD = BD[BD['BehaviorType'] != 'ViewSalePageCategory']
BD = BD[BD['BehaviorType'] != 'Fav']
BD = BD[BD['BehaviorType'] != 'Search']
BD = BD.dropna()
```

新增行為年份、行為月份、行為時間等欄位資料：
```python
BD['HitDateTime'] = pd.to_datetime(BD['HitDateTime'], errors = 'coerce')
BD['buy_year'] = pd.DatetimeIndex(BD['HitDateTime']).year
BD['buy_month'] = pd.DatetimeIndex(BD['HitDateTime']).month
BD['buy_hour'] = pd.DatetimeIndex(BD['HitDateTime']).hour
```

#### Member Data
移除含有Na值的資料：
```python
MD = MD.dropna()
```
計算會員年齡以及註冊年數，並移除不合理的年齡資料：
```python
MD['Birthday'] = pd.to_datetime(MD['Birthday'], errors = 'coerce')
MD['birth_year'] = pd.DatetimeIndex(MD['Birthday']).year
MD = MD[MD['birth_year'] > 1900]
MD = MD[MD['birth_year'] < 2019]
MD['age'] = 2019 - MD['birth_year']

MD['RegisterDate'] = pd.to_datetime(MD['RegisterDate'], errors = 'coerce')
MD['year_of_reg'] = 2019 - pd.DatetimeIndex(MD['RegisterDate']).year
```

#### Combined Data
將OnlineMemberId設為Member data和Behavior data的Index，並合併兩表：
```python
BD = BD.set_index('OnlineMemberId')
MD = MD.set_index('OnlineMemberId')

# combine BD and MD
data = BD.join(MD)
```
新增Purchase欄位：
```python
Purchase = []
for behavior in data['BehaviorType']:
    if behavior == 'Purchase':
        Purchase.append(1)
    else:
        Purchase.append(0)

data['Purchase'] = Purchase
del Purchase
```

因未購買資料比數遠大於購買資料，為避免模型偏頗，因此對未購買資料進行隨機抽樣，使兩者比例為3:1
```python
not_buy = data[data['Purchase'] == 0]
not_buy_sample = not_buy.sample(frac=0.06)

data = data[data['Purchase'] == 1]
data = pd.concat([data, not_buy_sample], axis=0)
del not_buy, noy_buy_sample
```
#### Create Dummy variables
將類別變數調整成Dummy variables，並新增對應欄位：
```python
final_train = pd.get_dummies(data, columns=['TrafficSourceCategory','RegisterSourceTypeDef',
                                            'Gender', 'SourceType','MemberCardLevel'])                       
```

#### 切割資料集
```python
X = final_train.loc[:, final_train.columns != 'Purchase']
y = final_train.loc[:, final_train.columns == 'Purchase']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
```

### Model
Logistic Regression
* cols為一包含所有欲加進Model中之變數名稱的list，list中的變數名稱必須和final_train中的欄位名稱完全相同，亦可自行刪減變數。
```python
cols = X_train.columns

X = X_train[cols]
y = y_train['Purchase']
```
### Output
Fit the regression and print summary report:
```python
logit_model=sm.Logit(y,X)
result=logit_model.fit()
print(result.summary2())
```


## 廣告效益分析(2)
### File Name
Advertisement analysis 2.ipynb
### Preprocessing
####  A.移除非會員資料
```python
non_member=[]
for i in range(len(fb_view)):
    online_id=fb_view.iloc[i]['OnlineMemberId']
    if str(online_id)=='nan':
        non_member.append(i)
fb_member_view=fb_view.drop(fb_view.index[non_member])
```
####  B.建立一個dictionary，將OnlineMemberID對應到UUID
```python
online2uuid=dict()
for i in range(len(Member)):
    onlineID=Member.iloc[i]['OnlineMemberId']
    if onlineID not in online2uuid.keys():
        online2uuid[onlineID]=Member.iloc[i]['UUID']
```
計算客人的年紀（age）
```python
def calculate_age(born):
    today = date.today()
    return today.year - born.year - ((today.month, today.day) < (born.month, born.day))
```
####  C.記下每筆交易資料發生的月份
```python
for i in range(len(Order)):
    #將日期從string的形式轉成datetime
    strdate=Order.iloc[i]['TradesDate']
    datee = datetime.datetime.strptime(strdate, "%Y/%m/%d")
    month=datee.month
    month_column.append(month)
```
####  D.新增“Buy”欄位，對應至Order data，若客人此次瀏覽後兩週內有購買行為，則將Buy設為1，未購買則設為0
```python
#針對每次的瀏覽，去找他後來是否有買
#second_sample是從behavioral data中，隨機抽取從FB導流，進入Web/APP從事瀏覽行為的資料。並將second_sample以UUID進行排序，這樣在查詢此人是否有購買紀錄時，只要檢查完自己UUID的資料即可確認，縮短判斷時間
for i in range(len(second_sample)):
    print('number {}'.format(i))
    onlineID=second_sample.iloc[i]['OnlineMemberId']
    if onlineID not in online2uuid.keys():
        buy.append(0)
    else:
        uuid=online2uuid[onlineID] #將OnlineMemberID轉成UUID
        find_person=False
        view_time=second_sample.iloc[i]['datetime']
        for j in range(len(new_order_noreturn)):
            if new_order_noreturn.iloc[j]['UUID']==uuid:
                find_person=True
                buy_time=new_order_noreturn.iloc[j]['datetime']
                dis=(buy_time-view_time).days
                if dis>=0 and dis<=14:
                    #判斷是否在兩週內有購買
                    buy.append(1)
                    break
                elif j==len(new_order_noreturn)-1:
                    buy.append(0)
                    break
            elif find_person==True:
                #此情況表示檢查完自己UUID的資料了，但是沒有一筆是在瀏覽後的14天內
                buy.append(0)
                break
            elif j==len(new_order_noreturn)-1 and find_person==False:
                #此情況表示此人沒有任何的購買紀錄，以致於find_person的值仍為True
                buy.append(0)
                break
```
### Split Train and Test Data
```python
X_train, X_test, y_train, y_test = train_test_split(X_second, y_second, test_size=0.2, random_state=42)
```

### Model and Output
PartA. Logistic Regression，瞭解不同解釋變數對於購買與否的影響
* cols為一包含欲加進Model中之變數名稱的list，list中的變數名稱必須和second_sample中的欄位名稱完全相同，亦可自行刪減變數。
```python
three_cols=['Female','Age','APP','SessionNumber','ProductPrice','month','frequency','Qty','average'] 
X = X_train[cols]
y = y_train['Buy']

Fit the regression and print summary report:
```python
logit_model=sm.Logit(y,X)
result=logit_model.fit()
print(result.summary2())
```
PartB. Naive Bayse,SVM,Random Forest,KNN，依照所選用的X變數預測瀏覽商品後兩週內是否購買
```python
print("===== NB =====")
gnb = GaussianNB()
gnb.fit(X_train, y_train)
result2 = gnb.predict(X_test)


print("===== SVM =====")
my_svm = SVC()
my_svm.fit(X_train, y_train)
result1 = my_svm.predict(X_test)


print("===== RF =====")
clf = RandomForestClassifier()
clf.fit(X_train, y_train)
result4 = clf.predict(X_test)


print("===== KNN =====")
neigh = KNeighborsClassifier()
neigh.fit(X_train, y_train)
result3 = neigh.predict(X_test)
```

### 消費者行為分析
### File Name
customer_behavior_analysis.ipynb

### Setting
確認以下設定後，即可“Restart and Run All”
* 引入檔案需注意檔案路徑
* 呼叫draw_tree()輸出圖檔時，需注意資料夾存在

### Model
* 使用的 model
如果要新增其他model，則要同時增加於names和classifiers
```python
names = [
    "Naive Bayes",
    "Random Forest",
    "Decision Tree",
    "AdaBoost"
]
classifiers = [
    GaussianNB(),
    RandomForestClassifier(max_depth=5, n_estimators=16, max_features=3, random_state=5),
    DecisionTreeClassifier(max_depth=5, random_state=5),
    AdaBoostClassifier(),
]
```

* Training and Testing
``` python
for name, clf in zip(names, classifiers):
    clf.fit(X_train, y_train)
    y_predict = clf.predict(X_test)
    evals = precision_recall_fscore_support(y_test, y_predict, average='weighted')
    precision, recall, f1score = "{:.4f}".format(evals[0]), "{:.4f}".format(evals[1]), "{:.4f}".format(evals[2])
    if name == "Random Forest":
        draw_tree(clf, "./tree chart/rf({}).png".format(n))
```

* Plot the Tree
```python
def draw_tree(clf, output_name):
    dot_data  = tree.export_graphviz(
                clf.estimators_[0], out_file=None, 
                feature_names=limit_vocab, class_names={0: "Not Buy", 1: "Buy"},
                filled=True, rounded=True, special_characters=True
            )
    graph = pydotplus.graph_from_dot_data(dot_data)
    graph.write_png(output_name)
    print(output_name, "saved!")
```
### Output
* png files (for the first estimator of Random Forest result)
* evaluations.csv (record all results)