---
GA: UA-159972578-2
---
###### tags: `R` `Data Processing` `資料前處理`
# R 常用技巧
Other Reference:
1. [Wrangling (Rpubs Note)](http://rpubs.com/RitaTang/WranglingNote)
2. [Visualization](/@ritatang242/SJF_dAR37)
3. [Visualization (Rpubs Note)](https://rpubs.com/RitaTang/Visualization)
4. [Text Mining (Preprocessing)](/@ritatang242/B1qbL3yLL)
5. [Text Mining (Chinese)](/PFrfv9NJSuy7frOV-wuJoA?both)
6. [PCA 主成分分析 (Principal Component Analysis)](/@ritatang242/r1lDWqiUL)
# Package, initialization
```{r}
pacman::p_load()
rm(list = ls()); gc()
```
# Encode
```{r}
a = cbind(df$area, df$product_name)
Encoding(a) = 'UTF-8' # 只能放vector
```
# Deal with NA
| 符號 | 定義 |
| -------- | -------- |
| NA |missing or undefined data |
| NULL | empty object (e.g. null/empty lists) |
| Inf and -Inf | positive and negative infinity |
| NaN | results that cannot be reasonably defined |
+ 刪除NA
```{r}
df = na.omit(df)
# equivalent: total %>% na.omit()
mean(data, na.rm = T)
```
```{r}
a = airquality[, colSums(is.na(airquality))==0]
# 先找出誰是NA，再計算每個變數的NA總值來確認誰沒有NA，以此去除NA
```
+ 跳過NA
```{r}
mean(data, na.action="na.pass")
```
+ 把NA變o
```{r}
mx[is.na(mx)] = 0
```
+ 將NA變為一個類別
```{r}
Reduce(rbind, Map(function(x) read.csv(x, na.strings = '.',
stringsAsFactors = F)[, -1], c('cars1.csv','cars2.csv')))
# 創造一個function做Map，一次讀取cars1和cars2兩個資料集(變成一個list)
# 並將值為"."的替換成NA
# 把這個list用rbind和reduce合在一起
```
# Date Format
```{r}
as.Date("2018-10-10", format = "%Y-%m-%d")
as.character(r$BOARD_DATE) %>% as.Date(.,"%Y%m%d")
as.Date(rail_df$BOARD_DATE %>% as.character(), format = '%Y%m%d')
```
# Data Frame Type
+ data.frame
+ 預覽全部資料
+ data.frame會將字串資料轉換成factor
+ data.table
+ data.table不會
+ 沒有rowname
+ 語法使用list()
```{r}
dt[, list(A, C)]
# equivalent: df[, c(1,3)]
```
+ tibble
+ 方便查詢(可以直接預覽前10筆)
+ 處理快速
# Read & Write
+ 套件介紹
+ library(foreign) 可以處理非csv的檔案,如sas
+ library(XML)
+ library(DBI) 讀取關聯式資料庫
+ library(RMySQL) 讀取SQL資料庫
+ Read Data
+ fread (速度最快/大量處理)
+ csv
+ library(data.table)
+ read_csv (速度中等)
+ csv, xlsx, xls
+ library(readr)
+ read.csv (速度最慢)
+ csv
+ read_file
+ txt
+ library(readr)
+ read.table
+ txt
```{r}
library(data.table)
fread("TaiwanRailway.csv", sep = '\t', encoding="big5")
read.table("TaiwanRailway.csv", header = T, stringsAsFactors = F)
xmlToDataFrame("Desktop/A_lvr_land_A.XML")
```
+ Write R.data
```{r}
save(想儲存的變數名稱, file = "../final.rdata") # 將指定的環境變數保存
save.image() # save everthing in current environment
load("../final.rdata")
```
+ Write Data
+ fwrite
+ write.csv
+ saveXML
+ write.foreign
```{r}
fwrite("TaiwanRailway.csv")
write.csv("TaiwanRailway.csv")
saveXML(xml, "test.xml")
write.foreign()
```
# Function
```{r}
predtr = predict(mod4, tr)
MAE = function(n,y,y_hat) {
return( (1/n) * sum(abs(y-y_hat)) )
}
MAE(nrow(tr), tr$Salary, predtr) # 223.9275
# (1/nrow(tr))*sum(abs(tr$Salary-predtr)) # 223.9275
```
# 排序
+ sort
+ 回傳排序後(A-F)本身實際值
```
x = c("D","A","C","F","B","E")
sort(x, decreasing = F)
```
> [1] "A" "B" "C" "D" "E" "F"
+ rank
+ 回傳排序後(A-F)的排名, 依照排序前的順序打印
```{r}
rank(x)
```
> [1] 4 1 3 6 2 5
> D: rank 4, A: rank 1
+ oder
+ 回傳排序後(A-F)在排序前的索引
```{r}
order(x, decreasing = F)
```
> [1] 2 5 3 1 6 4
> A: index[2], B: index[5]
+ arrange (check Dplyr)
+ 只能對data.frame做
# Dplyr
<iframe width=100%, height=500px src="https://hackmd.io/@ritatang242/BkyUfSKrL">
</iframe>
# Join
+ merge
+ left_join
+ right_join
+ inner_join
+ all_join
+ semi_join
# SQL
```{r}
sqldf::sqldf("select * from A")
where # filter的概念
group by # group_by
is not null # 非NA值
order by x limit 5 # order,照x排列取五個
```
# apply 系列
## 1. apply
```{r}
apply(data, 1, FUN)
# margin: 1 means "row"; 2 means "column"
```
## 2. lapply
+ 取代迴圈
+ 回傳list
```{r}
lapply(data, FUN)
lapply(c(sum,mean,prob), FUN = function(f) f(data))
```
## 3. sapply
+ 取代迴圈
+ 回傳matrix
```{r}
sapply(data, FUN)
```
## 4. tapply
+ 對類別做function計算
```{r}
tapply(data, INDEX = iris$Species, FUN = distinct_counts)
# INDEX: 分類
# tapply() 函數是融入 table() 函數功能的形式。
```
# aggregate
```{r}
aggregate(price ~ cut + color, data = diamonds, mean)
# 根據某個cut和color進行分群，求price平均數
aggregate(x=mtcars$mpg, by=list("cyl"= mtcars$cyl), FUN=mean)
# cyl x
# 1 4 26.66364
# 2 6 19.74286
# 3 8 15.10000
```
# by
```{r}
by(data=mtcars$mpg, INDICES=list("cyl"= mtcars$cyl), FUN=mean)
# cyl: 4
# [1] 26.66364
# cyl: 6
# [1] 19.74286
# cyl: 8
# [1] 15.1
```
# Map Reduce
## Map
```{r}
Map(FUN, df$x)
```
與lapply頗像，讓完整的df中的不同obs.分別執行同樣的function，再一起存入一個list
```{r}
genKPercentile = function(q1, q2, q3, q4){
pct = Map(function(x) {function(y) quantile(y, x/100)}, c(q1, q2, q3, q4))
names(pct) <- Map(function(z) paste0("pct_", z), c(q1, q2, q3, q4))
return(pct)
}
pcts = genKPercentile(25,75,50,10)
pcts$pct_50(mtcars$mpg) # 50%; 19.2
pcts$pct_10(mtcars$mpg) # 10%; 14.34
```
## Reduce
```{r}
Reduce(f, x, init, right = FALSE, accumulate = FALSE)
```
+ 將一個向量按相鄰的兩個元素依次作二元函数運算(一層包一層的疊加)，最后输出结果。
+ Reduce的基本參數有兩個，參數1是二元函数，参數2是運算向量
+ 參數註解：
+ f： 有兩個參數的函数对象
+ x： 向量
+ init： 一個標準值，類型和x向量的元素相同
+ right： 當值為FALSE時，從向量的左側開始，依次向右側取出元素傳给函数；當值為TRUE時，則從向量的右側開始
+ accumulate：當值為TRUE時，會呈現逐步運算的结果，若為預設值，以上運算都只會输出最终结果
```{r}
x = c(12,25,3,8)
Reduce(sum, x) # 48 # 運算過程:sum(sum(sum(12,25),3),8)
Reduce(sum, x, accumulate = T) # 37 40 48
```
```{r}
Reduce(sum,1:5,10) # 25
# 運算過程如下:
a <- sum(10,1)
b <- sum(a,2)
c <- sum(b,3)
d <- sum(c,4)
sum(d,5)
```
## mapReduce
```{r}
mapReduce(分組變量, 指定函數, data = 數據框名）
```
+ 與前面说的Map和Reduce功能完全不同，mapReduce提供了按分组變量運行指定函数的方式
+ 實現分組統計的一種簡便、直接的方式，且能同時指定多個函數
+ 回傳結果為一矩陣，同aggregate和sapply功能，但mapReduce可以實現平行運算，是為其優點
```
> dt
sex age weight
1 m 27 45.6
2 f 25 55.9
3 m 40 49.0
4 f 28 59.5
5 m 38 53.5
6 f 32 48.9
7 m 36 45.9
8 f 26 53.4
9 m 32 48.4
10 f 31 54.2
```
按性別分組，計算統計指標：平均年齡，平均體重，年齡標準差，體重標準差
```{r}
mapReduce(sex, mean(age),mean(weight),sd(age),sd(weight), data = dt)
# [,1] [,2] [,3] [,4]
# f 28.4 54.38 3.049590 3.858368
# m 34.6 48.48 5.176872 3.179151
```
# Identify item
## identical
辨認兩者是否完全相等
```{r}
identical(AY$cust_id, subset(AX, train)$cust_id)
# [1] TRUE
```
## unique
找出不重複的值(組合)
```{r}
unique(cust[,c(1:3)]) #看cust第一到第三的欄位共有幾種pattern
unique(cust$cust_id) #找不重複的所有cust_id
```
## duplicated
一個個判斷是否為重複，第一次出現是新值，第二次再出現一樣的即為重複。
```{r}
a <- c(rep("A", 3), rep("B", 3), rep("C",2))
b <- c(1,1,2,4,1,1,2,2)
df <-data.frame(a,b)
```
```{r}
duplicated(df)
# [1] FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE
```
```{r}
df[duplicated(df), ]
# a b
# 2 A 1
# 6 B 1
# 8 C 2
```
```{r}
df[!duplicated(df), ]
# a b
# 1 A 1
# 3 A 2
# 4 B 4
# 5 B 1
# 7 C 2
```
# Data Reshaping
## melt/gather(寬的→長的)
```{r}
melt(data=md, id.vars=c("ID","Time"))
# id.vars即uid, 類似group_by
md = gather(reshape_data, key="ketCol", value="valCol", 3, 4)
```
## cast/speard(長的→寬的)
```{r}
dcast(data=md, formula= ID + Time ~ variable, value.var="value")
spread(reshape_data, key="ketCol", value="valCol", 3, 4)
```
# case_when
利用case_when()函數將連續值變為區間類別
```{r}
case_when(Age>=18 ~ "Adult",
Age<18 ~ "Child",
is.na(Age) ~ "Unknown")
```
# switch
```{r}
switch(指定執行第幾行/哪個名稱的程式碼,
第一行：做A,
第二行：做B,
第三行：做C,...)
switch("first", first = 1 + 1, second = 1 + 2, third = 1 + 3)
# [1] 2
switch(3, "first", "second", "third", "fourth")
# [1] "third"
```
Sign in with Wallet
Connect another wallet