---
# System prepended metadata

title: WAF Health monitor每日報表
tags: [Trouble shooting]

---

#  WAF Health monitor每日報表

主要是為了解決日常查詢log習慣，因此開發一個定時寄信報表排程。

每日寄件的報表程式佈建於ubuntu server 上的一個自動化排程perl script。
* VirtualHost: 140.113.220.167
* 相關模組：
```
Time::Local;　　　　　　　　　　# system time
DateTime::Format::ISO8601;  # define timestamp
MIME::Lite;     　　　　　　　# sending mail...
```

*  Scripts 路徑:
`/home/netsys/WAF_daily_report/waf_daily_report_ver-20201130.pl`

*  排程: 
可查詢 `crontab –l` 指令，目前是設定在每日上午 7:00執行一次。

```
netsys@waflog [~] $ crontab -l
# For more information see the manual pages of crontab(5) and cron(8)

# m h  dom mon dow   command
00 07 * * *  /usr/bin/perl /home/netsys/WAF_daily_report/waf_daily_report_ver-20201130.pl  >> /home/netsys/WAF_daily_report/waf_praseing_20201130.log
```


#### 程式碼分解:

* ##### 日期擷取  : 擷取系統今天與昨天時間(範本)
1. 今天
```
sub spGetYesterdaysDate1 {
my ($sec, $min, $hour, $mday, $mon, $year) = localtime();
my $yesterday_midday=timelocal(0,0,12,$mday,$mon,$year) - 24*60*60;
($sec, $min, $hour, $mday, $mon, $year) = localtime($yesterday_midday);
my @abbr = qw( Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec );
my $YesterdaysDate = sprintf "%02s %2d ", $abbr[$mon], $mday;
return $YesterdaysDate;
}
```
2. 昨天
```

sub spGetYesterdaysDate2 {
my ($sec, $min, $hour, $mday, $mon, $year) = localtime();
my $yesterday_midday=timelocal(0,0,12,$mday,$mon,$year) - 24*60*60;
($sec, $min, $hour, $mday, $mon, $year) = localtime($yesterday_midday);
my $YesterdaysDate2;
my $YesterdaysDate2 = sprintf "%4d%02d%02s", $year+1900, $mon+1, $mday;
return $YesterdaysDate2;
}
```


* #### perl 執行shell指令執行
把log 從設備挖到 server這邊，處理完之後刪掉，以免日久累積大量檔案
```
my $pwd=qx(pwd);
my $scp_cmd="指令用字串呈現";
system($scp_cmd); 
```


* ##### 資料過濾
用 `grep` 一步一步過濾，撈出想要的log，主要根據log 字串特徵。
觀察log結構之後，主要特徵包含　`EVENT DEVICE`、`0-PPE-0`、`140.113`.

[另法]在shell搭配　`awk` 、`sed `也可。

```
my $cmd="grep '". $key_word."' ". $pwd."/ns.log.".$date_yesterday ."|  grep 'Device \"server' - | grep 'EVENT DEVICE' - | grep '0-PPE-0' - | grep '140.113.' - ";

print "\$cmd=$cmd\n";
my $raw = qx($cmd);

my @RAW=split/\n/,$raw;   
print "  total are : ".@RAW."\n";   


if(@RAW != 0){}else{ die };　　　
```



 * ### 資料格式排列
這一部分主要包含用array儲存重新排列拼貼的資料、時間計算、用hash去存資料。

1. log結構其實有些資訊室也規律性，且希望擷取出來重新排列的。用　`substr()　`或`　split()`　去取出想要的片段，再把他照順去排好用　`Array`存放。

2. 使用　`DateTime::Format::ISO8601`模組，因為時間的進位制不是10進位。沒辦法直接做計算，因此需要把時間轉換成一個值，方便後面去做相減計算。後面呈現需要再把他轉乘時間格式。

4. 為了同時保有資料與要被計算的值之間的連結，用 `%hash` 去以 `$key => $value` 存時間和轉換成的數值　 


```
my %relat;   ###  for event down
my @matrix;  ## for parse array
my @order;   ###  key order 


foreach $_(@RAW){
    s/"//;
    
    #［ event time & date capture］
    my $time=substr($_,7,8);
    my $state=(split'-',$_)[-1];
    my $year=substr($date_yesterday,0,4);
    my $month=substr($date_yesterday,-4,2);
    my $date=substr($date_yesterday,-2,2);

    # ［ip & port capture］
    my $network=(split'\(',$_)[0];
    my $string2=(split'_',$network)[-1];
	my ($ip, $port)=(split/:/,$string2);
	
    # ［make time format for caulate time value］
	my $key=$state ."\t". $ip ."\t". $port ."\t". $year . $month . $date ."\t". $time;
	my ($hr, $min, $sec) = (split/:/,$time)[0,1,2];
	my $dt=$year."-".$month."-".$date."T".$hr.":".$min.":".$sec."Z";

	  my $dtt = DateTime::Format::ISO8601->parse_datetime( $dt );
	  my $value = $dtt->epoch();
	  $relat{$key}=$value;

      
      push (@matrix,$value."\t".$key);
      push(@order,$key);
      print "$value\t$key\n";
}
```




* ### 　過濾出來的　log 進行　duration　計算
抓down 和　up的時間點，去計算與時間格式轉換和分類。 
foreach()內第一層＆第二層if（）去算事件down和up時間，第三層if（）去判對分類server IP

```
foreach my $timstamp_key(@matrix){
   if ($timstamp_key =~ /State DOWN/)　　　　
   {
      
      my($timstamp, $status_a, $ip_a, $port_a, $date_a, $timing_a)=(split'\t',$timstamp_key)[0,1,2,3,4,5]; 

      for (my $i=0; $i<=$#order;$i++)
      {
          my $timstamp_compare=$order[$i];
          my $up_timestamp=$relat{$timstamp_compare};
          #print "       !$timstamp_compare\n";
          my($status_b, $ip_b, $port_b, $date_b, $timing_b)=(split'\t',$timstamp_compare)[0,1,2,3,4];
            
            if($ip_b =~ m/$ip_a/ )
            {
                 # if status is up and ip:port same to compare_key
                if( $port_b =~ m/$port_a/ )
                {

                    if($status_b =~ m/State UP/)
                    {
                            my $duration_sec =  $up_timestamp - $timstamp_key;
                            my $duration_format = time_compute($duration_sec);
                            my $domain=$ip_domain{$ip_a};
                            my $html="";
                                    if ($ip_b =~ m/140.113.41./)
                                    {

                                            if($ip_b =~ m/140.113.41.241/){

                                            print "技發組(TD)\t"."$ip_a:$port_a\t$domain\t$date_a $timing_a\t$date_b $timing_b\t$duration_format\n";
                                            $html="<tr>\n<th>技發組(TD)</th>\n<th>".$ip_a.":".$port_a."</th>\n<th>".$domain."</th>\n<th>"."$date_a $timing_a"."</th>\n<th>"."$date_b $timing_b"."</th>\n<th>".$duration_format."</th>\n</tr>";
                                            push @all_group,$html;
                                            

                                            }else{

                                            print "校資組(MIS)\t"."$ip_a:$port_a\t$domain\t$date_a $timing_a\t$date_b $timing_b\t$duration_format\n";
                                            $html="<tr>\n<th>校資組(MIS)</th>\n<th>".$ip_a.":".$port_a."</th>\n<th>".$domain."</th>\n<th>"."$date_a $timing_a"."</th>\n<th>"."$date_b $timing_b"."</th>\n<th>".$duration_format."</th>\n</tr>";
                                            push @all_group,$html;
                                            
                                            }
                                    
                                    }
                                    elsif($ip_b =~ m/140.113.40./)
                                    {
                                            print "校資組(MIS)\t"."$ip_a:$port_a\t$domain\t$date_a $timing_a\t$date_b $timing_b\t$duration_format\n";
                                            $html="<tr>\n<th>校資組(MIS)</th>\n<th>".$ip_a.":".$port_a."</th>\n<th>".$domain."</th>\n<th>"."$date_a $timing_a"."</th>\n<th>"."$date_b $timing_b"."</th>\n<th>".$duration_format."</th>\n</tr>";
                                            push @all_group,$html;
                                            
                                    }else{

					    if($ip_b =~ m/140.113.9.153/){

                                            print "校資組(MIS)\t"."$ip_a:$port_a\t$domain\t$date_a $timing_a\t$date_b $timing_b\t$duration_format\n";
                                            $html="<tr>\n<th>校資組(MIS)</th>\n<th>".$ip_a.":".$port_a."</th>\n<th>".$domain."</th>\n<th>"."$date_a $timing_a"."</th>\n<th>"."$date_b $timing_b"."</th>\n<th>".$duration_format."</th>\n</tr>";
                                            push @all_group,$html;
                                            

                                            }else{

                                            print "技發組(TD)\t"."$ip_a:$port_a\t$domain\t$date_a $timing_a\t$date_b $timing_b\t$duration_format\n";
                                            $html="<tr>\n<th>技發組(TD)</th>\n<th>".$ip_a.":".$port_a."</th>\n<th>".$domain."</th>\n<th>"."$date_a $timing_a"."</th>\n<th>"."$date_b $timing_b"."</th>\n<th>".$duration_format."</th>\n</tr>";
                                            push @all_group,$html;
                                        
					    
					    }
                                    }
                          
                            delete $relat{$timstamp_compare};                 
                            $order[$i]="";
			     $num = scalar keys %relat;                       
                            last;
                    }else{   }

                    
                }
            }
      }
            
    }else{   # do nothing      }
    
}
```



* #### 時間轉換
時間的近位制。把計算出來的時間長度坐進為換算用。
```
sub time_compute{

    my $time = shift;
    my $day=int($time/60/60/24);
    my $hr1=int($time/60/60);
    my $hr=int(($time/60/60)-($day*24));
    my $min=int(($time-60*60*$hr1)/60);
    my $sec=int($time-60*60*24*$day-60*60*$hr-60*$min);

        if ($time >=     24*60*60)    { sprintf "$day day $hr hr $min min $sec sec";}
        elsif ($time >=        60*60) { sprintf "$hr hr $min min $sec sec";}
        elsif ($time >=           60) { sprintf "$min min $sec sec";}
        else                          { sprintf "$sec sec";}
}
```



* ### 　 寄信模組　MIME::Lite
perl　其實有很多寄信模組可以使用。選擇　MIME::Lite，他是把信的內容用html在呈現，比較可以做一點設計。使用　MIME::Lite　套件來設計信的結構分成２塊，寄件資訊和內容本體。
1. 寄件資訊　：　`$msg = MIME::Lite->new();`
```
 my $msg = MIME::Lite->new(
　　　From    => 'netsys@nctu.edu.tw', 　　　　　　　　　　　　　 　　　＃寄件
   　To      => 'closeguan@nctu.edu.tw',  　　　　　　　　　　　　　　＃收件
　　　Cc      => 's6437094@gmail.com',  　　　　　　　　　　　　　　　　＃副本
    Subject => '［WAF daily report］ :all　group　WAF　report',  　＃標題
    Type    => 'multipart/related
);
```
２.信件內容：
裡面要寫的內容都在 `$msg->attach();`　，可以用　`html`　的語法去調整呈現。
 style 屬性來指定該 HTML 元素的樣式
```
$msg->attach(
    Type => 'text/html',
    Data => qq{
            <h4>Dear, all,</h4>

            <p>      
            </p>


		<table style="width:80%" border="1" >
            
				  <caption><font size="5" face="verdana">[$date_yesterday all_group_WAF_report]</font></caption>
				  <tr bgcolor=#b0e4f2>
				      $header
				  </tr>
				  
                  
				     $all_group_html　　　＃把排列好的資料直接貼上
                  

		</table>

               <p>-------------</p>
               <p>Please feel free to bring up any suggestion or problem you may have.</p>
               <p>Sincerely,</p>

               <p>
               Division of NetSys<br><br>
               Contact person : Yu-Chen Huang(黃俞禎)<br>
               Contact number : 03-5712121 # 52861
               
               </p>


	},

);

$msg->send();

$msg->print(\*STDOUT);　　
```




# report 範本


![](https://i.imgur.com/XZThVvw.png)



# 用Ｒ繪製周報統計圖表


![](https://i.imgur.com/i8eGgJK.png)


### 1. 累積次數
上圖呈現近雙週waf health monitor 監控次數統計呈現，以縱軸呈現日期，橫軸呈偵測到的現事件次數，並以顏色區分主機。圖中waf health monitor 監控預設參數，是以每5秒進行一次syn-ack 詢問，兩次未於2秒內未收到 ack 回應，則會等待30秒後才接續syn-ack 詢問。
```
#1 import library
library(ggplot2)
library(ggpubr)


# import data as data frame
df <- read.table("2020_0316-0329_00.txt", header = T)
dd <- c("03/16(Mon.)", "03/17(Tue.)", "03/18(Wen.)", "03/19(Thr.)", "05/20(Fri.)", "03/21(Sat.)", "03/22(Sun.)","03/23(Mon.)", "03/24(Tue.)","03/25(Wen.)", "03/26(Thr.)", "03/27(Fri.)", "03/28(Sat.)", "03/29(Sun.)")

# polt display
ggplot(df, aes(x=Date, fill=domain)) +
  geom_bar(width = 0.5, position = "stack") +
  geom_text(stat="count", 
            aes(label=..count..), 
            position = position_stack(vjust = 0.5), 
            color=I("black"), 
            size=3) +
  labs(title = "WAF daily report",
       subtitle = "2020/3/16 ~ 2020/3/29",
       x = "Date", y = "Domain Frequency") +
  scale_x_discrete(labels = dd) +
  coord_flip() +
  theme(legend.position = "bottom")
```

### 2. 發生時間分布點
WAF HTTP/HTTPS 服務偵測圖，根據一週個時間點鎖呈現服務中段時間分佈，僅呈現偵測到中斷時間大於30秒之資訊，圖示描述如下：
* 以[顏色]區分不同[服務(service ip) ]、以圖形[大小]表示[down的時間]。
* 以[形狀]表示服務隸屬[組別]，三角形▲表示[校務資訊組]、圓形●表示[技術發展組]。

![](https://i.imgur.com/PNbL7bW.png)
```
#1) import library

library(ggplot2)
library(chron)
library(ggrepel)

#2) load data

raw <- read.table("2020_0316-0329_00.txt", header = T)

#3) select duration > 30以上的資料
v1 <- raw$Date[raw$durations>30]
v2 <- raw$Division[raw$durations>30]
v3 <- raw$ip[raw$durations>30]
v4 <- raw$port[raw$durations>30]
v5 <- raw$domain[raw$durations>30]
v6 <- raw$down_time[raw$durations>30]
v7 <- raw$up_time[raw$durations>30]
v8 <- raw$durations[raw$durations>30]
all <- data.frame(v1, v2, v3, v4, v5, v6, v7, v8)
names(all) <- names(raw)

#4) x & y 軸上的刻度
dd <- c("03/16(Mon.)", "03/17(Tue.)", "03/18(Wen.)", "03/19(Thr.)", "05/20(Fri.)", "03/21(Sat.)", "03/22(Sun.)","03/23(Mon.)", "03/24(Tue.)","03/25(Wen.)", "03/26(Thr.)", "03/27(Fri.)", "03/28(Sat.)", "03/29(Sun.)")
tt <- c(
  "00:00","01:00","02:00","03:00","04:00","05:00","06:00","07:00","08:00","09:00","10:00","11:00","12:00","13:00","14:00","15:00","16:00","17:00","18:00","19:00","20:00","21:00","22:00","23:00","24:00"
)

#5) 採用灰色background
theme_set(theme_gray())

#6) plot 
  #6.1 basic plot

g <- ggplot(all, aes(x=times(all$down_time), y=all$Date)) +
  geom_point(aes(color=as.factor(all$domain), shape = all$Division, size = all$durations)) +
  facet_wrap( ~ all$port, nrow=2) 

  #6.2 座標範圍
  # 時間資料與數值資料的轉換: 01:00:00 = 1/24
  # 記得修該時間軸範圍時也要修改軸上的標籤tt

  g1 <- g +
        coord_cartesian(xlim=c(0/24,24/24)) +  
        scale_x_continuous(breaks=seq(0/24,24/24,1/24), labels = tt)  +
        scale_y_discrete(labels = dd) 

  #6.3 標題與座標軸文字
  # 記得修改subtitle的日期

  g2 <- g1 + 
        labs(title = "WAF daily report", 
             subtitle = "2020 3/16~3/29", 
             y = "Date \n", 
             x = "Down Time") +
        theme(axis.text.x = element_text(size = 12, 
                                   angle = 45, 
                                   vjust = 1.2, 
                                   hjust = 1.2),
                                　　axis.title.x = element_text(size = 14),
                                　　axis.text.y = element_text(size = 14),
                                　　axis.title.y = element_text(size = 18),
                                　　plot.title=element_text(size=10, 
                                　　face="bold", 
                                　　family="American Typewriter",
                                　　color="blue",
                                　　hjust=0.5,
                                　　lineheight=1.2),
                                   plot.subtitle=element_text(size=15, 
                                   family="American Typewriter",
                                   face="bold",
                                   hjust=0.5)
  ) 

#6.4 圖例

  g3 <- g2 + 
        labs(color="domain\n", size="duration (sec)\n", shape="divisions\n") +
        theme(legend.position="bottom", 
        legend.box = "horizontal", 
        legend.background = element_rect(fill="gray",  
        size=0.5, linetype="solid",colour ="black")) +
        guides(shape = guide_legend(order = 1, ncol=1), 
        size = guide_legend(order = 2, ncol=1), 
        color = guide_legend(order=3, ncol=3)) 

#6.5 將down time為5:00之後的點標上domain name

   v5[times(all$down_time) < times("05:00:00")] <- NA
#v5[times(all$down_time) < times("05:00:00")] <- NA

g4 <- g3 +
        geom_label_repel(aes(label = v5),
                             box.padding = 0.35, 
                             point.padding = 0.5,
                             segment.color = 'grey50') 

#7) save plot

g4

ggsave(file="0plot.pdf",device="pdf" ,width=35.7 , heigh=23 , unit="cm")
```




---

# WAF 區架構圖

目前cloud區與ＷＡＦ區的網路架構。


![](https://i.imgur.com/T8JnyhP.png)

# NetScaaler crontab

```
root@ns# crontab -l
0 1  * * * sh /flash/nsconfig/script/nslogren.sh
#0 */1 * * * /usr/sbin/ntpdate -s 118.163.81.61
```


# NetScaaler script view

確認　`/flash/nsconfig/script/nslogren.sh`

```
#!/bin/sh
checkdate=`date -v -1d  +%Y%m%d`
checkmonth=`date -v -1d +%Y%m`
checkcurrentmonth=`date +%Y%m`

mv /var/log/ns.log.0.gz /var/log/ns.log.$checkdate.gz


find /var/nslog/ -iregex "/var/nslog/newnslog.[0-9]\{1,3\}.*.gz" -exec ls -l -D '%Y%m%d-%H%M' {} \; | nawk '{print "mv "$7" /var/nslog/newnslog."$6".tar.gz" }' | /usr/bin/bash
find /var/nslog/*.gz   -ctime +180 -exec rm {} \;

echo 0 > /var/nslog/nslog.nextfile

if [ ! -d "/var/log/$checkmonth" ] && [ $checkmonth != $checkcurrentmonth ] ; then
  mkdir /var/log/$checkmonth
  mv /var/log/ns.log.$checkmonth* /var/log/$checkmonth
  cp /var/log/$checkmonth/ns.log.$checkdate.gz /var/log
fi

crontab -l > /flash/nsconfig/cron.txt
```

新增一條，讓換月時候不會再 /var/log 抓不到東西。
cp /var/log/$checkmonth/ns.log.$checkdate.gz /var/log

