Fluentd

tags: QCT Data Center

Install td-agent

sudo curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
td-agent -c CONFIG_PATH -v 

Elasticsearch Output Plugin

• Install Elasticsearch

docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.3.0

• Install td-agent Elasticsearch plugin

sudo td-agent-gem install elasticsearch

# /etc/td-agent/td-agent.conf 
<match {bmc,system.**}>
  @type elasticsearch
  host 10.103.3.89
  port 9200
  index_name nfvd
  include_timestamp true
</match>

• REST APIs
  • List all indices
  ​​curl http://10.103.3.89:9200/_cat/indices?v
  

  • Delete index
  ​​curl -X DELETE http://10.103.3.89:9200/fluentd | jq
  

  • Check logs
  ​​curl http://10.103.3.89:9200/fluentd/_search | jq
  

  ​​curl http://10.103.3.89:9200/fluentd/_search?q=FIELD_VALUE | jq
  

  ​​curl http://10.103.11.100:9200/nfvd/_search?q=SerialNo:QTFCR2725007B | jq
  

• Data format

2019-08-21T11:34:25+08:00	system.loadavg	{"key1":"0.00","key2":"0.00","key3":"0.00"}

每筆紀錄要有時間欄位，且格式2019-08-21T11:34:25+08:00要一樣。

Syslog Input Plugin

• in_syslog is included in Fluentd's core.

# /etc/td-agent/td-agent.conf 
<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  tag log.system
  protocol_type tcp
</source>

• Rsyslog client

# /etc/rsyslog.conf should include
$IncludeConfig /etc/rsyslog.d/*.conf

# /etc/rsyslog.d/00-td-agent.conf should include
*.*        @@10.103.3.84:5140

sudo service rsyslog restart

To forward messages to another host via UDP, prepend the hostname with the at sign (@). To forward it via plain tcp, prepend two at signs (@@). To forward via RELP, prepend the string :omrelp: in front of the hostname.

• Check log

logger test
tail -f /var/log/syslog