Windowsのプロセス単位のCPU使用率アラートをfluentdを使ってやってみる…の続き

2014.08.27 追記

Release 0.10.49 で入っているTextFormatterを使って、plaintextformatterを使ってるところを無理やりjson固定でのTextFormatterに入れ替えることで、末尾にある文字コード変換のエラーは解消できるようになりました。
fluentd/ChangeLog at master · fluent/fluentd · GitHub


前回(Windowsのプロセス単位のCPU使用率アラートをfluentdを使ってやってみる - メモ帳みたいなもの)の続きです。
やってること自体はfluentdを使っている人であればありふれた内容なのでほぼ自分用のメモです。

  • 「1時間単位での min/max/avg も出したくなってくる。」の部分とグラフ化
  • windowsのイベントログを受ける部分の追加
nxlog.conf

im_msvistalogはExec でsyslog形式に変換して流しても動きましたが、情報が冗長という感じだったためim_fileと同じ形にしました。
fluent側で受けるtagを分けるために、Outputを追加。
どうも重複して利用がダメそうで、結局流れでInput/Processor/Output/Routeをすべて追加。
こんなんで良いのかな…。

 define ROOT C:\Program Files (x86)\nxlog
 Moduledir %ROOT%\modules
 CacheDir %ROOT%\data
 Pidfile %ROOT%\data\nxlog.pid
 SpoolDir %ROOT%\data
 LogFile %ROOT%\data\nxlog.log

 <Extension syslog>
   Module      xm_syslog
 </Extension>

 <Extension json>
   Module      xm_json
 </Extension>

 <Input in>
   Module im_file
   File "D:\work\winfluent\srclog\log.txt"
   SavePos TRUE
   InputType LineBased
 </Input>

 <Input ev>
   Module      im_msvistalog
#   Exec        $Message = to_json(); to_syslog_bsd();
   SavePos TRUE
   ReadFromLast TRUE
 </Input>

 <Processor t>
   Module pm_transformer
   OutputFormat syslog_bsd
   Exec $Message=(": "+$raw_event);
 </Processor>

 <Processor t_ev>
   Module pm_transformer
   OutputFormat syslog_bsd
   Exec $Message=($raw_event);
 </Processor>

 <Output out>
   Module om_udp
   Host xxx.xxx.xxx.xxx
   Port 55514
 </Output>

 <Output out_ev>
   Module om_udp
   Host xxx.xxx.xxx.xxx
   Port 55515
 </Output>

 <Route r>
   Path in => t => out
 </Route>
 
 <Route r_ev>
   Path ev => t_ev => out_ev
 </Route>
追加プラグイン
/usr/lib64/fluent/ruby/bin/gem install fluent-plugin-numeric-monitor
/usr/lib64/fluent/ruby/bin/gem install fluent-plugin-growthforecast
入れてるけどまだ使ってないプラグイン
/usr/lib64/fluent/ruby/bin/gem install fluent-plugin-file-alternative
/usr/lib64/fluent/ruby/bin/gem install fluent-plugin-datacounter
/usr/lib64/fluent/ruby/bin/gem install fluent-plugin-redeliver
td-agent.conf

im_fileとim_msvistalogでtagを変えたかったため、受けポートを増やして対応。
2回目のrewrite_tag_filterで、growthforecastに投げる際に「tag_for section」するとタグにPIDが入っているのが却って邪魔になったのでタグからPIDは外した。notifier のタグにPIDが入るのとメリット・デメリット差し引きでどちらかを取るか…という感じではある。

<source>
  type syslog
  protocol_type udp
  port 55514
  tag  winps
</source>

<source>
  type syslog
  protocol_type udp
  port 55515
  tag  winev
</source>

##################################################################
<match winev.**>
  type copy
  <store>
    type file
    path /var/log/td-agent/arch/winev
    time_slice_format  %Y%m%d
    buffer_type file
    buffer_path /var/log/td-agent/buffer/winev/
    buffer_chunk_limit 100m
    flush_interval 5s
  </store>

</match>


##################################################################
<match winps.**>
  type copy
  <store>
    type file
    path /var/log/td-agent/arch/winps
    buffer_type file
    buffer_path /var/log/td-agent/buffer/winps/
    buffer_chunk_limit 100m
    flush_interval 5s
  </store>

  <store>
    type filter
    all deny
    allow message: /firefox/, message: /Idle/
  </store>

</match>

<match filtered.**>
  type rewrite_tag_filter
  rewriterule1  host  ^(.+)$  filterrewrited.$1.winps
  remove_tag_prefix filtered
</match>
<match filterrewrited.**>
  type parser
  remove_prefix filterrewrited
  add_prefix winproc
  format /^(?<Name>[^ ]* +\d+) +(?<Cpu>\d+) +(?<Thd>\d+) +(?<Hnd>\d+) +(?<Priv>\d+) +(?<CpuTime>.+) +(?<ElapsTime>.+)$/
  key_name message
  suppress_parse_error_log true
</match>

<match winproc.**>
  type rewrite_tag_filter
  rewriterule1  Name  ^([^ ]*) +(\d+)$  $1.${tag}
  remove_tag_prefix winproc
</match>
<match firefox.**>
  type copy
  <store>
    type notifier
    <def>
      pattern     firefox
      check       numeric_upward
      warn_threshold 5
      crit_threshold 10
      target_keys Cpu
    </def>
  </store>
  <store>
    type numeric_monitor
    count_interval 60
    aggregate tag
    output_per_tag yes
    tag_prefix monitor
    monitor_key Cpu
    output_key_prefix cpu_stat
    percentiles 50
  </store>
</match>
<match Idle.**>
  type copy
  <store>
    type notifier
    <def>
      pattern     Idle
      check       numeric_downward
      warn_threshold 95
      crit_threshold 80
      target_keys Cpu
    </def>
  </store>
  <store>
    type numeric_monitor
    count_interval 60
    aggregate tag
    output_per_tag yes
    tag_prefix monitor
    monitor_key Cpu
    output_key_prefix cpu_stat
    percentiles 50
  </store>
</match>

##################################################################
<match notification.**>
  type stdout
#  type     mail
#  host     localhost
#  port     25
#  from     FROM
#  to       TO
#  subject  fluentd notification
#  out_keys pattern,target_tag,target_key,level,value,message_time
</match>

#<match monitor.**>
#  type stdout
#</match>
<match monitor.**>
  type copy
  <store>
    type stdout
  </store>
  <store>
    type growthforecast
    remove_prefix monitor
    gfapi_url http://localhost:5125/api/
    service cpustat
    tag_for section
    name_keys cpu_stat_max,cpu_stat_min,cpu_stat_avg,cpu_stat_percentile_50
  </store>
</match>
その他

fluent-plugin-file-alternative でイベントログをファイル保存しようとすると文字コード変換部でエラー。
ちょっと追いきれてないので、とりあえずメモだけ。out_file であれば問題なくファイルに保存できました。

2014-05-07 14:10:07 +0900 [warn]: emit transaction failed  error_class=Encoding::UndefinedConversionError error=#<Encoding::UndefinedConversionError: "\xE3" from ASCII-8BIT to UTF-8>
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-mixin-plaintextformatter-0.2.6/lib/fluent/mixin/plaintextformatter.rb:85:in `encode'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-mixin-plaintextformatter-0.2.6/lib/fluent/mixin/plaintextformatter.rb:85:in `to_json'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-mixin-plaintextformatter-0.2.6/lib/fluent/mixin/plaintextformatter.rb:85:in `stringify_record'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-mixin-plaintextformatter-0.2.6/lib/fluent/mixin/plaintextformatter.rb:115:in `format'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/output.rb:527:in `block in emit'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/event.rb:54:in `call'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/event.rb:54:in `each'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/output.rb:518:in `emit'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/match.rb:36:in `emit'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/engine.rb:152:in `emit_stream'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/engine.rb:132:in `emit'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin/in_syslog.rb:199:in `emit'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin/in_syslog.rb:173:in `receive_data'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin/in_syslog.rb:245:in `call'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin/in_syslog.rb:245:in `on_read'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/io.rb:108:in `on_readable'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/io.rb:170:in `on_readable'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run_once'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run'
  2014-05-07 14:10:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin/in_syslog.rb:118:in `run'
2014-05-07 14:10:07 +0900 [error]: syslog failed to emit error="\"\\xE3\" from ASCII-8BIT to UTF-8" error_class="Encoding::UndefinedConversionError" tag="winev.user.info" record="{\"host\":\"desktop-PC\",\"ident\":\"Service_Control_Manager\",\"pid\":\"672\",\"message\":\": 2014-05-06 20:59:12 nsr-PC INFO 7036 WWAN AutoConfig \u30B5\u30FC\u30D3\u30B9\u306F \u5B9F\u884C\u4E2D \u72B6\u614B\u306B\u79FB\u884C\u3057\u307E\u3057\u305F\u3002\\r\"}"