Logstash input 플러그인 jdbc 버그?

강의 중에 있었던 일. 6.8.3 버전으로 mysql 연동을 진행하는데 jdbc 드라이버를 찾지 못한다. 이전 강의에서는 정상적으로 동작했는데? 일단 stackoverflow 형님들이 알려준대로 드라이버를 'logstash-core/lib/jars' 경로에 복사해서 해결.

집에 와서 같은 버전, 같은 설정으로 테스트해봤다. 잘 됨.-_-

D:\ELK\logstash-6.8.3\bin> .\logstash -f D:\test.conf
Sending Logstash logs to D:/ELK/logstash-6.8.3/logs which is now configured via log4j2.properties
[2019-12-22T13:05:57,287][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"D:/ELK/logstash-6.8.3/data/queue"}
[2019-12-22T13:05:57,305][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"D:/ELK/logstash-6.8.3/data/dead_letter_queue"}
[2019-12-22T13:05:57,411][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-12-22T13:05:57,425][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.3"}
[2019-12-22T13:05:57,457][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"3d869cdf-ad6f-4e56-8009-e188582eddf0", :path=>"D:/ELK/logstash-6.8.3/data/uuid"}
[2019-12-22T13:06:05,028][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-12-22T13:06:05,296][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"}
[2019-12-22T13:06:05,360][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-22T13:06:05,816][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
D:/ELK/logstash-6.8.3/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated
[2019-12-22T13:07:01,693][INFO ][logstash.inputs.jdbc     ] (0.005986s) SELECT version()
[2019-12-22T13:07:01,729][INFO ][logstash.inputs.jdbc     ] (0.003593s) SELECT a.timestamp, b.sig_name from event a, signature b where a.signature = b.sig_id and a.timestamp > '1970-01-01 00:00:00'
D:/ELK/logstash-6.8.3/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
      "sig_name" => "icmp-request",
    "@timestamp" => 2019-12-21T11:07:01.774Z,
      "@version" => "1",
     "timestamp" => 2018-10-10T10:48:01.000Z
}

jdbc_driver_library 경로를 일부러 틀리게 주면 에러가 재현된다. 뭐지?

input {
  jdbc {
    jdbc_driver_library => "D:/WRONG_PATH/mysql-connector-java-5.1.42-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://192.168.56.100:3306/snort"
    jdbc_user => "root"
    jdbc_password => "password"
    schedule => "* * * * *"
    statement => "
SELECT a.timestamp, b.sig_name 
from event a, signature b 
where a.signature = b.sig_id 
and a.timestamp > :sql_last_value
"
  }
}

filter {

}

output {
  stdout {}
}

다음은 에러 메시지.

D:\ELK\logstash-6.8.3\bin> .\logstash -f D:\test.conf
Sending Logstash logs to D:/ELK/logstash-6.8.3/logs which is now configured via log4j2.properties
[2019-12-22T13:16:32,282][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-12-22T13:16:32,306][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.3"}
[2019-12-22T13:16:39,541][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-12-22T13:16:39,786][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"}
[2019-12-22T13:16:39,857][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-22T13:16:40,280][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
D:/ELK/logstash-6.8.3/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated
{ 2012 rufus-scheduler intercepted an error:
  2012   job:
  2012     Rufus::Scheduler::CronJob "* * * * *" {}
  2012   error:
  2012     2012
  2012     LogStash::ConfigurationError
  2012     com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?

이거 말고도 CentOS7 환경에서 7.5.0 버전은 로그스태시 서비스 기동 시 pipelines.yml에 지정된 파이프라인 설정을 못읽어오더라. 근데 또 집에서는 잘 됨(..) 강의장에 마가 꼈나? 지우고 다시 깔아볼 걸 그랬나?

관련 글

• Logstash input jdbc error
• mysql-connector-java 8 버그
• Elasticsearch 활용(Snort 데이터베이스 연동)