데이터 분석 수준을 높이려다 보면 하나의 데이터를 이렇게도 바꿔보고, 저렇게도 바꿔보는 과정을 거치게 되는데, 이때 보통은 원본을 보존하기 위해 복사본을 사용한다.
filter { mutate { copy => { "원본" => "복사본" } }}다음은 테스트 파이프라인과 실행 결과.
filter { mutate { remove_field => [ "@timestamp", "@version", "path", "host" ] copy => { "message" => "message_copy" } }}[2021-01-14T22:17:15,337][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}{ "message_copy" => "index.php?a=1&b=&c=3&B=4\r", "message" => "index.php?a=1&b=&c=3&B=4\r"}그런데 mutate의 copy 옵션은 한 번에 하나의 복사본만을 만들 수 있다. 다음과 같은 구문을 실행하면,
filter { mutate { remove_field => [ "@timestamp", "@version", "path", "host" ] copy => { "message" => "message_copy1" "message" => "message_copy2" } }}Duplicate keys found 에러와 함께 파이프라인 실행 실패.
[2021-01-14T22:19:07,462][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Duplicate keys found in your configuration: [\"message\"]\nAt line: 12, column 11 (byte 206)\nafter input {\r\n\tfile {\r\n\t\tpath => \"d:/test.log\"\r\n\t\tstart_position => \"beginning\"\r\n\t\tsincedb_path => \"nul\"\r\n\t}\r\n}\r\n\r\nfilter {\r\n\tmutate {\r\n\t\tremove_field => [ \"@timestamp\", \"@version\", \"path\", \"host\" ]\r\n\t\tcopy => {", :backtrace=>["D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/compiler/lscl.rb:200:in `validate!'", "D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/compiler/lscl.rb:218:in `expr'", "D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/compiler/lscl.rb:150:in `expr'", "org/jruby/RubyArray.java:2577:in `map'", "D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/compiler/lscl.rb:114:in `expr_attributes'", "D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/compiler/lscl.rb:92:in `expr'", "org/jruby/RubyArray.java:2577:in `map'", "D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/compiler/lscl.rb:85:in `expr'", "D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/compiler/lscl.rb:64:in `block in compile'", "org/jruby/RubyArray.java:1809:in `each'", "D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/compiler/lscl.rb:62:in `compile'", "D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/compiler.rb:36:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/pipeline_action/reload.rb:53:in `execute'", "D:/ELK/logstash-7.10.1/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}복사본을 2개 이상 만들려면 mutate 필터를 한 번 더 선언해야 한다.
filter { mutate { remove_field => [ "@timestamp", "@version", "path", "host" ] copy => { "message" => "message_copy1" } }
mutate { copy => { "message" => "message_copy2" } }}[2021-01-14T22:23:13,322][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}{ "message_copy2" => "index.php?a=1&b=&c=3&B=4\r", "message_copy1" => "index.php?a=1&b=&c=3&B=4\r", "message" => "index.php?a=1&b=&c=3&B=4\r"}이게 귀찮을 때 모든 필터 플러그인에서 사용할 수 있는 공통 옵션 중 add_field가 좋은 대안이 된다.
filter { mutate { remove_field => [ "@timestamp", "@version", "path", "host" ] add_field => { "message_copy1" => "%{message}" "message_copy2" => "%{message}" } }}[2021-01-14T22:29:12,362][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}{ "message_copy2" => "index.php?a=1&b=&c=3&B=4\r", "message_copy1" => "index.php?a=1&b=&c=3&B=4\r", "message" => "index.php?a=1&b=&c=3&B=4\r"}"SOURCE" => "TARGET" 구조인 copy와 달리 "TARGET" => "SOURCE" 구조라 좀 헷갈리지만 파이프라인을 좀 더 단순하게 만들 수 있다.
관련 글
댓글 없음:
댓글 쓰기