Filebeat의 multi path

filebeat 설정.

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - d:\firewall.log
  prospector.scanner.fingerprint.enabled: false

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

output.console:
  pretty: true

processors:
  - include_fields:
      fields: ["log.file.path", "message"]

{
  "@timestamp": "2025-06-27T13:24:11.549Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.0"
  },
  "log": {
    "file": {
      "path": "d:\\firewall.log"
    }
  },
  "message": "2025-06-24 15:44:30 ALLOW UDP 192.168.31.177 142.250.207.100 64118 443 0 - - - - - - - SEND"
}

연동 데이터를 추가하면?

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - d:\firewall.log
    - d:\auditbeat.log

{
  "@timestamp": "2025-06-27T13:25:50.441Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.0"
  },
  "log": {
    "file": {
      "path": "d:\\auditbeat.log"
    }
  },
  "message": "{\"@timestamp\":\"2025-06-25T09:40:17.236Z\",\"@metadata\":{\"beat\":\"auditbeat\",\"type\":\"_doc\",\"version\":\"8.12.0\"},\"event\":{\"kind\":\"event\",\"category\":[\"process\"],\"type\":[\"end\"],\"action\":\"process_stopped\",\"module\":\"system\",\"dataset\":\"process\"},\"ecs\":{\"version\":\"8.0.0\"},\"host\":{\"name\":\"MHKANG\"},\"agent\":{\"version\":\"8.12.0\",\"ephemeral_id\":\"013c425b-7951-4670-ab30-7432708042cd\",\"id\":\"5d624028-d6db-4bf3-9930-2739c9b72407\",\"name\":\"MHKANG\",\"type\":\"auditbeat\"},\"process\":{\"parent\":{\"pid\":1636},\"hash\":{\"sha1\":\"119965512252f4da4ac86887562e0d4ad4f2d4c3\"},\"entity_id\":\"1vfinYF9dclwaAOM\",\"pid\":216868,\"executable\":\"C:\\\\Splunk\\\\bin\\\\python3.exe\",\"start\":\"2025-06-25T09:40:05.187Z\",\"name\":\"python3.exe\",\"args\":[\"C:\\\\Splunk\\\\bin\\\\Python3.exe\",\"C:\\\\Splunk\\\\bin\\\\runScript.py\",\"rest_vocabularies.Vocabularies\"],\"working_directory\":\"C:\\\\Splunk\\\\bin\"},\"message\":\"Process python3.exe (PID: 216868) by user NT AUTHORITY\\\\SYSTEM STOPPED\",\"user\":{\"group\":{\"id\":\"S-1-5-18\"},\"name\":\"NT AUTHORITY\\\\SYSTEM\",\"id\":\"S-1-5-18\"},\"service\":{\"type\":\"system\"}}"
}

추가된 데이터만 가져온다. 항상 그렇지는 않은데 왔다갔다 하면서 하나만 가져온다. 공식문서는 여러 개 지정할 수 있는 것처럼 설명하는데 버그인가? 다음 설정도 마찬가지.

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - d:\*.log

filebeat 시작 후 업데이트된 내용이 없어서라고 하기엔 일단 하나는 가져오니까(..) 물론 filestream input 선언을 따로 해주면 가져오긴 한다.

filebeat.inputs:
- type: filestream
  id: my-filestream-id1
  enabled: true
  paths:
    - d:\firewall.log
  prospector.scanner.fingerprint.enabled: false

- type: filestream
  id: my-filestream-id2
  enabled: true
  paths:
    - d:\auditbeat.log
  prospector.scanner.fingerprint.enabled: false  

{
  "@timestamp": "2025-06-27T13:30:19.791Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.0"
  },
  "log": {
    "file": {
      "path": "d:\\firewall.log"
    }
  },
  "message": "2025-06-24 15:44:30 ALLOW UDP 192.168.31.177 142.250.207.100 64118 443 0 - - - - - - - SEND"
}
{
  "@timestamp": "2025-06-27T13:30:19.791Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.0"
  },
  "log": {
    "file": {
      "path": "d:\\auditbeat.log"
    }
  },
  "message": "{\"@timestamp\":\"2025-06-25T09:40:17.236Z\",\"@metadata\":{\"beat\":\"auditbeat\",\"type\":\"_doc\",\"version\":\"8.12.0\"},\"event\":{\"kind\":\"event\",\"category\":[\"process\"],\"type\":[\"end\"],\"action\":\"process_stopped\",\"module\":\"system\",\"dataset\":\"process\"},\"ecs\":{\"version\":\"8.0.0\"},\"host\":{\"name\":\"MHKANG\"},\"agent\":{\"version\":\"8.12.0\",\"ephemeral_id\":\"013c425b-7951-4670-ab30-7432708042cd\",\"id\":\"5d624028-d6db-4bf3-9930-2739c9b72407\",\"name\":\"MHKANG\",\"type\":\"auditbeat\"},\"process\":{\"parent\":{\"pid\":1636},\"hash\":{\"sha1\":\"119965512252f4da4ac86887562e0d4ad4f2d4c3\"},\"entity_id\":\"1vfinYF9dclwaAOM\",\"pid\":216868,\"executable\":\"C:\\\\Splunk\\\\bin\\\\python3.exe\",\"start\":\"2025-06-25T09:40:05.187Z\",\"name\":\"python3.exe\",\"args\":[\"C:\\\\Splunk\\\\bin\\\\Python3.exe\",\"C:\\\\Splunk\\\\bin\\\\runScript.py\",\"rest_vocabularies.Vocabularies\"],\"working_directory\":\"C:\\\\Splunk\\\\bin\"},\"message\":\"Process python3.exe (PID: 216868) by user NT AUTHORITY\\\\SYSTEM STOPPED\",\"user\":{\"group\":{\"id\":\"S-1-5-18\"},\"name\":\"NT AUTHORITY\\\\SYSTEM\",\"id\":\"S-1-5-18\"},\"service\":{\"type\":\"system\"}}"
}

인덱스를 따로 저장하고 싶으면 index 옵션 추가.

filebeat.inputs:
- type: filestream
  id: my-filestream-id1
  enabled: true
  paths:
    - d:\firewall.log
  prospector.scanner.fingerprint.enabled: false
  index: "firewall-%{+yyyy}"

- type: filestream
  id: my-filestream-id2
  enabled: true
  paths:
    - d:\auditbeat.log
  prospector.scanner.fingerprint.enabled: false
  index: "auditlog-%{+yyyy}" 

커스텀 인덱스 만드는 거 쉬웠구나. 공식 문서를 꼼꼼히 읽지 않으면 언제나 조져지는 건 나(..)

관련 글

• Filebeat 9.0의 변화
• Winlogbeat 8.12의 변화