재시작이 필요한 윈도우 업데이트를 할 때마다 꽤 잦은 빈도로 스플렁크 서비스가 올라오지 않는다. 수동으로 스타트해봐도,
C:\Splunk\bin> .\splunk.exe start
Splunk> 4TW
Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket access_log apache history iislog main secure_log summary winevent Done Checking filesystem compatibility... Done Checking conf files for problems... Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 2: display.visualizations.custom.wordcloud_app.wordcloud.useColors (value: true). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 3: display.visualizations.custom.wordcloud_app.wordcloud.alignmentMode (value: horizontal). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 4: display.visualizations.custom.wordcloud_app.wordcloud.backgroundColor (value: #fff). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 5: display.visualizations.custom.wordcloud_app.wordcloud.colorBy (value: colorMode). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 6: display.visualizations.custom.wordcloud_app.wordcloud.colorMode (value: categorical). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 7: display.visualizations.custom.wordcloud_app.wordcloud.numOfBins (value: 3). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 8: display.visualizations.custom.wordcloud_app.wordcloud.minColor (value: #f7bc38). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 9: display.visualizations.custom.wordcloud_app.wordcloud.maxColor (value: #d93f3c). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 10: display.visualizations.custom.wordcloud_app.wordcloud.splunkTastic (value: false). Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug' Done Checking default conf files for edits... Validating installed files against hashes from 'C:\Splunk\splunk-8.2.0-e053ef3c985f-windows-64-manifest'File 'C:\Splunk\etc/apps/python_upgrade_readiness_app/default/app.conf' changed.File 'C:\Splunk\etc/apps/python_upgrade_readiness_app/default/collections.conf' changed.File 'C:\Splunk\etc/apps/python_upgrade_readiness_app/default/inputs.conf' changed.File 'C:\Splunk\etc/apps/python_upgrade_readiness_app/default/restmap.conf' changed.File 'C:\Splunk\etc/apps/python_upgrade_readiness_app/default/web.conf' changed.Could not open 'C:\Splunk\etc/apps/splunk_essentials_8_2/default/app.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/alert_actions.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/app.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/authorize.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/collections.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/commands.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/inputs.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/props.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/restmap.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/securegateway.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/web.conf': Problems were found, please review your files and move customizations to localAll preliminary checks passed.
Starting splunk server daemon (splunkd)...
Splunkd: Starting (pid 12180)
Timed out waiting for splunkd to start.splunkd 데몬이 올라오지 않음. C:\Splunk\var\log\splunk\splunkd.log를 살펴 보니,
02-13-2022 13:58:59.550 +0900 INFO loader [0 MainThread] - win-service: Starting as a Windows service: will run various system checks first...02-13-2022 13:58:59.550 +0900 INFO loader [0 MainThread] - win-service: Splunk starting as a local administrator02-13-2022 13:58:59.550 +0900 INFO loader [0 MainThread] - Automatic migration of modular inputs02-13-2022 13:59:02.564 +0900 INFO loader [0 MainThread] - win-service: Command pre-flight-checks ran successfully.02-13-2022 13:59:03.445 +0900 INFO loader [0 MainThread] - win-service: Command check-xml-files ran successfully.02-13-2022 13:59:03.445 +0900 INFO ServerConfig [0 MainThread] - My GUID is F9F1BC6F-AFC1-4679-8218-49899735443002-13-2022 13:59:03.445 +0900 INFO ServerConfig [0 MainThread] - My server name is "MHKANG".02-13-2022 13:59:03.445 +0900 INFO ServerConfig [0 MainThread] - Found no site defined in server.conf02-13-2022 13:59:03.445 +0900 INFO ServerConfig [0 MainThread] - Found no hostname options in server.conf. Will attempt to use default for now.02-13-2022 13:59:03.445 +0900 INFO ServerConfig [0 MainThread] - Host name option is "".02-13-2022 13:59:03.445 +0900 INFO ServerConfig [0 MainThread] - My hostname is "MHKANG".02-13-2022 13:59:03.461 +0900 INFO ServerConfig [0 MainThread] - SSL session cache path enabled 0 session timeout on SSL server 300.00002-13-2022 13:59:03.461 +0900 INFO ServerConfig [0 MainThread] - Setting HTTP server compression state=on02-13-2022 13:59:03.461 +0900 INFO ServerConfig [0 MainThread] - Setting HTTP client compression state=1 (true)02-13-2022 13:59:03.461 +0900 INFO ServerConfig [0 MainThread] - Splunk is starting with EC-SSC disabled02-13-2022 13:59:03.461 +0900 FATAL HTTPServer [0 MainThread] - Could not bind to port 8089최종적으로 'Could not bind to port 8089' 메시지 발생. 8089는 스플렁크 서비스 관리 포트. 사용 가능한데 왜?
C:\Users\Administrator>netstat -an | findstr 8089
C:\Users\Administrator>8089에서 8090으로 변경.
[httpServer]disableDefaultPort = true[settings]mgmtHostPort = 127.0.0.1:8090설정 변경 후, 스플렁크 스타트.
C:\Splunk\bin> .\splunk.exe start
Splunk> 4TW
Checking prerequisites... Management port has been set disabled; the web UI cannot work. Checking http port [8000]: open Management port has been set disabled; cli support for this configuration is currently incomplete. Checking configuration... Done. Checking critical directories... Done Checking indexes... (skipping validation of index paths because not running as LocalSystem) Validated: _audit _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket access_log apache history iislog main secure_log summary winevent Done Checking filesystem compatibility... Done Checking conf files for problems... Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 2: display.visualizations.custom.wordcloud_app.wordcloud.useColors (value: true). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 3: display.visualizations.custom.wordcloud_app.wordcloud.alignmentMode (value: horizontal). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 4: display.visualizations.custom.wordcloud_app.wordcloud.backgroundColor (value: #fff). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 5: display.visualizations.custom.wordcloud_app.wordcloud.colorBy (value: colorMode). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 6: display.visualizations.custom.wordcloud_app.wordcloud.colorMode (value: categorical). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 7: display.visualizations.custom.wordcloud_app.wordcloud.numOfBins (value: 3). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 8: display.visualizations.custom.wordcloud_app.wordcloud.minColor (value: #f7bc38). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 9: display.visualizations.custom.wordcloud_app.wordcloud.maxColor (value: #d93f3c). Invalid key in stanza [default] in C:\Splunk\etc\apps\wordcloud_app\default\savedsearches.conf, line 10: display.visualizations.custom.wordcloud_app.wordcloud.splunkTastic (value: false). Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug' Done Checking default conf files for edits... Validating installed files against hashes from 'C:\Splunk\splunk-8.2.0-e053ef3c985f-windows-64-manifest'File 'C:\Splunk\etc/apps/python_upgrade_readiness_app/default/app.conf' changed.File 'C:\Splunk\etc/apps/python_upgrade_readiness_app/default/collections.conf' changed.File 'C:\Splunk\etc/apps/python_upgrade_readiness_app/default/inputs.conf' changed.File 'C:\Splunk\etc/apps/python_upgrade_readiness_app/default/restmap.conf' changed.File 'C:\Splunk\etc/apps/python_upgrade_readiness_app/default/web.conf' changed.Could not open 'C:\Splunk\etc/apps/splunk_essentials_8_2/default/app.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/alert_actions.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/app.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/authorize.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/collections.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/commands.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/inputs.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/props.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/restmap.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/securegateway.conf': Could not open 'C:\Splunk\etc/apps/splunk_secure_gateway/default/web.conf': Problems were found, please review your files and move customizations to localAll preliminary checks passed.
Starting splunk server daemon (splunkd)...
Splunkd: Starting (pid 12504)Done
Waiting for web server at http://127.0.0.1:8000 to be available.........................이번엔 웹이 감감 무소식. C:\Splunk\var\log\splunk\web_service.log를 보니,
2022-02-13 14:02:48,896 ERROR [62050cf4aa2a64c7fc1c8] __init__:522 - Socket error communicating with splunkd (error=[WinError 10061] ?€??而댄벂?곗뿉???곌껐??嫄곕??덉쑝誘€濡??곌껐?섏? 紐삵뻽?듬땲??, path = /services/server/infosplunkd가 웹과 통신하지 못한다. 스플렁크 웹은 8000 포트를 사용하는데, 해당 포트 바인딩이 안 되는 듯. 8089와 마찬가지로 포트 충돌 상황은 아닌데(..) web.conf의 'httpport = 8000' 설정을 8888로 변경해봤다.
C:\Users\Administrator>netstat -anb
활성 연결
프로토콜 로컬 주소 외부 주소 상태[splunkd.exe] TCP 127.0.0.1:5467 127.0.0.1:8888 SYN_SENTsplunkd가 연결을 시도하지만 웹이 응답하지 않음. 희한한 게 분명 splunkd는 실행이 됐다고 하는데 변경된 관리 포트 사용 흔적이 없다.
C:\Splunk\bin>splunk.exe statusSplunkd: Running (pid 14308)
C:\Splunk\bin>netstat -an | findstr 8090
C:\Splunk\bin>윈도우를 업데이트할 때마다 같은 문제가 자주 발생하는 걸 보면 아무래도 업데이트가 윈도우 안정성을 해치는 부분이 있는 것 같다. 엘라스틱도 종종 안 올라옴.
[2022-02-10T09:38:17,908][ERROR][o.e.b.Bootstrap ] [node-1] Exceptionorg.elasticsearch.transport.BindTransportException: Failed to bind to 127.0.0.1:[9300-9400]재밌는 건 서로 충돌하는 포트가 없음에도 엘라스틱이 실행되면 스플렁크가 실행되지 않고, 스플렁크가 실행되면 엘라스틱이 실행되지 않는다. 둘 다 먹통이 되는 경우는 없음. 윈도우 종료 후 재시작하면 대부분 해결되니 그나마 다행.
관련 글
댓글 없음:
댓글 쓰기