엘라스틱 노드 에러(bootstrap.memory_lock)

JVM 힙 메모리의 디스크 스왑은 노드 성능 및 안정성을 해친다고 한다. 그래서 엘라스틱은 스왑 방지 옵션을 제공함.

설정 변경 후, 향상된 성능을 기대하며 재시작. 하지만 실패.

[2021-01-16T15:30:18,018][WARN ][o.e.b.JNANatives         ] [Centos7] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2021-01-16T15:30:18,021][WARN ][o.e.b.JNANatives         ] [Centos7] This can result in part of the JVM being swapped out.
[2021-01-16T15:30:18,021][WARN ][o.e.b.JNANatives         ] [Centos7] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2021-01-16T15:30:18,022][WARN ][o.e.b.JNANatives         ] [Centos7] These can be adjusted by modifying /etc/security/limits.conf, for example:
  # allow user 'elasticsearch' mlockall
  elasticsearch soft memlock unlimited
  elasticsearch hard memlock unlimited
[2021-01-16T15:30:18,022][WARN ][o.e.b.JNANatives         ] [Centos7] If you are logged in interactively, you will have to re-login for the new limits to take effect.
                 .                 .                 .            
                 .                 .                 .      
                 .                 .                 .                  
[2021-01-16T15:30:27,246][ERROR][o.e.b.Bootstrap          ] [Centos7] node validation exception
[1] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked

해당 옵션을 사용하고 싶으면 limits.conf를 수정하라는 경고가 보인다. 시키는대로 수정.

하지만 여전히 안 된다. 매뉴얼을 자세히 보니 해당 경고는 압축 파일 버전으로 설치했을 때 필요한 설정.

Configuring system settings

Where to configure systems settings depends on which package you have used to install Elasticsearch, and which operating system you are using.

When using the .zip or .tar.gz packages, system settings can be configured:

• temporarily with ulimit, or
• permanently in /etc/security/limits.conf.

When using the RPM or Debian packages, most system settings are set in the system configuration file. However, systems which use systemd require that system limits are specified in a systemd configuration file.

RPM으로 설치했을 때는 다른 설정 변경이 필요한데, 이것도 서비스 관리를 init이 하느냐, systemd가 하느냐에 따라 다르다. 

Systemd configuration

When using the RPM or Debian packages on systems that use systemd, system limits must be specified via systemd.

The systemd service file (/usr/lib/systemd/system/elasticsearch.service) contains the limits that are applied by default.

To override them, add a file called /etc/systemd/system/elasticsearch.service.d/override.conf (alternatively, you may run sudo systemctl edit elasticsearch which opens the file automatically inside your default editor). Set any changes in this file, such as:

[Service]
LimitMEMLOCK=infinity

Once finished, run the following command to reload units:

sudo systemctl daemon-reload

centos는 v7부터 systemd를 사용하니 elasticsearch.service를 수정해주면 됨.

관련 글

• 엘라스틱 노드 에러(low disk watermark)
• 엘라스틱 노드 에러(node.max_local_storage_nodes)