VIC Containers running Elasticsearch are unresponsive

Products

• VMware vSphere ESXi

Issue/Introduction

• Deploying elasticsearch container on VCH fails to respond.
• The output of docker logs is similar to:

# docker -H 10.43.18.11:2376 --tls logs <container-id>

[2017-09-08T16:16:29,334][INFO ][o.e.n.Node ] [elasticsearch] version[5.5.2], pid[222], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/4.4.67-1.ph1-esx/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b15]

[2017-09-08T16:16:29,334][INFO ][o.e.n.Node ] [elasticsearch] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]

### lines omitted for brevity ###

[2017-09-08T16:16:32,785][INFO ][o.e.n.Node ] [elasticsearch] initialized

[2017-09-08T16:16:32,785][INFO ][o.e.n.Node ] [elasticsearch] starting ...

[2017-09-08T16:16:32,931][INFO ][o.e.t.TransportService ] [elasticsearch] publish_address {172.16.0.5:9300}, bound_addresses {172.16.0.5:9300}

[2017-09-08T16:16:32,943][INFO ][o.e.b.BootstrapChecks ] [elasticsearch] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks

ERROR: [2] bootstrap checks failed

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

[2017-09-08T16:16:32,957][INFO ][o.e.n.Node ] [elasticsearch] stopping ...

Environment

VMware vSphere Integrated Containers 1.0.x

Cause

Elasticsearch in production mode enforces the bootstrap parameters. If these parameters are not meet the service will not start.

Resolution

This is a known issue affecting vSphere Integrated Containers 1.1.

Currently, there is no resolution.

Workaround:

To work around this issue, use one of these options:

• Run elasticsearch in development mode so the bootstrap checks are not enforced. For more information, see Elasticsearch Devlopment mode.
• When running the container, have the container wait in a loop until the max_map_count = 262144 before continuing to start elasticsearch application. Later, use docker exec to change the max_map_count as the root user. 

Example:

docker run -d --name mycontainer -m 8GB docker.elastic.co/elasticsearch/elasticsearch:5.5.2 sh -c 'until [ $(cat /proc/sys/vm/max_map_count) -eq 262144 ]; do echo .;sleep 1;done;elasticsearch'

docker exec mycontainer sh -c "echo 262144 > /proc/sys/vm/max_map_count"

Additional Information

https://github.com/vmware/vic/issues/5353