{"id":706,"date":"2020-02-10T14:22:26","date_gmt":"2020-02-10T19:22:26","guid":{"rendered":"https:\/\/crossan007.dev\/blog\/?p=706"},"modified":"2020-02-10T14:24:34","modified_gmt":"2020-02-10T19:24:34","slug":"recovering-from-full-elasticsearch-nodes","status":"publish","type":"post","link":"https:\/\/crossan007.dev\/blog\/monitoring\/recovering-from-full-elasticsearch-nodes\/","title":{"rendered":"Recovering from full Elasticsearch nodes"},"content":{"rendered":"\n
Tweet<\/a><\/div>\n\n

Recently I ran out of space on a 5 node Elasticsearch cluster. Events were not being indexed, and Logstash had amassed a 10GB disk-backed queue. It was not pretty<\/p>\n\n\n\n

I discovered that the fifth node was configured incorrectly and was storing the ES data on one of the smaller disk partitions. I stopped the Elasticsearch service on this node while I formulated a plan.<\/p>\n\n\n\n

Unfortunately, I didn’t have the time (or confidence) to move the entire \/var<\/code> directory to the large partition (which happened to be serving the \/home<\/code> folder: mounted as \/dev\/mapper\/centos-home<\/code>), so I instead created a new folder at \/home\/elasticsearch<\/code> (so it would be on the large partition), and “symlinked”\/var\/elasticsearch<\/code> to the new home folder on the larger partition ln -s \/home\/elasticsearch\/elasticsearch \/var\/lib\/elasticsearch<\/code><\/p>\n\n\n\n

After creating the Symlink, I started the Elasticsearch service, and watched the logs. After some time, I noticed that there were still no primary shards assigned to this new nodes (despite it being the only node with disk space utilization below the threshold), so I dug in a bit more<\/p>\n\n\n\n

This is where I learned about \/_cluster\/allocation\/explain <\/code> which provides details about why<\/strong> certain shards may have an allocation problem<\/a>. Ah ha! After 5 failed attempts to unassigned shards to my new node, Elasticsearch just needed a little kick to re run the allocation process: I opened up the Kibana console, and ran POST \/_cluster\/reroute?retry_failed=true<\/code> to force the algorithm to re-evaluate the location of shards<\/a><\/p>\n\n\n\n

Within about 90 seconds, the Elasticsearch cluster began rerouting all of the unassigned shards, and my logstash disk-queue began to shrink as the events poured into the freshly allocated shards on my new node. <\/p>\n\n\n\n

Problem solved. <\/p>\n\n\n\n

Stay tuned for next week when I pay off the technical debt incurred by placing my Elasticsearch shards on a symlink \ud83d\ude2c<\/p>\n","protected":false},"excerpt":{"rendered":"

Recently I ran out of space on a 5 node Elasticsearch cluster. Events were not being indexed, and Logstash had amassed a 10GB disk-backed queue. It was not pretty I discovered that the fifth node was configured incorrectly and was storing the ES data on one of the smaller disk partitions. I stopped the Elasticsearch … Continue reading Recovering from full Elasticsearch nodes<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[91],"tags":[240,244,210,25,243],"yoast_head":"\n\n\n\n\n\n\n\n\n\n\n\n\t\n\t\n\t\n