【Elasticsearch】NODE_LEFTが消えない

GCE上にESインスタンスを３つつくってNode3台のクラスター構成にしている

これがあるときストレージがいっぱいになったことで一つのNodeがNODE_LEFTのステータスになってしまった

$ curl -XGET host:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason

...
index-1   3 r UNASSIGNED NODE_LEFT
...

この復旧までにおこなったメモ

まずはシャードの自動割り当てを止める

curl -XPUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "transient": {
    "cluster.routing.allocation.enable": "none"
  }
}'

そして、レプリカシャード数を０に

curl -XPUT "localhost:9200/scored-vacation-rental-v9-vrbo/_settings" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index": {
      "number_of_replicas": 0
    }
  }
}'

その後、NODE_LEFTステータスになっているインスタンスに入って、ESを起動させてみたがエラーになって起動しなかった

$ sudo systemctl start elasticsearch

その時のログ

Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested:
 at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]];
        at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086)
        at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.
        at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
        at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:43)
Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.initial_master_nodes
 at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]
        at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204)
        at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198)
        at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
        at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109)
        ... 10 more
Exception in thread "main" 2022-07-10 15:57:43,551 main ERROR No Log4j 2 configuration file found. Usin
SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: JsonParseException[Duplica
 at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]];
        at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086)
        at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.
        at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
        at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.initial_master_nodes
 at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]
        at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204)
        at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198)
        at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
        at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109)
        ... 9 moreJul 10 15:57:35 es7-staylist-elasticsearch-8t1x systemd-entrypoint[21075]: Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested:
 at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]];
        at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086)
        at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.
        at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
        at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:43)
Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.initial_master_nodes
 at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]
        at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204)
        at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198)
        at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
        at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109)
        ... 10 more
Exception in thread "main" 2022-07-10 15:57:43,551 main ERROR No Log4j 2 configuration file found. Usin
SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: JsonParseException[Duplica
 at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]];
        at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086)
        at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.
        at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
        at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.initial_master_nodes
 at [Source: (sun.nio.ch.ChannelInputStream); line: 34, column: 29]
        at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204)
        at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198)
        at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
        at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109)
        ... 9 more

これはインスタンス作り直したほうが早そうだったので、GCEのインスタンスを停止し再起動したらうまくESも起動された

レプリカシャード数とシャードの自動割り当てを元に戻して復旧を完了した