Elastic Shell 101 - Advanced Features

What is it

Elastic Shell is an open source project that includes a set of command line utilities and run as a client to help you manage Elasticsearch. It is entirely written in Bash Shell.

“Elastic Shell 101” is a series of posts that tells you how to use it.

In this post, I will use upgrade command as an example to demonstrate some advanced features of Elastic Shell such as auto completion, logging, dry run, dependency check.

Auto completion

To remember all the Elastic Shell commands, sub-commands, options, and arguments may not be very easy. There are a couple of ways that can help.

You can run command with --help option to read the help information. Or, you don’t have to input all the command arguments because Elastic Shell can tell you what are the next available sub commands or arguments that you can use based on the context.

Other than that, you can use auto completion. For example, if we type elash, then double tabs, it will give you a suggestion list for the available commands that you can use.

bash-4.4# elash 
index     reindex   snapshot  upgrade   

If you type elash up, then it will auto-complete that for you because only the upgrade command is started from up. Continue to type double tabs, it will continue to give you the suggestions.

bash-4.4# elash upgrade 
cat          rolling      config       --ui-dialog  --help       
full         report       --ui-text    --dry-run    

With all these different ways, you can learn how to run the Elastic Shell commands by yourself!

Logging

Elastic Shell prints logs to the console when you run its commands. It also supports logging to file. This leverages on syslogd and is disabled by default.

To enable it, if you are using the Elastic Shell Docker image, just simply start syslogd. Then all set. The log file called elash.log can be found at /var/log.

Let’s run a full upgrade and see what it logs to the file. The Elastic Shell upgrade command has a sub-command called report. It is used to report the current upgrade status of the Elasticsearch cluster. In our case, we can see there are three nodes in the cluster. The current Elasticsearch version is 5.5.1. We are going to upgrade them to the target Elasticsearch version 6.3.2.

bash-4.4# elash upgrade report 

Report of Upgrades
------------------------------------------------------------------------------
                          Host       From    Current         To   Upgraded
------------------------------------------------------------------------------
   http://elasticsearch-1:9200      5.5.1      5.5.1      6.3.2         no
   http://elasticsearch-2:9200      5.5.1      5.5.1      6.3.2         no
   http://elasticsearch-3:9200      5.5.1      5.5.1      6.3.2         no
------------------------------------------------------------------------------

Commands/Options:
  cat
  full
  rolling
  report
  config

The source version, target version, and the hostnames of the cluster nodes are configured in main.properties file located at Elastic Shell config folder. They can also be overridden by environment variables.

Run upgrade

Let’s run upgrade full to launch a full cluster restart upgrade that requires a full cluster restart during the upgrade. You will see, Elastic Shell will disable shard allocation at first, then perform a synced flush which is configurable.

bash-4.4# elash upgrade full

Set Shard Allocation
------------------------------------------------------------------------------
{
  "acknowledged": true,
  "persistent": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "none"
        }
      }
    }
  },
  "transient": {}
}

Perform Synced Flush
------------------------------------------------------------------------------
{
  "_shards": {
    "total": 0,
    "successful": 0,
    "failed": 0
  }
}

Then, it will wait for the cluster to be stopped. During the time, let’s stop the cluster and start a new one which have the Elasticsearch with target version installed. After it’s started, let’s go back to the terminal window that is running Elastic Shell.

Progress
------------------------------------------------------------------------------
Thu Feb 21 06:50:20 UTC 2019 waiting for the cluster to be stopped......[done]
Thu Feb 21 06:50:25 UTC 2019 waiting for the cluster to be started...................[done]
Thu Feb 21 06:50:45 UTC 2019 the cluster has been restarted

Set Shard Allocation
------------------------------------------------------------------------------
{
  "acknowledged": true,
  "persistent": {},
  "transient": {}
}

Progress
------------------------------------------------------------------------------
Thu Feb 21 06:50:46 UTC 2019 waiting for the cluster to be green.[done]
Thu Feb 21 06:50:46 UTC 2019 the cluster has become green

You will see the cluster restart has been detected by Elastic Shell, and after the new cluster is fully started, Elastic Shell will re-enable the shard allocation, then wait for the cluster healthiness becomes green.

Now, let’s go to check the log file. It shows what command we choose, and all steps that Elastic Shell performed with every single request it sends.

bash-4.4# cat /var/log/elash.log 
Feb 21 06:59:18 ab097271ac11 local7.info root: ##################################################
Feb 21 06:59:18 ab097271ac11 local7.info root: # elash started at Thu Feb 21 06:59:18 UTC 2019
Feb 21 06:59:18 ab097271ac11 local7.info root: ##################################################
Feb 21 06:59:18 ab097271ac11 local7.info root: Current arg: full, available options: cat full rolling report config
Feb 21 06:59:18 ab097271ac11 local7.info root: User choice: choice=full
Feb 21 06:59:18 ab097271ac11 local7.info root: curl --show-error -X GET http://elasticsearch-1:9200/ 2>&1 --silent --header Content-Type: application/json
Feb 21 06:59:19 ab097271ac11 local7.info root: curl --show-error -X PUT http://elasticsearch-1:9200/_cluster/settings 2>&1 --silent --data {
...

Dry run

It is important to note that some operations such as upgrade, reindex, delete index are dangerous. To avoid mistake, you can run Elastic Shell in dry run mode before run it in production environment. In order to do so, you just need to add --dry-run option.

For example, let’s try to delete github index in dry run mode. By default, you will see the returned response is mocked.

bash-4.4# elash index --dry-run delete github

Delete github
------------------------------------------------------------------------------
(dry run...)

Commands/Options:
  companydatabase
  github
  ...

This is true because Elastic Shell doesn’t really touch the target server. All HTTP calls made to Elasticsearch cluster are mocked and they do not go through the network traffic. After dry run is finished, usually, you can open the log file to check the curl command details being involved and see if that is what you expect. For example, to check if the hostname, port number, or the name of the index to be deleted is correct.

In some cases that are more complicated, such as upgrade, we cannot simply return the same mock response all the time. For example, our upgrade report command will actually send request to Elasticsearch cluster and read the cluster information to know the current version, so that can decide whether or not a cluster node has been upgraded. In such case, we can not simply return a fixed mock response.

Elastic Shell provides a configuration file called dryrun.properties. It’s in the Elastic Shell config folder. We can use that file to configure whatever expected result for every single request.

Before check dryrun.properties, let’s run upgrade report command at first, and check its logs.

Feb 21 07:15:01 ab097271ac11 local7.info root: ##################################################
Feb 21 07:15:01 ab097271ac11 local7.info root: # elash started at Thu Feb 21 07:15:01 UTC 2019
Feb 21 07:15:01 ab097271ac11 local7.info root: ##################################################
Feb 21 07:15:01 ab097271ac11 local7.info root: Current arg: report, available options: cat full rolling report config
Feb 21 07:15:01 ab097271ac11 local7.info root: User choice: choice=report
Feb 21 07:15:01 ab097271ac11 local7.info root: curl --show-error -X GET http://elasticsearch-1:9200/ 2>&1 --silent --header Content-Type: application/json
Feb 21 07:15:01 ab097271ac11 local7.info root: curl --show-error -X GET http://elasticsearch-2:9200/ 2>&1 --silent --header Content-Type: application/json
Feb 21 07:15:01 ab097271ac11 local7.info root: curl --show-error -X GET http://elasticsearch-3:9200/ 2>&1 --silent --header Content-Type: application/json
Feb 21 07:15:01 ab097271ac11 local7.info root: Report of Upgrades
Feb 21 07:15:01 ab097271ac11 local7.info root: ------------------------------------------------------------------------------
Feb 21 07:15:01 ab097271ac11 local7.info root:                           Host       From    Current         To   Upgraded
Feb 21 07:15:01 ab097271ac11 local7.info root: ------------------------------------------------------------------------------
Feb 21 07:15:01 ab097271ac11 local7.info root:    http://elasticsearch-1:9200      5.5.1      6.3.2      6.3.2        yes
Feb 21 07:15:01 ab097271ac11 local7.info root:    http://elasticsearch-2:9200      5.5.1      6.3.2      6.3.2        yes
Feb 21 07:15:01 ab097271ac11 local7.info root:    http://elasticsearch-3:9200      5.5.1      6.3.2      6.3.2        yes
Feb 21 07:15:01 ab097271ac11 local7.info root: ------------------------------------------------------------------------------
Feb 21 07:15:01 ab097271ac11 local7.info root: all nodes have been upgraded from 5.5.1 to 6.3.2
Feb 21 07:15:01 ab097271ac11 local7.info root: ##################################################
Feb 21 07:15:01 ab097271ac11 local7.info root: # elash stopped at Thu Feb 21 07:15:01 UTC 2019
Feb 21 07:15:01 ab097271ac11 local7.info root: ##################################################

You will see there are three HTTP calls involved in this command. Each one was sent to a different node to get server information which includes the Elasticsearch version. We can add the three calls into dryrun.properties, with the expected versions.

bash-4.4# cat elash/config/dryrun.properties 
# dry run upgrade report
http://elasticsearch-1:9200/           ={ "version": { "number": "5.5.1" }  }
http://elasticsearch-2:9200/           ={ "version": { "number": "5.6.12" }  }
http://elasticsearch-3:9200/           ={ "version": { "number": "6.3.2" }  }

As above, we force the first node to return 5.5.1, the second to return 5.6.12, and the third to return 6.3.2. Then rerun it in dry run mode.

bash-4.4# elash upgrade --dry-run report 

Report of Upgrades
------------------------------------------------------------------------------
                          Host       From    Current         To   Upgraded
------------------------------------------------------------------------------
   http://elasticsearch-1:9200      5.5.1      5.5.1      6.3.2         no
   http://elasticsearch-2:9200      5.5.1     5.6.12      6.3.2    partial
   http://elasticsearch-3:9200      5.5.1      6.3.2      6.3.2        yes
------------------------------------------------------------------------------

Now, you see, even we’ve upgraded all the cluster nodes, it still reported only the last node has been upgraded. This is because all version information is coming from our dryrun.properties file rather than the real Elasticsearch cluster.

Dependency check

There is one more thing to note. Elastic Shell has a few dependencies to make it work. For example:

• curl is used to send the HTTP request
• jq is a great tool used to parse or prettify JSON data
• dialog is used to display dialogs and menus when run in interactive mode

If you use the Elastic Shell Docker image, they are installed out of the box. If not, then you may need to install them by yourself.

Most of the dependencies are optional. Elastic Shell will check its depdencies when it’s launched. If they are not installed, there will be alternatives or feature restricted. For example:

• If curl is not installed, the tool will be launched in dry run mode.
• If jq is not installed, the JSON output will not be prettified and some features that require to parse JSON data may be disabled.
• If dialog is not there, the --ui-dialog option will not be available.

You will be prompted with these information as warnings when launch Elastic Shell. For example, in my case, when I run Elastic Shell on my local machine instead of the Docker container. I can see two warnings:

$ ./main.sh index cat
main.sh: (warn) dependency 'jq' not found, some features may not be available
main.sh: (warn) dependency 'dialog' not found, dialog mode disabled

Commands/Options:
  indices
  shards
  nodes
  ...

This is because I don’t have jq and dialog installed on my local machine.

Any question about this post or Elastic Shell, feel free to leave comments or drop email at morningspace@yahoo.com.