./diagnostics.sh --host 2775abprd8230d55d11e5edc86752260dd.us-east-1.aws.found.io -u elastic -p --port 9243 --ssl --type api --noVerify

Customizing What Is Collected

The Config Directory

All configuration used by the utility is located the config /config under the folder created when the diagnostic utility was unzipped. These can be modified to change some behaviors in the diagnostic utility. The *-rest.yml files all contain queries that executed against the cluster being diagnosed. They are versioned, the the Elasticsearch calls have additional modifiers that can be used to further customize the retrievals. The diags.yml file has generalized configuration information, and scrub.yml can be used to drive the sanitization function.

Removing Or Modifying Calls

To prevent a call from being executed or modify the results via the syntax, simple comment out, remove or change the entry. You can also add a completely different entry. Make sure that the key you use for that call does not overlap with another one already used. The file name of the output that will be packaged in the diag will be derived from that key.

Preventing Retries

At times you may want to compress the time frames for a diagnostic run and do not want multiple retry attempts if the first one fails. These will only be executed if a REST call within the configuration file has a retry: true parameter in its configuration. If this setting exists simply comment it out or set it to false to disable the retry.

Executing Scripted Runs

Executing the diagnostic via a script passing in all parameters at a time but passwords must currently be sent in via plain text so it is not recommended unless you have the proper security mechanisms in place to safeguard your credentials. The parameters:
--passwordText, --pkiPassText, --proxyPassText, --pkiPassText, --remotePassText, and --keyFilePassText can be used instead of their switch parameter equivalents to send in a value rather than prompt for a masked password. These are not displayed via the help or on the command line options table because we do not encourage their use unless you absolutely need to have this functionality.


Elasticsearch Deployed In Docker Containers

During execution, the diagnostic will attempt to determine whether any of the nodes in the cluster are running within Docker containers, particularly the node targeted via the host name. If one or more nodes on that targeted host are running in Docker containers, an additional set of Docker specific diagnostics such as inspect, top, and info, as well as obtaining the logs. This will be done for every discovered container on the host(not just ones containing Elasticsearch). In addition, when it is possible to determine if the calls are valid, the utility will also attempt to make the usual system calls to the host OS running the containers.

If errors occur when attempting to obtain diagnostics from Elasticsearch nodes or Logstash processes running within Docker containers, consider running with the --type api, logstash-api to verify that the configuration is not causing issues with the system call or log extraction modules in the diagnostic. This should allow the REST API subset to be successfully collected.

Running From A Docker Container

When the diagnostic is deployed within a Docker container it will recognize the enclosing environment and disable the types local and local-logstash. These modes of operation require the diagnostic to verify that it is running on the same host as the process it is investigating because of the ways in which system calls and file operations are handled. Docker containers muddy the waters, so to speak, in this case making this difficult if not impossible. So for the sake of reliability, once the diagnostic is deployed within Docker it will always function as if it were a remote component. The only options available will be remote, logstash-remote, remote, and api.

There are a number of options for interacting with applications running within Docker containers. The easiest way to run the diagnostic is simply to perform a docker run -it which opens a pseudo TTY. At that point you can interface with the diagnostic in the same way as you would when it was directly installed on the host. If you look in the /docker directory in the diagnostic distribution you will find a sample script named diagnostic-container-exec.sh that contains an example of how to do this.

   docker run -it -v ~/docker-diagnostic-output:/diagnostic-output  support-diagnostics-app  bash

For the diagnostic to work seamlessly from within a container, there must be a consistent location where files can be written. The default location when the diagnostic detects that it is deployed in Docker will be a volume named diagnostic-output. If you examine the above script you will notice that it mounts that volume to a local directory on the host where the diagnostic loaded Docker container resides. In this case it is a folder named docker-diagnostic-output in the home directory of the user account running the script. Temp files and the eventual diagnostic archive will be written to this location. You may change the volume if you adjust the explicit output directory whenever you run the diagnostic, but given that you are mapping the volume to local storage that creates a possible failure point. Therefore it's recommended you leave the diagnostic-output volume name as is and simply adjust the local mapping.

File Sanitization Utility

Overview - How Sanitization Works

In some cases the information collected by the diagnostic may have content that cannot be viewed by those outside the organization. IP addresses and host names, for instance. The diagnostic contains functionality that allows one to replace this content with values they choose contained in a configuration file. It will process a diagnostic archive file by file, replacing the entries in the config with a configured substitute value.

It is run via a separate execution script, and can process any valid Elasticsearch cluster diagnostic archive produced by Support Diagnostics 6.4 or greater. It can also process a single file. It does not need to be run on the same host that produced the diagnostic. Or by the same version number that produced the archive as long as it is a supported version. Logstash diagnostics are not supported at this time, although you may process those using the single file by file functionality for each entry.

It will go through each file line by line checking the content. If you are only concerned about IP addresses, you do not have to configure anything. The utility will automatically obfuscate all node id's node names, IPv4, IPv6 and MAC addresses. It is important to note this because as it does this, it will generate a new random IP value and cache it to use every time it encounters that same IP later on. So that the same obfuscated value will be consistent across diagnostic files. This ensures that you can differentiate between occurrences of discrete nodes in the cluster. If you replace all the IP addresses with a global XXX.XXX.XXX.XXX mask you will lose the ability to see which node did what.

After it has checked for IP and MAC addresses it will use any configured tokens. If you include a configuration file of supplied string tokens, any occurrence of that token will be replaced with a generated replacement. As with IP's this will be consistent from file to file but not between runs. It supports explicit string literal replacement or regexes that match a broader set of criteria. An example configuration file (scrub.yml) is included in the root installation directory as an example for creating your own tokens.

Running The Santizer