Elasticsearch Maven Plugin Build Status

A Maven 3.1+ plugin to run instances of Elasticsearch version 5+ during the integration test phase of a build. Instances are started in forked processes using the runforked goal. They are terminated using the stop goal and, for extra peace of mind, using a JVM shutdown hook.

Each instance is installed in ${project.build.directory}/elasticsearch${instanceIndex}.

For Elasticsearch version 1.x.x and 2.x.x support, see version 1.x and 2.x of this plugin.

Because the plugin uses the new Eclipse based Aether framework, it only works with Maven 3.1.0 and above. See this discussion for details.

Usage

The Elasticsearch behaviour and properties can be configured through the following plugin configuration parameters:

To use the plugin, include the following in your pom.xml file and modify the configuration as needed:

<plugin>
    <groupId>com.github.alexcojocaru</groupId>
    <artifactId>elasticsearch-maven-plugin</artifactId>
    <!--
        THE PLUGIN VERSION; FOR THE LIST OF AVAILABLE VERSIONS, SEE
        https://github.com/alexcojocaru/elasticsearch-maven-plugin/releases
    -->
    <version>6.13</version>
    <configuration>
        <!-- THE ELASTICSEARCH VERSION; REPLACE WITH THE VERSION YOU NEED -->
        <version>7.2.0</version>
        <clusterName>test</clusterName>
        <transportPort>9300</transportPort>
        <httpPort>9200</httpPort>
    </configuration>
    <executions>
        <!--
            The elasticsearch maven plugin goals are by default bound to the
            pre-integration-test and post-integration-test phases
        -->
        <execution>
            <id>start-elasticsearch</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>runforked</goal>
            </goals>
        </execution>
        <execution>
            <id>stop-elasticsearch</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>stop</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Environment variables

The environment variables to set before starting each Elasticsearch instance.

Environment variables may be referenced from the Elasticsearch configuration, but the main use case for defining environment variables is to set JAVA_HOME, so that Elasticsearch is run with a different JDK than the one used to run Maven.

The way to define environment variables is as follows:

<plugin>
    <groupId>com.github.alexcojocaru</groupId>
    <artifactId>elasticsearch-maven-plugin</artifactId>
    <version>6.0</version>
    <configuration>
        <clusterName>test</clusterName>
        <transportPort>9300</transportPort>
        <httpPort>9200</httpPort>
        ...
        <environmentVariables>
            <SOME_CUSTOM_VARIABLE>somevalue</SOME_CUSTOM_VARIABLE>
            <JAVA_HOME>${path-to-java-home-configured-through-maven-profiles}</JAVA_HOME>
        </environmentVariables>
        ...
    </configuration>
    <executions>
        ...
    </executions>
</plugin>

Plugins

A list of Elasticsearch plugins can be provided to the elasticsearch-maven-plugin. They will be installed into each Elasticsearch instance inside the plugins directory using the --batch option, before the instance gets started.

The way to enable plugins is as follows:

<plugin>
    <groupId>com.github.alexcojocaru</groupId>
    <artifactId>elasticsearch-maven-plugin</artifactId>
    <version>6.0</version>
    <configuration>
        <clusterName>test</clusterName>
        <transportPort>9300</transportPort>
        <httpPort>9200</httpPort>
        ...
        <plugins>
            <plugin>
                <uri>analysis-icu</uri>
            </plugin>
            <plugin>
                <uri>https://github.com/alexcojocaru/plugin.zip</uri>
                <esJavaOpts>-Djavax.net.ssl.trustStore=/home/alex/trustStore.jks</esJavaOpts>
            </plugin>
            <plugin>
                <uri>file:///home/alex/foo.zip</uri>
            </plugin>
        </plugins>
        ...
    </configuration>
    <executions>
        ...
    </executions>
</plugin>

The plugin tag takes 2 parameters:

Instance settings

Instance settings are applied to each corresponding elasticsearch instance (via -E on the commandline) during startup. If the list is smaller then instanceCount no extra settings are applied to the remaining instances. If it's larger, the extra items are ignored.

Example:

<plugin>
    <groupId>com.github.alexcojocaru</groupId>
    <artifactId>elasticsearch-maven-plugin</artifactId>
    <version>6.1</version>
    <configuration>
        ...
        <instanceCount>2</instanceCount>
        <instanceSettings>
            <properties>
                <node.name>First</node.name>
                <node.attr.data_type>ingest</node.attr.data_type>
            </properties>
            <properties>
                <node.name>Second</node.name>
                <node.attr.data_type>search</node.attr.data_type>
            </properties>
        </instanceSettings>
        ...
    </configuration>
    <executions>
        ...
    </executions>
</plugin>

Initialization scripts

A comma-separated list of initialization script files can be provided using the pathInitScript parameter of the plugin, in which case they will be executed against the local Elasticsearch cluster. The file extension defines the file format: json for JSON format, anything else for custom format, both formats can be used at the same time.

The script paths will be trimmed and executed in the order they are provided. For example:

<pathInitScript>script1.json,script2.script</pathInitScript>

is equivalent to

<pathInitScript>
    script1.json,
    script2.script
</pathInitScript>

And the plugin will always execute script1.json before script2.script

JSON format

The provided JSON files should contain a list of requests to be sent, one by one, to the Elasticsearch cluster. Each request definition has three properties:

Example (see the src/main/test/resources/init.json file for a more complete example):

To send a POST request to http://localhost:9200/test_index/test_type/_mapping, followed by a DELETE request to http://localhost:9200/test_index/test_type/1:

[
    {
        "method": "POST",
        "path": "test_index/test_type/_mapping",
        "payload": {
            "test_type": {
                "properties": {
                    "name": {
                        "type": "keyword"
                    },
                    "lastModified": {
                        "type": "date"
                    }
                }
            }
        }
    },
    {
        "method": "DELETE",
        "path": "test_index/test_type/1"
    }
]

Custom format

Each line defines a request to be sent to the Elasticsearch cluster, and it has three parts separated by colon:

Note: Empty lines are ignored, as well as lines starting with the '#' sign.

Examples (see the src/it/runforked-with-init-script/init.script file for a more complete example):

FAQ

Error: Could not find or load main class (on OSX 10.13.6)

There seems to be an issue when starting certain versions of Elasticsearch (eg. 5.6.8) on OSX 10.13.6, directly or via the plugin. The issue is caused by the incorrect quoting of the -cp argument on the Java command built by the bin/elasticsearch script inside the Elasticsearch package. A workaround is described here. In summary, set the ES_JVM_OPTIONS environment variable to -cp "./target/elasticsearch0/lib/*" in the IDE's run configuration or on the shell environment where maven/Elasticsearch is executed.

Error: Could not reserve enough space for x object heap (on Win 10)

This has the same root cause as the OSX specific error described above, and can be fixed using the same workaround.

Node is killed when running in TravisCI

When running your build job in TravisCI, it can happen that your node is being killed without any notice. To fix that you may have to modify the .travis.yml file as follows:

sudo: true
before_script:
  - sudo sysctl -w vm.max_map_count=262144

Error: java.lang.RuntimeException: can not run elasticsearch as root (in Docker)

When running the build in Docker, depending on the Docker image, the current user in the container maybe be the root user and, because of this, Elasticsearch will fail to start. The fix is to use a Docker image which does not use the root user. See this discussion for details.

Avoid downloading a plugin from the Internet repeatedly

When you want to run integration tests with a given plugin (eg. reindex-client), elasticsearch-maven-plugin will run behind the scene a command like bin/elasticsearch-plugin install reindex-client which will download the plugin from the Internet at every execution.

You can use some Maven magic to avoid the download by first using maven-dependency-plugin to download the plugin as an artifact which will be stored in your local .m2 directory, then copy from there to your project target directory, eg.

mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get \
    -DrepoUrl=https://repo1.maven.org/maven2 \
    -Dartifact=org.elasticsearch.plugin:reindex-client:5.4.2

Then just tell the elasticsearch-maven-plugin to use the local URI.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>3.0.0</version>
    <executions>
        <execution>
            <id>integ-setup-dependencies-plugins</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>copy</goal>
            </goals>
            <configuration>
                <artifactItems>
                    <artifactItem>
                        <groupId>org.elasticsearch.plugin</groupId>
                        <artifactId>reindex-client</artifactId>
                        <version>5.4.2</version>
                        <type>zip</type>
                    </artifactItem>
                </artifactItems>
                <useBaseVersion>true</useBaseVersion>
                <outputDirectory>${project.build.directory}/integration-tests/plugins</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>
<plugin>
    <groupId>com.github.alexcojocaru</groupId>
    <artifactId>elasticsearch-maven-plugin</artifactId>
    <version>5.7</version>
    <configuration>
        <version>5.4.2</version>
        <plugins>
            <plugin>
                <uri>file://${project.build.directory}/integration-tests/plugins/reindex-client-5.4.2.zip</uri>
            </plugin>
        </plugins>
    </configuration>
</plugin>

Why does the plugin uses a lock file and why does it need to open a server socket?

The plugin, at runtime, looks for the Elasticsearch artifact in the local maven repository and, if not found, it downloads it into the system temp directory and installs it from there into the local maven repo. When several Elasticsearch maven plugins are executed at the same time on a single machine (eg. during parallel builds), and the ES artifact hasn't already been installed in the local repo, it is not desirable that each plugin downloads and installs the same artifact; that is because the ES artifact is a large file (~ 250 Mb), and downloading the same file multiple times would be a waste of bandwidth.

Because of this, the plugin is capable to synchronize with other plugins running at the same time on the same machine, so that a single one downloads and installs, while the rest wait for the master plugin to complete in order to move forward with using the artifact. The synchronization mechanism is described by this diagram.

This is related to this plugin issue.

Integration Tests

The integration tests exist as maven plugins in the src/it directory and are executed via the maven invoker plugin (see the pom.xml for details on its configuration).

Each test has a maven like structure, with a pom.xml and a src/test/java directory containing the test sources.

The properties common between all tests are defined in the invoker plugin config under the "testProperties" tag.

During the integration test phase, the invoker plugin copies the tests to target/it and executes the "clean" and "verify" goals on each them. They are executed in a separate process with a brand new maven config, defined using the following two invoker plugin properties: localRepositoryPath (set to target/local-repo) and settingsFile (set to src/it/settings.xml). The invoker configuration also defines a pre-build hook script and a post-build hook script to run before and after executing the test. These are groovy scripts which each test directory must contain.

The structure of an integration test

Because the integration tests are executed as maven projects, they have a maven-like file structure.

pom.xml

The pom.xml is generic and does not contain anything specific to any test - it defines the test project dependencies and which goals to execute on the elasticsearch maven plugin.

setup.groovy

The pre-build hook script (setup.groovy) does the plugin and context configuration, by using the ItSetup util to create a map of plugin properties and to save them to the test.properties file in the test directory (to be picked up by the Java tests via the methods in ItBase). The properties are also set on the context, for some are needed by the post-build hook script. Defining the number of ES instances is required in the groovy script. The ES cluster name and the HTTP and transport protocol ports are randomly generated by ItSetup.generateProperties to avoid clashes between integration tests. Any other properties to be passed over to the plugin can be added to the props map (see src/it/runforked-auto-create-index-false/setup.groovy for an example).

verify.groovy

The standard verification done here is that the ES base directory(ies) were created and that the ES instance(s) are not running (via the ItVerification util). This file uses some of the plugin properties set on the context by the pre-build hook script.

Java test file(s)

The actual tests are defined in java files in the src/test/java directory under each integration test directory (eg. src/it/runforked-auto-create-index-false/src/test/java/com/github/alexcojocaru/mojo/elasticsearch/v2/AutoCreateIndexTest.java). They will be compiled and executed during the maven invoker plugin execution of the integration test maven project. All java tests should extend com.github.alexcojocaru.mojo.elasticsearch.v2.ItBase to get the clusterName and httpPort read from the context (ie. the "test.properties" file created by the pre-build hook script) and the ES client set up.

NOTE: It is not possible to execute such a test case in an IDE, due to the lack of context (the test properties must be set in the props file by executing the groovy script, the elasticsearch maven plugin must be running, etc).

How to write new tests

Copy one of the existing integration tests and modify as needed. It will be picked up by the invoker plugin due to the wildcard definition in the plugin config in pom.xml.

How to run single integration test

Set the integrationTest env variable to the integration test name when running maven, eg:

$ mvn clean verify -DintegrationTest=runforked-defaults-es6

How to debug an integration test

There are two ways to obtain more information during the execution of an integration test:

Debugging the maven execution

To have the invoker plugin output detailed information about the integration test execution, change the debug attribute of the invoker plugin configuration (in the pom.xml) to true.

Debugging the elasticsearch maven plugin execution during the integration test

Set the es.logLevel property of the plugin to DEBUG, by adding

props.put("es.logLevel", "DEBUG");

to the setup.groovy file for the integration test you want to debug.