The Open Distro for Elasticsearch Anomaly Detection plugin enables you to leverage Machine Learning based algorithms to automatically detect anomalies as your log data is ingested. Combined with Alerting, you can monitor your data in near real time and automatically send alert notifications . With an intuitive Kibana interface and a powerful API, it is easy to set up, tune, and monitor your anomaly detectors.
Anomaly detection is using Random Cut Forest (RCF) algorithm for detecting anomalous data points.
Anomaly detections run a scheduled job based on job-scheduler.
You should use anomaly detection plugin with the same version of Open Distro Alerting plugin. You can also create a monitor based on the anomaly detector. A scheduled monitor run checks the anomaly detection results regularly and collects anomalies to trigger alerts based on custom trigger conditions.
Please see our documentation.
To build from command line set JAVA_HOME to point to a JDK 14 before running ./gradlew
This package uses the Gradle build system. Gradle comes with excellent documentation that should be your first stop when trying to figure out how to operate or modify the build. we also use the Elastic build tools for Gradle. These tools are idiosyncratic and don't always follow the conventions and instructions for building regular Java code using Gradle. Not everything in this package will work the way it's described in the Gradle documentation. If you encounter such a situation, the Elastic build tools source code is your best bet for figuring out what's going on.
Currently we just put RCF jar in lib as dependency. Plan to publish to Maven and we can import it later. Before publishing to Maven, you can still build this package directly and find source code in RCF Github package.
./gradlew buildbuilds and tests
./gradlew :runlaunches a single node cluster with the AD (and job-scheduler) plugin installed
./gradlew :integTestlaunches a single node cluster with the AD (and job-scheduler) plugin installed and runs all integration tests
./gradlew :integTest --tests="**.test execute foo"runs a single integration test class or method
./gradlew spotlessApplyformats code. And/or import formatting rules in
When launching a cluster using one of the above commands logs are placed in
/build/cluster/run node0/elasticsearch-<version>/logs. Though the logs are teed to the console, in practices it's best to check the actual log file.
Currently, the only IDE we support is IntelliJ IDEA. It's free, it's open source, it works. The gradle tasks above can also be launched from IntelliJ's Gradle toolbar and the extra parameters can be passed in via the Launch Configurations VM arguments.
Sometimes it's useful to attach a debugger to either the ES cluster or the integ tests to see what's going on. When running unit tests you can just hit 'Debug' from the IDE's gutter to debug the tests. To debug code running in an actual server run:
./gradlew :integTest --debug-jvm # to start a cluster and run integ tests OR ./gradlew :run --debug-jvm # to just start a cluster that can be debugged
The ES server JVM will launch suspended and wait for a debugger to attach to
localhost:8000 before starting the ES server.
To debug code running in an integ test (which exercises the server from a separate JVM) run:
./gradlew -Dtest.debug :integTest
The test runner JVM will start suspended and wait for a debugger to attach to
localhost:5005 before running the tests.
Sometimes you need to launch a cluster with more than one Elasticsearch server process.
You can do this by running
./gradlew run -PnumNodes=<numberOfNodesYouWant>
You can also debug a multi-node cluster, by using a combination of above multi-node and debug steps.
But, you must set up debugger configurations to listen on each port starting from
5005 and increasing by 1 for each node.
We welcome you to get involved in development, documentation, testing the anomaly detection plugin. See our contribution guidelines and join in.
This project has adopted an Open Source Code of Conduct.
If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page. Please do not create a public GitHub issue.
See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.
Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.