Aggregates Plugin for Graylog

Build Status

Required Graylog version: 2.5.0 and later - NOT BACKWARDS COMPATIBLE WITH OLDER GRAYLOG VERSIONS.

Note: When upgrading from 1.x.x to version 2.0.0 of the plugin, it's required to modify existing rules (regarding streams and alerting) and existing report schedules (regarding receivers). See the screenshots below.

The Aggregates Plugin for Graylog enables users to execute term searches and get notified when the given criteria are met. Currently, there are the following alert conditions in Graylog:

However, these conditions will not be sufficient to match the following scenario:

Send an alert when someone fails to login from the same source ip 3 or more times in the past minute.

This scenario is actually very useful in a security context, but with the built-in alert conditions, it's not possible to match exactly this condition. The part "from the same source ip" is difficult to match. It would take an aggregate search that groups by value and returns the count per value. That's what the plugin aims to do. You can configure rules as shown in the screenshots below.

Create / edit a rule

Generated Alert Condition

Alert Overview

Rule overview

Report schedule overview

Create / edit a report schedule

Report example

Define the Aggregates Email Callback

Email example

Usage

Permissions

Use the Aggregates tab in the web interface of Graylog to define rules with alert criteria. For non-admin users, there are the following permissions that should be configured (via the REST API) to be able to fully (or partly) operate the plugin:

Note: To be able to view the list of rules, non-admin users need both the aggregate_rules:read and the aggregate_report_schedules:read permissions.

Each rule can be configured to be executed on a particular stream. For the latter option to be present, the user needs to be able to have at least the following permissions:

Sending alerts

Since version 2.0.0, the plugin integrates tightly with the Notifications and Alert Conditions within Graylog. You can define a notification on a stream as you would normally do. The plugin creates an Alert Condition when creating an Aggregate Rule and it keeps the condition in sync with the rule after updates. In previous versions, you could only send emails, but now you can also use the HTTP Alarm Callback for instance.

New in version 2.3.0 is the Aggregates Email Alarm Callback. This callback works in the same way as the normal Email Alarm Callback, but a table (inserted at the ${matchedTermsTable} placeholder) is added to the template, as shown in the screenshot. The table contains the found values and their number of occurrences, with links to the respective search query. This callback sends an HTML email, so you can customize the layout by using HTML tags and CSS in the email template.

Note 1: If you delete the Alert Condition, the plugin re-creates it, except when you disable the rule.
Note 2: Enabling the message backlog can inflict a performance penalty, as the backlog has to be assembled from the found terms, using separate searches. Use with care.
Note 3: Alert Conditions are created under the user admin.
Note 4: For the Aggregates Email Alarm Callback, only email receivers can be defined, user receivers are not supported.

Reporting

In the rule overview, there's an option (checkbox) to include rule history in a report. This report is a PDF file that contains a bar chart for every rule, summing up the total number of hits for that rule per period. The grid for the chart is automatically determined based on the total amount of time. The report is tailored per alert receiver, which means that a receipient will only receive charts for the rules subscribed to.

When creating or editing a rule, the schedule(s) for generating report(s) can be supplied. For configuring a schedule, you should supply a name, a valid Cron expression using the Drools syntax and the timespan, e.g. the amount of history you wish to incorporate in the report. Since version 2.0.0 of the plugin, receivers of reports are defined on the report schedule, not on the aggregates rule anymore.

Note: The maximum timespan determines the overall retention. So, if you have a report schedule that takes a year of history, the retention of hits will be a year. This might influence your MongoDB storage needs.

Changelog

2.4.0

2.3.0

2.2.4

2.2.3

2.2.2

2.2.1

2.1.1

2.1.0

2.0.0

Getting started

This project is using Maven 3 and requires Java 8 or higher.

Plugin Release

We are using the maven release plugin:

$ mvn release:prepare
[...]
$ mvn release:perform

This sets the version numbers, creates a tag and pushes to GitHub. Travis CI will build the release artifacts and upload to GitHub automatically.

Installation

Download the plugin and place the .jar file in your Graylog plugin directory. The plugin directory is the plugins/ folder relative from your graylog-server directory by default and can be configured in your graylog.conf file.

Restart graylog-server and you are done.

Development

You can improve your development experience for the web interface part of your plugin dramatically by making use of hot reloading. To do this, do the following: