Build Status

Promagent

Prometheus Monitoring for Java Web Applications without Modifying their Source Code.

The promagent-maven-plugin is a tool for creating custom Java agents for Prometheus monitoring. The Java agents instrument Java Web Applications with Prometheus metrics without modifying the applications' source code. The agents use the Byte Buddy bytecode manipulation library to insert Prometheus metrics during application startup.

The Promagent code repository contains two projects:

The example agent was tested with Tomcat for the Spring Boot example and with the Wildfly application server for the Java EE example.

Example

screenshot

Downloading and Compiling the Example Agent

Clone Promagent from GitHub:

git clone https://github.com/fstab/promagent.git
cd promagent

The promagent-api and promagent-maven-plugin are not on Maven Central yet. Run the following commands to make them available locally (in ~/.m2/repository/):

cd promagent-framework
mvn clean install
cd ..

Compile the example agent. This should create the file ./promagent-example/target/promagent.jar:

cd promagent-example
mvn clean verify
cd ..

Spring Boot Demo

The following runs with Java 8 and was not tested with Java 9 yet.

Download and compile a Spring Boot Getting Started application.

git clone https://github.com/spring-guides/gs-accessing-data-rest.git
cd gs-accessing-data-rest/complete
mvn clean package
cd ../..

Run the Spring Boot application with the Promagent attached.

java \
    -javaagent:promagent/promagent-example/target/promagent.jar=port=9300 \
    -jar gs-accessing-data-rest/complete/target/gs-accessing-data-rest-0.1.0.jar

Go to http://localhost:8080 to view the Spring Boot application, go to http://localhost:9300/metrics to view the Prometheus metrics.

Java EE Demo on Wildfly

_This demo runs with Java 8. For a Java 9 version, see JAVA_9_DEMO.md._

Download and compile a Wildfly Quickstart application.

git clone https://github.com/wildfly/quickstart.git
cd quickstart/kitchensink
mvn clean package
cd ../..

Download and extract the Wildfly application server.

curl -O http://download.jboss.org/wildfly/10.1.0.Final/wildfly-10.1.0.Final.tar.gz
tar xfz wildfly-10.1.0.Final.tar.gz

Run the Wildfly application server with the Promagent attached.

cd wildfly-10.1.0.Final
LOGMANAGER_JAR=$(find $(pwd) -name 'jboss-logmanager-*.jar')
export JAVA_OPTS="
    -Xbootclasspath/p:${LOGMANAGER_JAR}
    -Djboss.modules.system.pkgs=org.jboss.logmanager,io.promagent.agent
    -Djava.util.logging.manager=org.jboss.logmanager.LogManager
    -javaagent:../promagent/promagent-example/target/promagent.jar=port=9300
"
./bin/standalone.sh

In a new Shell window, deploy the quickstart application.

cd wildfly-10.1.0.Final
./bin/jboss-cli.sh --connect --command="deploy ../quickstart/kitchensink/target/kitchensink.war"

Go to http://localhost:8080/kitchensink to view the quickstart application, go to http://localhost:9300/metrics to view the Prometheus metrics.

Creating your Own Agent

A Promagent is implemented as a set of Hooks. A Hook is a Java class meeting the following requirements:

The best way to get started is to have a look at the ServletHook and JdbcHook in the promagent-example.

A simple Hook counting the number of Servlet requests looks as follows:

@Hook(instruments = "javax.servlet.Servlet")
public class ServletHook {

    private final Counter servletRequestsTotal;

        public ServletHook(MetricsStore metricsStore) {
            servletRequestsTotal = metricsStore.createOrGet(new MetricDef<>(
                    "servlet_requests_total",
                    (name, registry) -> Counter.build()
                        .name(name)
                        .help("Total number of Servlet requests.")
                        .register(registry)
            ));
        }

        @After(method = "service")
        public void after(ServletRequest request, ServletResponse response) {
            httpRequestsTotal.inc();
        }
}

To build a Promagent project with Maven, you need two entries in the pom.xml. First, the promagent-api must be included as a dependency:

<dependency>
    <groupId>io.promagent</groupId>
    <artifactId>promagent-api</artifactId>
    <version>1.0-SNAPSHOT</version>
    <scope>provided</scope> <!-- provided at runtime by the internal agent implementation -->
</dependency>

Second, the promagent-maven-plugin that creates an agent JAR:

<build>
    <finalName>promagent</finalName>
    <plugins>
        <plugin>
            <groupId>io.promagent</groupId>
            <artifactId>promagent-maven-plugin</artifactId>
            <version>1.0-SNAPSHOT</version>
            <executions>
                <execution>
                    <id>promagent</id>
                    <phase>package</phase>
                    <goals>
                        <goal>build</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

With these two things included, mvn clean package should produce a working Java agent in target/promagent.jar.

A Hook's Life Cycle

A Hook's lifecycle depends on whether the instrumented method call is a nested call or not. A call is nested when the method is called by another method that was instrumented with the same hook. This happens for example if a Servlet's service() method calls another Servlet's service() method.

By default, nested calls are ignored and the Hook is only invoked for the outer call. In the Servlet example, this is the intended behavior, because it guarantees that each HTTP request is counted only once, even even if a Servlet internally calls another Servlet to handle the request.

If the Hook is defined with @Hook(skipNestedCalls = false) the Hook will be invoked for all nested calls, not only for the outer call.

For each outer call, a new Hook instance is created. If the Hook implements both a @Before and an @After method, the same instance is used for @Before and @After. That way, you can set a start time as a member variable in the @Before method, and use it in the @After method to calculate the duration of the call.

For nested calls, the Hook instance from the outer call is re-used. That way, you can put data into member variables in order to pass that data down the call stack.

The Hook's Constructor Parameter

Most applications use static variables to maintain Prometheus metrics, as described in the Prometheus Client Library for Java documentation:

# Doesn't work with Promagent
static final Counter counter = Counter.build()
    .name("requests_total")
    .help("Total requests.")
    .register();

Unfortunately, static variables are maintained per deployment in an application server. When an application is re-deployed, a new instance of the same Counter is created, which causes conflicts in the Prometheus registry (as the Prometheus registry is maintained by Promagent, it survives re-deployments). Moreover, it is impossible to instrument a mix of internal modules (like an internal Servlet in the JAX-RS implementation) and deployments (like Servlets in a WAR file) with static variables.

To prevent this, Promagent requires Hooks to use the MetricsStore to maintain metrics:

# This is the correct way with Promagent
Counter counter = metricsStore.createOrGet(new MetricDef<>(
                    "requests_total",
                    (name, registry) -> Counter.build()
                        .name(name)
                        .help("Total requests.")
                        .register(registry)
            ));

The Promagent library will take care that the Counter is created only once, and that the Counter instance is re-used across multiple deployments and internal modules in an application server.

Hook Annotations

Using Labels

The Prometheus server internally stores one time series for each observed set of label values. The time series database in the Prometheus server can easily handle thousands of different time series, but millions of different time series could be a problem. Therefore, it is important to keep the number of different label values relatively small. Unique user IDs, timestamps, or session keys should not be used as label values.

The promagent-example strips HTTP URLs and SQL queries to make sure that there are not too many different label values:

Of course, replacing path parameters and SQL values is application specific. The promagent-example implements a very simple replacement in ServletHook.stripPathParameters() and JdbcHook.stripValues(), but you probably need to customize these methods for your application.

Running Docker Tests

The promagent-example project contains an alternative Maven configuration in pom-with-docker-tests.xml. This configuration uses the docker-maven-plugin to create Docker images and run integration tests against Docker containers.

The Wildfly tests can be run as follows:

cd promagent-example
mvn -f pom-with-docker-tests.xml clean verify -Pwildfly
cd ..

The Spring Boot tests can be run as follows:

cd promagent-example
mvn -f pom-with-docker-tests.xml clean verify -Pspring
cd ..

The first run takes a while, because the Docker images need to be built. Once the images are available on the local systems, runs are significantly faster.

Exposing Metrics

Promagent supports three different ways of exposing metrics to the Prometheus server:

Status

This is a demo project. The main goal is to learn the internals of bytecode manipulation and class loading in Java application servers. I am planning to work on the following:

The promagent-api and promagent-maven-plugin are not yet available on Maven Central, but they will be uploaded when the API becomes a bit more stable.

If you want to write your own agent and are looking for examples of methods you might want to instrument, look at related projects, like inspectIT (hooks are configured here) or stagemonitor.

Resources

Thank You ConSol

This project is supported as part of the R&D activities at ConSol Software GmbH. See the ConSol Labs Blog for more info.