Pub/Sub Emulator for Kafka

This project implements a gRPC server that satisfies the Cloud Pub/Sub API as an emulation layer on top of an existing Kafka cluster configuration. The emulator is exposed as a standalone Java application with a mandatory configuration passed as an argument at runtime. It is fully compatible with the latest versions of the Google Cloud Client libraries and is explicitly tested against the Java client library.

Build Status Coverage Codacy Badge

Building and Running

Checkout the source and then build the self-contained JAR with mvn package. If you wish to run the integration test suite, which starts an in-memory Kafka cluster, run mvn verify. The self-contained JAR is then available at ./target/kafka-pubsub-emulator-<version>.jar.

Standalone JAR

When invoking the standalone JAR, you must specify the location of a JSON-formatted configuration file using the -c command line argument.

A sample configuration can be found at ./demo/benchmark/config/config.json. This configuration assumes you have Kafka accessible via three hosts, kafka-0, kafka-1, and kafka-2 each at port 9092.

You must also provide a JSON-formatted file containing Pub/Sub project definitions using the -p command line argument. These entries define the projects available to the emulator's clients, along with the Topics and Subscriptions that will be present when the server is started. This file must be writeable if your clients will be creating, deleting, or modifying Topics and Subscriptions.

A sample configuration can be found at ./demo/benchmark/config/pubsub.json. This provides for a single project named performance-testing, 4 Topics, and one Subscription per topic.

java -jar target/kafka-pubsub-emulator-<POM version>.jar \
  -c <path to your server configuration file> \
  -p <path to your Pub/Sub file>

Most likely, you will need to specify your own configuration file as described in the Configuration Options section below.

Docker

To execute the emulator using Docker, you must provide it with the necessary configurations. For an example, see the ./demo/benchmark/docker-containers.sh script.

export PATH_TO_CONFIGS="/path/to/configuration-files/"

docker build -t kafka-pubsub-emulator:<POM version> .

docker run --mount type=bind,src=${PATH_TO_CONFIGS},dst=/etc/config \
  -d kafka-pub-sub-emulator:<POM version> \
  -c /etc/config/config.json -p /etc/config/pubsub.json

Kubernetes

Updates coming soon...

Configuration Options

The Pub/Sub Emulator server needs to be started with a JSON-based configuration file that indicates the server port, optional TLS settings, and the information needed to connect to a Kafka cluster. See src/main/proto/config.proto for the full structure.

Required Options

Optional Options

Pub/Sub Configuration Repository

The Pub/Sub configuration must be bootstrapped when the emulator is started. Currently, there is a single implementation of the PubSubRepository interface, which supports reading from and writing back to a file. This file must be provided with the -p command-line argument and should be a JSON-formatted Protocol buffer message matching the PubSub message type.

A valid configuration must have at least one Project. Each Project can have many Topics, each of which may have one or more Subscriptions to it. Within a Topic, the kafka_topic property can be used in situations where the Kafka topic being bound to has a different name than what will be exposed through the emulator.

Connecting with Google Cloud Client Library for Java

To connect client applications, you can use the official Google Cloud Platform client libraries for your preferred language. The examples below assume you are using the Java libraries. These settings can be adapted for other languages.

Setting explicit CredentialsProvider and ChannelProvider

By default, the client library attempts to use your Google Cloud Project credentials as explained in the Authentication docs, and connects to pubsub.googleapis.com. Since the emulator does not run in GCP, you'll need to specify a custom CredentialsProvider and create a new Channel bound to the emulator's host and port.

Below is an example that will create a Publisher client connected to the emulator server running on the local machine at port 8080 using a plaintext connection.

ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 8080)
        .usePlaintext(true)
        .build();
Publisher publisher = Publisher.newBuilder(ProjectTopicName.of("my-project", "my-topic")
        .setChannelProvider(
            FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel)))
        .setCredentialsProvider(new NoCredentialsProvider())
        .build();

If your emulator server is using SSL/TLS, you will need to create a secure Channel using a slightly different instantiation pattern.

File certificate = new File("path-to-certificate.crt");
ManagedChannel channel;
try {
  channel =
      NettyChannelBuilder.forAddress("localhost", 8080)
          .maxInboundMessageSize(100000)
          .sslContext(GrpcSslContexts.forClient().trustManager(certificate).build())
          .build();
} catch (SSLException e) {
  System.err.println("Unable to create SSL channel " + e.getMessage());
}
Publisher publisher = Publisher.newBuilder(ProjectTopicName.of("my-project", "my-topic")
        .setChannelProvider(
            FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel)))
        .setCredentialsProvider(new NoCredentialsProvider())
        .build();

One difference between how the client library behaves with the emulator vs. Cloud Pub/Sub is that by default, clients connected to Cloud Pub/Sub open multiple channels (1 per CPU core). Currently, it's not possible to specify that type of multi-channel configuration with the emulator without writing custom Channel code.

For further reference, consult the examples in the integration tests or look at the demos for various platforms.