Kafka-on-Pulsar (aka KoP) was developed to support Kafka protocol natively on Apache Pulsar.
KoP brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the KoP protocol handler to your existing Pulsar cluster, you can now migrate your existing Kafka applications and services to Pulsar without modifying the code. This enables Kafka applications to leverage Pulsar’s powerful features, such as:
KoP unlocks new use cases by integrating two popular messaging and event streaming ecosystems together. Customers can leverage advantages from each ecosystem and build a truly unified event streaming platform with Apache Pulsar to accelerate the development of real-time applications and services.
The following figure illustrates how is the Kafka-on-Pulsar protocol handler was implemented within Pulsar.
KoP is part of StreamNative Platform. Please visit StreamNative Docs for more details.
If you already have a Apache Pulsar cluster running, it is very straightforward to enable Kafka-on-Pulsar on your existing Pulsar cluster by downloading and installing the KoP protocol handler to Pulsar brokers.
You can download the released KoP protocol handler from here.
Alternatively, you can also build the KoP protocol handler from source code.
git clone https://github.com/streamnative/kop.git cd kop
build the project.
mvn clean install -DskipTests
the nar file can be found at this location.
As mentioned previously, the KoP protocol handler is a plugin that can be installed to the Pulsar brokers.
All what you need to do is to configure the Pulsar broker to run the KoP protocol handler as a plugin, that is,
add configurations in Pulsar's configuration file, such as
Set the configuration of the KoP protocol handler.
Add the following properties and set their values in Pulsar configuration file, such as
Regarding topic auto create partition type, if you are not using StreamNative Platform, please set it to
|Property||Set it to the following value||Default value|
||Location of KoP NAR file||./protocols|
messagingProtocols=kafka protocolHandlerDirectory=./protocols allowAutoTopicCreationType=partitioned
Set Kafka service listeners.
The hostname in listeners should be the same as Pulsar broker's
After you have installed the KoP protocol handler to Pulsar broker, you can restart the Pulsar brokers to load KoP.
Download the Kafka 2.0.0 release and untar it.
tar -xzf kafka_2.11-2.0.0.tgz cd kafka_2.11-2.0.0
Use a Kafka producer and a Kafka consumer to verify.
In Kafka’s binary, there is a command-line producer and consumer.
Run the command-line producer and send a few messages to the server.
> bin/kafka-console-producer.sh --broker-list [pulsar-broker-address]:9092 --topic test This is a message This is another message
Kafka has a command-line consumer dumping out messages to standard output.
> bin/kafka-console-consumer.sh --bootstrap-server [pulsar-broker-address]:9092 --topic test --from-beginning This is a message This is another message
See Configure KoP for more details.
KoP supports TLS encryption and integrates with Pulsar's authentication and authorization providers seamlessly.
See Secure KoP for more details.
For the complete documentation of KoP, please checkout StreamNative Documentation.