Nakadi Event Broker

Build Status codecov.io Codacy Badge

Nakadi is a distributed event bus broker that implements a RESTful API abstraction on top of Kafka-like queues, which can be used to send, receive, and analyze streaming data in real time, in a reliable and highly available manner.

One of the most prominent use cases of Nakadi is to decouple micro-services by building data streams between producers and consumers.

Main users of nakadi are developers and analysts. Nakadi provides features like REST based integration, multi consumer, ordered delivery, interactive UI, fully managed, security, ensuring data quality, abstraction of big data technology, and push model based consumption.

Nakadi is in active developement and is currently in production inside Zalando as the backbone of our microservices sending millions of events daily with a throughput of more than hundreds gigabytes per second. In one line, Nakadi is a high-scalability data-stream for enterprise engineering teams.

Nakadi Deployment Diagram

More detailed information can be found on our website.

Project goal

The goal of Nakadi (ნაკადი means stream in Georgian) is to provide an event broker infrastructure to:

Development status

Presentations

Features

Read more about latest development on the releases page.

Additional features that we plan to cover in the future are:

Related projects

The zalando-nakadi organisation contains many useful related projects like

How to contribute to Nakadi

Read our contribution guidelines on how to submit issues and pull requests, then get Nakadi up and running locally using Docker:

Dependencies

The Nakadi server is a Java 8 Spring Boot application. It uses Kafka 1.1.1 as its broker and PostgreSQL 9.5 as its supporting database.

Nakadi requires recent versions of docker and docker-compose. In particular, docker-compose >= v1.7.0 is required. See Install Docker Compose for information on installing the most recent docker-compose version.

The project is built with Gradle. The ./gradlew wrapper script will bootstrap the right Gradle version if it's not already installed.

Install

To get the source, clone the git repository.

git clone https://github.com/zalando/nakadi.git

Building

The gradle setup is fairly standard, the main tasks are:

Some other useful tasks are:

For working with an IDE, the eclipse IDE task is available and you'll be able to import the build.gradle into Intellij IDEA directly.

Running a Server

From the project's home directory you can start Nakadi via Gradle:

./gradlew startNakadi

This will build the project and run docker compose with 4 services:

To stop the running Nakadi server:

./gradlew stopNakadi

Using Nakadi and its API

Please read the manual for the full API usage details.

Creating Event Types

The Nakadi API allows the publishing and consuming of events over HTTP. To do this the producer must register an event type with the Nakadi schema registry.

This example shows a minimalistic undefined category event type with a wildcard schema:

curl -v -XPOST http://localhost:8080/event-types -H "Content-type: application/json" -d '{
  "name": "order.ORDER_RECEIVED",
  "owning_application": "order-service",
  "category": "undefined",
  "schema": {
    "type": "json_schema",
    "schema": "{ \"additionalProperties\": true }"
  }
}'

Note: This is not a recommended category and schema. It should be used only for testing.

You can read more about this in the manual.

Consuming Events

You can open a stream for an event type via the events sub-resource:

curl -v http://localhost:8080/event-types/order.ORDER_RECEIVED/events

HTTP/1.1 200 OK

{"cursor":{"partition":"0","offset":"82376-000087231"},"events":[{"order_number": "ORDER_001"}]}
{"cursor":{"partition":"0","offset":"82376-000087232"}}
{"cursor":{"partition":"0","offset":"82376-000087232"},"events":[{"order_number": "ORDER_002"}]}
{"cursor":{"partition":"0","offset":"82376-000087233"},"events":[{"order_number": "ORDER_003"}]}

You will see the events when you publish them from another console for example. The records without events field are Keep Alive messages.

Note: the low-level API should be used only for debugging. It is not recommended for production systems. For production systems, please use the Subscriptions API.

Publishing Events

Events for an event type can be published by posting to its "events" collection:

curl -v -XPOST http://localhost:8080/event-types/order.ORDER_RECEIVED/events \
 -H "Content-type: application/json" \
 -d '[{
    "order_number": "24873243241"
  }, {
    "order_number": "24873243242"
  }]'

HTTP/1.1 200 OK  

Read more in the manual.

Contributing

Nakadi accepts contributions from the open-source community.

Please read CONTRIBUTING.md.

Please also note our CODE_OF_CONDUCT.md.

Contact

This email address serves as the main contact address for this project.

Bug reports and feature requests are more likely to be addressed if posted as issues here on GitHub.

License

Please read the full LICENSE

The MIT License (MIT) Copyright © 2015 Zalando SE, https://tech.zalando.com

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.