Event-Driven microservices with Kafka Streams

A skeleton example of how to use Kafka Streams with Event-Driven microservices.

This is an application that generates a simple football statistics like match scores, team and player rankings. It should be a little more comprehensible than a typical code examples based on orders, payments etc.

 

architecture

You can think of this application as a supplement to the basic code examples of Kafka Streams. Kafka Streams is used here extensively for many purposes like data processing, building materialized views, persisting the domain model and exposing the output of services. Furthermore, there is no traditional database, local state stores instead (PostgreSQL plays a role of legacy system only).

This codebase is trying to apply the idea of stateful streams processing with Kafka and Kafka Streams. These blog posts are a great introduction to that concept:

Services

Additional modules:

Events and Streams

Each event represents a state change that occurred to the system.

Event Fields
MatchScheduled match id, season, home club, away club, matchDate
MatchStarted match id, home club, away club
GoalScored goal id, match id, minute, scorer, scored for
CardReceived card id, match id, minute, receiver, type (yellow/red)
MatchFinished match id
PlayerStartedCareer palyer id, name

These events are the source for stream processing. For example, in order to determine the result of a match, you must join MatchStarted and GoalScored streams and then count the goals (see the code).

Domain Model

model

Kafka Topics

Topics are organized under logical groups as follows:

The prefixes above aim to separate the application topics from the internal Kafka Streams and Debezium topics.

topics

REST Endpoints

There are only two REST endpoits and both of them are accessed from the outside of the system:

How to run

Requirements: Java 10, Docker with at least 6 GB memory limit.

  1. Add kafka bootstrap address to your /etc/hosts:
    127.0.0.1 kafka
    127.0.0.1 postgres
    127.0.0.1 connect
    127.0.0.1 football-match
    127.0.0.1 football-player
    127.0.0.1 football-view-basic
    127.0.0.1 football-view-top
    127.0.0.1 football-ui
  2. Build microservices and Docker containers:
    cd football
    mvn install
  3. Build and run the demo application:

    cd football-tests
    mvn install -DskipTests
    java -jar target/football-tests-0.0.1-SNAPSHOT-jar-with-dependencies.jar

    It will start Docker containers and wait until all microservices are ready.

    If the default startup timeout (180 sec) will be insufficient, try to increase it:

    java -jar target/football-tests-0.0.1-SNAPSHOT-jar-with-dependencies.jar 400
  4. Wait a minute and launch http://football-ui:18080/. In a few seconds you should see some statistics updated in real time:

 

demo

The input is historical data from English Football League Championship. The historical events (goal scored, card received etc.) are played back every few seconds.