Changestream

CircleCI codecov

Changestream sources object-level change events from a MySQL Replication Master (configured for row-based replication) by streaming the MySQL binlog and transforming change events into JSON strings that can be published anywhere.

Currently, Amazon Simple Queuing Service (SQS), Amazon Simple Notification Service and Amazon S3 are supported with optional client-side message encryption via AES.

Documentation

Getting Started

Requirements

Setup

First, start MySQL with

docker-compose up

(or, for the hard way without Docker, follow the bespoke setup instructions).

You will then have access to a MySQL instance on localhost port 3306 with a root password of "password", and Changestream configured to listen for events.

Changestream is available on Docker Hub.

Then build and run Changestream using the sbt CLI:

$ sbt run

Building

Docker Image (requires a docker host to build)

Note: The Dockerfile will be written to target/docker/Dockerfile, and the docker image will be added to your local docker repo.

$ sbt docker

Debian Package (requires dpkg-deb, written to target/*.deb)

$ sbt debian:packageBin

Jar Package (written to target/scala-2.11/*.jar)

$ sbt package

Contributing

Testing

Changestream is 100% unit and integration tested, with end-to-end testing on CircleCI with MySQL and Amazon SNS.

Mysql Test Config

In order to run tests locally you need to configure test.conf with a working MySQL connection configuration.

Make sure that your test user has permissions to create and drop databases and tables. By default, Changestream uses the changestream_test table for tests.

AWS Test Config

To run the integration tests, as well as the unit tests on the SqsActor and SnsActor, you also need to ensure that you have configured your AWS tokens to be available from the environment, and that your AWS tokens have access to create and add messages to queues (SQS) and create and publish to topics (SNS).

AWS_ACCESS_KEY_ID=<access_key_id>
AWS_SECRET_ACCESS_KEY=<secret_access_key>
AWS_REGION=<your_region>
AWS_S3_BUCKET=<your_bucket>
AWS_S3_KEY_PREFIX=<some_key_prefix>

Running Tests

Once that's all set up, you can either run

$ sbt test

from the command line, or run the test command in IntelliJ (with the Scala and SBT plugins installed).

ScalaStyle

Changestream uses ScalaStyle for code linting--it is not automatic. Before submitting a PR, please take a look at the output of the sbt scalastyle command and correct any issues you find.

License

Changestream is available under the MIT License.

Authors

Peter Fry (@racerpeter) and the Mavengineering team.

Inspiration

Changestream was inspired in a large part by mardambey/mypipe by Hisham Mardam-Bey. The first proof-of-concept for changestream was a fork of mypipe - thank you Hisham.