Replicates data changes from MySQL binlog to HBase or Kafka. In case of HBase, preserves the previous data versions. HBase storage is intended for auditing and analysis of historical data. In addition, special daily-changes tables can be maintained in HBase, which are convenient for fast and cheap imports from HBase to Hive. Replication to Kafka is intended for easy real-time access to a stream of data changes.
This readme file provides basic introduction on how to get started. For more details, refer to official documentation at mysql-time-machine.
mvn clean packagefrom the root of the
replicatorrepository to build the MySQL Replicator distribution that will be used later;
images/replicator_testing/input/replicator/directory inside the
container_build.shscript from the
images/replicator_testing/directory inside the
docker imagesto verify that
replicator-testingimage has been built successfully;
Replicator assumes that there is a pre-installed environment in which it can run. This environment consists of:
Easiest way to test drive the replicator is to use docker to locally create this needed environment. In addition to docker you will need docker-compose installed locally.
git clone https://github.com/mysql-time-machine/docker.git cd docker/docker-compose/replicator_testing
Start all containers (mysql, kafka, graphite, replicator, zookeeper)
Now, in another terminal, you can connect to the replicator container
./attach_to_replicator cd /replicator
This folder contains:
There are the steps to initialize the environemnt and start the replication with latest 0.16 version:
./00_setup # will unznip the jars from distribution ./01_seed # will initialize mysql with random data ./02_run_016 # will start the replication
Random data in the seed step has been inserted in pre-created database 'test' in precreated table 'sometable'. The provided mysql instance is configured to use RBR and binlogs are active.
mysql --host=mysql --user=root --pass=mysqlPass mysql> use test; mysql> show tables; +----------------+ | Tables_in_test | +----------------+ | sometable | +----------------+ 1 row in set (0.00 sec)
As the replication is running, you can observe the replication statistics at graphite dashboard: http://localhost/dashboard/
Packaging the project is really simple with Maven
mvn clean package
Will generate a tar.gz distribution as output. The distribution will contain all dependencies.
To deploy a new version to Maven central it's enough executing
mvn clean deploy -P release
If previous step didn't work is probably because you don't have a SonaType account or a published GPG key. Follow these steps:
Now you should be in conditions to deploy the project.
Bosko Devetak [email protected]
Carlos Tasada [ctasada]
Dmitrii Tcyganov [dtcyganov]
Evgeny Dmitriev [dmitrieveu]
Fabricio Damasceno [forlando]
Gaurav Kohli [gauravkohli]
Greg Franklin [gregf1]
Islam Hassan [ishassan]
Mikhail Dutikov [mikhaildutikov]
Muhammad Abbady [muhammad-abbady]
Philippe Bruhat (BooK) [book]
Pavel Salimov [chcat]
Pedro Silva [pedros]
Raynald Chung [raynald]
Rares Mirica [mrares]
Steve Aurigema [steveauri]
Replicator was originally developed for Booking.com. With approval from Booking.com, the code and specification were generalized and published as Open Source on github, for which the author would like to express his gratitude.
Copyright (C) 2015, 2016, 2017, 2018, 2019 by Author and Contributors
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.