SpinalTap is a general-purpose reliable Change Data Capture (CDC) service, capable of detecting data mutations with low-latency across different data sources, and propagating them as standardized events to downstream consumers. SpinalTap has become an integral component in Airbnb's infrastructure platform and data processing pipeline, on which several critical applications are reliant on. More information from our engineering blog.
SpinalTap does not support message authentication yet as part of its current open-source offering.
Message authentication is a requirement to enforce authenticity and accessibility of messages propagated in the message bus, as well as source verification of transmitted messages. Given SpinalTap is by design a source & destination agnostic library, we believe the burden of supplying a message authentication workflow lies on the message bus solution used, or as an extension to the destination component implement adopted. This is inline with the usage within Airbnb’s infrastructure, where SpinalTap is just another client to our internal message transport which in itself should supply message authentication as a general offering for all its clients. If the need arises in the future, message authentication can be seamlessly integrated within the library as a potential supported feature.
https://thrift.apache.org/docs/install/
cd spinaltap-standalone
../gradlew shadowjar
Recommend the version 0.9.0.1 or above (https://kafka.apache.org/downloads).
cd kafka
./gradle build
MySQL server should be configured to use Row-based
binlog:
[mysqld]
log-bin=mysql-bin-changelog
binlog-format=ROW
server-id=1
zk-connection-string: localhost:2181
zk-namespace: spinaltap-standalone
kafka-config:
bootstrap_servers: localhost:9092
mysql-user: spinaltap
mysql-password: spinaltap
mysql-server-id: 12345
mysql-schema-store:
host: schema-store.xxx
port: 3306
database: schema-store
archive-database: schema-store-archive
mysql-sources:
- name: mysql-db-1
host: localhost
port: 3306
host_role: MASTER
socket_timeout_seconds: -1
schema_version_enabled: true
initial_binlog_position:
fileName: mysql-bin-changelog.001234
position: 4
nextPosition: 4
tables:
- test:users
- test:places
destination:
pool_size: 2
buffer_size: 1000
- name: mysql-db-2
host: mysql-db-2.xxx
port: 3306
tables:
- db1:table1
destination:
buffer_size: 1000
boostrap_servers
is required.SELECT
, REPLICATION SLAVE
, REPLICATION CLIENT
, SHOW VIEW
permissions are required. MySQL schema store is a MySQL instance which is used to store SpinalTap MySQL table schemas and their version history. When this feature is enabled, SpinalTap will track table schema changes and save the schema version snapshots into the schema store.
MASTER
, REPLICA
, or MIGRATION
. The corresponding Kafka topic prefixes are different. The default value is MASTER
.false
.<database_name>:<table_name>
.cd kafka
bin/zookeeper-server-start.sh config/zookeeper.properties
cd kafka
bin/kafka-server-start.sh config/server.properties
java -jar build/libs/spinaltap-standalone-all.jar spinaltap_standalone.yaml