dynamo-cassandra-proxy

Build state: build

dynamo-cassandra-proxy consists of a scalable proxy layer that sits between your app and Apache Cassandra. See the design summary in docs

It provides compatibility with the DynamoDB SDK which allows existing DynamoDB applications to read and write data to Cassandra without application changes.

It also supports the ability to sync DynamoDB tables with cassandra via DynamoDB Streams.

Config

Create your yaml based on the template:

cp conf/dynamo-cassandra-proxy.yaml.template conf/dynamo-cassandra-proxy.yaml

The following are the options supported by the proxy:

Option Description
streamsEnabled When set to true it enables the proxy to pull live data from an existing dynamodb table
dynamoRegion Only needed when streaming is enabled, region your dynamodb table is in
dyanmoAccessKey Only needed when streaming is enabled, used to connect to dynamodb streams
dyanmoSecretKey Only needed when streaming is enabled, used to connect to dynamodb streams
awsDynamodbEndpoint Only needed when streaming is enabled, used to connect to dynamodb streams
contactPoints Contact points to connect to Apache Cassandra(TM) cluster. If you are using the docker option just leave localhost
dockerCassandra When set to true it will stand up Cassandra in your local docker. Ensure the docker deamon is installed and running and your user has access to run docker ps

To run locally

Clone:

git clone [email protected]:datastax/dynamo-cassandra-proxy.git

Build:

mvn package

Run the app: whether you point the proxy at your own c* cluster or you rely on the proxy to stand up a cassandra node using the cassandraDocker option in the yaml. You can run the code locally by:

java -Ddw.contactPoints="$contactPoints" -cp target/dynamodb-cassandra-proxy-0.1.0.jar com.datastax.powertools.dcp.DCProxyApplication server conf/dynamo-cassandra-proxy.yaml

The proxy will come up and listen on port :8080. In your dynamodb application, just point your app to <host>:8080 in the SDK. A sample connection string (in Java) should look as follows:

        ClientConfiguration config = new ClientConfiguration();
        config.setMaxConnections(dynamodbMaxConnections);;
        String dynamodbEndpoint = "localhost:8080"
        String signinRegion = "dummy"
        AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder
                .EndpointConfiguration(protocol + "://" + dynamodbEndpoint, signinRegion);
        ddbBuilder = AmazonDynamoDBClientBuilder.standard()
                .withClientConfiguration(config)
                .withEndpointConfiguration(endpointConfiguration);

Note, MaxConnections is the main lever to get the AWS SDK to perform beyond very basic levels. We have tested this up to the maxium of 50 and it appears to scale almost linearly all the way up on a medum sized box. If you are doing some benchmarking and are looking to try to saturate a cassandra cluster, crank this value up.

To run via docker-compose

Build the app

mvn package

Build and run the docker containerst st

docker-compose up

To run in local k8s

Set up cassandra config map:

kubectl create configmap cassandra-config \

--from-file=common/cassandra/conf-dir/resources/cassandra/conf

Apply k8s yaml:

kubectl apply -f k8s-local/proxy-suite.yaml

At this point, your pods should look as follows:

$ kubectl get pods                                                                                     [2:34:13]
NAME                  READY   STATUS              RESTARTS   AGE
cassandra-0           1/1     Running             0          2m35s
cassandra-1           1/1     Running             0          168s
cassandra-3           1/1     Running             0          123s
dynamo-cass-proxy-0   1/1     Running             4          63s

To terminate your deploymet run:

kubectl delete -f k8s-local/proxy-suite.yaml

To run on GKE

Create cluster

If you do not have a GKE cluster yet, create one per the gcloud docs or use this sample commands:

create:

gcloud container clusters create dynamo-proxy-cluster \
    --cluster-version=1.12.5-gke.10 --zone us-west1-b \
    --machine-type n1-standard-4  --num-nodes 1

configure kubectl:

gcloud container clusters get-credentials k8-12-5-10-gke-n1-std-4 --zone us-west1-b

Deploy the proxy

Set up configMap

kubectl create configmap cassandra-config \
    --from-file=common/cassandra/conf-dir/resources/cassandra/conf

Apply k8s yaml:

kubectl apply -f gke/proxy-suite.yaml

The workload will appear on your google console like so (ensure you're viewing the right project):

gke

To terminate your deploymet run:

kubectl delete -f gke/proxy-suite.yaml

Contributing

A good place to start might be fleshing out your own Translator. For details on translators see Translators in the docs

MVP Roadmap:

Check means currently completed:

Other features not yet implemented:

License

This project is licensed under the Apache Public License 2.0