Coverage Status

Datasource-Receiver

Datasource-Receiver is a library that allows the user to read data with SparkSQL from Datasources and then insert the data in Spark Streaming

Requirements

This library requires Spark 1.5+, Scala 2.10+

Using the library

There are two ways of using Datasource-Receiver library:

The first one is to add the next dependency in your pom.xml:

  <dependency>
    <groupId>com.stratio.receiver</groupId>
    <artifactId>datasource</artifactId>
    <version>LATEST</version>
  </dependency>

The other one is to clone the full repository and build the project:

  git clone https://github.com/Stratio/Datasource-Receiver.git
  mvn clean install

The basic idea is receive data in Spark Streaming with Spark Sql:

SparkSQL -> SparkStreaming

Some use cases are monitoring tables, migrating data, generate streaming data from batch data... The use of SparkSQL have the advantage that the library is polyglot and can read data from multiple databases, like JDBC, Cassandra, ElasticSearch, MongoDB ... The user must add the java jar with the connector library and then is possible to use it in the initial query. Per example:

Distributed Approach with SparkSQL

The distributed approach is based in SparkSQL, on each streaming window the receiver execute one query with SparkSQL and generate one DataFrame and the partitions of this DataFrame corresponds with the DStream partitions.

Datasource Receiver distributed with SparkSQL

With the receiver is possible to monitoring tables in the datasources, the user specify one offset field and on each window the offset is updated to generate the new incremental query.

Incremental Queries

The receiver have implemented one Object for specify the offset options, limit the results and generate stop conditions. One example:

   InputSentences(
         query = "select * from tableName",
         offsetConditions = OffsetConditions(OffsetField(name = "followersCount")),
         stopConditions = StopConditions(stopWhenEmpty = true, finishContextWhenEmpty = true),
         initialStatements = Seq("CREATE TEMPORARY TABLE tableName USING com.stratio.datasource.mongodb OPTIONS (host 'localhost', database 'sparta', collection 'streamtrigger')")
   )

Build

There are two modules to package with different versions of Spark

Scala API

Java API

Spark Parameters Options

Parameter Description Optional
stopGracefully Stop Streaming Context gracefully Yes (default: true)
stopSparkContext Stop SparkContext when stop Streaming Yes (default: false)
storageLevel Max time to receive messages Yes (default: MEMORY_ONLY)
rememberDuration Remember duration for Spark Dstreams Yes (default: 60s)

License

Licensed to STRATIO (C) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The STRATIO (C) licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.