Real-Time Analytics with Spark Streaming

A solution describing data-processing design pattern for streaming data through Kinesis and Spark Streaming at real-time.

Overview

In today’s world, you need to continually innovate and improve your product to meet your customers’ needs. Require to provide an analytic ecosystem that scales with the business needs. Need to provide options in New Analytics Technologies by focusing on enabling new analytic approaches including programmatic Data Analysis. Analytics with kinesis and spark streaming helps to reliably process real-time data.

Processing vast amount of semi-structured data is a challenge between at what instance the data is collected and it availability for presentation. The common delay is due to requirement to validate or identify the coarse data. Often able to react immediately to new data is significant. AWS tool like EMR and Kinesis is used to process semi-structured or unstructured data from data streams at real-time. The real-time data processing is achieved with AWS Services such as Amazon Kinesis, Amazon EMR (Spark), Amazon S3.

When processing large amounts of semi-structured data, there is always a delay between the point when data is collected and its availability in dashboards. Often the delay results from the need to validate or at least identify coarse data. In some cases, however, being able to react immediately to new data is more important than being 100 percent certain of the data’s validity.

The AWS tool most frequently used to deal with large volumes of semi-structured or unstructured data is Amazon Elastic MapReduce (Amazon EMR). Stream or real-time processing, the processing of a constant flux of data, in real time, is possible with a Lambda Architecture solution that includes Amazon Kinesis, Amazon Simple Storage Service (Amazon S3), Spark Streaming, and Spark SQL on top of an Amazon EMR cluster.

Abstract

The solution focuses on functional areas such as real-time data from Kinesis is processed with spark streaming and data visualization is done with Zeppelin. The solution supports three options in spark submit mode

* Deploy a default DemoApp application
* Deploying a custom spark application for real-time streaming on an EMR cluster
* Visualize the real-time data and process the spark application code from Zeppelin UI

This design deploys a spark application written in high-level programming language Scala on a EMR cluster in an Amazon VPC, which is a logically isolated section of the Amazon Web Services (AWS). All the components of EMR with the spark application are deployed into the VPC and within a private subnet. The EMR web Interfaces are enabled by Port forwarding from a bastion host which internally forwards to the EMR master node.

Cloudformation Templates

* aws-vpc.template
* producer.template
* real-time-analytics-spark-streaming.template
* real-time-analytics-spark-streaming-demo.template (To demo a streaming application)

Known issues

EMR security groups

When deleting the solution stack, the Amazon EMR cluster does not delete the following security groups (since they reference one another): ElasticMapReduce-Slave-Private, ElasticMapReduce-ServiceAccess, and ElasticMapReduce-Master-Private. This will cause the VPC stack deletion to fail, since a VPC cannot be deleted while there are still available security groups.

To remove all inbound and outbound rules for a security group, use the AWS Command Line Interface (AWS CLI) and enter the following commands, which must be executed for each group listed above. Using this method requires jq, a lightweight command-line JSON processor, to be installed.

SECURITY_GROUP_ID=<REPLACE_ME>
aws ec2 describe-security-groups --group-ids $SECURITY_GROUP_ID --output json | jq '.SecurityGroups[0].IpPermissions' > IpPermissions.json
aws ec2 describe-security-groups --group-ids $SECURITY_GROUP_ID --output json | jq '.SecurityGroups[0].IpPermissionsEgress' > IpPermissionsEgress.json
aws ec2 revoke-security-group-ingress --group-id $SECURITY_GROUP_ID --ip-permissions file://IpPermissions.json
aws ec2 revoke-security-group-egress --group-id $SECURITY_GROUP_ID --ip-permissions file://IpPermissionsEgress.json

Once all rules have been removed, the security group can be deleted:

aws ec2 delete-security-group --group-id $SECURITY_GROUP_ID

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.