• Example Search
• Project Search

• Java
• Python
• JavaScript
• TypeScript
• C++
• Scala
• Blog

Project: learn-spark (GitHub Link)

• learn-spark-master
  • LICENSE
  • source-code
    • learn-spark
      • src
        • main
          • resources
            • tags_sample.json
            • question_tags_10K.csv
          • scala
            • com
              • allaboutscala
                • learn
                  • spark
                    • sql
                      • SparkSQL_Tutorial.scala
                    • functions
                      • Tutorial_04_ExplodeJson.scala
                      • Tutorial_07_CheckDataFrameColumnExist.scala
                      • Tutorial_05_ContatenateDataFrameColumnWise.scala
                      • Tutorial_08_SplitArrayColumn.scala
                      • Tutorial_15_DataFrameDropNull.scala
                      • Tutorial_11_DataFrameFirstRow.scala
                      • Tutorial_06_DataFrameColumnArrayContains.scala
                      • Tutorial_10_DataFrameColumnWithUDF.scala
                      • Tutorail_10_DataFrameConstantColumn.scala
                      • Tutorial_09_RenameDataFrameColumn.scala
                      • Tutorial_02_GetDataFrameColumnNames.scala
                      • Tutorial_03_DataFrameColumnNamesAndDataTypes.scala
                      • Tutorial_01_CreateDataFrameFromTuple.scala
                      • Tutorial_14_DataFrameStringFunctions.scala
                      • Tutorial_13_DataFrame_Column_Hashing.scala
                      • Tutorial_12_Formatting_DataFrame_Column.scala
                    • utils
                      • Context.scala
                    • dataframe
                      • DataFrame_Tutorial.scala
                      • DataFrameOperations.scala
                      • DataFrameStatistics_Tutorial.scala
      • project
        • build.properties
        • plugins.sbt
      • build.sbt
      • .gitignore
  • README.md
  • .gitignore

Source Code For Scala Tutorials From www.allaboutscala.com/big-data/spark/

Introduction

The Scala programming language is rapidly growing in popularity! Sadly, most of the online tutorials do not provide a step-by-step guide :(

At www.allaboutscala.com, we provide a complete beginner's tutorial to help you learn Scala in small, simple and easy steps.

• Tutorials are organized such that they allow you to learn gradually.
• Tutorials are written by taking into account the complete beginner.
• Tutorials will make you proficient with the same professional tools used by the Scala experts.

Learn Spark

• Code snippets for Learn Spark
• For additional details, please visit www.allaboutscala.com

The examples below are the source code for Spark Tutorials from allaboutscala.com

Project Setup

• StackOverflow dataset
• Add Apache Spark 2.0 SBT dependencies
• Bootstrap a SparkSession

DataFrame SQL Query

• DataFrame Introduction
• Create a DataFrame from reading a CSV file
• DataFrame schema
• Select columns from a dataframe
• Filter by column value of a dataframe
• Count rows of a dataframe
• SQL like query
• Multiple filter chaining
• SQL IN clause
• SQL Group By
• SQL Group By with filter
• SQL order by
• Cast columns to specific data type
• Operate on a filtered dataframe
• DataFrame Join
• Join and select columns
• Join on explicit columns
• Inner Join
• Left Outer Join
• Right Outer Join
• Distinct

Spark SQL

• Spark SQL Introduction
• Register temp table from dataframe
• List all tables in Spark's catalog
• List catalog tables using Spark SQL
• Select columns
• Filter by column value
• Count number of rows
• SQL like
• SQL where with and clause
• SQL IN clause
• SQL Group By
• SQL Group By with having clause
• SQL Order by
• Typed columns, filter and create temp table
• SQL Inner Join
• SQL Left Outer Join
• SQL Right Outer Join
• SQL Distinct
• Register User Defined Function (UDF)

DataFrame Statistics

• DataFrame Statistics Introduction
• Create DataFrame from CSV
• Average
• Maximum
• Minimum
• Mean
• Sum
• Group by query with statistics
• DataFrame Statistics using describe() method
• Correlation
• Covariance
• Frequent Items
• Crosstab
• Stratified sampling using sampleBy
• Approximate Quantile
• Bloom Filter
• Count Min Sketch
• Sampling With Replacement

DataFrame Operations

• DataFrame Operations Introduction
• Setup DataFrames
• Convert DataFrame row to Scala case class
• DataFrame row to Scala case class using map()
• Create DataFrame from collection
• DataFrame Union
• DataFrame Intersection
• Append column to DataFrame using withColumn()

Spark Functions

• Create DataFrame from Tuples
• Get DataFrame column names
• DataFrame column names and types
• Nested Json into DataFrame using explode method
• Concatenate DataFrames using join() method
• Search DataFrame column using array_contains() method
• Check DataFrame column exists
• Split DataFrame Array column
• Rename DataFrame column
• Create DataFrame constant column
• DataFrame new column with User Defined Function (UDF)
• DataFrame first row
• Format DataFrame column
• DataFrame column hashing
• DataFrame String functions
• DataFrame drop null

Stay in touch via Facebook and Twitter for upcoming tutorials!

Contact

Nadim Bahadoor at http://allaboutscala.com/contact/

License

Apache 2.0 License - see the LICENSE.TXT file