sev7e0/wow-spark

Project: wow-spark (GitHub Link)

wow-spark-master
- src
  - main
    - resources
      - log4j.properties
      - Stage划分算法.md
      - sparkresource
        users.avro
        users.parquet
        device.json
        people.csv
        student.txt
        user.avsc
        people.txt
        full_user.avsc
        kv1.txt
        student-question.txt
        users.dat
        users
        people.json
        employees.json
      - Worker相关分析
        Worker.scala
        DriverRunner.scala
        ExecutorRunner.scala
      - core
        SparkContext.md
        readme.md
      - Master相关分析
        Master源码.code
    - scala
      - com
        sev7e0
        wow
        sql
        A_3_AvroDataSource.scala
        A_0_LoadSaveFunction.scala
        A_5_Json.scala
        A_8_MyAverage.scala
        A_1_DataFrameTest.scala
        A_9_MyAverageByAggregator.scala
        A_6_HiveTables.scala
        A_7_Parquet.scala
        README.md
        A_4_JoinDF.scala
        A_2_DataSetTest.scala
        kafka
        st_56980436.java
        IntCountByStructured.scala
        CountIntByStreaming.scala
        CountIntByStreaming_java.java
        README.md
        spark-streaming消费kafka的两种方式.md
        utils
        TestUtils.scala
        JavaSparkContextUtil.java
        hbase
        Hdfs2HBase.scala
        demo.java
        elasticsearch
        SparkStreamingWriteToEs.java
        SparkStreamingReadFromEs.java
        LocalConfiguration.java
        scala
        7-Functional.scala
        package.scala
        A_10_ClassConstructor.scala
        2-Pattern.scala
        6-FutureAndPromise.scala
        1-Extractor.scala
        TypeClass.scala
        4-Either.scala
        2-PatternMatch.scala
        demo.scala
        3-ErrorHandling.scala
        5-Future.scala
        A_9_InnerMethod.scala
        8-PartialApplicationFunctions.scala
        README.md
        8-DependencyInjection.scala
        DependencyClass.scala
        TypeClass2.scala
        Associated.scala
        delta
        QuickStart.scala
        README.md
        structured_streaming
        A_2_JoinOperations.scala
        A_6_ContinuousProcessing.scala
        A_4_StreamingQuery.scala
        A_3_StreamDeduplication.scala
        A_5_StreamingWordWcount.scala
        A_7_WriteStream.scala
        A_2_JoinOperation.scala
        A_1_WindowOperation.scala
        A_1_BasicOperation.scala
        README.md
        App.scala
        core
        scala
        TransformationOperation_Scala.scala
        PV.scala
        LineCount_Scala.scala
        SecondarySortKey.scala
        UV.scala
        ActionOperation_Scala.scala
        SecondarySort.scala
        TopN_Scala.scala
        BroadCastVariable_Scala.scala
        java
        PartitionerTestJava.java
        ActionOperation_Java.java
        SecondarySortKey.java
        SortWordCount_Java.java
        CheckPointJava.java
        AccumulatorVariable_Java.java
        TopNGroup_Java.java
        SecondarySort_Java.java
        README.md
        WordCountLocal.java
        Persist.java
        BroadCastVariable_Java.java
        TransformationOperation_Java.java
        TopN_Java.java
        LineCount_Java.java
        spark_streaming
        QueueStream.scala
        StreamingUV.scala
        7_RecoverableNetworkWordCount.scala
        5_DataFrameAndSql.scala
        NetworkWordCount.scala
        3_TransformationsDStream.scala
        2_DiscretizedStreams.scala
        3_TransformationDStream.scala
        README.md
        spark turning.md
        4_OutputOperationDStream.scala
        1_InitializingStreamingContext.scala
        StreamingLogger.scala
        StreamingDistinct.scala
        6_Checkpoint.scala
  - test
    - scala
      - com
        sev7e0
        wow
        ScalaTest.scala
- pom.xml
- LICENSE
- run.sh
- README.md
- .gitignore

wow-spark

Spark modules and Scala practice, At the same time, have some source code analysis.

Spark自学记录手册

Environment

	tool	version
1	Java	8
2	Scala	2.12.8
3	Maven	3.5.4
4	Spark	2.4.0
5	Kafka	2.2.0

wow-spark

Environment

List

Spark Core

Spark SQL

Structured Streaming

Spark + Kafka

Spark Streaming

Delta Lake

Spark Source Code Analysis

Scala

MyBlog