• Example Search
• Project Search

• Java
• Python
• JavaScript
• TypeScript
• C++
• Scala
• Blog

• example-spark-scala-read-and-write-from-hdfs-master
  • src
    • main
      • scala
        • io
          • saagie
            • example
              • hdfs
                • Main.scala
  • LICENSE
  • project
    • build.properties
    • assembly.sbt
    • plugins.sbt
  • README.md
  • build.sbt
  • .gitignore

Example for readind & writing into hdfs with Spark Scala

Package for saagie : sbt clean assembly and get the package in target.

Usage in local :

• sbt clean assembly
• spark-submit --class=io.saagie.example.hdfs.Main example-spark-scala-read-and-write-from-hdfs-assembly-1.1.jar "hdfs://hdfshost:8020/"

Usage in Saagie :

• sbt clean assembly (in local, to generate jar file)
• create new Spark Job
• upload the jar (target/scala-2.11/example-spark-scala-read-and-write-from-hdfs-assembly-1.1.jar)
• Replace MyClass with the full class name (ex : io.saagie.example.hdfs.Main)
• copy URL from HDFS connection details panel and add it in first argument in the command line
• choose Spark 2.1.0
• choose resources you want to allocate to the job
• create and launch.