Yggdrasil is a more efficient way in Apache Spark to train decision trees for large depths and datasets with a high number of features. For depths greater than 10, Yggdrasil is an order of magnitude faster than Spark MLlib v1.6.0.
Add the dependency to your SBT project by adding the following to build.sbt
(see the Spark Packages
listing for
spark-submit and Maven instructions):
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"
libraryDependencies += "fabuzaid21" % "yggdrasil" % "1.0"Then use Yggdrasil as follows:
import org.apache.spark.ml.tree.impl.YggdrasilClassifier // YgddrasilRegressor
// Identical to the Spark MLlib Decision Tree API
val dt = new YggdrasilClassifier()
.setFeaturesCol("indexedFeatures")
.setLabelCol(labelColName)
.setMaxDepth(params.maxDepth)
.setMaxBins(params.maxBins)
.setMinInstancesPerNode(params.minInstancesPerNode)
.setMinInfoGain(params.minInfoGain)
.setCacheNodeIds(params.cacheNodeIds)
.setCheckpointInterval(params.checkpointInterval)