Java library and command-line application for converting Apache Spark ML pipelines to PMML.
feature.Binarizer
feature.Bucketizer
feature.ChiSqSelectorModel
(the result of fitting a feature.ChiSqSelector
)feature.ColumnPruner
feature.CountVectorizerModel
(the result of fitting a feature.CountVectorizer
)feature.IDFModel
(the result of fitting a feature.IDF
)feature.ImputerModel
(the result of fitting a feature.Imputer
)feature.IndexToString
feature.Interaction
feature.MaxAbsScalerModel
(the result of fitting a feature.MaxAbsScaler
)feature.MinMaxScalerModel
(the result of fitting a feature.MinMaxScaler
)feature.NGram
feature.OneHotEncoderModel
(the result of fitting a feature.OneHotEncoder
)feature.PCAModel
(the result of fitting a feature.PCA
)feature.QuantileDiscretizer
feature.RegexTokenizer
feature.RFormulaModel
(the result of fitting a feature.RFormula
)feature.SQLTransformer
case when
and if
.+
, -
, *
and /
.<
, <=
, ==
, >=
and >
.and
, or
and not
.abs
, ceil
, exp
, floor
, ln
, log10
, pow
and rint
.regexp_replace
and rlike
.concat
, lower
, substring
, trim
and upper
.boolean
, cast
, double
, int
and string
.in
, isnull
, isnotnull
, negative
and positive
.feature.StandardScalerModel
(the result of fitting a feature.StandardScaler
)feature.StopWordsRemover
feature.StringIndexerModel
(the result of fitting a feature.StringIndexer
)feature.Tokenizer
feature.VectorAssembler
feature.VectorAttributeRewriter
feature.VectorIndexerModel
(the result of fitting a feature.VectorIndexer
)feature.VectorSizeHint
feature.VectorSlicer
classification.DecisionTreeClassificationModel
classification.GBTClassificationModel
classification.LinearSVCModel
classification.LogisticRegressionModel
classification.MultilayerPerceptronClassificationModel
classification.NaiveBayesModel
classification.RandomForestClassificationModel
clustering.KMeansModel
regression.DecisionTreeRegressionModel
regression.GBTRegressionModel
regression.GeneralizedLinearRegressionModel
regression.LinearRegressionModel
regression.RandomForestRegressionModel
PipelineModel
HasPredictionCol#getPredictionCol()
) of earlier clustering, classification and regression models.HasProbabilityCol#getProbabilityCol()
) of earlier classification models.tuning.CrossValidatorModel
tuning.TrainValidationSplitModel
JPMML-SparkML library JAR file (together with accompanying Java source and Javadocs JAR files) is released via Maven Central Repository.
The current version is 1.6.1 (23 June, 2020).
<dependency>
<groupId>org.jpmml</groupId>
<artifactId>jpmml-sparkml</artifactId>
<version>1.6.1</version>
</dependency>
Compatibility matrix:
Apache Spark version | JPMML-SparkML branch | Status |
---|---|---|
1.5.X and 1.6.X | 1.0.X |
Archived |
2.0.X | 1.1.X |
Archived |
2.1.X | 1.2.X |
Archived |
2.2.X | 1.3.X |
Archived |
2.3.X | 1.4.X |
Active |
2.4.X | 1.5.X |
Active |
3.0.X | master |
Active |
JPMML-SparkML depends on the latest and greatest version of the JPMML-Model library, which is in conflict with the legacy version that is part of Apache Spark version 2.0.X, 2.1.X and 2.2.X distributions.
This conflict is documented in SPARK-15526. For possible resolutions, please switch from this README.md file to the README.md file of some earlier JPMML-SparkML development branch.
Enter the project root directory and build using Apache Maven:
mvn clean install
The build produces two JAR files:
target/jpmml-sparkml-1.6-SNAPSHOT.jar
- Library JAR file.target/jpmml-sparkml-executable-1.6-SNAPSHOT.jar
- Example application JAR file.Fitting a Spark ML pipeline that only makes use of supported Transformer types:
DataFrame irisData = ...;
StructType schema = irisData.schema();
RFormula formula = new RFormula()
.setFormula("Species ~ .");
DecisionTreeClassifier classifier = new DecisionTreeClassifier()
.setLabelCol(formula.getLabelCol())
.setFeaturesCol(formula.getFeaturesCol());
Pipeline pipeline = new Pipeline()
.setStages(new PipelineStage[]{formula, classifier});
PipelineModel pipelineModel = pipeline.fit(irisData);
Converting the Spark ML pipeline to PMML using the org.jpmml.sparkml.PMMLBuilder
builder class:
PMML pmml = new PMMLBuilder(schema, pipelineModel)
.build();
// Viewing the result
JAXBUtil.marshalPMML(pmml, new StreamResult(System.out));
The example application JAR file contains an executable class org.jpmml.sparkml.Main
, which can be used to convert a pair of serialized org.apache.spark.sql.types.StructType
and org.apache.spark.ml.PipelineModel
objects to PMML.
The example application JAR file does not include Apache Spark runtime libraries. Therefore, this executable class must be executed using Apache Spark's spark-submit
helper script.
For example, converting a pair of Spark ML schema and pipeline serialization files src/test/resources/schema/Iris.json
and src/test/resources/pipeline/DecisionTreeIris.zip
, respectively, to a PMML file DecisionTreeIris.pmml
:
spark-submit --master local --class org.jpmml.sparkml.Main target/jpmml-sparkml-executable-1.6-SNAPSHOT.jar --schema-input src/test/resources/schema/Iris.json --pipeline-input src/test/resources/pipeline/DecisionTreeIris.zip --pmml-output DecisionTreeIris.pmml
Getting help:
spark-submit --master local --class org.jpmml.sparkml.Main target/jpmml-sparkml-executable-1.6-SNAPSHOT.jar --help
JPMML-SparkML is licensed under the terms and conditions of the GNU Affero General Public License, Version 3.0.
If you would like to use JPMML-SparkML in a proprietary software project, then it is possible to enter into a licensing agreement which makes JPMML-SparkML available under the terms and conditions of the BSD 3-Clause License instead.
JPMML-SparkML is developed and maintained by Openscoring Ltd, Estonia.
Interested in using Java PMML API software in your company? Please contact [email protected]