Java Code Examples for org.apache.spark.sql.SparkSession#Builder

The following examples show how to use org.apache.spark.sql.SparkSession#Builder . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: IcebergSourceBenchmark.java    From iceberg with Apache License 2.0 5 votes vote down vote up 
protected void setupSpark(boolean enableDictionaryEncoding) {
  SparkSession.Builder builder = SparkSession.builder()
          .config("spark.ui.enabled", false);
  if (!enableDictionaryEncoding) {
    builder.config("parquet.dictionary.page.size", "1")
            .config("parquet.enable.dictionary", false)
            .config(TableProperties.PARQUET_DICT_SIZE_BYTES, "1");
  }
  builder.master("local");
  spark = builder.getOrCreate();
  Configuration sparkHadoopConf = spark.sessionState().newHadoopConf();
  hadoopConf.forEach(entry -> sparkHadoopConf.set(entry.getKey(), entry.getValue()));
}
Example 2
Source File: SparkSessionUtil.java    From jpmml-sparkml with GNU Affero General Public License v3.0 5 votes vote down vote up 
static
public SparkSession createSparkSession(){
	SparkSession.Builder builder = SparkSession.builder()
		.appName("test")
		.master("local[1]")
		.config("spark.ui.enabled", false);

	SparkSession sparkSession = builder.getOrCreate();

	SparkContext sparkContext = sparkSession.sparkContext();
	sparkContext.setLogLevel("ERROR");

	return sparkSession;
}
Example 3
Source File: SparkFactoryImpl.java    From beakerx with Apache License 2.0 5 votes vote down vote up 
private Optional<SparkSessionBuilder> getBuilderFromUser(Object result) {
  if (result instanceof SparkConf) {
    return of(sparkSessionBuilderFactory.newInstance((SparkConf) result));
  } else if (result instanceof SparkSession.Builder) {
    return of(sparkSessionBuilderFactory.newInstance((SparkSession.Builder) result));
  } else {
    return Optional.empty();
  }
}
Example 4
Source File: SparkSessionBuilderImpl.java    From beakerx with Apache License 2.0 5 votes vote down vote up 
private Field getOptionsField(SparkSession.Builder sparkSessionBuilder) {
  Field[] declaredFields = sparkSessionBuilder.getClass().getDeclaredFields();
  Optional<Field> options = Arrays.stream(declaredFields).filter(f -> f.getName().contains("options")).findFirst();
  if (options.isPresent()) {
    return options.get();
  }
  throw new RuntimeException("SparkSession.builder does not contain 'options' field.");
}
Example 5
Source File: HoodieExampleSparkUtils.java    From hudi with Apache License 2.0 4 votes vote down vote up 
public static SparkSession buildSparkSession(String appName, Map<String, String> additionalConfigs) {

    SparkSession.Builder builder = SparkSession.builder().appName(appName);
    additionalConfigs.forEach(builder::config);
    return builder.getOrCreate();
  }
Example 6
Source File: SparkSessionBuilderFactoryImpl.java    From beakerx with Apache License 2.0 4 votes vote down vote up 
@Override
public SparkSessionBuilder newInstance(SparkSession.Builder builder) {
  return new SparkSessionBuilderImpl(builder);
}
Example 7
Source File: SparkSessionBuilderImpl.java    From beakerx with Apache License 2.0 4 votes vote down vote up 
public SparkSessionBuilderImpl(SparkSession.Builder sparkSessionBuilder) {
  this.sparkSessionBuilder = sparkSessionBuilder;
}
Example 8
Source File: SparkSessionBuilderFactoryMock.java    From beakerx with Apache License 2.0 4 votes vote down vote up 
@Override
public SparkSessionBuilder newInstance(SparkSession.Builder builder) {
  return new SparkSessionBuilderImpl(builder);
}
Example 9
Source File: SparkSessionBuilderFactory.java    From beakerx with Apache License 2.0 votes vote down vote up 
SparkSessionBuilder newInstance(SparkSession.Builder builder);