org.apache.parquet.avro.AvroWriteSupport Java Examples

The following examples show how to use org.apache.parquet.avro.AvroWriteSupport. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: ParquetUtils.java    From nifi with Apache License 2.0 6 votes vote down vote up 
public static void applyCommonConfig(Configuration conf, ParquetConfig parquetConfig) {
    if (parquetConfig.getAvroReadCompatibility() != null) {
        conf.setBoolean(AvroReadSupport.AVRO_COMPATIBILITY,
                parquetConfig.getAvroReadCompatibility().booleanValue());
    }

    if (parquetConfig.getAvroAddListElementRecords() != null) {
        conf.setBoolean(AvroSchemaConverter.ADD_LIST_ELEMENT_RECORDS,
                parquetConfig.getAvroAddListElementRecords().booleanValue());
    }

    if (parquetConfig.getAvroWriteOldListStructure() != null) {
        conf.setBoolean(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
                parquetConfig.getAvroWriteOldListStructure().booleanValue());
    }
}
Example #2
Source File: Parquet.java    From iceberg with Apache License 2.0 5 votes vote down vote up 
@SuppressWarnings("unchecked")
private <T> WriteSupport<T> getWriteSupport(MessageType type) {
  if (writeSupport != null) {
    return (WriteSupport<T>) writeSupport;
  } else {
    return new AvroWriteSupport<>(
        type,
        ParquetAvro.parquetAvroSchema(AvroSchemaUtil.convert(schema, name)),
        ParquetAvro.DEFAULT_MODEL);
  }
}
Example #3
Source File: Parquet.java    From iceberg with Apache License 2.0 5 votes vote down vote up 
@SuppressWarnings("unchecked")
private <T> WriteSupport<T> getWriteSupport(MessageType type) {
  if (writeSupport != null) {
    return (WriteSupport<T>) writeSupport;
  } else {
    return new AvroWriteSupport<>(
        type,
        ParquetAvro.parquetAvroSchema(AvroSchemaUtil.convert(schema, name)),
        ParquetAvro.DEFAULT_MODEL);
  }
}
Example #4
Source File: ParquetHdfsFileSink.java    From components with Apache License 2.0 5 votes vote down vote up 
@Override
protected void configure(Job job, KV<Void, IndexedRecord> sample) {
    super.configure(job, sample);
    IndexedRecord record = (IndexedRecord) sample.getValue();
    AvroWriteSupport.setSchema(job.getConfiguration(), record.getSchema());
    ParquetOutputFormat.setCompression(job, CompressionCodecName.SNAPPY);
}
Example #5
Source File: AvroParquetWriterBuilder.java    From datacollector with Apache License 2.0 4 votes vote down vote up 
protected WriteSupport<T> getWriteSupport(Configuration conf) {
  return new AvroWriteSupport((new AvroSchemaConverterLogicalTypesPre19(conf)).convert(this.schema), this.schema, this.model);
}
Example #6
Source File: AvroParquetOutputFormat.java    From parquet-mr with Apache License 2.0 4 votes vote down vote up 
public AvroParquetOutputFormat() {
  super(new AvroWriteSupport<T>());
}
Example #7
Source File: AvroParquetOutputFormat.java    From parquet-mr with Apache License 2.0 2 votes vote down vote up 
/**
 * Set the Avro schema to use for writing. The schema is translated into a Parquet
 * schema so that the records can be written in Parquet format. It is also
 * stored in the Parquet metadata so that records can be reconstructed as Avro
 * objects at read time without specifying a read schema.
 * @param job a job
 * @param schema a schema for the data that will be written
 * @see org.apache.parquet.avro.AvroParquetInputFormat#setAvroReadSchema(org.apache.hadoop.mapreduce.Job, org.apache.avro.Schema)
 */
public static void setSchema(Job job, Schema schema) {
  AvroWriteSupport.setSchema(ContextUtil.getConfiguration(job), schema);
}
Example #8
Source File: AvroParquetOutputFormat.java    From parquet-mr with Apache License 2.0 2 votes vote down vote up 
/**
 * Sets the {@link AvroDataSupplier} class that will be used. The data
 * supplier provides instances of {@link org.apache.avro.generic.GenericData}
 * that are used to deconstruct records.
 *
 * @param job a {@link Job} to configure
 * @param supplierClass a supplier class
 */
public static void setAvroDataSupplier(
    Job job, Class<? extends AvroDataSupplier> supplierClass) {
  AvroWriteSupport.setAvroDataSupplier(ContextUtil.getConfiguration(job),
      supplierClass);
}