Java Code Examples for org.apache.flink.fs.s3.common.writer.S3AccessHelper

The following examples show how to use org.apache.flink.fs.s3.common.writer.S3AccessHelper. These examples are extracted from open source projects. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source Project: Flink-CEPplus   Source File: FlinkS3FileSystem.java    License: Apache License 2.0 5 votes vote down vote up
/**
 * Creates a FlinkS3FileSystem based on the given Hadoop S3 file system.
 * The given Hadoop file system object is expected to be initialized already.
 *
 * <p>This constructor additionally configures the entropy injection for the file system.
 *
 * @param hadoopS3FileSystem The Hadoop FileSystem that will be used under the hood.
 * @param entropyInjectionKey The substring that will be replaced by entropy or removed.
 * @param entropyLength The number of random alphanumeric characters to inject as entropy.
 */
public FlinkS3FileSystem(
		org.apache.hadoop.fs.FileSystem hadoopS3FileSystem,
		String localTmpDirectory,
		@Nullable String entropyInjectionKey,
		int entropyLength,
		@Nullable S3AccessHelper s3UploadHelper,
		long s3uploadPartSize,
		int maxConcurrentUploadsPerStream) {

	super(hadoopS3FileSystem);

	if (entropyInjectionKey != null && entropyLength <= 0) {
		throw new IllegalArgumentException("Entropy length must be >= 0 when entropy injection key is set");
	}

	this.entropyInjectionKey = entropyInjectionKey;
	this.entropyLength = entropyLength;

	// recoverable writer parameter configuration initialization
	this.localTmpDir = Preconditions.checkNotNull(localTmpDirectory);
	this.tmpFileCreator = RefCountedTmpFileCreator.inDirectories(new File(localTmpDirectory));
	this.s3AccessHelper = s3UploadHelper;
	this.uploadThreadPool = Executors.newCachedThreadPool();

	Preconditions.checkArgument(s3uploadPartSize >= S3_MULTIPART_MIN_PART_SIZE);
	this.s3uploadPartSize = s3uploadPartSize;
	this.maxConcurrentUploadsPerStream = maxConcurrentUploadsPerStream;
}
 
Example 2
Source Project: flink   Source File: FlinkS3FileSystem.java    License: Apache License 2.0 5 votes vote down vote up
/**
 * Creates a FlinkS3FileSystem based on the given Hadoop S3 file system.
 * The given Hadoop file system object is expected to be initialized already.
 *
 * <p>This constructor additionally configures the entropy injection for the file system.
 *
 * @param hadoopS3FileSystem The Hadoop FileSystem that will be used under the hood.
 * @param entropyInjectionKey The substring that will be replaced by entropy or removed.
 * @param entropyLength The number of random alphanumeric characters to inject as entropy.
 */
public FlinkS3FileSystem(
		org.apache.hadoop.fs.FileSystem hadoopS3FileSystem,
		String localTmpDirectory,
		@Nullable String entropyInjectionKey,
		int entropyLength,
		@Nullable S3AccessHelper s3UploadHelper,
		long s3uploadPartSize,
		int maxConcurrentUploadsPerStream) {

	super(hadoopS3FileSystem);

	if (entropyInjectionKey != null && entropyLength <= 0) {
		throw new IllegalArgumentException("Entropy length must be >= 0 when entropy injection key is set");
	}

	this.entropyInjectionKey = entropyInjectionKey;
	this.entropyLength = entropyLength;

	// recoverable writer parameter configuration initialization
	this.localTmpDir = Preconditions.checkNotNull(localTmpDirectory);
	this.tmpFileCreator = RefCountedTmpFileCreator.inDirectories(new File(localTmpDirectory));
	this.s3AccessHelper = s3UploadHelper;
	this.uploadThreadPool = Executors.newCachedThreadPool();

	Preconditions.checkArgument(s3uploadPartSize >= S3_MULTIPART_MIN_PART_SIZE);
	this.s3uploadPartSize = s3uploadPartSize;
	this.maxConcurrentUploadsPerStream = maxConcurrentUploadsPerStream;
}
 
Example 3
Source Project: flink   Source File: FlinkS3FileSystem.java    License: Apache License 2.0 5 votes vote down vote up
/**
 * Creates a FlinkS3FileSystem based on the given Hadoop S3 file system.
 * The given Hadoop file system object is expected to be initialized already.
 *
 * <p>This constructor additionally configures the entropy injection for the file system.
 *
 * @param hadoopS3FileSystem The Hadoop FileSystem that will be used under the hood.
 * @param entropyInjectionKey The substring that will be replaced by entropy or removed.
 * @param entropyLength The number of random alphanumeric characters to inject as entropy.
 */
public FlinkS3FileSystem(
		org.apache.hadoop.fs.FileSystem hadoopS3FileSystem,
		String localTmpDirectory,
		@Nullable String entropyInjectionKey,
		int entropyLength,
		@Nullable S3AccessHelper s3UploadHelper,
		long s3uploadPartSize,
		int maxConcurrentUploadsPerStream) {

	super(hadoopS3FileSystem);

	if (entropyInjectionKey != null && entropyLength <= 0) {
		throw new IllegalArgumentException("Entropy length must be >= 0 when entropy injection key is set");
	}

	this.entropyInjectionKey = entropyInjectionKey;
	this.entropyLength = entropyLength;

	// recoverable writer parameter configuration initialization
	this.localTmpDir = Preconditions.checkNotNull(localTmpDirectory);
	this.tmpFileCreator = RefCountedTmpFileCreator.inDirectories(new File(localTmpDirectory));
	this.s3AccessHelper = s3UploadHelper;
	this.uploadThreadPool = Executors.newCachedThreadPool();

	Preconditions.checkArgument(s3uploadPartSize >= S3_MULTIPART_MIN_PART_SIZE);
	this.s3uploadPartSize = s3uploadPartSize;
	this.maxConcurrentUploadsPerStream = maxConcurrentUploadsPerStream;
}
 
Example 4
Source Project: Flink-CEPplus   Source File: AbstractS3FileSystemFactory.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
protected abstract S3AccessHelper getS3AccessHelper(org.apache.hadoop.fs.FileSystem fs);
 
Example 5
Source Project: Flink-CEPplus   Source File: S3EntropyFsFactoryTest.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
	return null;
}
 
Example 6
Source Project: Flink-CEPplus   Source File: S3FileSystemFactory.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
	final S3AFileSystem s3Afs = (S3AFileSystem) fs;
	return new HadoopS3AccessHelper(s3Afs, s3Afs.getConf());
}
 
Example 7
Source Project: Flink-CEPplus   Source File: S3FileSystemFactory.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
	return null;
}
 
Example 8
Source Project: flink   Source File: AbstractS3FileSystemFactory.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
protected abstract S3AccessHelper getS3AccessHelper(org.apache.hadoop.fs.FileSystem fs);
 
Example 9
Source Project: flink   Source File: S3EntropyFsFactoryTest.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
	return null;
}
 
Example 10
Source Project: flink   Source File: S3FileSystemFactory.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
	final S3AFileSystem s3Afs = (S3AFileSystem) fs;
	return new HadoopS3AccessHelper(s3Afs, s3Afs.getConf());
}
 
Example 11
Source Project: flink   Source File: S3FileSystemFactory.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
	return null;
}
 
Example 12
Source Project: flink   Source File: AbstractS3FileSystemFactory.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
protected abstract S3AccessHelper getS3AccessHelper(org.apache.hadoop.fs.FileSystem fs);
 
Example 13
Source Project: flink   Source File: S3EntropyFsFactoryTest.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
	return null;
}
 
Example 14
Source Project: flink   Source File: S3FileSystemFactory.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
	final S3AFileSystem s3Afs = (S3AFileSystem) fs;
	return new HadoopS3AccessHelper(s3Afs, s3Afs.getConf());
}
 
Example 15
Source Project: flink   Source File: S3FileSystemFactory.java    License: Apache License 2.0 4 votes vote down vote up
@Nullable
@Override
protected S3AccessHelper getS3AccessHelper(FileSystem fs) {
	return null;
}