Java Code Examples for org.apache.flink.fs.s3.common.utils.RefCountedFSOutputStream

The following examples show how to use org.apache.flink.fs.s3.common.utils.RefCountedFSOutputStream. These examples are extracted from open source projects. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
/**
 * Single constructor to initialize all. Actual setup of the parts happens in the
 * factory methods.
 */
S3RecoverableFsDataOutputStream(
		RecoverableMultiPartUpload upload,
		FunctionWithException<File, RefCountedFile, IOException> tempFileCreator,
		RefCountedFSOutputStream initialTmpFile,
		long userDefinedMinPartSize,
		long bytesBeforeCurrentPart) {

	checkArgument(bytesBeforeCurrentPart >= 0L);

	this.upload = checkNotNull(upload);
	this.tmpFileProvider = checkNotNull(tempFileCreator);
	this.userDefinedMinPartSize = userDefinedMinPartSize;

	this.fileStream = initialTmpFile;
	this.bytesBeforeCurrentPart = bytesBeforeCurrentPart;
}
 
Example 2
/**
 * Adds a part to the uploads without any size limitations.
 *
 * <p>This method is non-blocking and does not wait for the part upload to complete.
 *
 * @param file The file with the part data.
 *
 * @throws IOException If this method throws an exception, the RecoverableS3MultiPartUpload
 *                     should not be used any more, but recovered instead.
 */
@Override
public void uploadPart(RefCountedFSOutputStream file) throws IOException {
	// this is to guarantee that nobody is
	// writing to the file we are uploading.
	checkState(file.isClosed());

	final CompletableFuture<PartETag> future = new CompletableFuture<>();
	uploadsInProgress.add(future);

	final long partLength = file.getPos();
	currentUploadInfo.registerNewPart(partLength);

	file.retain(); // keep the file while the async upload still runs
	uploadThreadPool.execute(new UploadTask(s3AccessHelper, currentUploadInfo, file, future));
}
 
Example 3
/**
 * Creates a snapshot of this MultiPartUpload, from which the upload can be resumed.
 *
 * <p>Data buffered locally which is less than
 * {@link org.apache.flink.fs.s3.common.FlinkS3FileSystem#S3_MULTIPART_MIN_PART_SIZE S3_MULTIPART_MIN_PART_SIZE},
 * and cannot be uploaded as part of the MPU and set to S3 as independent objects.
 *
 * <p>This implementation currently blocks until all part uploads are complete and returns
 * a completed future.
 */
@Override
public S3Recoverable snapshotAndGetRecoverable(@Nullable final RefCountedFSOutputStream incompletePartFile) throws IOException {

	final String incompletePartObjectName = safelyUploadSmallPart(incompletePartFile);

	// make sure all other uploads are complete
	// this currently makes the method blocking,
	// to be made non-blocking in the future
	awaitPendingPartsUpload();

	final String objectName = currentUploadInfo.getObjectName();
	final String uploadId = currentUploadInfo.getUploadId();
	final List<PartETag> completedParts = currentUploadInfo.getCopyOfEtagsOfCompleteParts();
	final long sizeInBytes = currentUploadInfo.getExpectedSizeInBytes();

	if (incompletePartObjectName == null) {
		return new S3Recoverable(objectName, uploadId, completedParts, sizeInBytes);
	} else {
		return new S3Recoverable(objectName, uploadId, completedParts, sizeInBytes, incompletePartObjectName, incompletePartFile.getPos());
	}
}
 
Example 4
@Nullable
private String safelyUploadSmallPart(@Nullable RefCountedFSOutputStream file) throws IOException {

	if (file == null || file.getPos() == 0L) {
		return null;
	}

	// first, upload the trailing data file. during that time, other in-progress uploads may complete.
	final String incompletePartObjectName = createIncompletePartObjectName();
	file.retain();
	try {
		s3AccessHelper.putObject(incompletePartObjectName, file.getInputFile());
	}
	finally {
		file.release();
	}
	return incompletePartObjectName;
}
 
Example 5
UploadTask(
		final S3AccessHelper s3AccessHelper,
		final MultiPartUploadInfo currentUpload,
		final RefCountedFSOutputStream file,
		final CompletableFuture<PartETag> future) {

	checkNotNull(currentUpload);

	this.objectName = currentUpload.getObjectName();
	this.uploadId = currentUpload.getUploadId();
	this.partNumber = currentUpload.getNumberOfRegisteredParts();

	// these are limits put by Amazon
	checkArgument(partNumber >= 1  && partNumber <= 10_000);

	this.s3AccessHelper = checkNotNull(s3AccessHelper);
	this.file = checkNotNull(file);
	this.future = checkNotNull(future);
}
 
Example 6
Source Project: flink   Source File: S3RecoverableFsDataOutputStream.java    License: Apache License 2.0 6 votes vote down vote up
/**
 * Single constructor to initialize all. Actual setup of the parts happens in the
 * factory methods.
 */
S3RecoverableFsDataOutputStream(
		RecoverableMultiPartUpload upload,
		FunctionWithException<File, RefCountedFile, IOException> tempFileCreator,
		RefCountedFSOutputStream initialTmpFile,
		long userDefinedMinPartSize,
		long bytesBeforeCurrentPart) {

	checkArgument(bytesBeforeCurrentPart >= 0L);

	this.upload = checkNotNull(upload);
	this.tmpFileProvider = checkNotNull(tempFileCreator);
	this.userDefinedMinPartSize = userDefinedMinPartSize;

	this.fileStream = initialTmpFile;
	this.bytesBeforeCurrentPart = bytesBeforeCurrentPart;
}
 
Example 7
Source Project: flink   Source File: RecoverableMultiPartUploadImpl.java    License: Apache License 2.0 6 votes vote down vote up
/**
 * Adds a part to the uploads without any size limitations.
 *
 * <p>This method is non-blocking and does not wait for the part upload to complete.
 *
 * @param file The file with the part data.
 *
 * @throws IOException If this method throws an exception, the RecoverableS3MultiPartUpload
 *                     should not be used any more, but recovered instead.
 */
@Override
public void uploadPart(RefCountedFSOutputStream file) throws IOException {
	// this is to guarantee that nobody is
	// writing to the file we are uploading.
	checkState(file.isClosed());

	final CompletableFuture<PartETag> future = new CompletableFuture<>();
	uploadsInProgress.add(future);

	final long partLength = file.getPos();
	currentUploadInfo.registerNewPart(partLength);

	file.retain(); // keep the file while the async upload still runs
	uploadThreadPool.execute(new UploadTask(s3AccessHelper, currentUploadInfo, file, future));
}
 
Example 8
Source Project: flink   Source File: RecoverableMultiPartUploadImpl.java    License: Apache License 2.0 6 votes vote down vote up
/**
 * Creates a snapshot of this MultiPartUpload, from which the upload can be resumed.
 *
 * <p>Data buffered locally which is less than
 * {@link org.apache.flink.fs.s3.common.FlinkS3FileSystem#S3_MULTIPART_MIN_PART_SIZE S3_MULTIPART_MIN_PART_SIZE},
 * and cannot be uploaded as part of the MPU and set to S3 as independent objects.
 *
 * <p>This implementation currently blocks until all part uploads are complete and returns
 * a completed future.
 */
@Override
public S3Recoverable snapshotAndGetRecoverable(@Nullable final RefCountedFSOutputStream incompletePartFile) throws IOException {

	final String incompletePartObjectName = safelyUploadSmallPart(incompletePartFile);

	// make sure all other uploads are complete
	// this currently makes the method blocking,
	// to be made non-blocking in the future
	awaitPendingPartsUpload();

	final String objectName = currentUploadInfo.getObjectName();
	final String uploadId = currentUploadInfo.getUploadId();
	final List<PartETag> completedParts = currentUploadInfo.getCopyOfEtagsOfCompleteParts();
	final long sizeInBytes = currentUploadInfo.getExpectedSizeInBytes();

	if (incompletePartObjectName == null) {
		return new S3Recoverable(objectName, uploadId, completedParts, sizeInBytes);
	} else {
		return new S3Recoverable(objectName, uploadId, completedParts, sizeInBytes, incompletePartObjectName, incompletePartFile.getPos());
	}
}
 
Example 9
Source Project: flink   Source File: RecoverableMultiPartUploadImpl.java    License: Apache License 2.0 6 votes vote down vote up
@Nullable
private String safelyUploadSmallPart(@Nullable RefCountedFSOutputStream file) throws IOException {

	if (file == null || file.getPos() == 0L) {
		return null;
	}

	// first, upload the trailing data file. during that time, other in-progress uploads may complete.
	final String incompletePartObjectName = createIncompletePartObjectName();
	file.retain();
	try {
		s3AccessHelper.putObject(incompletePartObjectName, file.getInputFile());
	}
	finally {
		file.release();
	}
	return incompletePartObjectName;
}
 
Example 10
Source Project: flink   Source File: RecoverableMultiPartUploadImpl.java    License: Apache License 2.0 6 votes vote down vote up
UploadTask(
		final S3AccessHelper s3AccessHelper,
		final MultiPartUploadInfo currentUpload,
		final RefCountedFSOutputStream file,
		final CompletableFuture<PartETag> future) {

	checkNotNull(currentUpload);

	this.objectName = currentUpload.getObjectName();
	this.uploadId = currentUpload.getUploadId();
	this.partNumber = currentUpload.getNumberOfRegisteredParts();

	// these are limits put by Amazon
	checkArgument(partNumber >= 1  && partNumber <= 10_000);

	this.s3AccessHelper = checkNotNull(s3AccessHelper);
	this.file = checkNotNull(file);
	this.future = checkNotNull(future);
}
 
Example 11
Source Project: flink   Source File: S3RecoverableFsDataOutputStream.java    License: Apache License 2.0 6 votes vote down vote up
/**
 * Single constructor to initialize all. Actual setup of the parts happens in the
 * factory methods.
 */
S3RecoverableFsDataOutputStream(
		RecoverableMultiPartUpload upload,
		FunctionWithException<File, RefCountedFileWithStream, IOException> tempFileCreator,
		RefCountedFSOutputStream initialTmpFile,
		long userDefinedMinPartSize,
		long bytesBeforeCurrentPart) {

	checkArgument(bytesBeforeCurrentPart >= 0L);

	this.upload = checkNotNull(upload);
	this.tmpFileProvider = checkNotNull(tempFileCreator);
	this.userDefinedMinPartSize = userDefinedMinPartSize;

	this.fileStream = initialTmpFile;
	this.bytesBeforeCurrentPart = bytesBeforeCurrentPart;
}
 
Example 12
Source Project: flink   Source File: RecoverableMultiPartUploadImpl.java    License: Apache License 2.0 6 votes vote down vote up
/**
 * Adds a part to the uploads without any size limitations.
 *
 * <p>This method is non-blocking and does not wait for the part upload to complete.
 *
 * @param file The file with the part data.
 *
 * @throws IOException If this method throws an exception, the RecoverableS3MultiPartUpload
 *                     should not be used any more, but recovered instead.
 */
@Override
public void uploadPart(RefCountedFSOutputStream file) throws IOException {
	// this is to guarantee that nobody is
	// writing to the file we are uploading.
	checkState(file.isClosed());

	final CompletableFuture<PartETag> future = new CompletableFuture<>();
	uploadsInProgress.add(future);

	final long partLength = file.getPos();
	currentUploadInfo.registerNewPart(partLength);

	file.retain(); // keep the file while the async upload still runs
	uploadThreadPool.execute(new UploadTask(s3AccessHelper, currentUploadInfo, file, future));
}
 
Example 13
Source Project: flink   Source File: RecoverableMultiPartUploadImpl.java    License: Apache License 2.0 6 votes vote down vote up
/**
 * Creates a snapshot of this MultiPartUpload, from which the upload can be resumed.
 *
 * <p>Data buffered locally which is less than
 * {@link org.apache.flink.fs.s3.common.FlinkS3FileSystem#S3_MULTIPART_MIN_PART_SIZE S3_MULTIPART_MIN_PART_SIZE},
 * and cannot be uploaded as part of the MPU and set to S3 as independent objects.
 *
 * <p>This implementation currently blocks until all part uploads are complete and returns
 * a completed future.
 */
@Override
public S3Recoverable snapshotAndGetRecoverable(@Nullable final RefCountedFSOutputStream incompletePartFile) throws IOException {

	final String incompletePartObjectName = safelyUploadSmallPart(incompletePartFile);

	// make sure all other uploads are complete
	// this currently makes the method blocking,
	// to be made non-blocking in the future
	awaitPendingPartsUpload();

	final String objectName = currentUploadInfo.getObjectName();
	final String uploadId = currentUploadInfo.getUploadId();
	final List<PartETag> completedParts = currentUploadInfo.getCopyOfEtagsOfCompleteParts();
	final long sizeInBytes = currentUploadInfo.getExpectedSizeInBytes();

	if (incompletePartObjectName == null) {
		return new S3Recoverable(objectName, uploadId, completedParts, sizeInBytes);
	} else {
		return new S3Recoverable(objectName, uploadId, completedParts, sizeInBytes, incompletePartObjectName, incompletePartFile.getPos());
	}
}
 
Example 14
Source Project: flink   Source File: RecoverableMultiPartUploadImpl.java    License: Apache License 2.0 6 votes vote down vote up
@Nullable
private String safelyUploadSmallPart(@Nullable RefCountedFSOutputStream file) throws IOException {

	if (file == null || file.getPos() == 0L) {
		return null;
	}

	// first, upload the trailing data file. during that time, other in-progress uploads may complete.
	final String incompletePartObjectName = createIncompletePartObjectName();
	file.retain();
	try {
		s3AccessHelper.putObject(incompletePartObjectName, file.getInputFile());
	}
	finally {
		file.release();
	}
	return incompletePartObjectName;
}
 
Example 15
Source Project: flink   Source File: RecoverableMultiPartUploadImpl.java    License: Apache License 2.0 6 votes vote down vote up
UploadTask(
		final S3AccessHelper s3AccessHelper,
		final MultiPartUploadInfo currentUpload,
		final RefCountedFSOutputStream file,
		final CompletableFuture<PartETag> future) {

	checkNotNull(currentUpload);

	this.objectName = currentUpload.getObjectName();
	this.uploadId = currentUpload.getUploadId();
	this.partNumber = currentUpload.getNumberOfRegisteredParts();

	// these are limits put by Amazon
	checkArgument(partNumber >= 1  && partNumber <= 10_000);

	this.s3AccessHelper = checkNotNull(s3AccessHelper);
	this.file = checkNotNull(file);
	this.future = checkNotNull(future);
}
 
Example 16
private static byte[] readFileContents(RefCountedFSOutputStream file) throws IOException {
	final byte[] content = new byte[MathUtils.checkedDownCast(file.getPos())];
	File inputFile = file.getInputFile();
	long bytesRead = new FileInputStream(inputFile).read(content, 0,  MathUtils.checkedDownCast(inputFile.length()));
	Assert.assertEquals(file.getPos(), bytesRead);
	return content;
}
 
Example 17
@Override
public RecoverableWriter.ResumeRecoverable snapshotAndGetRecoverable(RefCountedFSOutputStream incompletePartFile) throws IOException {
	lastPersistedIndex = uploadedContent.size();

	if (incompletePartFile.getPos() >= 0L) {
		byte[] bytes = readFileContents(incompletePartFile);
		uncompleted = Optional.of(bytes);
	}

	return null;
}
 
Example 18
@Override
public void uploadPart(RefCountedFSOutputStream file) throws IOException {
	numParts++;
	numBytes += file.getPos();

	uploadedContent.add(readFileContents(file));
}
 
Example 19
Source Project: flink   Source File: S3RecoverableFsDataOutputStreamTest.java    License: Apache License 2.0 5 votes vote down vote up
private static byte[] readFileContents(RefCountedFSOutputStream file) throws IOException {
	final byte[] content = new byte[MathUtils.checkedDownCast(file.getPos())];
	File inputFile = file.getInputFile();
	long bytesRead = new FileInputStream(inputFile).read(content, 0,  MathUtils.checkedDownCast(inputFile.length()));
	Assert.assertEquals(file.getPos(), bytesRead);
	return content;
}
 
Example 20
Source Project: flink   Source File: S3RecoverableFsDataOutputStreamTest.java    License: Apache License 2.0 5 votes vote down vote up
@Override
public RecoverableWriter.ResumeRecoverable snapshotAndGetRecoverable(RefCountedFSOutputStream incompletePartFile) throws IOException {
	lastPersistedIndex = uploadedContent.size();

	if (incompletePartFile.getPos() >= 0L) {
		byte[] bytes = readFileContents(incompletePartFile);
		uncompleted = Optional.of(bytes);
	}

	return null;
}
 
Example 21
Source Project: flink   Source File: S3RecoverableFsDataOutputStreamTest.java    License: Apache License 2.0 5 votes vote down vote up
@Override
public void uploadPart(RefCountedFSOutputStream file) throws IOException {
	numParts++;
	numBytes += file.getPos();

	uploadedContent.add(readFileContents(file));
}
 
Example 22
Source Project: flink   Source File: S3RecoverableFsDataOutputStreamTest.java    License: Apache License 2.0 5 votes vote down vote up
private static byte[] readFileContents(RefCountedFSOutputStream file) throws IOException {
	final byte[] content = new byte[MathUtils.checkedDownCast(file.getPos())];
	File inputFile = file.getInputFile();
	long bytesRead = new FileInputStream(inputFile).read(content, 0,  MathUtils.checkedDownCast(inputFile.length()));
	Assert.assertEquals(file.getPos(), bytesRead);
	return content;
}
 
Example 23
Source Project: flink   Source File: S3RecoverableFsDataOutputStreamTest.java    License: Apache License 2.0 5 votes vote down vote up
@Override
public RecoverableWriter.ResumeRecoverable snapshotAndGetRecoverable(RefCountedFSOutputStream incompletePartFile) throws IOException {
	lastPersistedIndex = uploadedContent.size();

	if (incompletePartFile.getPos() >= 0L) {
		byte[] bytes = readFileContents(incompletePartFile);
		uncompleted = Optional.of(bytes);
	}

	return null;
}
 
Example 24
Source Project: flink   Source File: S3RecoverableFsDataOutputStreamTest.java    License: Apache License 2.0 5 votes vote down vote up
@Override
public void uploadPart(RefCountedFSOutputStream file) throws IOException {
	numParts++;
	numBytes += file.getPos();

	uploadedContent.add(readFileContents(file));
}
 
Example 25
Source Project: Flink-CEPplus   Source File: RecoverableMultiPartUpload.java    License: Apache License 2.0 2 votes vote down vote up
/**
 * Creates a snapshot of this MultiPartUpload, from which the upload can be resumed.
 *
 * @param incompletePartFile The file containing the in-progress part which has not yet reached the minimum
 *                           part size in order to be uploaded.
 *
 * @return The {@link RecoverableWriter.ResumeRecoverable ResumeRecoverable} which
 * can be used to resume the upload.
 */
RecoverableWriter.ResumeRecoverable snapshotAndGetRecoverable(
		@Nullable final RefCountedFSOutputStream incompletePartFile) throws IOException;
 
Example 26
Source Project: Flink-CEPplus   Source File: RecoverableMultiPartUpload.java    License: Apache License 2.0 2 votes vote down vote up
/**
 * Adds a part to the uploads without any size limitations.
 *
 * @param file The file with the part data.
 *
 * @throws IOException If this method throws an exception, the RecoverableS3MultiPartUpload
 *                     should not be used any more, but recovered instead.
 */
void uploadPart(final RefCountedFSOutputStream file) throws IOException;
 
Example 27
Source Project: flink   Source File: RecoverableMultiPartUpload.java    License: Apache License 2.0 2 votes vote down vote up
/**
 * Creates a snapshot of this MultiPartUpload, from which the upload can be resumed.
 *
 * @param incompletePartFile The file containing the in-progress part which has not yet reached the minimum
 *                           part size in order to be uploaded.
 *
 * @return The {@link RecoverableWriter.ResumeRecoverable ResumeRecoverable} which
 * can be used to resume the upload.
 */
RecoverableWriter.ResumeRecoverable snapshotAndGetRecoverable(
		@Nullable final RefCountedFSOutputStream incompletePartFile) throws IOException;
 
Example 28
Source Project: flink   Source File: RecoverableMultiPartUpload.java    License: Apache License 2.0 2 votes vote down vote up
/**
 * Adds a part to the uploads without any size limitations.
 *
 * @param file The file with the part data.
 *
 * @throws IOException If this method throws an exception, the RecoverableS3MultiPartUpload
 *                     should not be used any more, but recovered instead.
 */
void uploadPart(final RefCountedFSOutputStream file) throws IOException;
 
Example 29
Source Project: flink   Source File: RecoverableMultiPartUpload.java    License: Apache License 2.0 2 votes vote down vote up
/**
 * Creates a snapshot of this MultiPartUpload, from which the upload can be resumed.
 *
 * @param incompletePartFile The file containing the in-progress part which has not yet reached the minimum
 *                           part size in order to be uploaded.
 *
 * @return The {@link RecoverableWriter.ResumeRecoverable ResumeRecoverable} which
 * can be used to resume the upload.
 */
RecoverableWriter.ResumeRecoverable snapshotAndGetRecoverable(
		@Nullable final RefCountedFSOutputStream incompletePartFile) throws IOException;
 
Example 30
Source Project: flink   Source File: RecoverableMultiPartUpload.java    License: Apache License 2.0 2 votes vote down vote up
/**
 * Adds a part to the uploads without any size limitations.
 *
 * @param file The file with the part data.
 *
 * @throws IOException If this method throws an exception, the RecoverableS3MultiPartUpload
 *                     should not be used any more, but recovered instead.
 */
void uploadPart(final RefCountedFSOutputStream file) throws IOException;