Java Code Examples for org.apache.flink.core.fs.RecoverableWriter#CommitRecoverable

The following examples show how to use org.apache.flink.core.fs.RecoverableWriter#CommitRecoverable . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: S3RecoverableFsDataOutputStreamTest.java    From flink with Apache License 2.0 6 votes vote down vote up 
@Override
public RecoverableFsDataOutputStream.Committer snapshotAndGetCommitter() throws IOException {
	lastPersistedIndex = uploadedContent.size();

	return new RecoverableFsDataOutputStream.Committer() {
		@Override
		public void commit() throws IOException {
			published = getPublishedContents();
			uploadedContent.clear();
			lastPersistedIndex = 0;
		}

		@Override
		public void commitAfterRecovery() throws IOException {
			if (published.length == 0) {
				commit();
			}
		}

		@Override
		public RecoverableWriter.CommitRecoverable getRecoverable() {
			return null;
		}
	};
}
Example 2
Source File: BucketTest.java    From flink with Apache License 2.0 5 votes vote down vote up 
private Bucket<String, String> getRestoredBucketWithOnlyPendingParts(final BaseStubWriter writer, final int numberOfPendingParts) throws IOException {
	final Map<Long, List<RecoverableWriter.CommitRecoverable>> completePartsPerCheckpoint =
			createPendingPartsPerCheckpoint(numberOfPendingParts);

	final BucketState<String> initStateWithOnlyInProgressFile =
			new BucketState<>("test", new Path(), 12345L, null, completePartsPerCheckpoint);
	return Bucket.restore(writer, 0, 1L, partFileFactory, rollingPolicy, initStateWithOnlyInProgressFile, new PartFileConfig());
}
Example 3
Source File: BucketState.java    From flink with Apache License 2.0 5 votes vote down vote up 
BucketState(
		final BucketID bucketId,
		final Path bucketPath,
		final long inProgressFileCreationTime,
		@Nullable final RecoverableWriter.ResumeRecoverable inProgressResumableFile,
		final Map<Long, List<RecoverableWriter.CommitRecoverable>> pendingCommittablesPerCheckpoint
) {
	this.bucketId = Preconditions.checkNotNull(bucketId);
	this.bucketPath = Preconditions.checkNotNull(bucketPath);
	this.inProgressFileCreationTime = inProgressFileCreationTime;
	this.inProgressResumableFile = inProgressResumableFile;
	this.committableFilesPerCheckpoint = Preconditions.checkNotNull(pendingCommittablesPerCheckpoint);
}
Example 4
Source File: BucketStateSerializer.java    From flink with Apache License 2.0 5 votes vote down vote up 
BucketStateSerializer(
		final SimpleVersionedSerializer<RecoverableWriter.ResumeRecoverable> resumableSerializer,
		final SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable> commitableSerializer,
		final SimpleVersionedSerializer<BucketID> bucketIdSerializer
) {
	this.resumableSerializer = Preconditions.checkNotNull(resumableSerializer);
	this.commitableSerializer = Preconditions.checkNotNull(commitableSerializer);
	this.bucketIdSerializer = Preconditions.checkNotNull(bucketIdSerializer);
}
Example 5
Source File: BucketStateSerializer.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
BucketStateSerializer(
		final SimpleVersionedSerializer<RecoverableWriter.ResumeRecoverable> resumableSerializer,
		final SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable> commitableSerializer,
		final SimpleVersionedSerializer<BucketID> bucketIdSerializer
) {
	this.resumableSerializer = Preconditions.checkNotNull(resumableSerializer);
	this.commitableSerializer = Preconditions.checkNotNull(commitableSerializer);
	this.bucketIdSerializer = Preconditions.checkNotNull(bucketIdSerializer);
}
Example 6
Source File: OutputStreamBasedPartFileWriter.java    From flink with Apache License 2.0 5 votes vote down vote up 
@Override
public PendingFile recoverPendingFile(final PendingFileRecoverable pendingFileRecoverable) throws IOException {
	final RecoverableWriter.CommitRecoverable commitRecoverable;

	if (pendingFileRecoverable instanceof OutputStreamBasedPendingFileRecoverable) {
		commitRecoverable = ((OutputStreamBasedPendingFileRecoverable) pendingFileRecoverable).getCommitRecoverable();
	} else if (pendingFileRecoverable instanceof OutputStreamBasedInProgressFileRecoverable) {
		commitRecoverable = ((OutputStreamBasedInProgressFileRecoverable) pendingFileRecoverable).getResumeRecoverable();
	} else {
		throw new IllegalArgumentException("can not recover from the pendingFileRecoverable");
	}
	return new OutputStreamBasedPendingFile(recoverableWriter.recoverForCommit(commitRecoverable));
}
Example 7
Source File: OutputStreamBasedPartFileWriter.java    From flink with Apache License 2.0 4 votes vote down vote up 
OutputStreamBasedPendingFileRecoverableSerializer(final SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable> commitSerializer) {
	this.commitSerializer = commitSerializer;
}
Example 8
Source File: BucketState.java    From flink with Apache License 2.0 4 votes vote down vote up 
Map<Long, List<RecoverableWriter.CommitRecoverable>> getCommittableFilesPerCheckpoint() {
	return committableFilesPerCheckpoint;
}
Example 9
Source File: BulkPartWriter.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
@Override
RecoverableWriter.CommitRecoverable closeForCommit() throws IOException {
	writer.flush();
	writer.finish();
	return super.closeForCommit();
}
Example 10
Source File: PartFileWriter.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
RecoverableWriter.CommitRecoverable closeForCommit() throws IOException {
	return currentPartStream.closeForCommit().getRecoverable();
}
Example 11
Source File: NoOpRecoverableWriter.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public RecoverableFsDataOutputStream.Committer recoverForCommit(RecoverableWriter.CommitRecoverable resumable) throws IOException {
	return null;
}
Example 12
Source File: S3Committer.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
@Override
public RecoverableWriter.CommitRecoverable getRecoverable() {
	return new S3Recoverable(objectName, uploadId, parts, totalLength);
}
Example 13
Source File: NoOpRecoverableWriter.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
@Override
public RecoverableFsDataOutputStream.Committer recoverForCommit(RecoverableWriter.CommitRecoverable resumable) throws IOException {
	return null;
}
Example 14
Source File: NoOpRecoverableWriter.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public RecoverableFsDataOutputStream.Committer recoverForCommit(RecoverableWriter.CommitRecoverable resumable) throws IOException {
	return null;
}
Example 15
Source File: NoOpCommitter.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
@Override
public RecoverableWriter.CommitRecoverable getRecoverable() {
	return null;
}
Example 16
Source File: S3Committer.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public RecoverableWriter.CommitRecoverable getRecoverable() {
	return new S3Recoverable(objectName, uploadId, parts, totalLength);
}
Example 17
Source File: NoOpCommitter.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public RecoverableWriter.CommitRecoverable getRecoverable() {
	return null;
}
Example 18
Source File: NoOpRecoverableWriter.java    From flink with Apache License 2.0 4 votes vote down vote up 
@Override
public SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable> getCommitRecoverableSerializer() {
	return null;
}
Example 19
Source File: HadoopS3RecoverableWriterITCase.java    From flink with Apache License 2.0 3 votes vote down vote up 
@Test
public void testCommitAfterRecovery() throws Exception {
	final Path path = new Path(basePathForTest, "part-0");

	final RecoverableWriter initWriter = getRecoverableWriter();

	final RecoverableFsDataOutputStream stream = initWriter.open(path);
	stream.write(bytesOf(testData1));

	stream.persist();
	stream.persist();

	// and write some more data
	stream.write(bytesOf(testData2));

	final RecoverableWriter.CommitRecoverable recoverable = stream.closeForCommit().getRecoverable();

	final byte[] serializedRecoverable = initWriter.getCommitRecoverableSerializer().serialize(recoverable);

	// get a new serializer from a new writer to make sure that no pre-initialized state leaks in.
	final RecoverableWriter newWriter = getRecoverableWriter();

	final SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable> deserializer = newWriter.getCommitRecoverableSerializer();
	final RecoverableWriter.CommitRecoverable recoveredRecoverable = deserializer.deserialize(deserializer.getVersion(), serializedRecoverable);

	final RecoverableFsDataOutputStream.Committer committer = newWriter.recoverForCommit(recoveredRecoverable);
	committer.commitAfterRecovery();

	Assert.assertEquals(testData1 + testData2, getContentsOfFile(path));
}
Example 20
Source File: HadoopS3RecoverableWriterITCase.java    From flink with Apache License 2.0 3 votes vote down vote up 
@Test
public void testCommitAfterRecovery() throws Exception {
	final Path path = new Path(basePathForTest, "part-0");

	final RecoverableWriter initWriter = getRecoverableWriter();

	final RecoverableFsDataOutputStream stream = initWriter.open(path);
	stream.write(bytesOf(testData1));

	stream.persist();
	stream.persist();

	// and write some more data
	stream.write(bytesOf(testData2));

	final RecoverableWriter.CommitRecoverable recoverable = stream.closeForCommit().getRecoverable();

	final byte[] serializedRecoverable = initWriter.getCommitRecoverableSerializer().serialize(recoverable);

	// get a new serializer from a new writer to make sure that no pre-initialized state leaks in.
	final RecoverableWriter newWriter = getRecoverableWriter();

	final SimpleVersionedSerializer<RecoverableWriter.CommitRecoverable> deserializer = newWriter.getCommitRecoverableSerializer();
	final RecoverableWriter.CommitRecoverable recoveredRecoverable = deserializer.deserialize(deserializer.getVersion(), serializedRecoverable);

	final RecoverableFsDataOutputStream.Committer committer = newWriter.recoverForCommit(recoveredRecoverable);
	committer.commitAfterRecovery();

	Assert.assertEquals(testData1 + testData2, getContentsOfFile(path));
}