com.amazonaws.services.s3.model.CompleteMultipartUploadResult Java Examples

The following examples show how to use com.amazonaws.services.s3.model.CompleteMultipartUploadResult. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: S3Service.java    From modeldb with Apache License 2.0 6 votes vote down vote up
@Override
public void commitMultipart(String s3Key, String uploadId, List<PartETag> partETags)
    throws ModelDBException {
  // Validate bucket
  Boolean exist = doesBucketExist(bucketName);
  if (!exist) {
    throw new ModelDBException("Bucket does not exists", io.grpc.Status.Code.INTERNAL);
  }
  CompleteMultipartUploadRequest completeMultipartUploadRequest =
      new CompleteMultipartUploadRequest(bucketName, s3Key, uploadId, partETags);
  try {
    CompleteMultipartUploadResult result =
        s3Client.completeMultipartUpload(completeMultipartUploadRequest);
    LOGGER.info("upload result: {}", result);
  } catch (AmazonS3Exception e) {
    if (e.getStatusCode() == HttpStatusCodes.STATUS_CODE_BAD_REQUEST) {
      LOGGER.info("message: {} additional details: {}", e.getMessage(), e.getAdditionalDetails());
      throw new ModelDBException(e.getErrorMessage(), io.grpc.Status.Code.FAILED_PRECONDITION);
    }
    throw e;
  }
}
 
Example #2
Source File: COSBlockOutputStream.java    From stocator with Apache License 2.0 6 votes vote down vote up
/**
 * This completes a multipart upload. Sometimes it fails; here retries are
 * handled to avoid losing all data on a transient failure.
 *
 * @param partETags list of partial uploads
 * @throws IOException on any problem
 */
private CompleteMultipartUploadResult complete(List<PartETag> partETags) throws IOException {
  int retryCount = 0;
  AmazonClientException lastException;
  String operation = String.format("Completing multi-part upload for key '%s',"
      + " id '%s' with %s partitions ",
      key, uploadId, partETags.size());
  do {
    try {
      LOG.debug(operation);
      return writeOperationHelper.completeMultipartUpload(uploadId, partETags);
    } catch (AmazonClientException e) {
      lastException = e;
    }
  }
  while (shouldRetry(operation, lastException, retryCount++));
  // this point is only reached if the operation failed more than
  // the allowed retry count
  throw translateException(operation, key, lastException);
}
 
Example #3
Source File: RecoverableMultiPartUploadImplTest.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up
@Override
public CompleteMultipartUploadResult commitMultiPartUpload(
		String key,
		String uploadId,
		List<PartETag> partETags,
		long length,
		AtomicInteger errorCount) throws IOException {
	return null;
}
 
Example #4
Source File: RecoverableMultiPartUploadImplTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Override
public CompleteMultipartUploadResult commitMultiPartUpload(
		String key,
		String uploadId,
		List<PartETag> partETags,
		long length,
		AtomicInteger errorCount) throws IOException {
	return null;
}
 
Example #5
Source File: S3OutputModuleMockTest.java    From attic-apex-malhar with Apache License 2.0 5 votes vote down vote up
private CompleteMultipartUploadResult completeMultiPart() throws IOException
{
  FileUtils.copyFile(inputFile, new File(outputDir + File.separator + FILE));
  CompleteMultipartUploadResult result = new CompleteMultipartUploadResult();
  result.setETag(outputDir);
  return result;
}
 
Example #6
Source File: S3FileMerger.java    From attic-apex-malhar with Apache License 2.0 5 votes vote down vote up
/**
 * Send the CompleteMultipartUploadRequest to S3 if all the blocks of a file are uploaded into S3.
 * @param keyName file to upload into S3
 */
private void verifyAndEmitFileMerge(String keyName)
{
  if (currentWindowId <= windowDataManager.getLargestCompletedWindow()) {
    return;
  }
  S3InitiateFileUploadOperator.UploadFileMetadata uploadFileMetadata = fileMetadatas.get(keyName);
  List<PartETag> partETags = uploadParts.get(keyName);
  if (partETags == null || uploadFileMetadata == null ||
      uploadFileMetadata.getFileMetadata().getNumberOfBlocks() != partETags.size()) {
    return;
  }

  if (partETags.size() <= 1) {
    uploadedFiles.add(keyName);
    LOG.debug("Uploaded file {} successfully", keyName);
    return;
  }

  CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName,
      keyName, uploadFileMetadata.getUploadId(), partETags);
  CompleteMultipartUploadResult result = s3Client.completeMultipartUpload(compRequest);
  if (result.getETag() != null) {
    uploadedFiles.add(keyName);
    LOG.debug("Uploaded file {} successfully", keyName);
  }
}
 
Example #7
Source File: COSAPIClient.java    From stocator with Apache License 2.0 5 votes vote down vote up
/**
 * Complete a multipart upload operation.
 * @param uploadId multipart operation Id
 * @param partETags list of partial uploads
 * @return the result
 * @throws AmazonClientException on problems
 */
CompleteMultipartUploadResult completeMultipartUpload(String uploadId,
    List<PartETag> partETags) throws AmazonClientException {
  LOG.debug("Completing multipart upload {} with {} parts",
      uploadId, partETags.size());
  return mClient.completeMultipartUpload(
      new CompleteMultipartUploadRequest(mBucket,
          key,
          uploadId,
          partETags));
}
 
Example #8
Source File: RecoverableMultiPartUploadImplTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Override
public CompleteMultipartUploadResult commitMultiPartUpload(
		String key,
		String uploadId,
		List<PartETag> partETags,
		long length,
		AtomicInteger errorCount) throws IOException {
	return null;
}
 
Example #9
Source File: TestUtil.java    From s3committer with Apache License 2.0 4 votes vote down vote up
private static CompleteMultipartUploadResult newResult(
    CompleteMultipartUploadRequest req) {
  return new CompleteMultipartUploadResult();
}
 
Example #10
Source File: HadoopS3AccessHelper.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public CompleteMultipartUploadResult commitMultiPartUpload(String destKey, String uploadId, List<PartETag> partETags, long length, AtomicInteger errorCount) throws IOException {
	return s3accessHelper.completeMPUwithRetries(destKey, uploadId, partETags, length, errorCount);
}
 
Example #11
Source File: HadoopS3AccessHelper.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up
@Override
public CompleteMultipartUploadResult commitMultiPartUpload(String destKey, String uploadId, List<PartETag> partETags, long length, AtomicInteger errorCount) throws IOException {
	return s3accessHelper.completeMPUwithRetries(destKey, uploadId, partETags, length, errorCount);
}
 
Example #12
Source File: DummyS3Client.java    From ignite with Apache License 2.0 4 votes vote down vote up
/** Unsupported Operation. */
@Override public CompleteMultipartUploadResult completeMultipartUpload(
    CompleteMultipartUploadRequest req) throws SdkClientException {
    throw new UnsupportedOperationException("Operation not supported");
}
 
Example #13
Source File: HadoopS3AccessHelper.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public CompleteMultipartUploadResult commitMultiPartUpload(String destKey, String uploadId, List<PartETag> partETags, long length, AtomicInteger errorCount) throws IOException {
	return s3accessHelper.completeMPUwithRetries(destKey, uploadId, partETags, length, errorCount);
}
 
Example #14
Source File: AmazonS3Mock.java    From Scribengin with GNU Affero General Public License v3.0 4 votes vote down vote up
@Override
public CompleteMultipartUploadResult completeMultipartUpload(CompleteMultipartUploadRequest request)
    throws AmazonClientException, AmazonServiceException {
  // TODO Auto-generated method stub
  return null;
}
 
Example #15
Source File: S3AccessHelper.java    From flink with Apache License 2.0 2 votes vote down vote up
/**
 * Finalizes a Multi-Part Upload.
 *
 * @param key the key identifying the object we finished uploading.
 * @param uploadId the id of the MPU.
 * @param partETags the list of {@link PartETag ETags} associated with this MPU.
 * @param length the size of the uploaded object.
 * @param errorCount a counter that will be used to count any failed attempts to commit the MPU.
 * @return The {@link CompleteMultipartUploadResult result} of the attempt to finalize the MPU.
 * @throws IOException
 */
CompleteMultipartUploadResult commitMultiPartUpload(String key, String uploadId, List<PartETag> partETags, long length, AtomicInteger errorCount) throws IOException;
 
Example #16
Source File: S3AccessHelper.java    From flink with Apache License 2.0 2 votes vote down vote up
/**
 * Finalizes a Multi-Part Upload.
 *
 * @param key the key identifying the object we finished uploading.
 * @param uploadId the id of the MPU.
 * @param partETags the list of {@link PartETag ETags} associated with this MPU.
 * @param length the size of the uploaded object.
 * @param errorCount a counter that will be used to count any failed attempts to commit the MPU.
 * @return The {@link CompleteMultipartUploadResult result} of the attempt to finalize the MPU.
 * @throws IOException
 */
CompleteMultipartUploadResult commitMultiPartUpload(String key, String uploadId, List<PartETag> partETags, long length, AtomicInteger errorCount) throws IOException;
 
Example #17
Source File: S3AccessHelper.java    From Flink-CEPplus with Apache License 2.0 2 votes vote down vote up
/**
 * Finalizes a Multi-Part Upload.
 *
 * @param key the key identifying the object we finished uploading.
 * @param uploadId the id of the MPU.
 * @param partETags the list of {@link PartETag ETags} associated with this MPU.
 * @param length the size of the uploaded object.
 * @param errorCount a counter that will be used to count any failed attempts to commit the MPU.
 * @return The {@link CompleteMultipartUploadResult result} of the attempt to finalize the MPU.
 * @throws IOException
 */
CompleteMultipartUploadResult commitMultiPartUpload(String key, String uploadId, List<PartETag> partETags, long length, AtomicInteger errorCount) throws IOException;