com.amazonaws.services.s3.transfer.Transfer Java Examples

The following examples show how to use com.amazonaws.services.s3.transfer.Transfer. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: PrestoS3FileSystem.java    From presto with Apache License 2.0 6 votes vote down vote up
private ProgressListener createProgressListener(Transfer transfer)
{
    return new ProgressListener()
    {
        private ProgressEventType previousType;
        private double previousTransferred;

        @Override
        public synchronized void progressChanged(ProgressEvent progressEvent)
        {
            ProgressEventType eventType = progressEvent.getEventType();
            if (previousType != eventType) {
                log.debug("Upload progress event (%s/%s): %s", bucket, key, eventType);
                previousType = eventType;
            }

            double transferred = transfer.getProgress().getPercentTransferred();
            if (transferred >= (previousTransferred + 10.0)) {
                log.debug("Upload percentage (%s/%s): %.0f%%", bucket, key, transferred);
                previousTransferred = transferred;
            }
        }
    };
}
 
Example #2
Source File: XferMgrProgress.java    From dlp-dataflow-deidentification with Apache License 2.0 6 votes vote down vote up
public static void showTransferProgress(Transfer xfer) {
  // print the transfer's human-readable description
  System.out.println(xfer.getDescription());
  // print an empty progress bar...
  printProgressBar(0.0);
  // update the progress bar while the xfer is ongoing.
  do {
    try {
      Thread.sleep(100);
    } catch (InterruptedException e) {
      return;
    }
    // Note: so_far and total aren't used, they're just for
    // documentation purposes.
    TransferProgress progress = xfer.getProgress();
    long so_far = progress.getBytesTransferred();
    long total = progress.getTotalBytesToTransfer();
    double pct = progress.getPercentTransferred();
    eraseProgressBar();
    printProgressBar(pct);
  } while (xfer.isDone() == false);
  // print the final state of the transfer.
  TransferState xfer_state = xfer.getState();
  System.out.println(": " + xfer_state);
}
 
Example #3
Source File: S3Util.java    From teamcity-s3-artifact-storage-plugin with Apache License 2.0 6 votes vote down vote up
@SuppressWarnings("UnusedReturnValue")
public static <T extends Transfer> Collection<T> withTransferManagerCorrectingRegion(@NotNull final Map<String, String> s3Settings,
                                                                                     @NotNull final WithTransferManager<T> withTransferManager) throws Throwable {
  try {
    return withTransferManager(s3Settings, withTransferManager);
  } catch (RuntimeException e) {
    final String correctRegion = extractCorrectedRegion(e);
    if (correctRegion != null) {
      LOGGER.debug("Running operation with corrected S3 region [" + correctRegion + "]", e);
      s3Settings.put(REGION_NAME_PARAM, correctRegion);
      return withTransferManager(s3Settings, withTransferManager);
    } else {
      throw e;
    }
  }
}
 
Example #4
Source File: S3DaoImpl.java    From herd with Apache License 2.0 6 votes vote down vote up
@Override
public S3FileTransferResultsDto downloadFile(final S3FileTransferRequestParamsDto params) throws InterruptedException
{
    LOGGER.info("Downloading S3 file... s3Key=\"{}\" s3BucketName=\"{}\" localPath=\"{}\"", params.getS3KeyPrefix(), params.getS3BucketName(),
        params.getLocalPath());

    // Perform the transfer.
    S3FileTransferResultsDto results = performTransfer(params, new Transferer()
    {
        @Override
        public Transfer performTransfer(TransferManager transferManager)
        {
            return s3Operations.download(params.getS3BucketName(), params.getS3KeyPrefix(), new File(params.getLocalPath()), transferManager);
        }
    });

    LOGGER
        .info("Downloaded S3 file to the local system. s3Key=\"{}\" s3BucketName=\"{}\" localPath=\"{}\" totalBytesTransferred={} transferDuration=\"{}\"",
            params.getS3KeyPrefix(), params.getS3BucketName(), params.getLocalPath(), results.getTotalBytesTransferred(),
            HerdDateUtils.formatDuration(results.getDurationMillis()));

    logOverallTransferRate(results);

    return results;
}
 
Example #5
Source File: S3S3Copier.java    From circus-train with Apache License 2.0 5 votes vote down vote up
@Override
public void transferStateChanged(Transfer transfer, TransferState state) {
  if (state == TransferState.Completed) {
    // NOTE: running progress doesn't seem to be reported correctly.
    // transfer.getProgress().getBytesTransferred() is always 0. Unsure what is the cause of this at this moment
    // so just printing total bytes when completed.
    LOG
        .debug("copied object from '{}/{}' to '{}/{}': {} bytes transferred", s3ObjectSummary.getBucketName(),
            s3ObjectSummary.getKey(), targetS3Uri.getBucket(), targetKey,
            transfer.getProgress().getTotalBytesToTransfer());
  }
}
 
Example #6
Source File: S3DaoImpl.java    From herd with Apache License 2.0 5 votes vote down vote up
@Override
public S3FileTransferResultsDto downloadDirectory(final S3FileTransferRequestParamsDto params) throws InterruptedException
{
    LOGGER.info("Downloading S3 directory to the local system... s3KeyPrefix=\"{}\" s3BucketName=\"{}\" localDirectory=\"{}\"", params.getS3KeyPrefix(),
        params.getS3BucketName(), params.getLocalPath());

    // Note that the directory download always recursively copies sub-directories.
    // To not recurse, we would have to list the files on S3 (AmazonS3Client.html#listObjects) and manually copy them one at a time.

    // Perform the transfer.
    S3FileTransferResultsDto results = performTransfer(params, new Transferer()
    {
        @Override
        public Transfer performTransfer(TransferManager transferManager)
        {
            return s3Operations.downloadDirectory(params.getS3BucketName(), params.getS3KeyPrefix(), new File(params.getLocalPath()), transferManager);
        }
    });

    LOGGER.info("Downloaded S3 directory to the local system. " +
            "s3KeyPrefix=\"{}\" s3BucketName=\"{}\" localDirectory=\"{}\" s3KeyCount={} totalBytesTransferred={} transferDuration=\"{}\"",
        params.getS3KeyPrefix(), params.getS3BucketName(), params.getLocalPath(), results.getTotalFilesTransferred(), results.getTotalBytesTransferred(),
        HerdDateUtils.formatDuration(results.getDurationMillis()));

    logOverallTransferRate(results);

    return results;
}
 
Example #7
Source File: S3DaoImpl.java    From herd with Apache License 2.0 5 votes vote down vote up
@Override
public S3FileTransferResultsDto uploadDirectory(final S3FileTransferRequestParamsDto params) throws InterruptedException
{
    LOGGER.info("Uploading local directory to S3... localDirectory=\"{}\" s3KeyPrefix=\"{}\" s3BucketName=\"{}\"", params.getLocalPath(),
        params.getS3KeyPrefix(), params.getS3BucketName());

    // Perform the transfer.
    S3FileTransferResultsDto results = performTransfer(params, new Transferer()
    {
        @Override
        public Transfer performTransfer(TransferManager transferManager)
        {
            return s3Operations.uploadDirectory(params.getS3BucketName(), params.getS3KeyPrefix(), new File(params.getLocalPath()), params.isRecursive(),
                new ObjectMetadataProvider()
                {
                    @Override
                    public void provideObjectMetadata(File file, ObjectMetadata metadata)
                    {
                        prepareMetadata(params, metadata);
                    }
                }, transferManager);
        }
    });

    LOGGER.info("Uploaded local directory to S3. " +
            "localDirectory=\"{}\" s3KeyPrefix=\"{}\" s3BucketName=\"{}\" s3KeyCount={} totalBytesTransferred={} transferDuration=\"{}\"",
        params.getLocalPath(), params.getS3KeyPrefix(), params.getS3BucketName(), results.getTotalFilesTransferred(), results.getTotalBytesTransferred(),
        HerdDateUtils.formatDuration(results.getDurationMillis()));

    logOverallTransferRate(results);

    return results;
}
 
Example #8
Source File: S3DaoImpl.java    From herd with Apache License 2.0 5 votes vote down vote up
@Override
public S3FileTransferResultsDto uploadFile(final S3FileTransferRequestParamsDto params) throws InterruptedException
{
    LOGGER.info("Uploading local file to S3... localPath=\"{}\" s3Key=\"{}\" s3BucketName=\"{}\"", params.getLocalPath(), params.getS3KeyPrefix(),
        params.getS3BucketName());

    // Perform the transfer.
    S3FileTransferResultsDto results = performTransfer(params, new Transferer()
    {
        @Override
        public Transfer performTransfer(TransferManager transferManager)
        {
            // Get a handle to the local file.
            File localFile = new File(params.getLocalPath());

            // Create and prepare the metadata.
            ObjectMetadata metadata = new ObjectMetadata();
            prepareMetadata(params, metadata);

            // Create a put request and a transfer manager with the parameters and the metadata.
            PutObjectRequest putObjectRequest = new PutObjectRequest(params.getS3BucketName(), params.getS3KeyPrefix(), localFile);
            putObjectRequest.setMetadata(metadata);

            return s3Operations.upload(putObjectRequest, transferManager);
        }
    });

    LOGGER.info("Uploaded local file to the S3. localPath=\"{}\" s3Key=\"{}\" s3BucketName=\"{}\" totalBytesTransferred={} transferDuration=\"{}\"",
        params.getLocalPath(), params.getS3KeyPrefix(), params.getS3BucketName(), results.getTotalBytesTransferred(),
        HerdDateUtils.formatDuration(results.getDurationMillis()));

    logOverallTransferRate(results);

    return results;
}
 
Example #9
Source File: S3DaoTest.java    From herd with Apache License 2.0 5 votes vote down vote up
/**
 * Test S3 file copy with an invalid KMS Id that will result in a cancelled transfer.
 */
@Test
public void testCopyFileInvalidKmsIdCancelled() throws InterruptedException
{
    // Put a 1 byte file in S3.
    s3Operations
        .putObject(new PutObjectRequest(storageDaoTestHelper.getS3LoadingDockBucketName(), TARGET_S3_KEY, new ByteArrayInputStream(new byte[1]), null),
            null);

    try
    {
        S3FileCopyRequestParamsDto transferDto = new S3FileCopyRequestParamsDto();
        transferDto.setSourceBucketName(storageDaoTestHelper.getS3LoadingDockBucketName());
        transferDto.setTargetBucketName(storageDaoTestHelper.getS3ExternalBucketName());
        transferDto.setSourceObjectKey(TARGET_S3_KEY);
        transferDto.setTargetObjectKey(TARGET_S3_KEY);
        transferDto.setKmsKeyId(MockS3OperationsImpl.MOCK_KMS_ID_CANCELED_TRANSFER);
        s3Dao.copyFile(transferDto);
        fail("An IllegalStateException was expected but not thrown.");
    }
    catch (IllegalStateException ex)
    {
        assertEquals("Invalid IllegalStateException message returned.",
            "The transfer operation \"" + MockS3OperationsImpl.MOCK_TRANSFER_DESCRIPTION + "\" did not complete successfully. " + "Current state: \"" +
                Transfer.TransferState.Canceled + "\".", ex.getMessage());
    }
}
 
Example #10
Source File: AWSTransferModel.java    From vocefiscal-android with Apache License 2.0 5 votes vote down vote up
public int getProgress() 
{
    Transfer transfer = getTransfer();
    if(transfer != null) 
    {
        int ret = (int)transfer.getProgress().getPercentTransferred();
        return ret;
    }
    return 0;
}
 
Example #11
Source File: S3DaoImpl.java    From herd with Apache License 2.0 4 votes vote down vote up
@Override
public S3FileTransferResultsDto copyFile(final S3FileCopyRequestParamsDto params) throws InterruptedException
{
    LOGGER
        .info("Copying S3 object... sourceS3Key=\"{}\" sourceS3BucketName=\"{}\" targetS3Key=\"{}\" targetS3BucketName=\"{}\"", params.getSourceObjectKey(),
            params.getSourceBucketName(), params.getTargetObjectKey(), params.getTargetBucketName());

    // Perform the copy.
    S3FileTransferResultsDto results = performTransfer(params, new Transferer()
    {
        @Override
        public Transfer performTransfer(TransferManager transferManager)
        {
            // Create a copy request.
            CopyObjectRequest copyObjectRequest =
                new CopyObjectRequest(params.getSourceBucketName(), params.getSourceObjectKey(), params.getTargetBucketName(), params.getTargetObjectKey());

            // If KMS Key ID is specified, set the AWS Key Management System parameters to be used to encrypt the object.
            if (StringUtils.isNotBlank(params.getKmsKeyId()))
            {
                copyObjectRequest.withSSEAwsKeyManagementParams(new SSEAwsKeyManagementParams(params.getKmsKeyId()));
            }
            // Otherwise, specify the server-side encryption algorithm for encrypting the object using AWS-managed keys.
            else
            {
                ObjectMetadata metadata = new ObjectMetadata();
                metadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
                copyObjectRequest.setNewObjectMetadata(metadata);
            }

            return s3Operations.copyFile(copyObjectRequest, transferManager);
        }
    });

    LOGGER.info("Copied S3 object. sourceS3Key=\"{}\" sourceS3BucketName=\"{}\" targetS3Key=\"{}\" targetS3BucketName=\"{}\" " +
            "totalBytesTransferred={} transferDuration=\"{}\"", params.getSourceObjectKey(), params.getSourceBucketName(), params.getTargetObjectKey(),
        params.getTargetBucketName(), results.getTotalBytesTransferred(), HerdDateUtils.formatDuration(results.getDurationMillis()));

    logOverallTransferRate(results);

    return results;
}
 
Example #12
Source File: AWSFiscalizacaoUpload.java    From vocefiscal-android with Apache License 2.0 4 votes vote down vote up
@Override
public Transfer getTransfer() 
{
	return mUpload; 
}
 
Example #13
Source File: AWSPictureUpload.java    From vocefiscal-android with Apache License 2.0 4 votes vote down vote up
@Override
public Transfer getTransfer() 
{
	return mUpload; 
}
 
Example #14
Source File: S3DaoImpl.java    From herd with Apache License 2.0 2 votes vote down vote up
/**
 * Perform a transfer using the specified transfer manager.
 *
 * @param transferManager the transfer manager.
 *
 * @return the transfer information for the transfer. This will typically be returned from an operation on the transfer manager (e.g. upload).
 */
public Transfer performTransfer(TransferManager transferManager);
 
Example #15
Source File: AWSTransferModel.java    From vocefiscal-android with Apache License 2.0 votes vote down vote up
public abstract Transfer getTransfer();