Java Code Examples for com.amazonaws.services.kinesis.model.LimitExceededException

The following examples show how to use com.amazonaws.services.kinesis.model.LimitExceededException. These examples are extracted from open source projects. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source Project: Flink-CEPplus   Source File: KinesisProxy.java    License: Apache License 2.0 5 votes vote down vote up
/**
 * Get metainfo for a Kinesis stream, which contains information about which shards this
 * Kinesis stream possess.
 *
 * <p>This method is using a "full jitter" approach described in AWS's article,
 * <a href="https://www.awsarchitectureblog.com/2015/03/backoff.html">
 *   "Exponential Backoff and Jitter"</a>.
 * This is necessary because concurrent calls will be made by all parallel subtask's fetcher.
 * This jitter backoff approach will help distribute calls across the fetchers over time.
 *
 * @param streamName the stream to describe
 * @param startShardId which shard to start with for this describe operation
 *
 * @return the result of the describe stream operation
 */
protected DescribeStreamResult describeStream(String streamName, @Nullable String startShardId)
		throws InterruptedException {
	final DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();
	describeStreamRequest.setStreamName(streamName);
	describeStreamRequest.setExclusiveStartShardId(startShardId);

	DescribeStreamResult describeStreamResult = null;

	// Call DescribeStream, with full-jitter backoff (if we get LimitExceededException).
	int attemptCount = 0;
	while (describeStreamResult == null) { // retry until we get a result
		try {
			describeStreamResult = kinesisClient.describeStream(describeStreamRequest);
		} catch (LimitExceededException le) {
			long backoffMillis = fullJitterBackoff(
					describeStreamBaseBackoffMillis,
					describeStreamMaxBackoffMillis,
					describeStreamExpConstant,
					attemptCount++);
			LOG.warn(String.format("Got LimitExceededException when describing stream %s. "
					+ "Backing off for %d millis.", streamName, backoffMillis));
			Thread.sleep(backoffMillis);
		} catch (ResourceNotFoundException re) {
			throw new RuntimeException("Error while getting stream details", re);
		}
	}

	String streamStatus = describeStreamResult.getStreamDescription().getStreamStatus();
	if (!(streamStatus.equals(StreamStatus.ACTIVE.toString())
			|| streamStatus.equals(StreamStatus.UPDATING.toString()))) {
		if (LOG.isWarnEnabled()) {
			LOG.warn(String.format("The status of stream %s is %s ; result of the current "
							+ "describeStream operation will not contain any shard information.",
					streamName, streamStatus));
		}
	}

	return describeStreamResult;
}
 
Example 2
Source Project: flink   Source File: KinesisProxy.java    License: Apache License 2.0 5 votes vote down vote up
/**
 * Get metainfo for a Kinesis stream, which contains information about which shards this
 * Kinesis stream possess.
 *
 * <p>This method is using a "full jitter" approach described in AWS's article,
 * <a href="https://www.awsarchitectureblog.com/2015/03/backoff.html">
 *   "Exponential Backoff and Jitter"</a>.
 * This is necessary because concurrent calls will be made by all parallel subtask's fetcher.
 * This jitter backoff approach will help distribute calls across the fetchers over time.
 *
 * @param streamName the stream to describe
 * @param startShardId which shard to start with for this describe operation
 *
 * @return the result of the describe stream operation
 */
protected DescribeStreamResult describeStream(String streamName, @Nullable String startShardId)
		throws InterruptedException {
	final DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();
	describeStreamRequest.setStreamName(streamName);
	describeStreamRequest.setExclusiveStartShardId(startShardId);

	DescribeStreamResult describeStreamResult = null;

	// Call DescribeStream, with full-jitter backoff (if we get LimitExceededException).
	int attemptCount = 0;
	while (describeStreamResult == null) { // retry until we get a result
		try {
			describeStreamResult = kinesisClient.describeStream(describeStreamRequest);
		} catch (LimitExceededException le) {
			long backoffMillis = fullJitterBackoff(
					describeStreamBaseBackoffMillis,
					describeStreamMaxBackoffMillis,
					describeStreamExpConstant,
					attemptCount++);
			LOG.warn(String.format("Got LimitExceededException when describing stream %s. "
					+ "Backing off for %d millis.", streamName, backoffMillis));
			Thread.sleep(backoffMillis);
		} catch (ResourceNotFoundException re) {
			throw new RuntimeException("Error while getting stream details", re);
		}
	}

	String streamStatus = describeStreamResult.getStreamDescription().getStreamStatus();
	if (!(streamStatus.equals(StreamStatus.ACTIVE.toString())
			|| streamStatus.equals(StreamStatus.UPDATING.toString()))) {
		if (LOG.isWarnEnabled()) {
			LOG.warn(String.format("The status of stream %s is %s ; result of the current "
							+ "describeStream operation will not contain any shard information.",
					streamName, streamStatus));
		}
	}

	return describeStreamResult;
}
 
Example 3
private List<Shard> getShardList(String stream, int retryCount) {
	List<Shard> shardList = new ArrayList<>();

	if (retryCount > configurationProperties.getDescribeStreamRetries()) {
		ResourceNotFoundException resourceNotFoundException = new ResourceNotFoundException(
				"The stream [" + stream + "] isn't ACTIVE or doesn't exist.");
		resourceNotFoundException.setServiceName("Kinesis");

		throw new ProvisioningException(
				"Kinesis org.springframework.cloud.stream.binder.kinesis.provisioning error",
				resourceNotFoundException);
	}

	ListShardsRequest listShardsRequest = new ListShardsRequest().withStreamName(stream);

	try {
		ListShardsResult listShardsResult = amazonKinesis.listShards(listShardsRequest);

		shardList.addAll(listShardsResult.getShards());

	}
	catch (LimitExceededException limitExceededException) {
		logger.info("Got LimitExceededException when describing stream [" + stream + "]. " + "Backing off for ["
				+ this.configurationProperties.getDescribeStreamBackoff() + "] millis.");

		try {
			Thread.sleep(this.configurationProperties.getDescribeStreamBackoff());
			getShardList(stream, retryCount++);
		}
		catch (InterruptedException ex) {
			Thread.currentThread().interrupt();
			throw new ProvisioningException(
					"The [describeStream] thread for the stream [" + stream + "] has been interrupted.", ex);
		}
	}

	return shardList;
}
 
Example 4
Source Project: beam   Source File: SimplifiedKinesisClient.java    License: Apache License 2.0 5 votes vote down vote up
public List<Shard> listShards(final String streamName) throws TransientKinesisException {
  return wrapExceptions(
      () -> {
        List<Shard> shards = Lists.newArrayList();
        String lastShardId = null;

        // DescribeStream has limits that can be hit fairly easily if we are attempting
        // to configure multiple KinesisIO inputs in the same account. Retry up to
        // LIST_SHARDS_DESCRIBE_STREAM_MAX_ATTEMPTS times if we end up hitting that limit.
        //
        // Only pass the wrapped exception up once that limit is reached. Use FluentBackoff
        // to implement the retry policy.
        FluentBackoff retryBackoff =
            FluentBackoff.DEFAULT
                .withMaxRetries(LIST_SHARDS_DESCRIBE_STREAM_MAX_ATTEMPTS)
                .withInitialBackoff(LIST_SHARDS_DESCRIBE_STREAM_INITIAL_BACKOFF);
        StreamDescription description = null;
        do {
          BackOff backoff = retryBackoff.backoff();
          Sleeper sleeper = Sleeper.DEFAULT;
          while (true) {
            try {
              description =
                  kinesis.describeStream(streamName, lastShardId).getStreamDescription();
              break;
            } catch (LimitExceededException exc) {
              if (!BackOffUtils.next(sleeper, backoff)) {
                throw exc;
              }
            }
          }

          shards.addAll(description.getShards());
          lastShardId = shards.get(shards.size() - 1).getShardId();
        } while (description.getHasMoreShards());

        return shards;
      });
}
 
Example 5
Source Project: beam   Source File: AmazonKinesisMock.java    License: Apache License 2.0 5 votes vote down vote up
@Override
public DescribeStreamResult describeStream(String streamName, String exclusiveStartShardId) {
  if (rateLimitDescribeStream-- > 0) {
    throw new LimitExceededException("DescribeStream rate limit exceeded");
  }
  int nextShardId = 0;
  if (exclusiveStartShardId != null) {
    nextShardId = parseInt(exclusiveStartShardId) + 1;
  }
  boolean hasMoreShards = nextShardId + 1 < shardedData.size();

  List<Shard> shards = new ArrayList<>();
  if (nextShardId < shardedData.size()) {
    shards.add(new Shard().withShardId(Integer.toString(nextShardId)));
  }

  HttpResponse response = new HttpResponse(null, null);
  response.setStatusCode(200);
  DescribeStreamResult result = new DescribeStreamResult();
  result.setSdkHttpMetadata(SdkHttpMetadata.from(response));
  result.withStreamDescription(
      new StreamDescription()
          .withHasMoreShards(hasMoreShards)
          .withShards(shards)
          .withStreamName(streamName));
  return result;
}
 
Example 6
Source Project: flink   Source File: KinesisProxy.java    License: Apache License 2.0 5 votes vote down vote up
/**
 * Get metainfo for a Kinesis stream, which contains information about which shards this
 * Kinesis stream possess.
 *
 * <p>This method is using a "full jitter" approach described in AWS's article,
 * <a href="https://www.awsarchitectureblog.com/2015/03/backoff.html">
 *   "Exponential Backoff and Jitter"</a>.
 * This is necessary because concurrent calls will be made by all parallel subtask's fetcher.
 * This jitter backoff approach will help distribute calls across the fetchers over time.
 *
 * @param streamName the stream to describe
 * @param startShardId which shard to start with for this describe operation
 *
 * @return the result of the describe stream operation
 */
protected DescribeStreamResult describeStream(String streamName, @Nullable String startShardId)
		throws InterruptedException {
	final DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();
	describeStreamRequest.setStreamName(streamName);
	describeStreamRequest.setExclusiveStartShardId(startShardId);

	DescribeStreamResult describeStreamResult = null;

	// Call DescribeStream, with full-jitter backoff (if we get LimitExceededException).
	int attemptCount = 0;
	while (describeStreamResult == null) { // retry until we get a result
		try {
			describeStreamResult = kinesisClient.describeStream(describeStreamRequest);
		} catch (LimitExceededException le) {
			long backoffMillis = fullJitterBackoff(
					describeStreamBaseBackoffMillis,
					describeStreamMaxBackoffMillis,
					describeStreamExpConstant,
					attemptCount++);
			LOG.warn(String.format("Got LimitExceededException when describing stream %s. "
					+ "Backing off for %d millis.", streamName, backoffMillis));
			Thread.sleep(backoffMillis);
		} catch (ResourceNotFoundException re) {
			throw new RuntimeException("Error while getting stream details", re);
		}
	}

	String streamStatus = describeStreamResult.getStreamDescription().getStreamStatus();
	if (!(streamStatus.equals(StreamStatus.ACTIVE.toString())
			|| streamStatus.equals(StreamStatus.UPDATING.toString()))) {
		if (LOG.isWarnEnabled()) {
			LOG.warn(String.format("The status of stream %s is %s ; result of the current "
							+ "describeStream operation will not contain any shard information.",
					streamName, streamStatus));
		}
	}

	return describeStreamResult;
}
 
Example 7
Source Project: beam   Source File: SimplifiedKinesisClientTest.java    License: Apache License 2.0 4 votes vote down vote up
@Test
public void shouldHandleLimitExceededExceptionForGetShardIterator() {
  shouldHandleGetShardIteratorError(
      new LimitExceededException(""), KinesisClientThrottledException.class);
}
 
Example 8
Source Project: beam   Source File: SimplifiedKinesisClientTest.java    License: Apache License 2.0 4 votes vote down vote up
@Test
public void shouldHandleLimitExceededExceptionForShardListing() {
  shouldHandleShardListingError(
      new LimitExceededException(""), KinesisClientThrottledException.class);
}
 
Example 9
Source Project: beam   Source File: SimplifiedKinesisClientTest.java    License: Apache License 2.0 4 votes vote down vote up
@Test
public void shouldHandleLimitExceededExceptionForGetBacklogBytes() {
  shouldHandleGetBacklogBytesError(
      new LimitExceededException(""), KinesisClientThrottledException.class);
}