com.amazonaws.services.kinesis.model.GetShardIteratorResult Java Examples

The following examples show how to use com.amazonaws.services.kinesis.model.GetShardIteratorResult. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: MockKinesisClient.java    From presto with Apache License 2.0 6 votes vote down vote up 
@Override
public GetShardIteratorResult getShardIterator(GetShardIteratorRequest getShardIteratorRequest)
        throws AmazonClientException
{
    ShardIterator iter = ShardIterator.fromStreamAndShard(getShardIteratorRequest.getStreamName(), getShardIteratorRequest.getShardId());
    if (iter == null) {
        throw new AmazonClientException("Bad stream or shard iterator!");
    }

    InternalStream theStream = this.getStream(iter.streamId);
    if (theStream == null) {
        throw new AmazonClientException("Unknown stream or bad shard iterator!");
    }

    String seqAsString = getShardIteratorRequest.getStartingSequenceNumber();
    if (seqAsString != null && !seqAsString.isEmpty() && getShardIteratorRequest.getShardIteratorType().equals("AFTER_SEQUENCE_NUMBER")) {
        int sequence = parseInt(seqAsString);
        iter.recordIndex = sequence + 1;
    }
    else {
        iter.recordIndex = 100;
    }

    GetShardIteratorResult result = new GetShardIteratorResult();
    return result.withShardIterator(iter.makeString());
}
Example #2
Source File: KinesisProxy.java    From flink with Apache License 2.0 6 votes vote down vote up 
private String getShardIterator(GetShardIteratorRequest getShardIteratorRequest) throws InterruptedException {
	GetShardIteratorResult getShardIteratorResult = null;

	int retryCount = 0;
	while (retryCount <= getShardIteratorMaxRetries && getShardIteratorResult == null) {
		try {
				getShardIteratorResult = kinesisClient.getShardIterator(getShardIteratorRequest);
		} catch (AmazonServiceException ex) {
			if (isRecoverableException(ex)) {
				long backoffMillis = fullJitterBackoff(
					getShardIteratorBaseBackoffMillis, getShardIteratorMaxBackoffMillis, getShardIteratorExpConstant, retryCount++);
				LOG.warn("Got recoverable AmazonServiceException. Backing off for "
					+ backoffMillis + " millis (" + ex.getClass().getName() + ": " + ex.getMessage() + ")");
				Thread.sleep(backoffMillis);
			} else {
				throw ex;
			}
		}
	}

	if (getShardIteratorResult == null) {
		throw new RuntimeException("Retries exceeded for getShardIterator operation - all " + getShardIteratorMaxRetries +
			" retry attempts failed.");
	}
	return getShardIteratorResult.getShardIterator();
}
Example #3
Source File: KinesisUtil.java    From datacollector with Apache License 2.0 6 votes vote down vote up 
public static List<com.amazonaws.services.kinesis.model.Record> getPreviewRecords(
    ClientConfiguration awsClientConfig,
    KinesisConfigBean conf,
    int maxBatchSize,
    GetShardIteratorRequest getShardIteratorRequest
) throws StageException {
  AmazonKinesis kinesisClient = getKinesisClient(awsClientConfig, conf);

  GetShardIteratorResult getShardIteratorResult = kinesisClient.getShardIterator(getShardIteratorRequest);
  String shardIterator = getShardIteratorResult.getShardIterator();

  GetRecordsRequest getRecordsRequest = new GetRecordsRequest();
  getRecordsRequest.setShardIterator(shardIterator);
  getRecordsRequest.setLimit(maxBatchSize);

  GetRecordsResult getRecordsResult = kinesisClient.getRecords(getRecordsRequest);
  return getRecordsResult.getRecords();
}
Example #4
Source File: SimplifiedKinesisClientTest.java    From beam with Apache License 2.0 6 votes vote down vote up 
@Test
public void shouldReturnIteratorStartingWithTimestamp() throws Exception {
  Instant timestamp = Instant.now();
  when(kinesis.getShardIterator(
          new GetShardIteratorRequest()
              .withStreamName(STREAM)
              .withShardId(SHARD_1)
              .withShardIteratorType(ShardIteratorType.AT_SEQUENCE_NUMBER)
              .withTimestamp(timestamp.toDate())))
      .thenReturn(new GetShardIteratorResult().withShardIterator(SHARD_ITERATOR));

  String stream =
      underTest.getShardIterator(
          STREAM, SHARD_1, ShardIteratorType.AT_SEQUENCE_NUMBER, null, timestamp);

  assertThat(stream).isEqualTo(SHARD_ITERATOR);
}
Example #5
Source File: SimplifiedKinesisClientTest.java    From beam with Apache License 2.0 6 votes vote down vote up 
@Test
public void shouldReturnIteratorStartingWithSequenceNumber() throws Exception {
  when(kinesis.getShardIterator(
          new GetShardIteratorRequest()
              .withStreamName(STREAM)
              .withShardId(SHARD_1)
              .withShardIteratorType(ShardIteratorType.AT_SEQUENCE_NUMBER)
              .withStartingSequenceNumber(SEQUENCE_NUMBER)))
      .thenReturn(new GetShardIteratorResult().withShardIterator(SHARD_ITERATOR));

  String stream =
      underTest.getShardIterator(
          STREAM, SHARD_1, ShardIteratorType.AT_SEQUENCE_NUMBER, SEQUENCE_NUMBER, null);

  assertThat(stream).isEqualTo(SHARD_ITERATOR);
}
Example #6
Source File: KinesisProxy.java    From flink with Apache License 2.0 6 votes vote down vote up 
private String getShardIterator(GetShardIteratorRequest getShardIteratorRequest) throws InterruptedException {
	GetShardIteratorResult getShardIteratorResult = null;

	int retryCount = 0;
	while (retryCount <= getShardIteratorMaxRetries && getShardIteratorResult == null) {
		try {
				getShardIteratorResult = kinesisClient.getShardIterator(getShardIteratorRequest);
		} catch (AmazonServiceException ex) {
			if (isRecoverableException(ex)) {
				long backoffMillis = fullJitterBackoff(
					getShardIteratorBaseBackoffMillis, getShardIteratorMaxBackoffMillis, getShardIteratorExpConstant, retryCount++);
				LOG.warn("Got recoverable AmazonServiceException. Backing off for "
					+ backoffMillis + " millis (" + ex.getClass().getName() + ": " + ex.getMessage() + ")");
				Thread.sleep(backoffMillis);
			} else {
				throw ex;
			}
		}
	}

	if (getShardIteratorResult == null) {
		throw new RuntimeException("Retries exceeded for getShardIterator operation - all " + getShardIteratorMaxRetries +
			" retry attempts failed.");
	}
	return getShardIteratorResult.getShardIterator();
}
Example #7
Source File: KinesisProxy.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up 
private String getShardIterator(GetShardIteratorRequest getShardIteratorRequest) throws InterruptedException {
	GetShardIteratorResult getShardIteratorResult = null;

	int retryCount = 0;
	while (retryCount <= getShardIteratorMaxRetries && getShardIteratorResult == null) {
		try {
				getShardIteratorResult = kinesisClient.getShardIterator(getShardIteratorRequest);
		} catch (AmazonServiceException ex) {
			if (isRecoverableException(ex)) {
				long backoffMillis = fullJitterBackoff(
					getShardIteratorBaseBackoffMillis, getShardIteratorMaxBackoffMillis, getShardIteratorExpConstant, retryCount++);
				LOG.warn("Got recoverable AmazonServiceException. Backing off for "
					+ backoffMillis + " millis (" + ex.getClass().getName() + ": " + ex.getMessage() + ")");
				Thread.sleep(backoffMillis);
			} else {
				throw ex;
			}
		}
	}

	if (getShardIteratorResult == null) {
		throw new RuntimeException("Retries exceeded for getShardIterator operation - all " + getShardIteratorMaxRetries +
			" retry attempts failed.");
	}
	return getShardIteratorResult.getShardIterator();
}
Example #8
Source File: AmazonKinesisMock.java    From beam with Apache License 2.0 5 votes vote down vote up 
@Override
public GetShardIteratorResult getShardIterator(GetShardIteratorRequest getShardIteratorRequest) {
  ShardIteratorType shardIteratorType =
      ShardIteratorType.fromValue(getShardIteratorRequest.getShardIteratorType());

  String shardIterator;
  if (shardIteratorType == ShardIteratorType.TRIM_HORIZON) {
    shardIterator = String.format("%s:%s", getShardIteratorRequest.getShardId(), 0);
  } else {
    throw new RuntimeException("Not implemented");
  }

  return new GetShardIteratorResult().withShardIterator(shardIterator);
}
Example #9
Source File: KinesisRecordSet.java    From presto-kinesis with Apache License 2.0 5 votes vote down vote up 
private void getIterator()
        throws ResourceNotFoundException
{
    GetShardIteratorRequest getShardIteratorRequest = new GetShardIteratorRequest();
    getShardIteratorRequest.setStreamName(split.getStreamName());
    getShardIteratorRequest.setShardId(split.getShardId());

    // Explanation: when we have a sequence number from a prior read or checkpoint, always use it.
    // Otherwise, decide if starting at a timestamp or the trim horizon based on configuration.
    // If starting at a timestamp, sue the session variable ITER_START_TIMESTAMP when given, otherwise
    // fallback on starting at ITER_OFFSET_SECONDS from timestamp.
    if (lastReadSeqNo == null) {
        // Important: shard iterator type AT_TIMESTAMP requires 1.11.x or above of the AWS SDK.
        if (SessionVariables.getIterFromTimestamp(session)) {
            getShardIteratorRequest.setShardIteratorType("AT_TIMESTAMP");
            long iterStartTs = SessionVariables.getIterStartTimestamp(session);
            if (iterStartTs == 0) {
                long startTs = System.currentTimeMillis() - (SessionVariables.getIterOffsetSeconds(session) * 1000);
                getShardIteratorRequest.setTimestamp(new Date(startTs));
            }
            else {
                getShardIteratorRequest.setTimestamp(new Date(iterStartTs));
            }
        }
        else {
            getShardIteratorRequest.setShardIteratorType("TRIM_HORIZON");
        }
    }
    else {
        getShardIteratorRequest.setShardIteratorType("AFTER_SEQUENCE_NUMBER");
        getShardIteratorRequest.setStartingSequenceNumber(lastReadSeqNo);
    }

    GetShardIteratorResult getShardIteratorResult = clientManager.getClient().getShardIterator(getShardIteratorRequest);
    shardIterator = getShardIteratorResult.getShardIterator();
}
Example #10
Source File: MockKinesisClient.java    From presto-kinesis with Apache License 2.0 5 votes vote down vote up 
@Override
public GetShardIteratorResult getShardIterator(GetShardIteratorRequest getShardIteratorRequest) throws AmazonServiceException, AmazonClientException
{
    ShardIterator iter = ShardIterator.fromStreamAndShard(getShardIteratorRequest.getStreamName(), getShardIteratorRequest.getShardId());
    if (iter != null) {
        InternalStream theStream = this.getStream(iter.streamId);
        if (theStream != null) {
            String seqAsString = getShardIteratorRequest.getStartingSequenceNumber();
            if (seqAsString != null && !seqAsString.isEmpty() && getShardIteratorRequest.getShardIteratorType().equals("AFTER_SEQUENCE_NUMBER")) {
                int sequence = Integer.parseInt(seqAsString);
                iter.recordIndex = sequence + 1;
            }
            else {
                iter.recordIndex = 100;
            }

            GetShardIteratorResult result = new GetShardIteratorResult();
            return result.withShardIterator(iter.makeString());
        }
        else {
            throw new AmazonClientException("Unknown stream or bad shard iterator!");
        }
    }
    else {
        throw new AmazonClientException("Bad stream or shard iterator!");
    }
}
Example #11
Source File: KinesisRecordSet.java    From presto with Apache License 2.0 5 votes vote down vote up 
private void getIterator()
        throws ResourceNotFoundException
{
    GetShardIteratorRequest getShardIteratorRequest = new GetShardIteratorRequest();
    getShardIteratorRequest.setStreamName(split.getStreamName());
    getShardIteratorRequest.setShardId(split.getShardId());

    // Explanation: when we have a sequence number from a prior read or checkpoint, always use it.
    // Otherwise, decide if starting at a timestamp or the trim horizon based on configuration.
    // If starting at a timestamp, use the session variable STARTING_TIMESTAMP when given, otherwise
    // fallback on starting at STARTING_OFFSET_SECONDS from timestamp.
    if (lastReadSequenceNumber == null) {
        if (isIteratorFromTimestamp(session)) {
            getShardIteratorRequest.setShardIteratorType("AT_TIMESTAMP");
            long iteratorStartTimestamp = getIteratorStartTimestamp(session);
            if (iteratorStartTimestamp == 0) {
                long startTimestamp = System.currentTimeMillis() - (getIteratorOffsetSeconds(session) * 1000);
                getShardIteratorRequest.setTimestamp(new Date(startTimestamp));
            }
            else {
                getShardIteratorRequest.setTimestamp(new Date(iteratorStartTimestamp));
            }
        }
        else {
            getShardIteratorRequest.setShardIteratorType("TRIM_HORIZON");
        }
    }
    else {
        getShardIteratorRequest.setShardIteratorType("AFTER_SEQUENCE_NUMBER");
        getShardIteratorRequest.setStartingSequenceNumber(lastReadSequenceNumber);
    }

    GetShardIteratorResult getShardIteratorResult = clientManager.getClient().getShardIterator(getShardIteratorRequest);
    shardIterator = getShardIteratorResult.getShardIterator();
}
Example #12
Source File: MockKinesisClient.java    From presto with Apache License 2.0 4 votes vote down vote up 
@Override
public GetShardIteratorResult getShardIterator(String streamName, String shardId, String shardIteratorType, String startingSequenceNumber)
        throws AmazonServiceException, AmazonClientException
{
    return null;
}
Example #13
Source File: AmazonKinesisMock.java    From beam with Apache License 2.0 4 votes vote down vote up 
@Override
public GetShardIteratorResult getShardIterator(
    String streamName, String shardId, String shardIteratorType) {
  throw new RuntimeException("Not implemented");
}
Example #14
Source File: AmazonKinesisMock.java    From beam with Apache License 2.0 4 votes vote down vote up 
@Override
public GetShardIteratorResult getShardIterator(
    String streamName, String shardId, String shardIteratorType, String startingSequenceNumber) {
  throw new RuntimeException("Not implemented");
}
Example #15
Source File: MockKinesisClient.java    From presto with Apache License 2.0 4 votes vote down vote up 
@Override
public GetShardIteratorResult getShardIterator(String streamName, String shardId, String shardIteratorType)
        throws AmazonServiceException, AmazonClientException
{
    return null;
}
Example #16
Source File: MockKinesisClient.java    From presto-kinesis with Apache License 2.0 4 votes vote down vote up 
@Override
public GetShardIteratorResult getShardIterator(String s, String s1, String s2) throws AmazonServiceException, AmazonClientException
{
    return null;
}
Example #17
Source File: MockKinesisClient.java    From presto-kinesis with Apache License 2.0 4 votes vote down vote up 
@Override
public GetShardIteratorResult getShardIterator(String s, String s1, String s2, String s3) throws AmazonServiceException, AmazonClientException
{
    return null;
}