Java Code Examples for kafka.api.OffsetRequest#EarliestTime

The following examples show how to use kafka.api.OffsetRequest#EarliestTime . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: SimpleConsumerThread.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
/**
 * For a set of partitions, if a partition is set with the special offsets {@link OffsetRequest#EarliestTime()}
 * or {@link OffsetRequest#LatestTime()}, replace them with actual offsets requested via a Kafka consumer.
 *
 * @param consumer The consumer connected to lead broker
 * @param partitions The list of partitions we need offsets for
 */
private static void requestAndSetEarliestOrLatestOffsetsFromKafka(
		SimpleConsumer consumer,
		List<KafkaTopicPartitionState<TopicAndPartition>> partitions) throws Exception {
	Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo = new HashMap<>();
	for (KafkaTopicPartitionState<TopicAndPartition> part : partitions) {
		if (part.getOffset() == OffsetRequest.EarliestTime() || part.getOffset() == OffsetRequest.LatestTime()) {
			requestInfo.put(part.getKafkaPartitionHandle(), new PartitionOffsetRequestInfo(part.getOffset(), 1));
		}
	}

	requestAndSetOffsetsFromKafka(consumer, partitions, requestInfo);
}
Example 2
Source File: Kafka08Fetcher.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
/**
 * Retrieve the behaviour of "auto.offset.reset" from the config properties.
 * A partition needs to fallback to "auto.offset.reset" as default offset when
 * we can't find offsets in ZK to start from in {@link StartupMode#GROUP_OFFSETS} startup mode.
 *
 * @param config kafka consumer properties
 * @return either OffsetRequest.LatestTime() or OffsetRequest.EarliestTime()
 */
private static long getInvalidOffsetBehavior(Properties config) {
	final String val = config.getProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "largest");
	if (val.equals("largest") || val.equals("latest")) { // largest is kafka 0.8, latest is kafka 0.9
		return OffsetRequest.LatestTime();
	} else {
		return OffsetRequest.EarliestTime();
	}
}
Example 3
Source File: SimpleConsumerThread.java    From flink with Apache License 2.0 5 votes vote down vote up 
/**
 * For a set of partitions, if a partition is set with the special offsets {@link OffsetRequest#EarliestTime()}
 * or {@link OffsetRequest#LatestTime()}, replace them with actual offsets requested via a Kafka consumer.
 *
 * @param consumer The consumer connected to lead broker
 * @param partitions The list of partitions we need offsets for
 */
private static void requestAndSetEarliestOrLatestOffsetsFromKafka(
		SimpleConsumer consumer,
		List<KafkaTopicPartitionState<TopicAndPartition>> partitions) throws Exception {
	Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo = new HashMap<>();
	for (KafkaTopicPartitionState<TopicAndPartition> part : partitions) {
		if (part.getOffset() == OffsetRequest.EarliestTime() || part.getOffset() == OffsetRequest.LatestTime()) {
			requestInfo.put(part.getKafkaPartitionHandle(), new PartitionOffsetRequestInfo(part.getOffset(), 1));
		}
	}

	requestAndSetOffsetsFromKafka(consumer, partitions, requestInfo);
}
Example 4
Source File: Kafka08Fetcher.java    From flink with Apache License 2.0 5 votes vote down vote up 
/**
 * Retrieve the behaviour of "auto.offset.reset" from the config properties.
 * A partition needs to fallback to "auto.offset.reset" as default offset when
 * we can't find offsets in ZK to start from in {@link StartupMode#GROUP_OFFSETS} startup mode.
 *
 * @param config kafka consumer properties
 * @return either OffsetRequest.LatestTime() or OffsetRequest.EarliestTime()
 */
private static long getInvalidOffsetBehavior(Properties config) {
	final String val = config.getProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "largest");
	if (val.equals("largest") || val.equals("latest")) { // largest is kafka 0.8, latest is kafka 0.9
		return OffsetRequest.LatestTime();
	} else {
		return OffsetRequest.EarliestTime();
	}
}
Example 5
Source File: KafkaUtilsTest.java    From storm-kafka-0.8-plus with Apache License 2.0 5 votes vote down vote up 
@Test
public void getOffsetFromConfigAndDontForceFromStart() {
    config.forceFromStart = false;
    config.startOffsetTime = OffsetRequest.EarliestTime();
    createTopicAndSendMessage();
    long latestOffset = KafkaUtils.getOffset(simpleConsumer, config.topic, 0, OffsetRequest.LatestTime());
    long offsetFromConfig = KafkaUtils.getOffset(simpleConsumer, config.topic, 0, config);
    assertThat(latestOffset, is(equalTo(offsetFromConfig)));
}
Example 6
Source File: KafkaUtilsTest.java    From storm-kafka-0.8-plus with Apache License 2.0 5 votes vote down vote up 
@Test
public void getOffsetFromConfigAndFroceFromStart() {
    config.forceFromStart = true;
    config.startOffsetTime = OffsetRequest.EarliestTime();
    createTopicAndSendMessage();
    long earliestOffset = KafkaUtils.getOffset(simpleConsumer, config.topic, 0, OffsetRequest.EarliestTime());
    long offsetFromConfig = KafkaUtils.getOffset(simpleConsumer, config.topic, 0, config);
    assertThat(earliestOffset, is(equalTo(offsetFromConfig)));
}