org.apache.flink.streaming.api.functions.co.KeyedBroadcastProcessFunction Java Examples

The following examples show how to use org.apache.flink.streaming.api.functions.co.KeyedBroadcastProcessFunction. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: ProcessFunctionTestHarnesses.java    From flink with Apache License 2.0 6 votes vote down vote up 
/**
 * Returns an initialized test harness for {@link KeyedBroadcastProcessFunction}.
 *
 * @param function instance of a {@link KeyedBroadcastProcessFunction} under test
 * @param descriptors broadcast state descriptors used in the {@code function}
 * @param <K> key type
 * @param <IN1> type of input stream elements
 * @param <IN2> type of broadcast stream elements
 * @param <OUT> type of output elements
 * @return {@link BroadcastOperatorTestHarness} wrapped around {@code function}
 */
public static <K, IN1, IN2, OUT>
KeyedBroadcastOperatorTestHarness<K, IN1, IN2, OUT> forKeyedBroadcastProcessFunction(
	final KeyedBroadcastProcessFunction<K, IN1, IN2, OUT> function,
	final KeySelector<IN1, K> keySelector,
	final TypeInformation<K> keyType,
	final MapStateDescriptor<?, ?>... descriptors)
	throws Exception {

	KeyedBroadcastOperatorTestHarness<K, IN1, IN2, OUT> testHarness =
		new KeyedBroadcastOperatorTestHarness<>(
			new CoBroadcastWithKeyedOperator<>(
				Preconditions.checkNotNull(function), Arrays.asList(descriptors)),
			keySelector,
			keyType,
			1,
			1,
			0);

	testHarness.open();
	return testHarness;
}
Example #2
Source File: CoBroadcastWithKeyedOperatorTest.java    From flink with Apache License 2.0 6 votes vote down vote up 
private static <KEY, IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final TypeInformation<KEY> keyTypeInfo,
		final KeySelector<IN1, KEY> keyKeySelector,
		final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx) throws Exception {

	return getInitializedTestHarness(
			keyTypeInfo,
			keyKeySelector,
			function,
			maxParallelism,
			numTasks,
			taskIdx,
			null);
}
Example #3
Source File: CoBroadcastWithKeyedOperatorTest.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up 
private static <KEY, IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final TypeInformation<KEY> keyTypeInfo,
		final KeySelector<IN1, KEY> keyKeySelector,
		final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx) throws Exception {

	return getInitializedTestHarness(
			keyTypeInfo,
			keyKeySelector,
			function,
			maxParallelism,
			numTasks,
			taskIdx,
			null);
}
Example #4
Source File: BroadcastConnectedStream.java    From flink with Apache License 2.0 6 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a {@link KeyedStream} and applies the given
 * {@link KeyedBroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link KeyedBroadcastProcessFunction} that is called for each element in the stream.
 * @param <KS> The type of the keys in the keyed stream.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <KS, OUT> SingleOutputStreamOperator<OUT> process(final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function) {

	TypeInformation<OUT> outTypeInfo = TypeExtractor.getBinaryOperatorReturnType(
			function,
			KeyedBroadcastProcessFunction.class,
			1,
			2,
			3,
			TypeExtractor.NO_INDEX,
			getType1(),
			getType2(),
			Utils.getCallLocationName(),
			true);

	return process(function, outTypeInfo);
}
Example #5
Source File: BroadcastConnectedStream.java    From flink with Apache License 2.0 6 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a {@link KeyedStream} and applies the given
 * {@link KeyedBroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link KeyedBroadcastProcessFunction} that is called for each element in the stream.
 * @param <KS> The type of the keys in the keyed stream.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <KS, OUT> SingleOutputStreamOperator<OUT> process(final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function) {

	TypeInformation<OUT> outTypeInfo = TypeExtractor.getBinaryOperatorReturnType(
			function,
			KeyedBroadcastProcessFunction.class,
			1,
			2,
			3,
			TypeExtractor.NO_INDEX,
			getType1(),
			getType2(),
			Utils.getCallLocationName(),
			true);

	return process(function, outTypeInfo);
}
Example #6
Source File: CoBroadcastWithKeyedOperatorTest.java    From flink with Apache License 2.0 6 votes vote down vote up 
private static <KEY, IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final TypeInformation<KEY> keyTypeInfo,
		final KeySelector<IN1, KEY> keyKeySelector,
		final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx) throws Exception {

	return getInitializedTestHarness(
			keyTypeInfo,
			keyKeySelector,
			function,
			maxParallelism,
			numTasks,
			taskIdx,
			null);
}
Example #7
Source File: BroadcastConnectedStream.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a {@link KeyedStream} and applies the given
 * {@link KeyedBroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link KeyedBroadcastProcessFunction} that is called for each element in the stream.
 * @param <KS> The type of the keys in the keyed stream.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <KS, OUT> SingleOutputStreamOperator<OUT> process(final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function) {

	TypeInformation<OUT> outTypeInfo = TypeExtractor.getBinaryOperatorReturnType(
			function,
			KeyedBroadcastProcessFunction.class,
			1,
			2,
			3,
			TypeExtractor.NO_INDEX,
			getType1(),
			getType2(),
			Utils.getCallLocationName(),
			true);

	return process(function, outTypeInfo);
}
Example #8
Source File: CoBroadcastWithKeyedOperator.java    From flink with Apache License 2.0 5 votes vote down vote up 
OnTimerContextImpl(
		final ExecutionConfig executionConfig,
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final TimerService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #9
Source File: CoBroadcastWithKeyedOperatorTest.java    From flink with Apache License 2.0 5 votes vote down vote up 
private static <KEY, IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final TypeInformation<KEY> keyTypeInfo,
		final KeySelector<IN1, KEY> keyKeySelector,
		final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx,
		final OperatorSubtaskState initState) throws Exception {

	final TwoInputStreamOperatorTestHarness<IN1, IN2, OUT>  testHarness =
			new KeyedTwoInputStreamOperatorTestHarness<>(
					new CoBroadcastWithKeyedOperator<>(
							Preconditions.checkNotNull(function),
							Collections.singletonList(STATE_DESCRIPTOR)),
					keyKeySelector,
					null,
					keyTypeInfo,
					maxParallelism,
					numTasks,
					taskIdx
			);

	testHarness.setup();
	testHarness.initializeState(initState);
	testHarness.open();

	return testHarness;
}
Example #10
Source File: CoBroadcastWithKeyedOperatorTest.java    From flink with Apache License 2.0 5 votes vote down vote up 
private static <KEY, IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final TypeInformation<KEY> keyTypeInfo,
		final KeySelector<IN1, KEY> keyKeySelector,
		final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> function) throws Exception {

	return getInitializedTestHarness(
			keyTypeInfo,
			keyKeySelector,
			function,
			1,
			1,
			0);
}
Example #11
Source File: CoBroadcastWithKeyedOperator.java    From flink with Apache License 2.0 5 votes vote down vote up 
OnTimerContextImpl(
		final ExecutionConfig executionConfig,
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final TimerService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #12
Source File: CoBroadcastWithKeyedOperator.java    From flink with Apache License 2.0 5 votes vote down vote up 
ReadOnlyContextImpl(
		final ExecutionConfig executionConfig,
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final TimerService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #13
Source File: CoBroadcastWithKeyedOperator.java    From flink with Apache License 2.0 5 votes vote down vote up 
ReadWriteContextImpl (
		final ExecutionConfig executionConfig,
		final KeyedStateBackend<KS> keyedStateBackend,
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final TimerService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.keyedStateBackend = Preconditions.checkNotNull(keyedStateBackend);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #14
Source File: BroadcastConnectedStream.java    From flink with Apache License 2.0 5 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a {@link KeyedStream} and applies the given
 * {@link KeyedBroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link KeyedBroadcastProcessFunction} that is called for each element in the stream.
 * @param outTypeInfo The type of the output elements.
 * @param <KS> The type of the keys in the keyed stream.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <KS, OUT> SingleOutputStreamOperator<OUT> process(
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final TypeInformation<OUT> outTypeInfo) {

	Preconditions.checkNotNull(function);
	Preconditions.checkArgument(inputStream1 instanceof KeyedStream,
			"A KeyedBroadcastProcessFunction can only be used on a keyed stream.");

	TwoInputStreamOperator<IN1, IN2, OUT> operator =
			new CoBroadcastWithKeyedOperator<>(clean(function), broadcastStateDescriptors);
	return transform("Co-Process-Broadcast-Keyed", outTypeInfo, operator);
}
Example #15
Source File: CoBroadcastWithKeyedOperatorTest.java    From flink with Apache License 2.0 5 votes vote down vote up 
private static <KEY, IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final TypeInformation<KEY> keyTypeInfo,
		final KeySelector<IN1, KEY> keyKeySelector,
		final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx,
		final OperatorSubtaskState initState) throws Exception {

	final TwoInputStreamOperatorTestHarness<IN1, IN2, OUT>  testHarness =
			new KeyedTwoInputStreamOperatorTestHarness<>(
					new CoBroadcastWithKeyedOperator<>(
							Preconditions.checkNotNull(function),
							Collections.singletonList(STATE_DESCRIPTOR)),
					keyKeySelector,
					null,
					keyTypeInfo,
					maxParallelism,
					numTasks,
					taskIdx
			);

	testHarness.setup();
	testHarness.initializeState(initState);
	testHarness.open();

	return testHarness;
}
Example #16
Source File: CoBroadcastWithKeyedOperatorTest.java    From flink with Apache License 2.0 5 votes vote down vote up 
private static <KEY, IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final TypeInformation<KEY> keyTypeInfo,
		final KeySelector<IN1, KEY> keyKeySelector,
		final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> function) throws Exception {

	return getInitializedTestHarness(
			keyTypeInfo,
			keyKeySelector,
			function,
			1,
			1,
			0);
}
Example #17
Source File: CoBroadcastWithKeyedOperator.java    From flink with Apache License 2.0 5 votes vote down vote up 
ReadOnlyContextImpl(
		final ExecutionConfig executionConfig,
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final TimerService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #18
Source File: CoBroadcastWithKeyedOperator.java    From flink with Apache License 2.0 5 votes vote down vote up 
ReadWriteContextImpl (
		final ExecutionConfig executionConfig,
		final KeyedStateBackend<KS> keyedStateBackend,
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final TimerService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.keyedStateBackend = Preconditions.checkNotNull(keyedStateBackend);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #19
Source File: BroadcastConnectedStream.java    From flink with Apache License 2.0 5 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a {@link KeyedStream} and applies the given
 * {@link KeyedBroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link KeyedBroadcastProcessFunction} that is called for each element in the stream.
 * @param outTypeInfo The type of the output elements.
 * @param <KS> The type of the keys in the keyed stream.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <KS, OUT> SingleOutputStreamOperator<OUT> process(
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final TypeInformation<OUT> outTypeInfo) {

	Preconditions.checkNotNull(function);
	Preconditions.checkArgument(inputStream1 instanceof KeyedStream,
			"A KeyedBroadcastProcessFunction can only be used on a keyed stream.");

	TwoInputStreamOperator<IN1, IN2, OUT> operator =
			new CoBroadcastWithKeyedOperator<>(clean(function), broadcastStateDescriptors);
	return transform("Co-Process-Broadcast-Keyed", outTypeInfo, operator);
}
Example #20
Source File: CoBroadcastWithKeyedOperatorTest.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
private static <KEY, IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final TypeInformation<KEY> keyTypeInfo,
		final KeySelector<IN1, KEY> keyKeySelector,
		final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx,
		final OperatorSubtaskState initState) throws Exception {

	final TwoInputStreamOperatorTestHarness<IN1, IN2, OUT>  testHarness =
			new KeyedTwoInputStreamOperatorTestHarness<>(
					new CoBroadcastWithKeyedOperator<>(
							Preconditions.checkNotNull(function),
							Collections.singletonList(STATE_DESCRIPTOR)),
					keyKeySelector,
					null,
					keyTypeInfo,
					maxParallelism,
					numTasks,
					taskIdx
			);

	testHarness.setup();
	testHarness.initializeState(initState);
	testHarness.open();

	return testHarness;
}
Example #21
Source File: CoBroadcastWithKeyedOperatorTest.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
private static <KEY, IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final TypeInformation<KEY> keyTypeInfo,
		final KeySelector<IN1, KEY> keyKeySelector,
		final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> function) throws Exception {

	return getInitializedTestHarness(
			keyTypeInfo,
			keyKeySelector,
			function,
			1,
			1,
			0);
}
Example #22
Source File: CoBroadcastWithKeyedOperator.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
OnTimerContextImpl(
		final ExecutionConfig executionConfig,
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final TimerService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #23
Source File: CoBroadcastWithKeyedOperator.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
ReadOnlyContextImpl(
		final ExecutionConfig executionConfig,
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final TimerService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #24
Source File: CoBroadcastWithKeyedOperator.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
ReadWriteContextImpl (
		final ExecutionConfig executionConfig,
		final KeyedStateBackend<KS> keyedStateBackend,
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final TimerService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.keyedStateBackend = Preconditions.checkNotNull(keyedStateBackend);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #25
Source File: BroadcastConnectedStream.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a {@link KeyedStream} and applies the given
 * {@link KeyedBroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link KeyedBroadcastProcessFunction} that is called for each element in the stream.
 * @param outTypeInfo The type of the output elements.
 * @param <KS> The type of the keys in the keyed stream.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <KS, OUT> SingleOutputStreamOperator<OUT> process(
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final TypeInformation<OUT> outTypeInfo) {

	Preconditions.checkNotNull(function);
	Preconditions.checkArgument(inputStream1 instanceof KeyedStream,
			"A KeyedBroadcastProcessFunction can only be used on a keyed stream.");

	TwoInputStreamOperator<IN1, IN2, OUT> operator =
			new CoBroadcastWithKeyedOperator<>(clean(function), broadcastStateDescriptors);
	return transform("Co-Process-Broadcast-Keyed", outTypeInfo, operator);
}
Example #26
Source File: CoBroadcastWithKeyedOperator.java    From flink with Apache License 2.0 4 votes vote down vote up 
public CoBroadcastWithKeyedOperator(
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors) {
	super(function);
	this.broadcastStateDescriptors = Preconditions.checkNotNull(broadcastStateDescriptors);
}
Example #27
Source File: CoBroadcastWithKeyedOperator.java    From flink with Apache License 2.0 4 votes vote down vote up 
public CoBroadcastWithKeyedOperator(
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors) {
	super(function);
	this.broadcastStateDescriptors = Preconditions.checkNotNull(broadcastStateDescriptors);
}
Example #28
Source File: CoBroadcastWithKeyedOperator.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
public CoBroadcastWithKeyedOperator(
		final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> function,
		final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors) {
	super(function);
	this.broadcastStateDescriptors = Preconditions.checkNotNull(broadcastStateDescriptors);
}