org.apache.flink.streaming.api.functions.co.BroadcastProcessFunction Java Examples

The following examples show how to use org.apache.flink.streaming.api.functions.co.BroadcastProcessFunction. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: ProcessFunctionTestHarnesses.java    From flink with Apache License 2.0 6 votes vote down vote up 
/**
 * Returns an initialized test harness for {@link BroadcastProcessFunction}.
 *
 * @param function instance of a {@link BroadcastProcessFunction} under test
 * @param descriptors broadcast state descriptors used in the {@code function}
 * @param <IN1> type of input stream elements
 * @param <IN2> type of broadcast stream elements
 * @param <OUT> type of output elements
 * @return {@link BroadcastOperatorTestHarness} wrapped around {@code function}
 */
public static <IN1, IN2, OUT>
BroadcastOperatorTestHarness<IN1, IN2, OUT> forBroadcastProcessFunction(
	final BroadcastProcessFunction<IN1, IN2, OUT> function,
	final MapStateDescriptor<?, ?>... descriptors)
	throws Exception {

	BroadcastOperatorTestHarness<IN1, IN2, OUT> testHarness =
		new BroadcastOperatorTestHarness<>(
			new CoBroadcastWithNonKeyedOperator<>(
				Preconditions.checkNotNull(function), Arrays.asList(descriptors)),
			1,
			1,
			0);
	testHarness.open();
	return testHarness;
}
Example #2
Source File: CoBroadcastWithNonKeyedOperatorTest.java    From flink with Apache License 2.0 6 votes vote down vote up 
private static <IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx,
		final OperatorSubtaskState initState,
		final MapStateDescriptor<?, ?>... descriptors) throws Exception {

	TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> testHarness = new TwoInputStreamOperatorTestHarness<>(
			new CoBroadcastWithNonKeyedOperator<>(
					Preconditions.checkNotNull(function),
					Arrays.asList(descriptors)),
			maxParallelism, numTasks, taskIdx
	);
	testHarness.setup();
	testHarness.initializeState(initState);
	testHarness.open();

	return testHarness;
}
Example #3
Source File: CoBroadcastWithNonKeyedOperatorTest.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up 
private static <IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx,
		final OperatorSubtaskState initState,
		final MapStateDescriptor<?, ?>... descriptors) throws Exception {

	TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> testHarness = new TwoInputStreamOperatorTestHarness<>(
			new CoBroadcastWithNonKeyedOperator<>(
					Preconditions.checkNotNull(function),
					Arrays.asList(descriptors)),
			maxParallelism, numTasks, taskIdx
	);
	testHarness.setup();
	testHarness.initializeState(initState);
	testHarness.open();

	return testHarness;
}
Example #4
Source File: BroadcastConnectedStream.java    From flink with Apache License 2.0 6 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a non-keyed {@link DataStream} and applies the given
 * {@link BroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link BroadcastProcessFunction} that is called for each element in the stream.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <OUT> SingleOutputStreamOperator<OUT> process(final BroadcastProcessFunction<IN1, IN2, OUT> function) {

	TypeInformation<OUT> outTypeInfo = TypeExtractor.getBinaryOperatorReturnType(
			function,
			BroadcastProcessFunction.class,
			0,
			1,
			2,
			TypeExtractor.NO_INDEX,
			getType1(),
			getType2(),
			Utils.getCallLocationName(),
			true);

	return process(function, outTypeInfo);
}
Example #5
Source File: BroadcastConnectedStream.java    From flink with Apache License 2.0 6 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a non-keyed {@link DataStream} and applies the given
 * {@link BroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link BroadcastProcessFunction} that is called for each element in the stream.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <OUT> SingleOutputStreamOperator<OUT> process(final BroadcastProcessFunction<IN1, IN2, OUT> function) {

	TypeInformation<OUT> outTypeInfo = TypeExtractor.getBinaryOperatorReturnType(
			function,
			BroadcastProcessFunction.class,
			0,
			1,
			2,
			TypeExtractor.NO_INDEX,
			getType1(),
			getType2(),
			Utils.getCallLocationName(),
			true);

	return process(function, outTypeInfo);
}
Example #6
Source File: CoBroadcastWithNonKeyedOperatorTest.java    From flink with Apache License 2.0 6 votes vote down vote up 
private static <IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx,
		final OperatorSubtaskState initState,
		final MapStateDescriptor<?, ?>... descriptors) throws Exception {

	TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> testHarness = new TwoInputStreamOperatorTestHarness<>(
			new CoBroadcastWithNonKeyedOperator<>(
					Preconditions.checkNotNull(function),
					Arrays.asList(descriptors)),
			maxParallelism, numTasks, taskIdx
	);
	testHarness.setup();
	testHarness.initializeState(initState);
	testHarness.open();

	return testHarness;
}
Example #7
Source File: BroadcastConnectedStream.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a non-keyed {@link DataStream} and applies the given
 * {@link BroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link BroadcastProcessFunction} that is called for each element in the stream.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <OUT> SingleOutputStreamOperator<OUT> process(final BroadcastProcessFunction<IN1, IN2, OUT> function) {

	TypeInformation<OUT> outTypeInfo = TypeExtractor.getBinaryOperatorReturnType(
			function,
			BroadcastProcessFunction.class,
			0,
			1,
			2,
			TypeExtractor.NO_INDEX,
			getType1(),
			getType2(),
			Utils.getCallLocationName(),
			true);

	return process(function, outTypeInfo);
}
Example #8
Source File: CoBroadcastWithNonKeyedOperator.java    From flink with Apache License 2.0 5 votes vote down vote up 
ReadOnlyContextImpl(
		final ExecutionConfig executionConfig,
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final ProcessingTimeService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #9
Source File: CoBroadcastWithNonKeyedOperatorTest.java    From flink with Apache License 2.0 5 votes vote down vote up 
private static <IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx,
		final MapStateDescriptor<?, ?>... descriptors) throws Exception {

	return getInitializedTestHarness(
			function,
			maxParallelism,
			numTasks,
			taskIdx,
			null,
			descriptors);
}
Example #10
Source File: CoBroadcastWithNonKeyedOperatorTest.java    From flink with Apache License 2.0 5 votes vote down vote up 
private static <IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final MapStateDescriptor<?, ?>... descriptors) throws Exception {

	return getInitializedTestHarness(
			function,
			1,
			1,
			0,
			descriptors);
}
Example #11
Source File: CoBroadcastWithNonKeyedOperator.java    From flink with Apache License 2.0 5 votes vote down vote up 
ReadOnlyContextImpl(
		final ExecutionConfig executionConfig,
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final ProcessingTimeService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #12
Source File: CoBroadcastWithNonKeyedOperator.java    From flink with Apache License 2.0 5 votes vote down vote up 
ReadWriteContextImpl(
		final ExecutionConfig executionConfig,
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final ProcessingTimeService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #13
Source File: BroadcastConnectedStream.java    From flink with Apache License 2.0 5 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a non-keyed {@link DataStream} and applies the given
 * {@link BroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link BroadcastProcessFunction} that is called for each element in the stream.
 * @param outTypeInfo The type of the output elements.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <OUT> SingleOutputStreamOperator<OUT> process(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final TypeInformation<OUT> outTypeInfo) {

	Preconditions.checkNotNull(function);
	Preconditions.checkArgument(!(inputStream1 instanceof KeyedStream),
			"A BroadcastProcessFunction can only be used on a non-keyed stream.");

	TwoInputStreamOperator<IN1, IN2, OUT> operator =
			new CoBroadcastWithNonKeyedOperator<>(clean(function), broadcastStateDescriptors);
	return transform("Co-Process-Broadcast", outTypeInfo, operator);
}
Example #14
Source File: CoBroadcastWithNonKeyedOperatorTest.java    From flink with Apache License 2.0 5 votes vote down vote up 
private static <IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx,
		final MapStateDescriptor<?, ?>... descriptors) throws Exception {

	return getInitializedTestHarness(
			function,
			maxParallelism,
			numTasks,
			taskIdx,
			null,
			descriptors);
}
Example #15
Source File: CoBroadcastWithNonKeyedOperatorTest.java    From flink with Apache License 2.0 5 votes vote down vote up 
private static <IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final MapStateDescriptor<?, ?>... descriptors) throws Exception {

	return getInitializedTestHarness(
			function,
			1,
			1,
			0,
			descriptors);
}
Example #16
Source File: CoBroadcastWithNonKeyedOperator.java    From flink with Apache License 2.0 5 votes vote down vote up 
ReadWriteContextImpl(
		final ExecutionConfig executionConfig,
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final ProcessingTimeService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #17
Source File: BroadcastConnectedStream.java    From flink with Apache License 2.0 5 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a non-keyed {@link DataStream} and applies the given
 * {@link BroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link BroadcastProcessFunction} that is called for each element in the stream.
 * @param outTypeInfo The type of the output elements.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <OUT> SingleOutputStreamOperator<OUT> process(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final TypeInformation<OUT> outTypeInfo) {

	Preconditions.checkNotNull(function);
	Preconditions.checkArgument(!(inputStream1 instanceof KeyedStream),
			"A BroadcastProcessFunction can only be used on a non-keyed stream.");

	TwoInputStreamOperator<IN1, IN2, OUT> operator =
			new CoBroadcastWithNonKeyedOperator<>(clean(function), broadcastStateDescriptors);
	return transform("Co-Process-Broadcast", outTypeInfo, operator);
}
Example #18
Source File: CoBroadcastWithNonKeyedOperatorTest.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
private static <IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final int maxParallelism,
		final int numTasks,
		final int taskIdx,
		final MapStateDescriptor<?, ?>... descriptors) throws Exception {

	return getInitializedTestHarness(
			function,
			maxParallelism,
			numTasks,
			taskIdx,
			null,
			descriptors);
}
Example #19
Source File: CoBroadcastWithNonKeyedOperatorTest.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
private static <IN1, IN2, OUT> TwoInputStreamOperatorTestHarness<IN1, IN2, OUT> getInitializedTestHarness(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final MapStateDescriptor<?, ?>... descriptors) throws Exception {

	return getInitializedTestHarness(
			function,
			1,
			1,
			0,
			descriptors);
}
Example #20
Source File: CoBroadcastWithNonKeyedOperator.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
ReadOnlyContextImpl(
		final ExecutionConfig executionConfig,
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final ProcessingTimeService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #21
Source File: CoBroadcastWithNonKeyedOperator.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
ReadWriteContextImpl(
		final ExecutionConfig executionConfig,
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates,
		final ProcessingTimeService timerService) {

	function.super();
	this.config = Preconditions.checkNotNull(executionConfig);
	this.states = Preconditions.checkNotNull(broadcastStates);
	this.timerService = Preconditions.checkNotNull(timerService);
}
Example #22
Source File: BroadcastConnectedStream.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
/**
 * Assumes as inputs a {@link BroadcastStream} and a non-keyed {@link DataStream} and applies the given
 * {@link BroadcastProcessFunction} on them, thereby creating a transformed output stream.
 *
 * @param function The {@link BroadcastProcessFunction} that is called for each element in the stream.
 * @param outTypeInfo The type of the output elements.
 * @param <OUT> The type of the output elements.
 * @return The transformed {@link DataStream}.
 */
@PublicEvolving
public <OUT> SingleOutputStreamOperator<OUT> process(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final TypeInformation<OUT> outTypeInfo) {

	Preconditions.checkNotNull(function);
	Preconditions.checkArgument(!(inputStream1 instanceof KeyedStream),
			"A BroadcastProcessFunction can only be used on a non-keyed stream.");

	TwoInputStreamOperator<IN1, IN2, OUT> operator =
			new CoBroadcastWithNonKeyedOperator<>(clean(function), broadcastStateDescriptors);
	return transform("Co-Process-Broadcast", outTypeInfo, operator);
}
Example #23
Source File: CoBroadcastWithNonKeyedOperator.java    From flink with Apache License 2.0 4 votes vote down vote up 
public CoBroadcastWithNonKeyedOperator(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors) {
	super(function);
	this.broadcastStateDescriptors = Preconditions.checkNotNull(broadcastStateDescriptors);
}
Example #24
Source File: CoBroadcastWithNonKeyedOperator.java    From flink with Apache License 2.0 4 votes vote down vote up 
public CoBroadcastWithNonKeyedOperator(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors) {
	super(function);
	this.broadcastStateDescriptors = Preconditions.checkNotNull(broadcastStateDescriptors);
}
Example #25
Source File: CoBroadcastWithNonKeyedOperator.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
public CoBroadcastWithNonKeyedOperator(
		final BroadcastProcessFunction<IN1, IN2, OUT> function,
		final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors) {
	super(function);
	this.broadcastStateDescriptors = Preconditions.checkNotNull(broadcastStateDescriptors);
}