org.apache.flink.runtime.iterative.concurrent.Broker Java Examples

The following examples show how to use org.apache.flink.runtime.iterative.concurrent.Broker. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: IterationHeadTask.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up
/**
 * The iteration head prepares the backchannel: it allocates memory, instantiates a {@link BlockingBackChannel} and
 * hands it to the iteration tail via a {@link Broker} singleton.
 **/
private BlockingBackChannel initBackChannel() throws Exception {

	/* get the size of the memory available to the backchannel */
	int backChannelMemoryPages = getMemoryManager().computeNumberOfPages(this.config.getRelativeBackChannelMemory());

	/* allocate the memory available to the backchannel */
	List<MemorySegment> segments = new ArrayList<MemorySegment>();
	int segmentSize = getMemoryManager().getPageSize();
	getMemoryManager().allocatePages(this, segments, backChannelMemoryPages);

	/* instantiate the backchannel */
	BlockingBackChannel backChannel = new BlockingBackChannel(new SerializedUpdateBuffer(segments, segmentSize,
		getIOManager()));

	/* hand the backchannel over to the iteration tail */
	Broker<BlockingBackChannel> broker = BlockingBackChannelBroker.instance();
	broker.handIn(brokerKey(), backChannel);

	return backChannel;
}
 
Example #2
Source File: AbstractIterativeTask.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up
/**
 * Creates a new solution set update output collector.
 *
 * <p>This collector is used by {@link IterationIntermediateTask} or {@link IterationTailTask} to update the
 * solution set of workset iterations. Depending on the task configuration, either a fast (non-probing)
 * {@link org.apache.flink.runtime.iterative.io.SolutionSetFastUpdateOutputCollector} or normal (re-probing)
 * {@link SolutionSetUpdateOutputCollector} is created.
 *
 * <p>If a non-null delegate is given, the new {@link Collector} will write back to the solution set and also call
 * collect(T) of the delegate.
 *
 * @param delegate null -OR- a delegate collector to be called by the newly created collector
 * @return a new {@link org.apache.flink.runtime.iterative.io.SolutionSetFastUpdateOutputCollector} or
 * {@link SolutionSetUpdateOutputCollector}
 */
protected Collector<OT> createSolutionSetUpdateOutputCollector(Collector<OT> delegate) {
	Broker<Object> solutionSetBroker = SolutionSetBroker.instance();

	Object ss = solutionSetBroker.get(brokerKey());
	if (ss instanceof CompactingHashTable) {
		@SuppressWarnings("unchecked")
		CompactingHashTable<OT> solutionSet = (CompactingHashTable<OT>) ss;
		return new SolutionSetUpdateOutputCollector<OT>(solutionSet, delegate);
	}
	else if (ss instanceof JoinHashMap) {
		@SuppressWarnings("unchecked")
		JoinHashMap<OT> map = (JoinHashMap<OT>) ss;
		return new SolutionSetObjectsUpdateOutputCollector<OT>(map, delegate);
	} else {
		throw new RuntimeException("Unrecognized solution set handle: " + ss);
	}
}
 
Example #3
Source File: IterationHeadTask.java    From flink with Apache License 2.0 6 votes vote down vote up
/**
 * The iteration head prepares the backchannel: it allocates memory, instantiates a {@link BlockingBackChannel} and
 * hands it to the iteration tail via a {@link Broker} singleton.
 **/
private BlockingBackChannel initBackChannel() throws Exception {

	/* get the size of the memory available to the backchannel */
	int backChannelMemoryPages = getMemoryManager().computeNumberOfPages(this.config.getRelativeBackChannelMemory());

	/* allocate the memory available to the backchannel */
	List<MemorySegment> segments = new ArrayList<MemorySegment>();
	int segmentSize = getMemoryManager().getPageSize();
	getMemoryManager().allocatePages(this, segments, backChannelMemoryPages);

	/* instantiate the backchannel */
	BlockingBackChannel backChannel = new BlockingBackChannel(new SerializedUpdateBuffer(segments, segmentSize,
		getIOManager()));

	/* hand the backchannel over to the iteration tail */
	Broker<BlockingBackChannel> broker = BlockingBackChannelBroker.instance();
	broker.handIn(brokerKey(), backChannel);

	return backChannel;
}
 
Example #4
Source File: AbstractIterativeTask.java    From flink with Apache License 2.0 6 votes vote down vote up
/**
 * Creates a new solution set update output collector.
 *
 * <p>This collector is used by {@link IterationIntermediateTask} or {@link IterationTailTask} to update the
 * solution set of workset iterations. Depending on the task configuration, either a fast (non-probing)
 * {@link org.apache.flink.runtime.iterative.io.SolutionSetFastUpdateOutputCollector} or normal (re-probing)
 * {@link SolutionSetUpdateOutputCollector} is created.
 *
 * <p>If a non-null delegate is given, the new {@link Collector} will write back to the solution set and also call
 * collect(T) of the delegate.
 *
 * @param delegate null -OR- a delegate collector to be called by the newly created collector
 * @return a new {@link org.apache.flink.runtime.iterative.io.SolutionSetFastUpdateOutputCollector} or
 * {@link SolutionSetUpdateOutputCollector}
 */
protected Collector<OT> createSolutionSetUpdateOutputCollector(Collector<OT> delegate) {
	Broker<Object> solutionSetBroker = SolutionSetBroker.instance();

	Object ss = solutionSetBroker.get(brokerKey());
	if (ss instanceof CompactingHashTable) {
		@SuppressWarnings("unchecked")
		CompactingHashTable<OT> solutionSet = (CompactingHashTable<OT>) ss;
		return new SolutionSetUpdateOutputCollector<OT>(solutionSet, delegate);
	}
	else if (ss instanceof JoinHashMap) {
		@SuppressWarnings("unchecked")
		JoinHashMap<OT> map = (JoinHashMap<OT>) ss;
		return new SolutionSetObjectsUpdateOutputCollector<OT>(map, delegate);
	} else {
		throw new RuntimeException("Unrecognized solution set handle: " + ss);
	}
}
 
Example #5
Source File: IterationHeadTask.java    From flink with Apache License 2.0 6 votes vote down vote up
/**
 * The iteration head prepares the backchannel: it allocates memory, instantiates a {@link BlockingBackChannel} and
 * hands it to the iteration tail via a {@link Broker} singleton.
 **/
private BlockingBackChannel initBackChannel() throws Exception {

	/* get the size of the memory available to the backchannel */
	int backChannelMemoryPages = getMemoryManager().computeNumberOfPages(this.config.getRelativeBackChannelMemory());

	/* allocate the memory available to the backchannel */
	List<MemorySegment> segments = new ArrayList<MemorySegment>();
	int segmentSize = getMemoryManager().getPageSize();
	getMemoryManager().allocatePages(this, segments, backChannelMemoryPages);

	/* instantiate the backchannel */
	BlockingBackChannel backChannel = new BlockingBackChannel(new SerializedUpdateBuffer(segments, segmentSize,
		getIOManager()));

	/* hand the backchannel over to the iteration tail */
	Broker<BlockingBackChannel> broker = BlockingBackChannelBroker.instance();
	broker.handIn(brokerKey(), backChannel);

	return backChannel;
}
 
Example #6
Source File: AbstractIterativeTask.java    From flink with Apache License 2.0 6 votes vote down vote up
/**
 * Creates a new solution set update output collector.
 *
 * <p>This collector is used by {@link IterationIntermediateTask} or {@link IterationTailTask} to update the
 * solution set of workset iterations. Depending on the task configuration, either a fast (non-probing)
 * {@link org.apache.flink.runtime.iterative.io.SolutionSetFastUpdateOutputCollector} or normal (re-probing)
 * {@link SolutionSetUpdateOutputCollector} is created.
 *
 * <p>If a non-null delegate is given, the new {@link Collector} will write back to the solution set and also call
 * collect(T) of the delegate.
 *
 * @param delegate null -OR- a delegate collector to be called by the newly created collector
 * @return a new {@link org.apache.flink.runtime.iterative.io.SolutionSetFastUpdateOutputCollector} or
 * {@link SolutionSetUpdateOutputCollector}
 */
protected Collector<OT> createSolutionSetUpdateOutputCollector(Collector<OT> delegate) {
	Broker<Object> solutionSetBroker = SolutionSetBroker.instance();

	Object ss = solutionSetBroker.get(brokerKey());
	if (ss instanceof CompactingHashTable) {
		@SuppressWarnings("unchecked")
		CompactingHashTable<OT> solutionSet = (CompactingHashTable<OT>) ss;
		return new SolutionSetUpdateOutputCollector<OT>(solutionSet, delegate);
	}
	else if (ss instanceof JoinHashMap) {
		@SuppressWarnings("unchecked")
		JoinHashMap<OT> map = (JoinHashMap<OT>) ss;
		return new SolutionSetObjectsUpdateOutputCollector<OT>(map, delegate);
	} else {
		throw new RuntimeException("Unrecognized solution set handle: " + ss);
	}
}