org.apache.flink.runtime.concurrent.FutureUtils.ConjunctFuture Java Examples

The following examples show how to use org.apache.flink.runtime.concurrent.FutureUtils.ConjunctFuture. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: ConjunctFutureTest.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up
/**
 * Tests that the conjunct future returns upon completion the collection of all future values
 * in the same order in which the futures were inserted.
 */
@Test
public void testConjunctFutureValue() throws Exception {
	final int numberFutures = 10;

	final List<CompletableFuture<Integer>> futures = new ArrayList<>(numberFutures);
	for (int i = 0; i < numberFutures; i++) {
		futures.add(new CompletableFuture<>());
	}

	ConjunctFuture<Collection<Number>> result = FutureUtils.combineAll(futures);

	final List<Tuple2<Integer, CompletableFuture<Integer>>> shuffledFutures = IntStream.range(0, futures.size())
		.mapToObj(index -> Tuple2.of(index, futures.get(index)))
		.collect(Collectors.toList());
	Collections.shuffle(shuffledFutures);

	for (Tuple2<Integer, CompletableFuture<Integer>> shuffledFuture : shuffledFutures) {
		assertThat(result.isDone(), is(false));
		shuffledFuture.f1.complete(shuffledFuture.f0);
	}

	assertThat(result.isDone(), is(true));

	assertThat(result.get(), is(equalTo(IntStream.range(0, numberFutures).boxed().collect(Collectors.toList()))));
}
 
Example #2
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 6 votes vote down vote up
/**
 * Tests that the conjunct future returns upon completion the collection of all future values
 * in the same order in which the futures were inserted.
 */
@Test
public void testConjunctFutureValue() throws Exception {
	final int numberFutures = 10;

	final List<CompletableFuture<Integer>> futures = new ArrayList<>(numberFutures);
	for (int i = 0; i < numberFutures; i++) {
		futures.add(new CompletableFuture<>());
	}

	ConjunctFuture<Collection<Number>> result = FutureUtils.combineAll(futures);

	final List<Tuple2<Integer, CompletableFuture<Integer>>> shuffledFutures = IntStream.range(0, futures.size())
		.mapToObj(index -> Tuple2.of(index, futures.get(index)))
		.collect(Collectors.toList());
	Collections.shuffle(shuffledFutures);

	for (Tuple2<Integer, CompletableFuture<Integer>> shuffledFuture : shuffledFutures) {
		assertThat(result.isDone(), is(false));
		shuffledFuture.f1.complete(shuffledFuture.f0);
	}

	assertThat(result.isDone(), is(true));

	assertThat(result.get(), is(equalTo(IntStream.range(0, numberFutures).boxed().collect(Collectors.toList()))));
}
 
Example #3
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 6 votes vote down vote up
/**
 * Tests that the conjunct future returns upon completion the collection of all future values
 * in the same order in which the futures were inserted.
 */
@Test
public void testConjunctFutureValue() throws Exception {
	final int numberFutures = 10;

	final List<CompletableFuture<Integer>> futures = new ArrayList<>(numberFutures);
	for (int i = 0; i < numberFutures; i++) {
		futures.add(new CompletableFuture<>());
	}

	ConjunctFuture<Collection<Number>> result = FutureUtils.combineAll(futures);

	final List<Tuple2<Integer, CompletableFuture<Integer>>> shuffledFutures = IntStream.range(0, futures.size())
		.mapToObj(index -> Tuple2.of(index, futures.get(index)))
		.collect(Collectors.toList());
	Collections.shuffle(shuffledFutures);

	for (Tuple2<Integer, CompletableFuture<Integer>> shuffledFuture : shuffledFutures) {
		assertThat(result.isDone(), is(false));
		shuffledFuture.f1.complete(shuffledFuture.f0);
	}

	assertThat(result.isDone(), is(true));

	assertThat(result.get(), is(equalTo(IntStream.range(0, numberFutures).boxed().collect(Collectors.toList()))));
}
 
Example #4
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Test
public void testConjunctOfNone() throws Exception {
	final ConjunctFuture<?> result = futureFactory.createFuture(Collections.<java.util.concurrent.CompletableFuture<Object>>emptyList());

	assertEquals(0, result.getNumFuturesTotal());
	assertEquals(0, result.getNumFuturesCompleted());
	assertTrue(result.isDone());
}
 
Example #5
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 5 votes vote down vote up
@Test
public void testConjunctOfNone() throws Exception {
	final ConjunctFuture<?> result = futureFactory.createFuture(Collections.<java.util.concurrent.CompletableFuture<Object>>emptyList());

	assertEquals(0, result.getNumFuturesTotal());
	assertEquals(0, result.getNumFuturesCompleted());
	assertTrue(result.isDone());
}
 
Example #6
Source File: ConjunctFutureTest.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up
@Test
public void testConjunctOfNone() throws Exception {
	final ConjunctFuture<?> result = futureFactory.createFuture(Collections.<java.util.concurrent.CompletableFuture<Object>>emptyList());

	assertEquals(0, result.getNumFuturesTotal());
	assertEquals(0, result.getNumFuturesCompleted());
	assertTrue(result.isDone());
}
 
Example #7
Source File: ExecutionGraph.java    From flink with Apache License 2.0 5 votes vote down vote up
private ConjunctFuture<Void> cancelVerticesAsync() {
	final ArrayList<CompletableFuture<?>> futures = new ArrayList<>(verticesInCreationOrder.size());

	// cancel all tasks (that still need cancelling)
	for (ExecutionJobVertex ejv : verticesInCreationOrder) {
		futures.add(ejv.cancelWithFuture());
	}

	// we build a future that is complete once all vertices have reached a terminal state
	return FutureUtils.waitForAll(futures);
}
 
Example #8
Source File: ExecutionGraph.java    From flink with Apache License 2.0 5 votes vote down vote up
private ConjunctFuture<Void> cancelVerticesAsync() {
	final ArrayList<CompletableFuture<?>> futures = new ArrayList<>(verticesInCreationOrder.size());

	// cancel all tasks (that still need cancelling)
	for (ExecutionJobVertex ejv : verticesInCreationOrder) {
		futures.add(ejv.cancelWithFuture());
	}

	// we build a future that is complete once all vertices have reached a terminal state
	return FutureUtils.waitForAll(futures);
}
 
Example #9
Source File: ExecutionGraph.java    From flink with Apache License 2.0 4 votes vote down vote up
/**
 * Fails the execution graph globally. This failure will not be recovered by a specific
 * failover strategy, but results in a full restart of all tasks.
 *
 * <p>This global failure is meant to be triggered in cases where the consistency of the
 * execution graph' state cannot be guaranteed any more (for example when catching unexpected
 * exceptions that indicate a bug or an unexpected call race), and where a full restart is the
 * safe way to get consistency back.
 *
 * @param t The exception that caused the failure.
 */
public void failGlobal(Throwable t) {
	if (!isLegacyScheduling()) {
		internalTaskFailuresListener.notifyGlobalFailure(t);
		return;
	}

	assertRunningInJobMasterMainThread();

	while (true) {
		JobStatus current = state;
		// stay in these states
		if (current == JobStatus.FAILING ||
			current == JobStatus.SUSPENDED ||
			current.isGloballyTerminalState()) {
			return;
		} else if (transitionState(current, JobStatus.FAILING, t)) {
			initFailureCause(t);

			// make sure no concurrent local or global actions interfere with the failover
			final long globalVersionForRestart = incrementGlobalModVersion();

			final CompletableFuture<Void> ongoingSchedulingFuture = schedulingFuture;

			// cancel ongoing scheduling action
			if (ongoingSchedulingFuture != null) {
				ongoingSchedulingFuture.cancel(false);
			}

			// we build a future that is complete once all vertices have reached a terminal state
			final ConjunctFuture<Void> allTerminal = cancelVerticesAsync();
			FutureUtils.assertNoException(allTerminal.handle(
				(Void ignored, Throwable throwable) -> {
					if (throwable != null) {
						transitionState(
							JobStatus.FAILING,
							JobStatus.FAILED,
							new FlinkException("Could not cancel all execution job vertices properly.", throwable));
					} else {
						allVerticesInTerminalState(globalVersionForRestart);
					}
					return null;
				}));

			return;
		}

		// else: concurrent change to execution state, retry
	}
}
 
Example #10
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public ConjunctFuture<?> createFuture(Collection<? extends java.util.concurrent.CompletableFuture<?>> futures) {
	return FutureUtils.combineAll(futures);
}
 
Example #11
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public ConjunctFuture<?> createFuture(Collection<? extends java.util.concurrent.CompletableFuture<?>> futures) {
	return FutureUtils.waitForAll(futures);
}
 
Example #12
Source File: ExecutionGraph.java    From flink with Apache License 2.0 4 votes vote down vote up
public void cancel() {

		assertRunningInJobMasterMainThread();

		while (true) {
			JobStatus current = state;

			if (current == JobStatus.RUNNING || current == JobStatus.CREATED || current == JobStatus.RESTARTING) {
				if (transitionState(current, JobStatus.CANCELLING)) {

					// make sure no concurrent local actions interfere with the cancellation
					final long globalVersionForRestart = incrementGlobalModVersion();

					final CompletableFuture<Void> ongoingSchedulingFuture = schedulingFuture;

					// cancel ongoing scheduling action
					if (ongoingSchedulingFuture != null) {
						ongoingSchedulingFuture.cancel(false);
					}

					final ConjunctFuture<Void> allTerminal = cancelVerticesAsync();
					allTerminal.whenComplete(
						(Void value, Throwable throwable) -> {
							if (throwable != null) {
								transitionState(
									JobStatus.CANCELLING,
									JobStatus.FAILED,
									new FlinkException(
										"Could not cancel job " + getJobName() + " because not all execution job vertices could be cancelled.",
										throwable));
							} else {
								// cancellations may currently be overridden by failures which trigger
								// restarts, so we need to pass a proper restart global version here
								allVerticesInTerminalState(globalVersionForRestart);
							}
						});

					return;
				}
			}
			// Executions are being canceled. Go into cancelling and wait for
			// all vertices to be in their final state.
			else if (current == JobStatus.FAILING) {
				if (transitionState(current, JobStatus.CANCELLING)) {
					return;
				}
			}
			else {
				// no need to treat other states
				return;
			}
		}
	}
 
Example #13
Source File: ExecutionGraph.java    From flink with Apache License 2.0 4 votes vote down vote up
/**
 * Suspends the current ExecutionGraph.
 *
 * <p>The JobStatus will be directly set to {@link JobStatus#SUSPENDED} iff the current state is not a terminal
 * state. All ExecutionJobVertices will be canceled and the onTerminalState() is executed.
 *
 * <p>The {@link JobStatus#SUSPENDED} state is a local terminal state which stops the execution of the job but does
 * not remove the job from the HA job store so that it can be recovered by another JobManager.
 *
 * @param suspensionCause Cause of the suspension
 */
public void suspend(Throwable suspensionCause) {

	assertRunningInJobMasterMainThread();

	if (state.isTerminalState()) {
		// stay in a terminal state
		return;
	} else if (transitionState(state, JobStatus.SUSPENDED, suspensionCause)) {
		initFailureCause(suspensionCause);

		// make sure no concurrent local actions interfere with the cancellation
		incrementGlobalModVersion();

		// cancel ongoing scheduling action
		if (schedulingFuture != null) {
			schedulingFuture.cancel(false);
		}
		final ArrayList<CompletableFuture<Void>> executionJobVertexTerminationFutures = new ArrayList<>(verticesInCreationOrder.size());

		for (ExecutionJobVertex ejv: verticesInCreationOrder) {
			executionJobVertexTerminationFutures.add(ejv.suspend());
		}

		final ConjunctFuture<Void> jobVerticesTerminationFuture = FutureUtils.waitForAll(executionJobVertexTerminationFutures);

		checkState(jobVerticesTerminationFuture.isDone(), "Suspend needs to happen atomically");

		jobVerticesTerminationFuture.whenComplete(
			(Void ignored, Throwable throwable) -> {
				if (throwable != null) {
					LOG.debug("Could not properly suspend the execution graph.", throwable);
				}

				onTerminalState(state);
				LOG.info("Job {} has been suspended.", getJobID());
			});
	} else {
		throw new IllegalStateException(String.format("Could not suspend because transition from %s to %s failed.", state, JobStatus.SUSPENDED));
	}
}
 
Example #14
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public ConjunctFuture<?> createFuture(Collection<? extends java.util.concurrent.CompletableFuture<?>> futures) {
	return FutureUtils.waitForAll(futures);
}
 
Example #15
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Test
public void testConjunctFutureCompletion() throws Exception {
	// some futures that we combine
	java.util.concurrent.CompletableFuture<Object> future1 = new java.util.concurrent.CompletableFuture<>();
	java.util.concurrent.CompletableFuture<Object> future2 = new java.util.concurrent.CompletableFuture<>();
	java.util.concurrent.CompletableFuture<Object> future3 = new java.util.concurrent.CompletableFuture<>();
	java.util.concurrent.CompletableFuture<Object> future4 = new java.util.concurrent.CompletableFuture<>();

	// some future is initially completed
	future2.complete(new Object());

	// build the conjunct future
	ConjunctFuture<?> result = futureFactory.createFuture(Arrays.asList(future1, future2, future3, future4));

	CompletableFuture<?> resultMapped = result.thenAccept(value -> {});

	assertEquals(4, result.getNumFuturesTotal());
	assertEquals(1, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	// complete two more futures
	future4.complete(new Object());
	assertEquals(2, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	future1.complete(new Object());
	assertEquals(3, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	// complete one future again
	future1.complete(new Object());
	assertEquals(3, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	// complete the final future
	future3.complete(new Object());
	assertEquals(4, result.getNumFuturesCompleted());
	assertTrue(result.isDone());
	assertTrue(resultMapped.isDone());
}
 
Example #16
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Override
public ConjunctFuture<?> createFuture(Collection<? extends java.util.concurrent.CompletableFuture<?>> futures) {
	return FutureUtils.combineAll(futures);
}
 
Example #17
Source File: ExecutionGraph.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up
public void cancel() {

		assertRunningInJobMasterMainThread();

		while (true) {
			JobStatus current = state;

			if (current == JobStatus.RUNNING || current == JobStatus.CREATED) {
				if (transitionState(current, JobStatus.CANCELLING)) {

					// make sure no concurrent local actions interfere with the cancellation
					final long globalVersionForRestart = incrementGlobalModVersion();

					final CompletableFuture<Void> ongoingSchedulingFuture = schedulingFuture;

					// cancel ongoing scheduling action
					if (ongoingSchedulingFuture != null) {
						ongoingSchedulingFuture.cancel(false);
					}

					final ArrayList<CompletableFuture<?>> futures = new ArrayList<>(verticesInCreationOrder.size());

					// cancel all tasks (that still need cancelling)
					for (ExecutionJobVertex ejv : verticesInCreationOrder) {
						futures.add(ejv.cancelWithFuture());
					}

					// we build a future that is complete once all vertices have reached a terminal state
					final ConjunctFuture<Void> allTerminal = FutureUtils.waitForAll(futures);
					allTerminal.whenComplete(
						(Void value, Throwable throwable) -> {
							if (throwable != null) {
								transitionState(
									JobStatus.CANCELLING,
									JobStatus.FAILED,
									new FlinkException(
										"Could not cancel job " + getJobName() + " because not all execution job vertices could be cancelled.",
										throwable));
							} else {
								// cancellations may currently be overridden by failures which trigger
								// restarts, so we need to pass a proper restart global version here
								allVerticesInTerminalState(globalVersionForRestart);
							}
						});

					return;
				}
			}
			// Executions are being canceled. Go into cancelling and wait for
			// all vertices to be in their final state.
			else if (current == JobStatus.FAILING) {
				if (transitionState(current, JobStatus.CANCELLING)) {
					return;
				}
			}
			// All vertices have been cancelled and it's safe to directly go
			// into the canceled state.
			else if (current == JobStatus.RESTARTING) {
				synchronized (progressLock) {
					if (transitionState(current, JobStatus.CANCELED)) {
						onTerminalState(JobStatus.CANCELED);

						LOG.info("Canceled during restart.");
						return;
					}
				}
			}
			else {
				// no need to treat other states
				return;
			}
		}
	}
 
Example #18
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 4 votes vote down vote up
@Test
public void testConjunctFutureCompletion() throws Exception {
	// some futures that we combine
	java.util.concurrent.CompletableFuture<Object> future1 = new java.util.concurrent.CompletableFuture<>();
	java.util.concurrent.CompletableFuture<Object> future2 = new java.util.concurrent.CompletableFuture<>();
	java.util.concurrent.CompletableFuture<Object> future3 = new java.util.concurrent.CompletableFuture<>();
	java.util.concurrent.CompletableFuture<Object> future4 = new java.util.concurrent.CompletableFuture<>();

	// some future is initially completed
	future2.complete(new Object());

	// build the conjunct future
	ConjunctFuture<?> result = futureFactory.createFuture(Arrays.asList(future1, future2, future3, future4));

	CompletableFuture<?> resultMapped = result.thenAccept(value -> {});

	assertEquals(4, result.getNumFuturesTotal());
	assertEquals(1, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	// complete two more futures
	future4.complete(new Object());
	assertEquals(2, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	future1.complete(new Object());
	assertEquals(3, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	// complete one future again
	future1.complete(new Object());
	assertEquals(3, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	// complete the final future
	future3.complete(new Object());
	assertEquals(4, result.getNumFuturesCompleted());
	assertTrue(result.isDone());
	assertTrue(resultMapped.isDone());
}
 
Example #19
Source File: ExecutionGraph.java    From flink with Apache License 2.0 4 votes vote down vote up
/**
 * Fails the execution graph globally. This failure will not be recovered by a specific
 * failover strategy, but results in a full restart of all tasks.
 *
 * <p>This global failure is meant to be triggered in cases where the consistency of the
 * execution graph' state cannot be guaranteed any more (for example when catching unexpected
 * exceptions that indicate a bug or an unexpected call race), and where a full restart is the
 * safe way to get consistency back.
 *
 * @param t The exception that caused the failure.
 */
public void failGlobal(Throwable t) {

	assertRunningInJobMasterMainThread();

	while (true) {
		JobStatus current = state;
		// stay in these states
		if (current == JobStatus.FAILING ||
			current == JobStatus.SUSPENDED ||
			current.isGloballyTerminalState()) {
			return;
		} else if (transitionState(current, JobStatus.FAILING, t)) {
			initFailureCause(t);

			// make sure no concurrent local or global actions interfere with the failover
			final long globalVersionForRestart = incrementGlobalModVersion();

			final CompletableFuture<Void> ongoingSchedulingFuture = schedulingFuture;

			// cancel ongoing scheduling action
			if (ongoingSchedulingFuture != null) {
				ongoingSchedulingFuture.cancel(false);
			}

			// we build a future that is complete once all vertices have reached a terminal state
			final ConjunctFuture<Void> allTerminal = cancelVerticesAsync();
			FutureUtils.assertNoException(allTerminal.handle(
				(Void ignored, Throwable throwable) -> {
					if (throwable != null) {
						transitionState(
							JobStatus.FAILING,
							JobStatus.FAILED,
							new FlinkException("Could not cancel all execution job vertices properly.", throwable));
					} else {
						allVerticesInTerminalState(globalVersionForRestart);
					}
					return null;
				}));

			return;
		}

		// else: concurrent change to execution state, retry
	}
}
 
Example #20
Source File: ExecutionGraph.java    From flink with Apache License 2.0 4 votes vote down vote up
/**
 * Suspends the current ExecutionGraph.
 *
 * <p>The JobStatus will be directly set to {@link JobStatus#SUSPENDED} iff the current state is not a terminal
 * state. All ExecutionJobVertices will be canceled and the onTerminalState() is executed.
 *
 * <p>The {@link JobStatus#SUSPENDED} state is a local terminal state which stops the execution of the job but does
 * not remove the job from the HA job store so that it can be recovered by another JobManager.
 *
 * @param suspensionCause Cause of the suspension
 */
public void suspend(Throwable suspensionCause) {

	assertRunningInJobMasterMainThread();

	if (state.isTerminalState()) {
		// stay in a terminal state
		return;
	} else if (transitionState(state, JobStatus.SUSPENDED, suspensionCause)) {
		initFailureCause(suspensionCause);

		// make sure no concurrent local actions interfere with the cancellation
		incrementGlobalModVersion();

		// cancel ongoing scheduling action
		if (schedulingFuture != null) {
			schedulingFuture.cancel(false);
		}
		final ArrayList<CompletableFuture<Void>> executionJobVertexTerminationFutures = new ArrayList<>(verticesInCreationOrder.size());

		for (ExecutionJobVertex ejv: verticesInCreationOrder) {
			executionJobVertexTerminationFutures.add(ejv.suspend());
		}

		final ConjunctFuture<Void> jobVerticesTerminationFuture = FutureUtils.waitForAll(executionJobVertexTerminationFutures);

		checkState(jobVerticesTerminationFuture.isDone(), "Suspend needs to happen atomically");

		jobVerticesTerminationFuture.whenComplete(
			(Void ignored, Throwable throwable) -> {
				if (throwable != null) {
					LOG.debug("Could not properly suspend the execution graph.", throwable);
				}

				onTerminalState(state);
				LOG.info("Job {} has been suspended.", getJobID());
			});
	} else {
		throw new IllegalStateException(String.format("Could not suspend because transition from %s to %s failed.", state, JobStatus.SUSPENDED));
	}
}
 
Example #21
Source File: ExecutionGraph.java    From flink with Apache License 2.0 4 votes vote down vote up
public void cancel() {

		assertRunningInJobMasterMainThread();

		while (true) {
			JobStatus current = state;

			if (current == JobStatus.RUNNING || current == JobStatus.CREATED) {
				if (transitionState(current, JobStatus.CANCELLING)) {

					// make sure no concurrent local actions interfere with the cancellation
					final long globalVersionForRestart = incrementGlobalModVersion();

					final CompletableFuture<Void> ongoingSchedulingFuture = schedulingFuture;

					// cancel ongoing scheduling action
					if (ongoingSchedulingFuture != null) {
						ongoingSchedulingFuture.cancel(false);
					}

					final ConjunctFuture<Void> allTerminal = cancelVerticesAsync();
					allTerminal.whenComplete(
						(Void value, Throwable throwable) -> {
							if (throwable != null) {
								transitionState(
									JobStatus.CANCELLING,
									JobStatus.FAILED,
									new FlinkException(
										"Could not cancel job " + getJobName() + " because not all execution job vertices could be cancelled.",
										throwable));
							} else {
								// cancellations may currently be overridden by failures which trigger
								// restarts, so we need to pass a proper restart global version here
								allVerticesInTerminalState(globalVersionForRestart);
							}
						});

					return;
				}
			}
			// Executions are being canceled. Go into cancelling and wait for
			// all vertices to be in their final state.
			else if (current == JobStatus.FAILING) {
				if (transitionState(current, JobStatus.CANCELLING)) {
					return;
				}
			}
			// All vertices have been cancelled and it's safe to directly go
			// into the canceled state.
			else if (current == JobStatus.RESTARTING) {
				synchronized (progressLock) {
					if (transitionState(current, JobStatus.CANCELED)) {
						onTerminalState(JobStatus.CANCELED);

						LOG.info("Canceled during restart.");
						return;
					}
				}
			}
			else {
				// no need to treat other states
				return;
			}
		}
	}
 
Example #22
Source File: ConjunctFutureTest.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up
@Override
public ConjunctFuture<?> createFuture(Collection<? extends java.util.concurrent.CompletableFuture<?>> futures) {
	return FutureUtils.waitForAll(futures);
}
 
Example #23
Source File: ConjunctFutureTest.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up
@Override
public ConjunctFuture<?> createFuture(Collection<? extends java.util.concurrent.CompletableFuture<?>> futures) {
	return FutureUtils.combineAll(futures);
}
 
Example #24
Source File: ConjunctFutureTest.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up
@Test
public void testConjunctFutureCompletion() throws Exception {
	// some futures that we combine
	java.util.concurrent.CompletableFuture<Object> future1 = new java.util.concurrent.CompletableFuture<>();
	java.util.concurrent.CompletableFuture<Object> future2 = new java.util.concurrent.CompletableFuture<>();
	java.util.concurrent.CompletableFuture<Object> future3 = new java.util.concurrent.CompletableFuture<>();
	java.util.concurrent.CompletableFuture<Object> future4 = new java.util.concurrent.CompletableFuture<>();

	// some future is initially completed
	future2.complete(new Object());

	// build the conjunct future
	ConjunctFuture<?> result = futureFactory.createFuture(Arrays.asList(future1, future2, future3, future4));

	CompletableFuture<?> resultMapped = result.thenAccept(value -> {});

	assertEquals(4, result.getNumFuturesTotal());
	assertEquals(1, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	// complete two more futures
	future4.complete(new Object());
	assertEquals(2, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	future1.complete(new Object());
	assertEquals(3, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	// complete one future again
	future1.complete(new Object());
	assertEquals(3, result.getNumFuturesCompleted());
	assertFalse(result.isDone());
	assertFalse(resultMapped.isDone());

	// complete the final future
	future3.complete(new Object());
	assertEquals(4, result.getNumFuturesCompleted());
	assertTrue(result.isDone());
	assertTrue(resultMapped.isDone());
}
 
Example #25
Source File: ExecutionGraph.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up
/**
 * Fails the execution graph globally. This failure will not be recovered by a specific
 * failover strategy, but results in a full restart of all tasks.
 *
 * <p>This global failure is meant to be triggered in cases where the consistency of the
 * execution graph' state cannot be guaranteed any more (for example when catching unexpected
 * exceptions that indicate a bug or an unexpected call race), and where a full restart is the
 * safe way to get consistency back.
 *
 * @param t The exception that caused the failure.
 */
public void failGlobal(Throwable t) {

	assertRunningInJobMasterMainThread();

	while (true) {
		JobStatus current = state;
		// stay in these states
		if (current == JobStatus.FAILING ||
			current == JobStatus.SUSPENDED ||
			current.isGloballyTerminalState()) {
			return;
		} else if (transitionState(current, JobStatus.FAILING, t)) {
			initFailureCause(t);

			// make sure no concurrent local or global actions interfere with the failover
			final long globalVersionForRestart = incrementGlobalModVersion();

			final CompletableFuture<Void> ongoingSchedulingFuture = schedulingFuture;

			// cancel ongoing scheduling action
			if (ongoingSchedulingFuture != null) {
				ongoingSchedulingFuture.cancel(false);
			}

			// we build a future that is complete once all vertices have reached a terminal state
			final ArrayList<CompletableFuture<?>> futures = new ArrayList<>(verticesInCreationOrder.size());

			// cancel all tasks (that still need cancelling)
			for (ExecutionJobVertex ejv : verticesInCreationOrder) {
				futures.add(ejv.cancelWithFuture());
			}

			final ConjunctFuture<Void> allTerminal = FutureUtils.waitForAll(futures);
			allTerminal.whenComplete(
				(Void ignored, Throwable throwable) -> {
					if (throwable != null) {
						transitionState(
							JobStatus.FAILING,
							JobStatus.FAILED,
							new FlinkException("Could not cancel all execution job vertices properly.", throwable));
					} else {
						allVerticesInTerminalState(globalVersionForRestart);
					}
				});

			return;
		}

		// else: concurrent change to execution state, retry
	}
}
 
Example #26
Source File: ExecutionGraph.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up
/**
 * Suspends the current ExecutionGraph.
 *
 * <p>The JobStatus will be directly set to {@link JobStatus#SUSPENDED} iff the current state is not a terminal
 * state. All ExecutionJobVertices will be canceled and the onTerminalState() is executed.
 *
 * <p>The {@link JobStatus#SUSPENDED} state is a local terminal state which stops the execution of the job but does
 * not remove the job from the HA job store so that it can be recovered by another JobManager.
 *
 * @param suspensionCause Cause of the suspension
 */
public void suspend(Throwable suspensionCause) {

	assertRunningInJobMasterMainThread();

	if (state.isTerminalState()) {
		// stay in a terminal state
		return;
	} else if (transitionState(state, JobStatus.SUSPENDED, suspensionCause)) {
		initFailureCause(suspensionCause);

		// make sure no concurrent local actions interfere with the cancellation
		incrementGlobalModVersion();

		// cancel ongoing scheduling action
		if (schedulingFuture != null) {
			schedulingFuture.cancel(false);
		}
		final ArrayList<CompletableFuture<Void>> executionJobVertexTerminationFutures = new ArrayList<>(verticesInCreationOrder.size());

		for (ExecutionJobVertex ejv: verticesInCreationOrder) {
			executionJobVertexTerminationFutures.add(ejv.suspend());
		}

		final ConjunctFuture<Void> jobVerticesTerminationFuture = FutureUtils.waitForAll(executionJobVertexTerminationFutures);

		checkState(jobVerticesTerminationFuture.isDone(), "Suspend needs to happen atomically");

		jobVerticesTerminationFuture.whenComplete(
			(Void ignored, Throwable throwable) -> {
				if (throwable != null) {
					LOG.debug("Could not properly suspend the execution graph.", throwable);
				}

				onTerminalState(state);
				LOG.info("Job {} has been suspended.", getJobID());
			});
	} else {
		throw new IllegalStateException(String.format("Could not suspend because transition from %s to %s failed.", state, JobStatus.SUSPENDED));
	}
}
 
Example #27
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 votes vote down vote up
ConjunctFuture<?> createFuture(Collection<? extends java.util.concurrent.CompletableFuture<?>> futures); 
Example #28
Source File: ConjunctFutureTest.java    From Flink-CEPplus with Apache License 2.0 votes vote down vote up
ConjunctFuture<?> createFuture(Collection<? extends java.util.concurrent.CompletableFuture<?>> futures); 
Example #29
Source File: ConjunctFutureTest.java    From flink with Apache License 2.0 votes vote down vote up
ConjunctFuture<?> createFuture(Collection<? extends java.util.concurrent.CompletableFuture<?>> futures);