Java Code Examples for org.apache.flink.configuration.ResourceManagerOptions#CONTAINERIZED_TASK_MANAGER_ENV_PREFIX

The following examples show how to use org.apache.flink.configuration.ResourceManagerOptions#CONTAINERIZED_TASK_MANAGER_ENV_PREFIX . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: ContaineredTaskManagerParameters.java    From flink with Apache License 2.0 6 votes vote down vote up 
/**
 * Computes the parameters to be used to start a TaskManager Java process.
 *
 * @param config The Flink configuration.
 * @param taskExecutorProcessSpec The resource specifics of the task executor.
 * @return The parameters to start the TaskManager processes with.
 */
public static ContaineredTaskManagerParameters create(
		Configuration config,
		TaskExecutorProcessSpec taskExecutorProcessSpec) {

	// obtain the additional environment variables from the configuration
	final HashMap<String, String> envVars = new HashMap<>();
	final String prefix = ResourceManagerOptions.CONTAINERIZED_TASK_MANAGER_ENV_PREFIX;

	for (String key : config.keySet()) {
		if (key.startsWith(prefix) && key.length() > prefix.length()) {
			// remove prefix
			String envVarKey = key.substring(prefix.length());
			envVars.put(envVarKey, config.getString(key, null));
		}
	}

	// done
	return new ContaineredTaskManagerParameters(taskExecutorProcessSpec, envVars);
}
Example 2
Source File: ContaineredTaskManagerParameters.java    From Flink-CEPplus with Apache License 2.0 5 votes vote down vote up 
/**
 * Computes the parameters to be used to start a TaskManager Java process.
 *
 * @param config The Flink configuration.
 * @param containerMemoryMB The size of the complete container, in megabytes.
 * @return The parameters to start the TaskManager processes with.
 */
public static ContaineredTaskManagerParameters create(
		Configuration config,
		long containerMemoryMB,
		int numSlots) {
	// (1) try to compute how much memory used by container
	final long cutoffMB = calculateCutoffMB(config, containerMemoryMB);

	// (2) split the remaining Java memory between heap and off-heap
	final long heapSizeMB = TaskManagerServices.calculateHeapSizeMB(containerMemoryMB - cutoffMB, config);
	// use the cut-off memory for off-heap (that was its intention)
	final long offHeapSizeMB = containerMemoryMB - heapSizeMB;

	// (3) obtain the additional environment variables from the configuration
	final HashMap<String, String> envVars = new HashMap<>();
	final String prefix = ResourceManagerOptions.CONTAINERIZED_TASK_MANAGER_ENV_PREFIX;

	for (String key : config.keySet()) {
		if (key.startsWith(prefix) && key.length() > prefix.length()) {
			// remove prefix
			String envVarKey = key.substring(prefix.length());
			envVars.put(envVarKey, config.getString(key, null));
		}
	}

	// done
	return new ContaineredTaskManagerParameters(
		containerMemoryMB, heapSizeMB, offHeapSizeMB, numSlots, envVars);
}
Example 3
Source File: ContaineredTaskManagerParameters.java    From flink with Apache License 2.0 5 votes vote down vote up 
/**
 * Computes the parameters to be used to start a TaskManager Java process.
 *
 * @param config The Flink configuration.
 * @param containerMemoryMB The size of the complete container, in megabytes.
 * @return The parameters to start the TaskManager processes with.
 */
public static ContaineredTaskManagerParameters create(
		Configuration config,
		long containerMemoryMB,
		int numSlots) {
	// (1) try to compute how much memory used by container
	final long cutoffMB = calculateCutoffMB(config, containerMemoryMB);

	// (2) split the remaining Java memory between heap and off-heap
	final long heapSizeMB = TaskManagerServices.calculateHeapSizeMB(containerMemoryMB - cutoffMB, config);
	// use the cut-off memory for off-heap (that was its intention)
	final long offHeapSizeMB = containerMemoryMB - heapSizeMB;

	// (3) obtain the additional environment variables from the configuration
	final HashMap<String, String> envVars = new HashMap<>();
	final String prefix = ResourceManagerOptions.CONTAINERIZED_TASK_MANAGER_ENV_PREFIX;

	for (String key : config.keySet()) {
		if (key.startsWith(prefix) && key.length() > prefix.length()) {
			// remove prefix
			String envVarKey = key.substring(prefix.length());
			envVars.put(envVarKey, config.getString(key, null));
		}
	}

	// done
	return new ContaineredTaskManagerParameters(
		containerMemoryMB, heapSizeMB, offHeapSizeMB, numSlots, envVars);
}