jMetalSP is a software platform for dynamic multi-objective Big Data Optimization. It combines the features of the jMetal multi-objective optimization framework with the Apache Spark cluster computing system. jMetal provides the optimization infrastructure for implementing both the Big Data optimization problems and the dynamic metaheuristic to solve them; Spark allows to manage the streaming data sources, allowing to effectively use the computing power of Hadoop clusters when processing large amounts of data.
Please, note that jMetalSP is a project in continuous development. If you have any question or suggestion of desirable features to be included, feel free to contact us. The current version is jMetalSP 2.0.
We are currently working on a redesign of the framework with the following ideas in mind:
The architecture of the current development version of jMetalSP (Version 1.1) is depicted in the following UML class diagram:
A jMetalSPApplication is composed of:
DynamicProblem class (the problem to be solved).DynamicAlgorithm class (the algorithm to use).StreamingDataSource objects, which process the incoming data and as a result they change make some change in the DynamicProblem.AlgorithmDataConsumer objects, which receive the results being generated by the DynamicAlgorithm.StreamingRuntime object to configure the streaming engine.The implementation of jMetalSP applies Java generics to ensure that all the componentes are compatible. The declaration of the jMetalSPApplication class and its main componentes is the following:
public class JMetalSPApplication<
S extends Solution<?>,
P extends DynamicProblem<S, ?>,
A extends DynamicAlgorithm<?, ? extends ObservedData<?>>> {
private List<StreamingDataSource<?>> streamingDataSourceList;
private List<DataConsumer<?>> algorithmDataConsumerList;
private StreamingRuntime streamingRuntime;
private P problem;
private A algorithm;
...
}This way, by using generics the Java compiler can check that all the components fit together.
InDM2 a new dynamic multi-objective optimization algorithm that allows the preferences of the decision maker (DM) to be incorporated into the search process. When solving a dynamic multi-objective optimization problem with InDM2, the DM can not only express her/his preferences by means of one or more reference points, which define the desired region of interest, but also those points can be also modified interactively.
The following example applications are included in the current development version:
DynamicContinuousApplication. Example of using NSGA-II, MOCell, SMPSO or WASF-GA to solve the FDA problems using the default streaming runtime, i.e. without SparkDynamicContinuousApplicationWithSpark. Example of using NSGA-II, MOCell, SMPSO or WASF-GA to solve the FDA problems using Spark.DynamicTSPApplication. Example of using NSGA-II or MOCell or SMPSO to solve a bi-objective TSP problem using the default streaming runtime, i.e. without Spark. The streaming data source simulates changes in the cost matrix (no external data source is used). This is a simplied version the algorithm presented in MOD 2016.NSGAIIRunner. This example, included in the paper to be presented in EMO 2017, shows how to configure the standard NSGA-II algorithm to solve a modified version of the ZDT1 problem using the Spark evaluator to evaluate the solutions of the population in parallel. InDM2RunnerForContinuousProblems. This example, included in the paper published in SWEVO 2018, shows how to configure the InDM2 with Interactive R-NSGA-II and Interactive WASFGA algorithms to solve FDA problems.InDM2RunnerForContinuousProblems3D. This example, included in the paper published in SWEVO 2018, shows how to configure the InDM2 with Interactive R-NSGA-II and Interactive WASFGA algorithms to solve FDA problems, showing results in a 3D plot.InDM2RunnerForNYTSP. This example, included in the paper to be presented in SWEVO 2018, shows how to configure the InDM2 with Interactive R-NSGA-II and Interactive WASFGA algorithms to solve TSP problem with traffic data from New York.InDM2RunnerForContinuousAutoUpdateProblems. This example is as InDM2RunnerForContinuousProblems but in this case the problem is updated after 25000 algorithm evaluations.DynamicContinuousApplicationAVRO. Example of using NSGA-II, MOCell, SMPSO or WASF-GA to solve the FDA problems using the default streaming runtime, i.e. without Spark and AVRO is used for interchange communications.DynamicContinuousApplicationFlink. This example is the same that DynamicContinuousApplication but with Flink as streaming runtime.DynamicContinuousApplicationFlinkkafka. This example is the same that DynamicContinuousApplicationFlink but using kafka for communicating components.DynamicContinuousApplicationFlinkkafkaAVRO. This example is the same that DynamicContinuousApplicationFlinkKafka but using AVRO adn Kafka for communicating components.DynamicContinuousApplicationWithSparkkafkaAVRO. Example of using NSGA-II, MOCell, SMPSO or WASF-GA to solve the FDA problems using Spark and Kafka-AVRO for communicating components.DSMPSORunnerForContinuousProblems. This example, shows how to configure the Dynamic SMPSO to solve FDA problems.RNSGAIIRunnerForContinuousProblems. This example, shows how to configure the Dynamic R-NSGA-II to solve FDA problems.SMPSORPRunnerForContinuousProblems. This example, shows how to configure the Dynamic SMPSO/RP to solve DF problems.In order to run the example DynamicContinuousApplicationWithSpark, it is necessary
to run first the CounterProvider program whose goal is to store continuously in a directory files containing the data (i.e., the value of a counter) that will read by Spark in streaming.
To run the examples that do not use Spark you need:
To execute the codes with Spark: