hyperspark-master
- src
  - main
    - resources
      - sol_ta089.txt
      - sol_ta066.txt
      - sol_ta112.txt
      - inst_ta012.txt
      - sol_ta068.txt
      - sol_ta116.txt
      - sol_ta058.txt
      - sol_ta021.txt
      - sol_ta085.txt
      - sol_ta064.txt
      - inst_ta117.txt
      - sol_ta093.txt
      - inst_ta014.txt
      - sol_ta096.txt
      - inst_ta046.txt
      - inst_ta053.txt
      - sol_ta083.txt
      - sol_ta100.txt
      - sol_ta041.txt
      - sol_ta099.txt
      - sol_ta084.txt
      - inst_ta045.txt
      - sol_ta037.txt
      - inst_ta115.txt
      - inst_ta039.txt
      - inst_ta079.txt
      - sol_ta001.txt
      - sol_ta033.txt
      - sol_ta075.txt
      - inst_ta108.txt
      - sol_ta053.txt
      - inst_ta005.txt
      - inst_ta080.txt
      - sol_ta023.txt
      - sol_ta120.txt
      - inst_ta106.txt
      - inst_ta075.txt
      - inst_ta028.txt
      - sol_ta117.txt
      - sol_ta080.txt
      - inst_ta059.txt
      - inst_ta015.txt
      - sol_ta047.txt
      - inst_ta058.txt
      - sol_ta069.txt
      - sol_ta102.txt
      - sol_ta092.txt
      - inst_ta047.txt
      - inst_ta064.txt
      - sol_ta025.txt
      - inst_ta074.txt
      - inst_ta087.txt
      - inst_ta084.txt
      - inst_ta017.txt
      - inst_ta086.txt
      - inst_ta034.txt
      - inst_ta066.txt
      - inst_ta065.txt
      - sol_ta022.txt
      - inst_ta107.txt
      - sol_ta057.txt
      - inst_ta095.txt
      - inst_ta112.txt
      - inst_ta025.txt
      - sol_ta090.txt
      - sol_ta016.txt
      - inst_ta076.txt
      - inst_ta011.txt
      - inst_ta024.txt
      - inst_ta104.txt
      - sol_ta010.txt
      - inst_ta041.txt
      - inst_ta109.txt
      - sol_ta094.txt
      - sol_ta024.txt
      - sol_ta042.txt
      - sol_ta008.txt
      - inst_ta101.txt
      - bestFoundSolutions.txt
      - inst_ta037.txt
      - sol_ta061.txt
      - sol_ta062.txt
      - inst_ta071.txt
      - sol_ta078.txt
      - sol_ta082.txt
      - inst_ta088.txt
      - sol_ta006.txt
      - sol_ta103.txt
      - inst_ta083.txt
      - inst_ta029.txt
      - inst_ta040.txt
      - inst_ta057.txt
      - inst_ta097.txt
      - inst_ta002.txt
      - sol_ta029.txt
      - inst_ta096.txt
      - sol_ta107.txt
      - inst_ta093.txt
      - inst_ta069.txt
      - inst_ta081.txt
      - sol_ta111.txt
      - inst_ta114.txt
      - sol_ta050.txt
      - inst_ta060.txt
      - inst_ta103.txt
      - sol_ta074.txt
      - sol_ta005.txt
      - inst_ta038.txt
      - inst_ta036.txt
      - sol_ta115.txt
      - inst_ta113.txt
      - sol_ta030.txt
      - sol_ta043.txt
      - sol_ta052.txt
      - inst_ta110.txt
      - sol_ta108.txt
      - inst_ta019.txt
      - inst_ta043.txt
      - sol_ta109.txt
      - sol_ta044.txt
      - inst_ta100.txt
      - inst_ta042.txt
      - inst_ta099.txt
      - sol_ta048.txt
      - sol_ta056.txt
      - inst_ta092.txt
      - sol_ta095.txt
      - inst_ta009.txt
      - inst_ta032.txt
      - sol_ta032.txt
      - sol_ta065.txt
      - sol_ta063.txt
      - sol_ta110.txt
      - sol_ta026.txt
      - sol_ta114.txt
      - sol_ta118.txt
      - inst_ta119.txt
      - sol_ta015.txt
      - sol_ta034.txt
      - sol_ta072.txt
      - inst_ta027.txt
      - sol_ta106.txt
      - inst_ta020.txt
      - inst_ta082.txt
      - sol_ta081.txt
      - sol_ta079.txt
      - inst_ta072.txt
      - inst_ta035.txt
      - inst_ta062.txt
      - sol_ta017.txt
      - inst_ta026.txt
      - inst_ta091.txt
      - sol_ta011.txt
      - sol_ta014.txt
      - inst_ta013.txt
      - inst_ta052.txt
      - sol_ta003.txt
      - inst_ta044.txt
      - sol_ta051.txt
      - inst_ta001.txt
      - inst_ta067.txt
      - inst_ta098.txt
      - sol_ta077.txt
      - sol_ta054.txt
      - sol_ta067.txt
      - inst_ta073.txt
      - inst_ta094.txt
      - sol_ta012.txt
      - sol_ta038.txt
      - sol_ta091.txt
      - inst_ta118.txt
      - sol_ta007.txt
      - inst_ta050.txt
      - inst_ta070.txt
      - inst_ta056.txt
      - sol_ta087.txt
      - inst_ta061.txt
      - inst_ta016.txt
      - sol_ta018.txt
      - inst_ta051.txt
      - inst_ta077.txt
      - inst_ta022.txt
      - inst_ta004.txt
      - sol_ta097.txt
      - sol_ta105.txt
      - sol_ta059.txt
      - sol_ta098.txt
      - sol_ta019.txt
      - sol_ta088.txt
      - inst_ta003.txt
      - sol_ta004.txt
      - sol_ta073.txt
      - inst_ta018.txt
      - inst_ta008.txt
      - sol_ta031.txt
      - inst_ta023.txt
      - sol_ta002.txt
      - inst_ta031.txt
      - sol_ta040.txt
      - inst_ta111.txt
      - sol_ta101.txt
      - inst_ta055.txt
      - inst_ta105.txt
      - inst_ta089.txt
      - inst_ta102.txt
      - inst_ta033.txt
      - sol_ta119.txt
      - inst_ta006.txt
      - sol_ta104.txt
      - inst_ta049.txt
      - inst_ta090.txt
      - sol_ta070.txt
      - inst_ta063.txt
      - inst_ta116.txt
      - sol_ta046.txt
      - inst_ta120.txt
      - inst_ta054.txt
      - sol_ta009.txt
      - sol_ta027.txt
      - inst_ta021.txt
      - inst_ta030.txt
      - sol_ta045.txt
      - inst_ta048.txt
      - inst_ta068.txt
      - sol_ta020.txt
      - sol_ta071.txt
      - inst_ta078.txt
      - inst_ta010.txt
      - sol_ta039.txt
      - sol_ta035.txt
      - sol_ta013.txt
      - sol_ta055.txt
      - inst_ta007.txt
      - sol_ta049.txt
      - sol_ta028.txt
      - sol_ta076.txt
      - inst_ta085.txt
      - sol_ta086.txt
      - sol_ta060.txt
      - sol_ta113.txt
      - sol_ta036.txt
    - scala
      - it
        polimi
        hyperh
        solution
        Solution.scala
        EvaluatedSolution.scala
        algorithms
        Algorithm.scala
        spark
        MapReduceHandler.scala
        StoppingCondition.scala
        FrameworkConf.scala
        DistributedDataset.scala
        Framework.scala
        SeedingStrategy.scala
        apps
        MainApp.scala
        TesterApp.scala
        YarnClusterApp.scala
        LocalRddApp.scala
        LocalTesterApp.scala
        CooperativeSame.scala
        LocalApp.scala
        experiments
        Experiment3.scala
        Experiment.scala
        Experiment1.scala
        Experiment2.scala
        types
        Types.scala
        problem
        Problem.scala
      - util
        FileManager.scala
        Random.scala
        TimeObject.scala
        ConsolePrinter.scala
        Performance.scala
        BestSolutionsGenerator.scala
        CustomLogger.scala
        RoundRobinIterator.scala
      - pfsp
        solution
        PfsSolution.scala
        PfsEvaluatedSolution.scala
        algorithms
        MMASAlgorithm.scala
        MMMASAlgorithm.scala
        ISAAlgorithm.scala
        HGAAlgorithm.scala
        ACOAlgorithm.scala
        SAAlgorithm.scala
        PACOAlgorithm.scala
        GAAlgorithm.scala
        TSABAlgorithm.scala
        NEHAlgorithm.scala
        TSAlgorithm.scala
        IGAlgorithm.scala
        neighbourhood
        NeighbourhoodOperator.scala
        util
        PermutationUtility.scala
        PfsSolutionParser.scala
        PfsProblemParser.scala
        DelphiInputConverter.scala
        spark
        SeedingStrategy.scala
        parallel
        PHGA.scala
        ParallelWork.scala
        problem
        PfsProblem.scala
  - test
    - resources
      - sol_ta089.txt
      - sol_ta066.txt
      - sol_ta112.txt
      - inst_ta012.txt
      - sol_ta068.txt
      - sol_ta116.txt
      - sol_ta058.txt
      - sol_ta021.txt
      - sol_ta085.txt
      - sol_ta064.txt
      - inst_ta117.txt
      - sol_ta093.txt
      - inst_ta014.txt
      - sol_ta096.txt
      - inst_ta046.txt
      - inst_ta053.txt
      - sol_ta083.txt
      - sol_ta100.txt
      - sol_ta041.txt
      - sol_ta099.txt
      - sol_ta084.txt
      - inst_ta045.txt
      - sol_ta037.txt
      - inst_ta115.txt
      - inst_ta039.txt
      - inst_ta079.txt
      - sol_ta001.txt
      - sol_ta033.txt
      - sol_ta075.txt
      - inst_ta108.txt
      - sol_ta053.txt
      - inst_ta005.txt
      - inst_ta080.txt
      - sol_ta023.txt
      - sol_ta120.txt
      - inst_ta106.txt
      - inst_ta075.txt
      - inst_ta028.txt
      - sol_ta117.txt
      - sol_ta080.txt
      - inst_ta059.txt
      - inst_ta015.txt
      - sol_ta047.txt
      - inst_ta058.txt
      - sol_ta069.txt
      - sol_ta102.txt
      - sol_ta092.txt
      - inst_ta047.txt
      - inst_ta064.txt
      - sol_ta025.txt
      - inst_ta074.txt
      - inst_ta087.txt
      - inst_ta084.txt
      - inst_ta017.txt
      - inst_ta086.txt
      - inst_ta034.txt
      - inst_ta066.txt
      - inst_ta065.txt
      - sol_ta022.txt
      - inst_ta107.txt
      - sol_ta057.txt
      - inst_ta095.txt
      - inst_ta112.txt
      - inst_ta025.txt
      - sol_ta090.txt
      - sol_ta016.txt
      - inst_ta076.txt
      - inst_ta011.txt
      - inst_ta024.txt
      - inst_ta104.txt
      - sol_ta010.txt
      - inst_ta041.txt
      - inst_ta109.txt
      - sol_ta094.txt
      - sol_ta024.txt
      - sol_ta042.txt
      - sol_ta008.txt
      - inst_ta101.txt
      - bestFoundSolutions.txt
      - inst_ta037.txt
      - sol_ta061.txt
      - sol_ta062.txt
      - inst_ta071.txt
      - sol_ta078.txt
      - sol_ta082.txt
      - inst_ta088.txt
      - sol_ta006.txt
      - sol_ta103.txt
      - inst_ta083.txt
      - inst_ta029.txt
      - inst_ta040.txt
      - inst_ta057.txt
      - inst_ta097.txt
      - inst_ta002.txt
      - sol_ta029.txt
      - inst_ta096.txt
      - sol_ta107.txt
      - inst_ta093.txt
      - inst_ta069.txt
      - inst_ta081.txt
      - sol_ta111.txt
      - inst_ta114.txt
      - sol_ta050.txt
      - inst_ta060.txt
      - inst_ta103.txt
      - sol_ta074.txt
      - sol_ta005.txt
      - inst_ta038.txt
      - inst_ta036.txt
      - sol_ta115.txt
      - inst_ta113.txt
      - sol_ta030.txt
      - sol_ta043.txt
      - sol_ta052.txt
      - inst_ta110.txt
      - sol_ta108.txt
      - inst_ta019.txt
      - inst_ta043.txt
      - sol_ta109.txt
      - sol_ta044.txt
      - inst_ta100.txt
      - inst_ta042.txt
      - inst_ta099.txt
      - sol_ta048.txt
      - sol_ta056.txt
      - inst_ta092.txt
      - sol_ta095.txt
      - inst_ta009.txt
      - inst_ta032.txt
      - sol_ta032.txt
      - sol_ta065.txt
      - sol_ta063.txt
      - sol_ta110.txt
      - sol_ta026.txt
      - sol_ta114.txt
      - sol_ta118.txt
      - inst_ta119.txt
      - sol_ta015.txt
      - sol_ta034.txt
      - sol_ta072.txt
      - inst_ta027.txt
      - sol_ta106.txt
      - inst_ta020.txt
      - inst_ta082.txt
      - sol_ta081.txt
      - sol_ta079.txt
      - inst_ta072.txt
      - inst_ta035.txt
      - inst_ta062.txt
      - sol_ta017.txt
      - inst_ta026.txt
      - inst_ta091.txt
      - sol_ta011.txt
      - sol_ta014.txt
      - inst_ta013.txt
      - inst_ta052.txt
      - sol_ta003.txt
      - inst_ta044.txt
      - sol_ta051.txt
      - inst_ta001.txt
      - inst_ta067.txt
      - inst_ta098.txt
      - sol_ta077.txt
      - sol_ta054.txt
      - sol_ta067.txt
      - inst_ta073.txt
      - inst_ta094.txt
      - sol_ta012.txt
      - sol_ta038.txt
      - sol_ta091.txt
      - inst_ta118.txt
      - sol_ta007.txt
      - inst_ta050.txt
      - inst_ta070.txt
      - inst_ta056.txt
      - sol_ta087.txt
      - inst_ta061.txt
      - inst_ta016.txt
      - sol_ta018.txt
      - inst_ta051.txt
      - inst_ta077.txt
      - inst_ta022.txt
      - inst_ta004.txt
      - sol_ta097.txt
      - sol_ta105.txt
      - sol_ta059.txt
      - sol_ta098.txt
      - sol_ta019.txt
      - sol_ta088.txt
      - inst_ta003.txt
      - sol_ta004.txt
      - sol_ta073.txt
      - inst_ta018.txt
      - inst_ta008.txt
      - sol_ta031.txt
      - inst_ta023.txt
      - sol_ta002.txt
      - inst_ta031.txt
      - sol_ta040.txt
      - inst_ta111.txt
      - sol_ta101.txt
      - inst_ta055.txt
      - inst_ta105.txt
      - inst_ta089.txt
      - inst_ta102.txt
      - inst_ta033.txt
      - sol_ta119.txt
      - inst_ta006.txt
      - sol_ta104.txt
      - inst_ta049.txt
      - inst_ta090.txt
      - sol_ta070.txt
      - inst_ta063.txt
      - inst_ta116.txt
      - sol_ta046.txt
      - inst_ta120.txt
      - inst_ta054.txt
      - sol_ta009.txt
      - sol_ta027.txt
      - inst_ta021.txt
      - inst_ta030.txt
      - sol_ta045.txt
      - inst_ta048.txt
      - inst_ta068.txt
      - sol_ta020.txt
      - sol_ta071.txt
      - inst_ta078.txt
      - inst_ta010.txt
      - sol_ta039.txt
      - sol_ta035.txt
      - sol_ta013.txt
      - sol_ta055.txt
      - inst_ta007.txt
      - sol_ta049.txt
      - sol_ta028.txt
      - sol_ta076.txt
      - inst_ta085.txt
      - sol_ta086.txt
      - sol_ta060.txt
      - sol_ta113.txt
      - sol_ta036.txt
    - scala
      - problem
        AlgorithmsTest.scala
- pom.xml
- output
  - experiment2.txt
- LICENSE
- .worksheet
  - src
    - it.polimi.hyperh.test.scala
    - it.polimi.hyperh.worksheets.testFramework.scala
    - it.polimi.hyperh.worksheets.test.scala
    - it.polimi.hyperh.testAlgorithms.scala
    - it.polimi.hyperh.worksheets.testAlgorithms.scala
    - test.scala
    - testAlgorithms.scala
- README.md
- .gitignore
- bash_scripts
  - experiment2.sh
  - runExp1Once.sh
  - logparser.sh
  - experiment3.sh
  - usefulAppInfo.sh
  - experiment1.sh
  - logcollector.sh
  - runExp2Once.sh

README

What is this repository for?

HyperSpark is a framework for running meta-heuristic algorithms on a cluster of commodity computers. The project is written in Scala and the cluster is managed by Spark.

The current version of the project implements the algorithms for Permutation Flowshop problems (PFSP), but it could be easily extended for any kind of problem.

The general idea is to use "Big Calculations" paradigm for running algorithms written by the user. What does that mean? It means that we perform some heavy computation algorithms on a cluster of machines, and we strive to optimize the usage of resources and time.

Goals of the project

Examine the performance of popular algorithms for PFSP when run in Scala/Spark environment
Show that the performance of one parallel run is more efficient than the performance of single algorithm run
Support the execution of both light-weight and CPU-intensive parallel algorithms.
Show that reusing the best solution found during a single parallel run for next runs is more efficient than just taking the best solution out of N independent runs.
Show that Scala/Spark environment suits better for implementation of cooperative algorithms than Hadoop's environment.
Simplicity: Show that the use of Scala/Spark environment is much simpler than using Hadoop environment, and that there is almost no setup of cluster using our framework.

How to use the framework?

The user writes its Problem, Solution, EvaluatedSolution and Algorithm classes.
Optionally, the user implements a StoppingCondition or uses an existing one "TimeExpired".
Optionally, the user implements a MapReduceHandler, or uses the default one (already set in FrameworkConf class).
The user specifies how many algorithms will be run in parallel to evaluate the problem.
The user specifies a stopping condition for parallel runs. (S)he uses the implemented TimeExpired stopping condition, or (s)he writes a custom stopping condition class.
What user could do is to define how the seeding solution (if exists) is provided to each of the algorithms that are run in parallel and how the results of the algorithms are combined (min, max, sum,etc.) and whether if the final result will be reused for another desired parallel run.

Implementation guide

Extend the "Problem" class (or use the existing one for PFSP, PFSProblem).
Extend the "Solution" and "EvaluatedSolution" class (or use the existing one for PFSP, PFSSolution and PFSEvaluatedSolution).
Implement evaluate(s: Solution) inside Problem class.
Extend the "Algorithm" trait and implement evaluate function signatures inside your custom algorithm class.
Extend the "StoppingCondition" and implement isSatisfied() function. Optionally, use already implemented "TimeExpired" stopping condition class.
Write your application that uses HyperSpark framework. For start, create a Scala object with main function. Inside the main write a FrameworkConf and provide it a problem, an algorithm, a parallelism multiplier, seeding solutions and the stopping condition for one parallel run. See it.polimi.hyperh.apps.LocalApp.scala example.
Use Framework.run(conf: FrameworkConf) or Framework.multipleRuns(conf: FrameworkConf, runs: Int) to get solution(s). There is an advanced option of using the best solution found of one parallel run as a seeding solution in the next iteration of a parallel run. This option can be enabled by creating/using an existing "SeedingStrategy" class, and setting in inside FrameworkConf before calling Framework.run method.
Inside MapReduceHandler.class there are two methods: hyperMap(...) - which runs the algorithm.evaluate over the problem, and hyperReduce(...) - which for now takes the solution with minimum value. For purposes other than PFSP problems, this class should be extended and than a custom MapReduceHandler should be set by using FrameworkConf.setMapReduceHandler(h: MapReduceHandler).

Algorithms implemented

NEH, Nawaz, Enscore and Ham (NEHAlgorithm)
Iterated Greedy (IGAlgorithm)
Genetic Algorithm, Reeves 1995 (GAAlgorithm)
Hybrid Genetic Algorithm, Zheng 2003 (HGAAlgorithm)
Simulated Annealing, Osman's addaption for PFSP 1989 (SAAlgorithm)
Improved Simulated Annealing, Xu and Oja (ISAAlgorithm)
Taboo Search, Taillard 1989 (TSAlgorithm)
Taboo Search with backjump tracking, Novicki and Smutnicki 1994 (TSABAlgorithm)
Max Min Ant System, Stutzle (MMASAlgorithm)
m-MMAS, Rajendran and Ziegler 2002 (MMMASAlgorithm)
PACO, Rajendran and Ziegler 2002 (PACOAlgorithm)

License info

The tool is available for use and improvement under a dual-licensing schema envisioning an Apache 2 license as well as a GPL-V3 License. Please contact the repository owners for further clarifications.