Nemo

Build Status Build Status

Nemo prerequisites and setup

Prerequisites

Installing Nemo

Running Beam applications

Running an external Beam application

./bin/run_external_app.sh \
    `pwd`/nemo_app/target/bd17f-1.0-SNAPSHOT.jar \
    -job_id mapreduce \
    -executor_json `pwd`/examples/resources/sample_executor_resources.json \
    -user_main MapReduce \
    -user_args "`pwd`/mr_input_data `pwd`/nemo_output/output_data"

Configurable options

Examples

## MapReduce example
./bin/run.sh \
    -job_id mr_default \
    -executor_json `pwd`/examples/resources/sample_executor_resources.json \
    -optimization_policy edu.snu.nemo.compiler.optimizer.policy.DefaultPolicy \
    -user_main edu.snu.nemo.examples.beam.MapReduce \
    -user_args "`pwd`/examples/resources/sample_input_mr `pwd`/examples/resources/sample_output_mr"

## YARN cluster example
./bin/run.sh \
    -deploy_mode yarn \
    -job_id mr_pado \
    -executor_json `pwd`/examples/resources/sample_executor_resources.json \
    -user_main edu.snu.nemo.examples.beam.MapReduce \
    -optimization_policy edu.snu.nemo.compiler.optimizer.policy.PadoPolicy \
    -user_args "hdfs://v-m:9000/sample_input_mr hdfs://v-m:9000/sample_output_mr"

Resource Configuration

-executor_json command line option can be used to provide a path to the JSON file that describes resource configuration for executors. Its default value is config/default.json, which initializes one of each Transient, Reserved, and Compute executor, each of which has one core and 1024MB memory.

Configurable options

Examples

[
  {
    "num": 12,
    "type": "Transient",
    "memory_mb": 1024,
    "capacity": 4
  },
  {
    "type": "Reserved",
    "memory_mb": 1024,
    "capacity": 2
  }
]

This example configuration specifies

Monitoring your job using web UI

Nemo Compiler and Engine can store JSON representation of intermediate DAGs.

Examples

./bin/run.sh \
    -job_id als \
    -executor_json `pwd`/examples/resources/sample_executor_resources.json \
    -user_main edu.snu.nemo.examples.beam.AlternatingLeastSquare \
    -optimization_policy edu.snu.nemo.compiler.optimizer.policy.PadoPolicy \
    -dag_dir "./dag/als" \
    -user_args "`pwd`/examples/resources/sample_input_als 10 3"