Pig Latin Compiler for Apache Spark / Flink

The goal of this project is to build a compiler for the Pig Latin dataflow language on modern data analytics platforms such as Apache Spark and Apache Flink. The project is not intented as a replacement or competitor of the official Pig compiler for Hadoop or its extensions such as PigSpork. Instead we have the following goals:

Installation

Clone & Update

Simply clone the git project.

Build

To build the project, in the project directory invoke

sbt package

This will build the (main) Pig compiler project as well as the shipped backends. (i.e. sparklib and flinklib)

There are several test cases included which should be passed: unit tests can be executed by sbt test, integration tests which compile and execute Pig scripts on Spark or Flink are executed by sbt it:test.

Note that building the compiler requires the most recent Spark and Flink jars, but they will be downloaded by sbt automatically.

If you want to use the compiler with the frontend scripts (see below), you have to build an assembly:

sbt assembly

Usage

We provide a simple wrapper script for processing Pig scripts. Just call it with

piglet --master local[4] --backend spark your_script.pig

To run this script you have to specify the full path to the platform distribution jar in the environment variable SPARK_JAR for Spark (e.g. spark-assembly-1.5.2-hadoop2.6.0.jar) and in FLINK_JAR (e.g. flink-dist_2.11-1.0.0.jar) for Flink. For Flink you also have to provide the path to the conf directory in FLINK_CONF_DIR.

An example for Spark could look like the following:

export SPARK_JAR=/opt/spark-1.6.0/assembly/target/scala-2.11/spark-assembly-1.6.0-hadoop2.6.0.jar
piglet --master local[4] --backend spark your_script.pig

The equivalent for Flink would be:

export FLINK_JAR=/opt/flink-1.0.0/build-target/lib/flink-dist_2.11-1.0.0.jar
export FLINK_CONF_DIR=/opt/flink-1.0.0/build-target/conf
piglet --master local[4] --backend flink your_script.pig

Note, that both for Spark and Flink you need a version built for Scala 2.11 (see e.g. Spark doc and Flink doc) and the same version used for building must also be used for execution. For Flink you have to run the start script found in the bin directory (e.g. /opt/flink-1.0.0/build-target/bin/start-local.sh) before executing scripts.

The following options are supported:

In addition, you can start an interactive Pig shell similar to Grunt:

piglet --interactive --backend spark

where Pig statements can be entered at the prompt and are executed as soon as a DUMP or STORE statement is entered. Furthermore, the schema can be printed using DESCRIBE.

Docker

Piglet can also be run as a Docker container. However, the image is not yet on DockerHub, so it has to be built manually:

sbt clean package assembly
docker build -t dbis/piglet .

Currently, the Docker image supports the Spark backend only.

To start the container, run:

docker run -it --rm --name piglet dbis/piglet

This uses the container's entrypoint which runs piglet. The above command will print the help message.

You can start the interactive mode, using -i option and enter your script.

docker run -it --rm --name piglet dbis/piglet -b spark -i

Alternatively, you can add your existing files into the container by mounting volumes and run the script in batch mode:

docker run -it --rm --name piglet -v /tmp/test.pig:/test.pig dbis/piglet -b spark /test.pig

As mentioned before, the container provides an entrypoint that executes piglet. In case you need a bash for that container, you need to overwrite the entrypoint:

docker run -it --rm --name piglet --entrypoint /bin/bash dbis/piglet

Configuration

To configure the program, we ship a configuration file. When starting the program for the first time, we will create our program home directory in your home directory and also copy the configuration file into this directory. More specifically, we will create a folder ~/.piglet (on *nix like systems) and copy the configuration file application.conf to this location.

If you update Piglet to a new version and the configuration file still exists from a previous version, a configuration exception might occur because we cannot find new configuration keys introduced by the new Piglet version in the existing config file. In such cases, you can start piglet with the -u (--update-config) option. This will force the override of your old configuration (make sure you have a backup if needed). Alternatively, you can simply remove the existing ~/.piglet/application.conf. This will also trigger the copy routine.

We use the Typesafe Config library.

Backends

As stated before, we support various backends that are used to execute the scripts. You can add your own backend by creating a jar file that contains the necessary configuration information and classes and adding it to the classpath (e.g. using the BACKEND_DIR variable).

More detailed information on how to create backends can be found in backends.md

Further Information