CircleCi

Telemetry-Airflow

Airflow is a platform to programmatically author, schedule and monitor workflows.

When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.

Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

Prerequisites

This app is built and deployed with docker and docker-compose.

Dependencies

Most Airflow jobs are thin wrappers that spin up an EMR cluster for running the job. Be aware that the configuration of the created EMR clusters depends on finding scripts in an S3 location configured by the SPARK_BUCKET variable. Those scripts are maintained in emr-bootstrap-spark and are deployed independently of this repository. Changes in behavior of Airflow jobs not explained by changes in the source of the Spark jobs or by changes in this repository could be due to changes in the bootstrap scripts.

Build Container

An Airflow container can be built with

make build

Export Credentials

For now, DAGs that use the Databricks operator won't parse until the following environment variables are set (see issue #501):

AWS_SECRET_ACCESS_KEY
AWS_ACCESS_KEY_ID
DB_TOKEN

Migrate Database

Airflow database migration is no longer a separate step for dev but is run by the web container if necessary on first run. That means, however, that you should run the web container (and the database container, of course) and wait for the database migrations to complete before running individual test commands per below. The easiest way to do this is to run make up and let it run until the migrations complete.

Testing

A single task, e.g. spark, of an Airflow dag, e.g. example, can be run with an execution date, e.g. 2018-01-01, in the dev environment with:

export DEV_USERNAME=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_ACCESS_KEY_ID=...
make run COMMAND="test example spark 20180101"

The DEV_USERNAME is a short string used to identify your EMR instances. This should be set to something like your IRC or Slack handle.

The container will run the desired task to completion (or failure). Note that if the container is stopped during the execution of a task, the task will be aborted. In the example's case, the Spark job will be terminated.

The logs of the task can be inspected in real-time with:

docker logs -f telemetryairflow_scheduler_1

You can see task logs and see cluster status on the EMR console

By default, the results will end up in the telemetry-test-bucket in S3. If your desired task depends on other views, it will expect to be able to find those results in telemetry-test-bucket too. It's your responsibility to run the tasks in correct order of their dependencies.

CAVEAT: When running the make run multiple times it can spin up multiple versions of the web container. It can also fail if you've never run make up to initialize the database. An alternative form of the above is to launch the containers and shell into the web container to run the airflow test command.

In one terminal launch the docker containers:

make up

Note: initializing the web container will run the airflow initdb/upgradedb

In another terminal shell into the web container, making sure to also supply the environment variables, then run the airflow test command:

export DEV_USERNAME=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
docker exec -ti -e DEV_USERNAME -e AWS_SECRET_ACCESS_KEY -e AWS_ACCESS_KEY_ID telemetry-airflow_web_1 /bin/bash
airflow test example spark 20180101

Adding dummy credentials

Tasks often require credentials to access external credentials. For example, one may choose to store API keys in an Airflow connection or variable. These variables are sure to exist in production but are often not mirrored locally for logistical reasons. Providing a dummy variable is the preferred way to keep the local development environment up to date.

In bin/run, please update the init_connections and init_variables with appropriate strings to prevent broken workflows. To test this, run bin/test-parse to check for errors. You may manually test this by restarting the orchestrated containers and checking for error messages within the main administration UI at localhost:8000.

Local Deployment

Assuming you're using macOS and Docker for macOS, start the docker service, click the docker icon in the menu bar, click on preferences and change the available memory to 4GB.

To deploy the Airflow container on the docker engine, with its required dependencies, run:

make up

You can now connect to your local Airflow web console at http://localhost:8000/.

All DAGs are paused by default for local instances and our staging instance of Airflow. In order to submit a DAG via the UI, you'll need to toggle the DAG from "Off" to "On". You'll likely want to toggle the DAG back to "Off" as soon as your desired task starts running.

Workaround for permission issues

Users on Linux distributions will encounter permission issues with docker-compose. This is because the local application folder is mounted as a volume into the running container. The Airflow user and group in the container is set to 10001.

To work around this, replace all instances of 10001 in Dockerfile.dev with the host user id.

sed -i "s/10001/$(id -u)/g" Dockerfile.dev

Testing Databricks Jobs

To run a job running on Databricks, run make up in the background. Follow this guide on generating a token and save this to a secure location. Export the token to a an environment variable:

export DB_TOKEN=<TOKEN>

Finally, run the testing command using docker-compose directly:

docker-compose exec web airflow test example spark 20180101

Testing Dev Changes

Note: This only works for telemetry-batch-view jobs

A dev changes can be run by simply changing the DEPLOY_TAG environment variable to whichever upstream branch you've pushed your local changes to.

Afterwards, you're going to need to:make clean and make build and nohup make up &

From there, you can either set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the Dockerfile and run make up to get a local UI and run from there, or you can follow the testing instructions above and use make run.

Testing GKE Jobs (including BigQuery-etl changes)

For now, follow the steps outlined here to create a service account: https://bugzilla.mozilla.org/show_bug.cgi?id=1553559#c1.

Enable that service account in Airflow with the following:

make build && make up
./bin/add_gcp_creds $GOOGLE_APPLICATION_CREDENTIALS

From there, connect to Airflow and enable your job.

Testing Dataproc Jobs

Dataproc jobs run on a self-contained Dataproc cluster, created by Airflow.

To test these, jobs, you'll need a sandbox account and corresponding service account. For information on creating that, see "Testing GKE Jobs". Your service account will need Dataproc and GCS permissions (and BigQuery, if you're connecting to it). Note: Dataproc requires "Dataproc/Dataproc Worker" as well as Compute Admin permissions. You'll need to ensure that the Dataproc API is enabled in your sandbox project.

Ensure that your dataproc job has a configurable project to write to. Set the project in the DAG entry to be configured based on development environment; see the ltv.py job for an example of that.

From there, run the following:

make build && make up
./bin/add_gcp_creds $GOOGLE_APPLICATION_CREDENTIALS google_cloud_airflow_dataproc

You can then connect to Airflow locally. Enable your DAG and see that it runs correctly.

Production Setup

Note: the canonical reference for production environment variables lives in a private repository.

When deploying to production make sure to set up the following environment variables:

Also, please set

Both values should be set by using the cryptography module's fernet tool that we've wrapped in a docker-compose call:

make secret

Run this for each key config variable, and don't use the same for both!

Debugging

Some useful docker tricks for development and debugging:

# Stop all docker containers:
docker stop $(docker ps -aq)

# Remove any leftover docker volumes:
docker volume rm $(docker volume ls -qf dangling=true)

# Purge docker volumes (helps with mysql container failing to start)
# Careful as this will purge all local volumes not used by at least one container.
docker volume prune

Failing CircleCI 'test-environment' check:

# These commands are from the bin/test-parse script (get_errors_in_listing)
# If --detach is unavailable,  make sure you are running the latest version of docker-compose
docker-compose up --detach

docker-compose logs --follow --tail 0 | sed -n '/\[testing_stage_0\]/q'

# Don't pipe to grep to see the full output including your errors
docker-compose exec web airflow list_dags

Triggering a task to re-run within the Airflow UI

Triggering backfill tasks using the CLI

CircleCI