A plugin for Apache Airflow that exposes REST endpoints for the Command Line Interfaces listed in the airflow documentation:
http://airflow.apache.org/cli.html
The plugin also includes other custom REST APIs.
flask_jwt_extended
module. Create the plugins folder if it doesn't exist.
The location you should put it is usually at {AIRFLOW_HOME}/plugins. The specific location can be found in your airflow.cfg file:
plugins_folder = /home/{USER_NAME}/airflow/plugins
Download the release code you want to deploy
Releases Available:
Branches Available:
URL to Download From:
Note: Each release/branch has its own README.md file that describes the specific options and steps you should take to deploy and configure. Verify the options available in each release/branch after you download it.
Unzip the file and move the contents of the plugins folder into your Airflow plugins directory
unzip airflow-rest-api-plugin-{RELEASE_VERSION_OR_BRANCH_NAME}.zip
cp -r airflow-rest-api-plugin-{RELEASE_VERSION_OR_BRANCH_NAME}/plugins/* {AIRFLOW_PLUGINS_FOLDER}
Update the base_url configuration in the airflow.cfg file to the public host or ip of the machine the Airflow instance is running on (Optional)
(Optional) Append the following content to the end of the {AIRFLOW_HOME}/airflow.cfg file to give you control over execution:
[rest_api_plugin]
# Logs global variables used in the REST API plugin when the plugin is loaded. Set to False by default to avoid too many logging messages.
# DEFAULT: False
log_loading = False
# Filters out loading messages from the standard out
# DEFAULT: True
filter_loading_messages_in_cli_response = True
# HTTP Header Name to be used for authenticating REST calls for the REST API Plugin
# DEFAULT: 'rest_api_plugin_http_token'
#rest_api_plugin_http_token_header_name = rest_api_plugin_http_token
# HTTP Token to be used for authenticating REST calls for the REST API Plugin
# DEFAULT: None
# Comment this out to disable Authentication
#rest_api_plugin_expected_http_token = changeme
(Optional) Setup Authentication for Security
a. Note: Requires that step #5 above be completed.
b. Follow the "Enabling Authentication" section below.
We need to replace the CSRF_ENABLED
attribute with WTF_CSRF_ENABLED
.
This change is required to support the POST method when RBAC is enabled with JWT.
Please add the following property in the {AIRFLOW_HOME}/webserver_config.py
file.
# Flask-WTF flag for CSRF
WTF_CSRF_ENABLED = False
Restart the Airflow Web Server
The REST API client supports a simple token based authentication mechanism where you can require users to pass in a specific http header to authenticate. By default this authentication mechanism is disabled but can be enabled with the "Setup" steps bellow.
Edit your {AIRFLOW_HOME}/airflow.cfg file
a. Under the [rest_api_plugin] section you added in step #5 of the "Deployment Instructions", uncomment the following configs:
rest_api_plugin_http_token_header_name = rest_api_plugin_http_token
rest_api_plugin_expected_http_token = changeme
(Optional) Update the 'rest_api_plugin_http_token_header_name' to the desired value
Change the value of 'rest_api_plugin_expected_http_token' to the desired token people should pass
Restart the Airflow Web Server
Once the steps above have been followed to enable authentication, users will need to pass a specific header along with their request to properly call the REST API. The header name is: rest_api_plugin_http_token
Example CURL Command:
curl --header "rest_api_plugin_http_token: changeme" http://{HOST}:{PORT}/admin/rest_api/api?api=version
In the event that you have authentication enabled and the user calling the REST Endpoint doesn't include the header, you will get the following response:
{
"call_time": "{TIMESTAMP}",
"http_response_code": 403,
"output": "Token Authentication Failed",
"response_time": "{TIMESTAMP}",
"status": "ERROR"
}
Plugin supports RBAC feature for Airflow versions 1.10.4 or higher.
rbac = True
in airflow.cfg
config file under webserver
category.airflow initdb
command which generates a new config file webserver_config.py
under the AIRFLOW_HOME.Security
menu in the Airflow UINote: Once you enable the RBAC support, it will create new set of Database Tables. So, you need to manually
migrate your existing users from users
table to the new table ab_users
. While migrating you need to specify
the role for each user. Please check Create User for more details.
Plugin works out of the box. If you need to access the plugin UI please follow the below steps.
Security
menumenu access on Rest API Plugin
and menu access on Admin
permissionsPlugin enables JWT Token based authentication for Airflow versions 1.10.4 or higher when RBAC support is enabled.
Generating the JWT access token
curl -XPOST http://{AIRFLOW_HOST}:{AIRFLOW_PORT}/api/v1/security/login -H "Content-Type: application/json"
-d '{"username":"username", "password":"password", "refresh":true, "provider": "db"}'
sample response which includes access_token and refresh_token.
{
"access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE1Njc3NzMxODcsIm5iZiI6MTU2Nzc3MzE4NywianRpIjoiYzFiOWRkNTgtNGY5OC00MTI5LTk1OTctMDc4YWJkOGM4ZjdkIiwiZXhwIjoxNTY3NzczNDg3LCJpZGVudGl0eSI6NiwiZnJlc2giOnRydWUsInR5cGUiOiJhY2Nlc3MifQ.pD1rCXt2NmjRlHxuoiLcZJ1RNUuHS_hV2Dudsqw6wCY",
"refresh_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE1Njc3NzMxODcsIm5iZiI6MTU2Nzc3MzE4NywianRpIjoiNmZlZTQ1MmItYzBkYy00NTc4LWFmZWItNTc3OTAwODAzN2FiIiwiZXhwIjoxNTcwMzY1MTg3LCJpZGVudGl0eSI6NiwidHlwZSI6InJlZnJlc2gifQ.Gjcr08qrDM4wCmKIJS8muJN_5AkGyYZfOQKH_HUKPqE"
}
By default, JWT access token is valid for 15 mins and refresh token is valid for 30 days. You can renew the access token with the help of refresh token as shown below.
Renewing the Access Token
curl -XPOST "http://{AIRFLOW_HOST}:{AIRFLOW_PORT}/api/v1/security/refresh" -H 'Authorization: Bearer <refresh_token>'
sample response returns the renewed access token as shown below.
{
"refresh_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE1Njc3NzI4NzUsIm5iZiI6MTU2Nzc3Mjg3NSwianRpIjoiMjE3Y2FhYzItYTQ1NS00MWMxLWI5OGUtOTRhYWU0OTcyZTljIiwiZXhwIjoxNTY3NzczMTc1LCJpZGVudGl0eSI6NSwiZnJlc2giOmZhbHNlLCJ0eXBlIjoiYWNjZXNzIn0.NoHaqc9caqk2B_cfutGO3U3Ih95lF0m9hvTPs5B_sE0"
}
Working with the rest_api_plugin and JWT Auth tokens
Pass the additional Authorization:Bearer <access_token>
header in the rest API request
curl -H 'Authorization:Bearer <access_token>' 'http://{AIRFLOW_HOST}:{AIRFLOW_PORT}/admin/rest_api/api?api=version'
If you have rest_api_plugin_http_token_header
authentication enabled then you need to pass both headers as shown below.
curl -H 'Authorization:Bearer <access_token>' -H 'rest_api_plugin_http_token: changeme' http://{AIRFLOW_HOST}:{AIRFLOW_PORT}/admin/rest_api/api?api=version
If JWT Authorization header is missing in the request you will get the following response. (Status Code: 401)
{ "msg": "Missing Authorization Header" }
If JWT access token expires you will get the following response. (Status Code: 401)
{ "msg": "Token has expired" }
If JWT access token is invalid you will get the following response. (Status Code: 422)
{ "msg": "Invalid payload padding" }
Please refer Flask-JWT-Extended module documentation for more details
Once you deploy the plugin and restart the web server, you can start to use the REST API. Bellow you will see the endpoints that are supported. In addition, you can also interact with the REST API from the Airflow Web Server. When you reload the page, you will see a link under the Admin tab called "REST API". Clicking on the link will navigate you to the following URL:
http://{AIRFLOW_HOST}:{AIRFLOW_PORT}/admin/rest_api/
This web page will show the Endpoints supported and provide a form for you to test submitting to them.
Gets the version of Airflow currently running. Supports both http GET and POST methods.
Available in Airflow Version: 1.0.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=version
Query Arguments:
None
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=version
Displays the version of this REST API Plugin you're using. Supports both http GET and POST methods.
Available in Airflow Version: None - Custom API
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=rest_api_plugin_version
Query Arguments:
None
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=rest_api_plugin_version
Render a task instance's template(s). Supports both http GET and POST methods.
Available in Airflow Version: 1.7.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=render
Query Arguments:
dag_id - string - The id of the dag
task_id - string - The id of the task
execution_date - string - The execution date of the DAG (Example: 2017-01-02T03:04:05)
subdir (optional) - string - File location or directory from which to look for the dag
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=render
CRUD operations on variables. Supports both http GET and POST methods.
Available in Airflow Version: 1.7.1 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=variables
Query Arguments:
set (optional) - Sets a variable. This requires passing the cmd
, key
and value
parameters. Please see the example below.
cmd=set
get (optional) - string - Get value of a variable
json (optional) - boolean - Deserialize JSON variable
default (optional) - string - Default value returned if variable does not exist
import (optional) - string - Import variables from JSON file
export (optional) - string - Export variables from JSON file
delete (optional) - string - Delete a variable
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=variables
For setting a variable like myVar1=myValue1
use
http://{HOST}:{PORT}/admin/rest_api/api?api=variables&cmd=set&key=myVar1&value=myValue1
List/Add/Delete connections. Supports both http GET and POST methods.
Available in Airflow Version: 1.8.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=connections
Query Arguments:
list (optional) - boolean - List all connections
add (optional) - string - Add a connection
delete (optional) - boolean - Delete a connection
conn_id (optional) - string - Connection id, required to add/delete a connection
conn_uri (optional) - string - Connection URI, required to add a connection
conn_extra (optional) - string - Connection 'Extra' field, optional when adding a connection
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=connections
Pauses a DAG. Supports both http GET and POST methods.
Available in Airflow Version: 1.7.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=pause
Query Arguments:
dag_id - string - The id of the dag
subdir (optional) - string - File location or directory from which to look for the dag
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=pause&dag_id=test_id
Resume a paused DAG. Supports both http GET and POST methods.
Available in Airflow Version: 1.7.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=unpause
Query Arguments:
dag_id - string - The id of the dag
subdir (optional) - string - File location or directory from which to look for the dag
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=unpause&dag_id=test_id
Returns the unmet dependencies for a task instance from the perspective of the scheduler. In other words, why a task instance doesn't get scheduled and then queued by the scheduler, and then run by an executor). Supports both http GET and POST methods.
Available in Airflow Version: 1.8.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=task_failed_deps
Query Arguments:
dag_id - string - The id of the dag
task_id - string - The id of the task
execution_date - string - The execution date of the DAG (Example: 2017-01-02T03:04:05)
subdir (optional) - string - File location or directory from which to look for the dag
Examples:
Triggers a Dag to Run. Supports both http GET and POST methods.
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=trigger_dag
Available in Airflow Version: 1.6.0 or greater
Query Arguments:
dag_id - The DAG ID of the DAG you want to trigger
run_id (Optional) - The RUN ID to use for the DAG run
conf (Optional) - Some configuration to pass to the DAG you trigger - (URL Encoded JSON)
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=trigger_dag&dag_id=test_id
Test a task instance. This will run a task without checking for dependencies or recording it's state in the database.
Available in Airflow Version: 0.1 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=test
Query Arguments:
dag_id - string - The id of the dag
task_id - string - The id of the task
execution_date - string - The execution date of the DAG (Example: 2017-01-02T03:04:05)
subdir (optional) - string - File location or directory from which to look for the dag
dryrun (optional) - boolean - Perform a dry run
task_params (optional) - string - Sends a JSON params dict to the task
Examples:
Get the status of a dag run. Supports both http GET and POST methods.
Available in Airflow Version: 1.8.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=dag_state
Query Arguments:
dag_id - string - The id of the dag
execution_date - string - The execution date of the DAG (Example: 2017-01-02T03:04:05)
subdir (optional) - string - File location or directory from which to look for the dag
Examples:
Run a single task instance. Supports both http GET and POST methods.
Available in Airflow Version: 1.0.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=run
Query Arguments:
dag_id - string - The id of the dag
task_id - string - The id of the task
execution_date - string - The execution date of the DAG (Example: 2017-01-02T03:04:05)
subdir (optional) - string - File location or directory from which to look for the dag
mark_success (optional) - boolean - Mark jobs as succeeded without running them
force (optional) - boolean - Ignore previous task instance state, rerun regardless if task already succeede
pool (optional) - boolean - Resource pool to use
cfg_path (optional) - boolean - Path to config file to use instead of airflow.cfg
local (optional) - boolean - Run the task using the LocalExecutor
ignore_all_dependencies (optional) - boolean - Ignores all non-critical dependencies, including ignore_ti_state and ignore_task_depsstore_true
ignore_dependencies (optional) - boolean - Ignore task-specific dependencies, e.g. upstream, depends_on_past, and retry delay dependencies
ignore_depends_on_past (optional) - boolean - Ignore depends_on_past dependencies (but respect upstream dependencies)
ship_dag (optional) - boolean - Pickles (serializes) the DAG and ships it to the worker
pickle (optional) - string - Serialized pickle object of the entire dag (used internally)
Examples:
List the tasks within a DAG. Supports both http GET and POST methods.
Available in Airflow Version: 0.1 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=list_tasks
Query Arguments:
dag_id - string - The id of the dag
tree (optional) - boolean - Tree view
subdir (optional) - string - File location or directory from which to look for the dag
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=list_tasks&dag_id=test_id
Run subsections of a DAG for a specified date range. Supports both http GET and POST methods.
Available in Airflow Version: 0.1 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=backfill
Query Arguments:
dag_id - string - The id of the dag
task_regex (optional) - string - The regex to filter specific task_ids to backfill (optional)
start_date (optional) - string - Override start_date YYYY-MM-DD. Either this or the end_date needs to be provided.
end_date (optional) - string - Override end_date YYYY-MM-DD. Either this or the start_date needs to be provided.
mark_success (optional) - boolean - Mark jobs as succeeded without running them
local (optional) - boolean - Run the task using the LocalExecutor
donot_pickle (optional) - boolean - Do not attempt to pickle the DAG object to send over to the workers, just tell the workers to run their version of the code.
include_adhoc (optional) - boolean - Include dags with the adhoc argument.
ignore_dependencies (optional) - boolean - Ignore task-specific dependencies, e.g. upstream, depends_on_past, and retry delay dependencies
ignore_first_depends_on_past (optional) - boolean - Ignores depends_on_past dependencies for the first set of tasks only (subsequent executions in the backfill DO respect depends_on_past)
subdir (optional) - string - File location or directory from which to look for the dag
pool (optional) - string - Resource pool to use
dry_run (optional) - boolean - Perform a dry run
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=backfill&dag_id=test_id
List all the DAGs. Supports both http GET and POST methods.
Available in Airflow Version: 0.1 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=list_dags
Query Arguments:
subdir (optional) - string - File location or directory from which to look for the dag
report (optional) - boolean - Show DagBag loading report
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=list_dags
Start a kerberos ticket renewer. Supports both http GET and POST methods.
Available in Airflow Version: 1.6.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=kerberos
Query Arguments:
principal (optional) - string - kerberos principal
keytab (optional) - string - keytab
pid (optional) - string - PID file location
daemon (optional) - boolean - Daemonize instead of running in the foreground
stdout (optional) - string - Redirect stdout to this file
stderr (optional) - string - Redirect stderr to this file
log-file (optional) - string - Location of the log file
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=kerberos
Start a Celery worker node. Supports both http GET and POST methods.
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=worker
Available in Airflow Version: 0.1 or greater
Query Arguments:
do_pickle (optional) - boolean - Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code.
queues (optional) - string - Comma delimited list of queues to serve
concurrency (optional) - string - The number of worker processes
pid (optional) - string - PID file location
daemon (optional) - boolean - Daemonize instead of running in the foreground
stdout (optional) - string - Redirect stdout to this file
stderr (optional) - string - Redirect stderr to this file
log-file (optional) - string - Location of the log file
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=worker
Start a Celery worker node. Supports both http GET and POST methods.
Available in Airflow Version: 1.0.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=flower
Query Arguments:
hostname (optional) - string - Set the hostname on which to run the server
port (optional) - string - The port on which to run the server
flower_conf (optional) - string - Configuration file for flower
broker_api (optional) - string - Broker api
pid (optional) - string - PID file location
daemon (optional) - boolean - Daemonize instead of running in the foreground
stdout (optional) - string - Redirect stdout to this file
stderr (optional) - string - Redirect stderr to this file
log-file (optional) - string - Location of the log file
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=flower
Start a scheduler instance. Supports both http GET and POST methods.
Available in Airflow Version: 1.0.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=scheduler
Query Arguments:
dag_id (optional) - string - The id of the dag
subdir (optional) - string - File location or directory from which to look for the dag
run-duration (optional) - string - Set number of seconds to execute before exiting
num_runs (optional) - string - Set the number of runs to execute before exiting
do_pickle (optional) - boolean - Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code.
pid (optional) - string - PID file location
daemon (optional) - boolean - Daemonize instead of running in the foreground
stdout (optional) - string - Redirect stdout to this file
stderr (optional) - string - Redirect stderr to this file
log-file (optional) - string - Location of the log file
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=scheduler
Get the status of a task instance. Supports both http GET and POST methods.
Available in Airflow Version: 1.0.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=task_state
Query Arguments:
dag_id - string - The id of the dag
task_id - string - The id of the task
execution_date - string - The execution date of the DAG (Example: 2017-01-02T03:04:05)
subdir (optional) - string - File location or directory from which to look for the dag
Examples:
CRUD operations on pools. Supports both http GET and POST methods.
Available in Airflow Version: 1.8.0 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=pool
Query Arguments:
set (optional) - Set pool name, slot count and description respectively. This requires passing the cmd
, pool_name
, slot_count
and description
parameters. Please see the example below.
cmd=set
get (optional) - string - Get pool info
delete (optional) - string - Delete a pool
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=pool
For setting a myTestpool
with a slot count of 10
and with myTestpoolDescription
description.
http://{HOST}:{PORT}/admin/rest_api/api?api=pool&cmd=set&pool_name=myTestpool&slot_count=10&description=myTestpoolDescription
Serve logs generate by worker. Supports both http GET and POST methods.
Available in Airflow Version: 0.1 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=serve_logs
Query Arguments:
None
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=serve_logs
Clear a set of task instance, as if they never ran. Supports both http GET and POST methods.
Available in Airflow Version: 0.1 or greater
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=clear
Query Arguments:
dag_id - string - The id of the dag
task_regex (optional) - string - The regex to filter specific task_ids to backfill (optional)
start_date (optional) - string - Override start_date YYYY-MM-DD
end_date (optional) - string - Override end_date YYYY-MM-DD
subdir (optional) - string - File location or directory from which to look for the dag
upstream (optional) - boolean - Include upstream tasks
downstream (optional) - boolean - Include downstream tasks
no_confirm (optional) - boolean - Do not request confirmation
only_failed (optional) - boolean - Only failed jobs
only_running (optional) - boolean - Only running jobs
exclude_subdags (optional) - boolean - Exclude subdags
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=clear
Deploy a new DAG. Supports both http GET and POST methods.
Available in Airflow Version: None - Custom API
POST - http://{HOST}:{PORT}/admin/rest_api/api?api=deploy_dag
POST Body Arguments:
dag_file - file - Python file to upload and deploy
force (optional) - boolean - Whether to forcefully upload the file if the file already exists or not
pause (optional) - boolean - The DAG will be forced to be paused when created and override the 'dags_are_paused_at_creation' config.
unpause (optional) - boolean - The DAG will be forced to be unpaused when created and override the 'dags_are_paused_at_creation' config.
Examples:
Header: multipart/form-data
URL: http://{HOST}:{PORT}/admin/rest_api/api?api=deploy_dag
Body: dag_file=path_to_file&force=on
CURL Example:
curl -X POST -H 'Content-Type: multipart/form-data' -F 'dag_file=@/path/to/dag.py' -F 'force=on' http://{HOST}:{PORT}/admin/rest_api/api?api=deploy_dag
Refresh a DAG. Supports both http GET and POST methods.
Available in Airflow Version: None - Custom API
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=refresh_dag
Query Arguments:
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=refresh_dag&dag_id=test_id
Refresh all DAGs. Supports both http GET and POST methods.
Available in Airflow Version: None - Custom API
GET - http://{HOST}:{PORT}/admin/rest_api/api?api=refresh_all_dags
Query Arguments:
None
Examples:
http://{HOST}:{PORT}/admin/rest_api/api?api=refresh_all_dags
The API's will all return a common response object. It is a JSON object with the following entries in it:
Sample (Result of calling the versions endpoint)
{
"airflow_cmd": "airflow version",
"arguments": {},
"call_time": "Tue, 29 Nov 2016 14:22:26 GMT",
"http_response_code": 200,
"output": "1.7.0",
"response_time": "Tue, 29 Nov 2016 14:27:59 GMT",
"status": "OK"
}