What is Sundial?
Sundial is a batch job scheduler for running Dockerized batch jobs on AWS Batch and AWS EMR (Elastic MapReduce)
Think of Sundial as an alternative to Chronos that uses Amazon Batch instead of Mesos and has some nice extra features.
Authors: CONTRIBUTORS.md
Features:
Set up Sundial service and Batch queue using Cloudformation template. See DEPLOY.md for details.
Submit your jobs using REST API. See description of job JSON format under JOBS.md
The latest API is hosted here: https://app.apibuilder.io/gilt/svc-sundial/latest A prettier version of the API can be seen at: http://ui-www.apibuilder.io.s3-website-us-east-1.amazonaws.com/org/gilt/app/svc-sundial
For development purposes it's possible to run sundial locally via the script ./start-dev.sh
.
Pre-requisites are:
.aws.env
file at the root of the project with the required env variables (See example below)The docker-compose.yml
file used to run sundial locally, contains almost all the config required to
run sundial on your laptop. Some env variables though, have to be setup on a case by case scenario.
An example is the following:
AWS_REGION=us-east-1
AWS_PROFILE=my-aws-profile
SUNDIAL_S3_BUCKET_DEV=sundial-dev-bucket
Once the cluster is created an easy way to deploy Sundial to your AWS account is via the Jenkins Pipeline plugin.
Create a Pipeline Project and paste the following in your Pipeline script.
Please note the two environmental variable required to customise the build:
DOCKER_REGISTRY='MY_DOCKER_REGISTRY'
CODEDEPLOY_S3_BUCKET='MY_S3_BUCKET'
pipeline {
agent any
environment {
DOCKER_REGISTRY = 'MY_DOCKER_REGISTRY'
CODEDEPLOY_S3_BUCKET = 'MY_S3_BUCKET'
}
stages {
stage('Code Checkout') {
steps {
git url: 'https://github.com/saksdirect/sundial.git'
}
}
stage('Deployment') {
steps {
sh """
./deploy.sh
"""
}
}
}
}
With the introduction of AWS Batch, we've decided to deprecate the ECS
executor. The new AWS Batch
allows a better, simpler and cheaper way of running batch jobs on aws.