GoCD plugin to keep pipelines and environments configuration in source-control in YAML. See this document to find out what are GoCD configuration repositories.
This is the second GoCD configuration plugin, which enhances some of shortcomings of JSON configuration plugin
", brackets {} and commas ,Step 1: GoCD versions newer than 17.8.0 already have the plugin bundled. You don't need to install anything.
If you're using GoCD version older than 17.8.0, you need to install the plugin in the GoCD server. Download it from
the releases page and place it on the GoCD server in
plugins/external directory.
Step 2: Follow the GoCD documentation to add a new configuration repository.
You can use the example repository at: https://github.com/tomzo/gocd-yaml-config-example.git.
In your config repo (tomzo/gocd-yaml-config-example.git in this case), ensure that your GoCD yaml config is suffixed with .gocd.yaml. Any file ending in .gocd.yaml is picked up by the plugin. Give it a minute or so for the polling to happen. Once that happens, you should see your pipeline(s) on your dashboard.
More examples are in test resources.
#ci.gocd.yaml
format_version: 9
environments:
testing:
environment_variables:
DEPLOYMENT: testing
secure_variables:
ENV_PASSWORD: "s&Du#@$xsSa"
pipelines:
- example-deploy-testing
- build-testing
pipelines:
mypipe1: # definition of mypipe1 pipeline
group: mygroup # note that the group name can contain only of alphanumeric & underscore characters
display_order: 10
label_template: "${mygit[:8]}"
lock_behavior: none
parameters: # list of parameters that can be configured for a pipeline
param1: value1
materials:
mygit: # this is the name of material, the name can contain only of alphanumeric & underscore characters
# keyword git says about type of material and url at once
git: http://my.example.org/mygit.git
branch: ci
myupstream: # this name does not matter, but there should be no 2 materials with the same name
# type is optional here, material type is implied based on presence of pipeline and stage fields
# type: dependency
pipeline: pipe2
stage: test
stages: # list of stages in order
- build: # name of stage
clean_workspace: true
jobs:
csharp: # name of the job
resources:
- net45
artifacts:
- build:
source: bin/
destination: build
- test:
source: tests/
destination: test-reports/
- test:
source: coverage.xml
tabs:
report: test-reports/index.html
tasks: # ordered list of tasks to execute in job csharp
- fetch:
pipeline: pipe2
stage: build
job: test
source: test-bin/
destination: bin/
- exec: # indicates type of task
command: make
arguments:
- "VERBOSE=true"
# shorthand for script-executor plugin
- script: ./build.sh ciThe default pattern is **/*.gocd.yaml and **/*.gocd.yml, which will recursively search the entire repository for all files ending with .gocd.yaml or .gocd.yml.
You can set a custom file pattern per configuration repository using the GoCD configuration UI:
Or in the config XML using <configuration>:
<config-repos>
<config-repo pluginId="yaml.config.plugin" id="repo1">
<git url="https://github.com/tomzo/gocd-yaml-config-example.git" />
<configuration>
<property>
<key>file_pattern</key>
<value>pipeline.gocd.yaml</value>
</property>
</configuration>
</config-repo>
</config-repos>You can validate if proposed GoCD YAML changes will be accepted by the server. Currently, 2 options are available:
You may find this introductory blog post useful.
There is an ongoing effort to allow in-depth validation of configuration before pushing configuration to the source control. This is provided by GoCD's preflight API and gocd-cli.
You have several options to configure validation tools on your workstation:
gocd-cli on your host.Either way you'll have gocd binary in your PATH or inside the docker container.
This will check general validity of the yaml file, without talking to the GoCD server:
gocd configrepo syntax --yaml pipeline.gocd.yamlThis command will parse and submit your yaml file to the configured GoCD server.
gocd configrepo preflight --yaml -r dotnet-dojo pipeline.gocd.yamlWhere -r is the configuration repository id, which you have earlier configured on GoCD server. You can check it on config repos page of your GoCD server, at /go/admin/config_repos. It is in the upper left corner of each config repo.
For new repositories, when you haven't added config repo to the server yet, you can also check if it is correct, just skip the -r argument.
gocd configrepo preflight --yaml pipeline.gocd.yamlSee official GoCD XML configuration reference for details about each element. Below is a reference of format supported by this plugin. Feel free to improve it!
Please note that it is now recommended to declare format_version in each gocd.yaml file, consistent across all your files.
Supports format_version value of 9. In this version, support of ignore_for_scheduling for dependency materials has been added. Setting this attribute will skip scheduling the pipeline when the dependency material has changed.
Using a newer format_version includes all the behavior of the previous versions too.
format_version: 9
pipelines:
...
environments:Supports format_version value of 7 and 8. In version 7, support for properties has been removed. In version 8, support for mingle as a tracking tool has been removed.
Using a newer format_version includes all the behavior of the previous versions too.
Supports format_version value of 6. In this version, support of allow_only_on_success attribute for approval in stage has been added. Setting this attribute to true will allow the stage to be manually triggered only if the previous stage has passed successfully.
Using a newer format_version includes all the behavior of the previous versions too.
format_version: 6
pipelines:
...
environments:Supports format_version value of 5. In this version, support of username and encrypted_password for git and hg material has been added. In addition to that, hg will also support branch attribute.
Using a newer format_version includes all the behavior of the previous versions too.
format_version: 9
pipelines:
...
environments:Supports format_version value of 4. In this version, support has been added to control the display order of pipelines.
This server version also supports format_version of 3 and 2. Using a newer format_version includes all the behavior of the previous versions too.
format_version: 4
pipelines:
...
environments:Supports format_version value of 3. In this version fetch artifact format was changed to include artifact_origin.
This server version also supports format_version of 2. Using a newer format_version includes all the behavior of the previous versions too.
format_version: 3
pipelines:
...
environments:Supports format_version value of 2. In this version pipeline locking behavior was changed.
format_version: 2
pipelines:
...
environments:Supports format_version value of 1. This is the initial version.
format_version: 1
pipelines:
...
environments:A minimal pipeline configuration must contain:
template - as a key in hashmypipe:
group: mygroup
materials:
mygit:
git: http://example.com/mygit.git
stages:
- build:
jobs:
build:
tasks:
- exec:
command: makeAll elements available on a pipeline object are:
grouplabel_templateminglesecure_variablestemplatepipe2:
group: group1
label_template: "foo-1.0-${COUNT}"
lock_behavior: none
tracking_tool:
link: "http://your-trackingtool/yourproject/${ID}"
regex: "evo-(\\d+)"
timer:
spec: "0 15 10 * * ? *"
environment_variables:
DEPLOYMENT: testing
secure_variables:
ENV_PASSWORD: "s&Du#@$xsSa"
materials:
...
stages:
...mypipe:
group: group1
label_template: "foo-1.0-${COUNT}"
lock_behavior: none
parameters:
param1: value
materials:
mygit:
git: http://example.com/mygit.git
template: template1Please note:
When format_version is 4 (see above), the order of display of pipelines on the GoCD dashboard can be influenced by setting the display_order property.
-1.display_order value, their order relative to each other will be indeterminate.mypipeline1:
group: group1
display_order: 10
mypipeline2:
group: group1
display_order: -10In the above example, since both pipelines are in the same group, pipeline2 will be shown ahead of pipeline1. If any pipelines are defined in the GoCD config XML, then they will appear in between these two pipelines.
tracking_tool:
link: "http://your-trackingtool/yourproject/${ID}"
regex: "evo-(\\d+)"timer:
spec: "0 15 10 * * ? *"
only_on_changes: yesSee Quartz about writing cron-like schedules.
For GoCD >= 17.12 and format_version: 2 and above:
lock_behavior: noneWhere lock_behavior is defined as in GoCD documentation can be one of:
lockOnFailure - same as locking: trueunlockWhenFinishednone - same locking: falseFor GoCD < 17.12 and format_version: 1:
locking: on
Where locking is a boolean.
A minimal stage must contain jobs: element or tasks: in single-job stage case.
build:
jobs:
firstJob:
...
secondJob:
...A custom stage:
test:
fetch_materials: yes
keep_artifacts: yes
clean_workspace: yes
approval:
type: manual
roles:
- manager
users:
- john
environment_variables:
TEST_NUM: 1
secure_variables:
PASSWORD: "!@ESsdD323#sdu"
jobs:
one:
...
two:
...keep_artifacts is set to true GoCD will never delete artifacts in this stage. This is equivalent of XML's artifactCleanupProhibited. By default this is false.Stage can have approval,
which is success by default. There are 2 ways to declare approval:
approval: manualIf you need to set associated users or roles:
approval:
type: manual
allow_only_on_success: true
roles:
- manager
users:
- johnYou can set allow_only_on_success to allow manual trigger only if the previous stage run is successful. The default value is false.
Job is a hash starting with jobs name:
test:
timeout: 5
run_instances: 7
environment_variables:
LD_LIBRARY_PATH: .
tabs:
test: results.xml
resources:
- linux
artifacts:
- test:
source: src
destination: dest
- build:
source: bin
- external:
id: docker-release-candidate
store_id: dockerhub
configuration:
options:
Image: gocd/gocd-demo
Tag: v${GO_PIPELINE_LABEL}
secure_options:
some_secure_property: "!@ESsdD323#sdu"
tasks:
...Note: timeout is added since 0.2.0 version of yaml plugin
Job configuration may define elastic agents profile id, as such:
elastic_profile_id: "docker.unit-test"It MUST NOT be specified along with resources.
Available in GoCD server since v16.12.0, yaml plugin 0.4.0.
There are 3 types of artifacts recognized by GoCD. Build and Test artifacts are stored on the GoCD server.
The source and the destination of the artifact that should be stored on the GoCD server must be specified.
- build:
source: bin
destination: binaries- test:
source: reports
destination: test-reportsArtifacts of type external are stored in an artifact store outside of GoCD.
The external artifact store's configuration must be created in the main GoCD config. Support for external artifact store config to be checked in as yaml is not available.
The external artifact store is referenced by the store_id. The build specific artifact details that the artifact plugin needs to publish the artifact is provided as configuration.
- external:
id: docker-release-candidate
store_id: dockerhub
configuration:
options:
Image: gocd/gocd-demo
Tag: v${GO_PIPELINE_LABEL}
secure_options:
some_secure_property: "!@ESsdD323#sdu"S3 plugin 2.x example usage:
- external:
id: pkg
store_id: s3-eu-west-1
configuration:
options:
Source: installers/target/
Destination: ${GO_ARTIFACT_LOCATOR}Part of job object can be number of job to runs:
run_instances: 7Or to run on all agents:
run_instances: allTabs are a hash with <tab-name>: <path> pairs.
Path should exist in GoCD servers artifacts.
tabs:
tests: test-reports/index.html
gauge: functional-reports/index.htmlDEPRECATION NOTICE: Since GoCD version 19.9 and format_version 7, properties are no longer supported
Job can have properties, declared as a hash:
properties:
cov1: # this is the name of property
source: test.xml
xpath: "substring-before(//report/data/all/coverage[starts-with(@type,\u0027class\u0027)]/@value, \u0027%\u0027)"
performance.ind1.mbps:
source: PerfTestReport.xml
xpath: "//PerformanceSuiteReport/WriteOnly/MBps"A common use case is that stage has only one job. This plugin provides a shorthand
to declared such stages - just omit the jobs: and job name from configuration tree.
You can then declare job and stage options on the same (stage) level:
stages:
- build:
approval: manual
resources:
- cpp
tasks:
- exec:
command: makeAbove configuration declares build stage with build job which executes make task.
Minimal configuration of a git pipeline material:
mygithttp://example.com/mygit.gitmygit:
git: http://example.com/mygit.gitAbove can be also written more explicitly:
mygit:
type: git
url: http://example.com/mygit.gitMore customized git material is possible:
gitMaterial1:
git: "http://my.git.repository.com"
branch: feature12
blacklist:
- externals/**/*.*
- tools/**/*.*
destination: dir1
auto_update: false
shallow_clone: trueSince GoCD >= 16.7.0 whitelist is also supported,
you can specify whitelist instead of blacklist, as such
gitMaterial1:
git: "[email protected]"
branch: "feature12"
whitelist:
- src/**/*.*For GoCD >= 19.4.0 and format_version: 5 and above:
You are advised to utilize username and encrypted_password for passing in material credentials as:
gitMaterial1:
git: "http://my.git.repository.com"
branch: feature12
username: my_username
encrypted_password: encrypted_valueencrypted_password you may specify password but encrypted_password makes more sense considering that the value is stored in SCM.attributes and url will result in a validation error e.g.
INVALID MERGED CONFIGURATION
Number of errors: 1+
1. Ambiguous credentials, must be provided either in URL or as attributes.;;
- For Config Repo: https://your.config.repo.url at cbb047d78c239ab23b9565099e800c6fe4cc0ancFor details about each option, see GoCD XML reference
svnMaterial1:
svn: "http://svn"
username: "user1"
encrypted_password: "encrypted_value"
check_externals: true
blacklist:
- tools
- lib
destination: destDir1
auto_update: falseInstead of encrypted_password you can specify password.
hgMaterial1:
hg: repos/myhg
blacklist:
- externals
- tools
destination: dir1
auto_update: false
username: my_username
encrypted_password: encrypted_value
branch: featureFor GoCD >= 19.4.0 and format_version: 5 and above:
You are advised to utilize username and encrypted_password for passing in material credentials as:
hgMaterial1:
hg: repos/myhg
username: my_username
encrypted_password: encrypted_valueencrypted_password you may specify password but encrypted_password makes more sense considering that the value is stored in SCM.attributes and url will result in a validation error e.g.
INVALID MERGED CONFIGURATION
Number of errors: 1+
1. Ambiguous credentials, must be provided either in URL or as attributes.;;
- For Config Repo: https://your.config.repo.url at cbb047d78c239ab23b9565099e800c6fe4cc0ancIn addition to that, you can also leverage branch attribute to specify the branch for material
hgMaterial1:
hg: repos/myhg
branch: featurep4Material1:
p4: "host.domain.com:12345"
username: johndoe
encrypted_password: encrypted_value
use_tickets: false
view: |
//depot/external... //ws/external...
//depot/tools... //ws/external...
blacklist:
- externals
- tools
auto_update: falseInstead of encrypted_password you can specify password.
TODO: - not supported by yaml plugin yet
myPluggableGit:
scm: someScmGitRepositoryId
destination: destinationDir
blacklist:
- dir1
- dir2Since GoCD >= 19.2.0 defining new pluggable materials that are not defined
in the GoCD server is supported.
myPluggableGit:
plugin_configuration:
id: plugin_Id
version: 1 #plugin version
options:
url: [email protected]:tomzo/gocd-yaml-config-plugin.git
destination: destinationDir
blacklist:
- dir1
- dir2This is a convenience for shorter and more consistent material declaration. When configuration repository is the same as one of pipeline materials, then you usually need to repeat definitions in XML and in JSON, for example:
materials:
foo:
git: "https://github.com/tomzo/gocd-json-config-example.git",
branch: ciAnd in server XML:
<config-repos>
<config-repo pluginId="yaml.config.plugin" id="repo1">
<git url="https://github.com/tomzo/gocd-json-config-example.git" branch="ci" />
</config-repo>
</config-repos>Notice that url and branch is repeated. This is inconvenient in case when you move repository, because it requires 2 updates, in code and in server XML.
Using configrepo material type, above repetition can be avoided,
last example can be refactored into:
materials:
foo:
type: configrepoServer interprets configrepo material in this way:
Clone the material configuration of the repository we are parsing as is in XML and replace name, destination and filters (whitelist/blacklist), then use the modified clone in place of
configrepomaterial.
To add a dependency on another pipeline stage:
mydependency:
pipeline: upstream-pipeline-1
stage: test
ignore_for_scheduling: falseNote: mydependency is the name of material - it must be unique
myapt:
package: apt-repo-idEvery task is a hash starting with its type.
Type can be exec, ant, nant, rake, fetch, plugin or script.
<type>:
<task-prop1>: <prop1-value>Optionally any task can have run_if and on_cancel.
run_if is a string. Valid values are passed, failed, anyon_cancel is a task object. Same rules apply as to tasks described on this page.
See rake example.exec:
run_if: any
working_directory: dir
command: make
arguments:
- -j3
- docs
- installant:
build_file: mybuild.xml
target: compile
run_if: passednant:
run_if: passed
working_directory: "script/build/123"
build_file: FilePath
target: Build
nant_path: NantExeA minimal rake task with default values is very short
rake:A complete rake example:
rake:
run_if: passed
working_directory: sample-project
build_file: SomeRakefile
target: build
on_cancel:
rake:
working_directory: sample-project
build_file: CancelRakefile
target: cancelfetch:
run_if: any
artifact_origin: gocd # default value
pipeline: pipe2
stage: upstream_stage
job: upstream_job
is_file: yes
source: result
destination: testartifact_origin if not specified then, defaults to gocd, which is consistent with previous format_version 1 and 2.pipeline if not specified, then defaults to current pipeline.fetch:
run_if: any
artifact_origin: external
pipeline: pipe2
stage: upstream_stage
job: upstream_job
artifact_id: upstream_artifact_id
configuration:
options:
DestOnAgent: foo
secure_options:
some_secure_property: "!@ESsdD323#sdu"S3 plugin 2.x example usage:
- fetch:
artifact_origin: external
stage: plan
job: plan
artifact_id: plan
configuration:
options:
IsFile: true
Destination: terraform
Source: deployment.tfplanplugin:
options:
ConverterType: jsunit
secure_options:
password: "ssd#%fFS*!Esx"
run_if: failed
configuration:
id: xunit.converter.task.plugin
version: 1Because script-executor plugin requires quite a lot of boiler-plate configuration there is a shorthand for defining tasks with it:
script: ./build.sh compileAbove is equivalent of
plugin:
options:
script: ./build.sh compile
configuration:
id: script-executor
version: 1You can declare multi-line scripts too:
script: |
./build.sh compile
make testAbove executes a 2-line script:
./build.sh compile
make testIf you want to execute a single long line script, but break it into multiple lines in YAML,
you can do this with > as such:
script: >
./build.sh compile &&
make testAbove executes a single line script:
./build.sh compile && make testNOTE: The agents should be a guid, which is currently impossible to get for user
testing:
environment_variables:
DEPLOYMENT: testing
secure_variables:
ENV_PASSWORD: "s&Du#@$xsSa"
pipelines:
- example-deploy-testing
- build-testing
agents:
- 123Environment variables can be declared in Environments, Pipelines, Stages and Jobs.
In YAML you have 2 keywords for secure (encrypted) variables and standard variables.
environment_variables:
DEPLOYMENT: testing
FOO: bar
secure_variables:
# this value is encrypted by Go's private key (Note in 16.7.0 there is no easy way to obtain such value yet)
MY_PASSWORD: "s&Du#@$xsSa"Parameters can be declared at the pipeline level.
parameters:
param1: value1
production: noFor versions of GoCD >= 17.1:
See the encryption API.
For versions of GoCD <= 16.12:
There is no easy way to generate obtain encrypted value from GoServer, alternatively you can login into go-server and execute the following command to generate encrypted value
sudo -u go bash -c 'echo -n 'YOUR-INPUT' | openssl enc -des-cbc -a -iv 0 -K $(cat /etc/go/cipher)'Among all configuration elements there are boolean values, which can be defined using any of the keywords below (as in yaml specs):
y|Y|yes|Yes|YES|true|True|TRUE|on|On|ONn|N|no|No|NO|false|False|FALSE|off|Off|OFFYAML Aliases (specification) are supported and provide a way to avoid duplication.
Aliases can be defined anywhere in the configuration as long as they are valid configuration elements.
- exec:
command: make
arguments:
- clean
- &verbose_arg "VERBOSE=true" # define the alias
- exec:
command: make
arguments:
- world
- *verbose_arg # use the aliasThere is also a dedicated top-level common section which allows you to have all aliases in one place and where you don't need to worry about correct placement within the configuration.
format_version: 4
common:
verbose_arg: &verbose_arg "VERBOSE=true"
build_tasks: &build_tasks
- exec:
command: make
arguments:
- clean
- exec:
command: make
arguments:
- world
pipelines:
pipe1:
stages:
- build:
jobs:
build:
tasks: *build_tasks
test:
tasks:
- *build_tasks # task list aliases can also be mixed with additional tasks in the same job
- exec:
command: make
arguments:
- testSimilarly stages can be re-used:
format_version: 4
common:
build_stages: &build_stages
- stage1:
jobs:
# ...
- stage2:
jobs:
# ...
pipelines:
pipeline_one:
stages: *build_stages
pipeline_two:
stages:
- *build_stages
- stage3:
jobs:
# ...Please note this brief overview of what is done by the plugin:
And this is done by the GoCD server:
To build and test this plugin, you'll need java jdk >= 8.
If you have local java environment, then you may run all tests and create a ready to use jar with:
./gradlew test jarYou don't need to setup java on your host, if you are fine with using docker and Dojo. This is actually how our GoCD builds the plugin:
dojo "gradle test jar"Assuming you already have a working docker, you can install dojo by placing the executable anywhere on the PATH.
DOJO_VERSION=0.6.0
# On Linux
wget -O dojo https://github.com/kudulab/dojo/releases/download/${DOJO_VERSION}/dojo_linux_amd64
# On OSX
# wget -O dojo https://github.com/kudulab/dojo/releases/download/${DOJO_VERSION}/dojo_darwin_amd64
sudo mv dojo /usr/local/bin
sudo chmod +x /usr/local/bin/dojoThen enter a docker container with java and gradle pre-installed, by running following command at the root of the project:
dojoWe use semantic versioning.
If you are submitting a new feature then please run a major version bump by
./tasks.sh set_version 0.X.0If you are submitting a fix, then do not change any versions as patch bump is made right after each release.
There are examples of yaml partials and their resulting json to be sent to GoCD server. If something is not working right we can always add a new case covering exact yaml that user has and json that we expect on server side.
Copyright 2019 Tomasz Sętkowski
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.