5.2 Tasks
The smallest configurable unit in a Concourse pipeline is a single task. A task can be thought of as a function from inputs
to outputs
that can either succeed or fail.
Going a bit further, ideally tasks are pure functions: given the same set of inputs, it should either always succeed with the same outputs or always fail. This is entirely up to your script's level of discipline, however. Flaky tests or dependencies on the internet are the most common source of impurity.
Once you have a running Concourse deployment, you can start configuring your tasks and executing them interactively from your terminal with the Fly commandline tool.
Once you've figured out your tasks's configuration, you can reuse it for a Job in your Pipeline.
Conventionally a task's configuration is placed in the same repository as the code it's testing, possibly under some ci
directory.
A task's configuration specifies the following:
platform: string
Required. The platform the task should run on. By convention, windows
, linux
, or darwin
are specified. This determines the pool of workers that the task can run against. The base deployment provides Linux workers.
Where resource
is:
type: string
Required. The type of the resource. Usually docker-image
.
params: object
Optional. A map of arbitrary configuration to forward to the resource. Refer to the resource type's documentation to see what it supports.
version: object
Optional. A specific version of the resource to fetch. This should be a map with string keys and values. If not specified, the latest version will be fetched.
Optional. The base image of the container, as provided by a resource definition.
You can use any resource that returns a filesystem in the correct format (a /rootfs
directory and a metadata.json
file in the top level) but normally this will be the Docker Image resource. If you'd like to make a resource of your own that supports this please use that as a reference implementation for now.
If you want to use an artifact source within the plan containing an image, you must set the image
in the plan step instead.
rootfs_uri: string
Optional. A string specifying the rootfs uri of the container, as interpreted by your worker's Garden backend.
image_resource
is a preferred way to specify base image and rootfs_uri
is not recommended. With rootfs_uri
image fetching is delegated to backend which does not guarantee image caching and might result in some permission errors. You should only use this if you cannot use image_resource
for some reason, and you know what you're doing.
inputs: [input]
Where input
is:
name: string
Required. The logical name of the input.
path: string
Optional. The path where the input will be placed. If not specified, the input's name
is used.
Paths are relative to the working directory of the task. Absolute paths are not respected.
optional: bool
Optional. If true
, then the input is not required by the task. The task may run even if this input is missing.
An optional
input that is missing will not appear in the current directory of the running task.
Optional. The set of artifacts used by task, determining which artifacts will be available in the current directory when the task runs.
These are satisfied by get
steps or outputs
of a previous task. These can also be provided by -i
with fly execute
.
If any required inputs are missing at run-time, then the task will error immediately.
outputs: [output]
Where output
is:
name: string
Required. The logical name of the output. The contents under path
will be made available to the rest of the plan under this name.
path: string
Optional. The path to a directory where the output will be taken from. If not specified, the output's name
is used.
Paths are relative to the working directory of the task. Absolute paths are not respected.
Note that this value must not overlap with any other inputs or outputs. Each output results in a new empty directory that your task should place artifacts in; if the path overlaps it'll clobber whatever files used to be there.
Optional. The artifacts produced by the task.
Each output configures a directory to make available to later steps in the build plan. The directory will be automatically created before the task runs, and the task should place any artifacts it wants to export in the directory.
Where cache
is:
path: string
Required. The path to a directory to be cached.
Paths are relative to the working directory of the task. Absolute paths are not respected.
Note that this value must not overlap with any other caches in the same task. Each cache results in a new empty directory that your task can place artifacts in; if the path overlaps it'll clobber whatever files used to be there.
Optional. The cached directories shared between task runs.
On the task's first run, all cache directories will be empty. It is the responsibility of the task to populate these directories with any artifacts to be cached. On subsequent runs, the cached directories will contain those artifacts.
Caches are scoped to the worker the task is run on, so you will not get a cache hit when subsequent builds run on different workers. This also means that caching is not intended to share state between workers, and your task should be able to run whether or not the cache is warmed.
Caches are also scoped to a particular task name inside of a pipeline's job. As a consequence, if the job name, step name or cache path are changed, the cache will not be used. This also means that caches do not exist for one-off builds.
run: run-config
Where run-config
is:
path: string
Required. The command to execute.
This is commonly a path to a script provided by one of the task's inputs, e.g. my-resource/scripts/test
. It could also be a command like bash
(respecting standard $PATH
lookup rules), or an absolute path to a file to execute, e.g. /bin/bash
.
args: [string]
Optional. Arguments to pass to the command. Note that when executed with Fly, any arguments passed to Fly are appended to this array.
dir: string
Optional. A directory, relative to the initial working directory, to set as the working directory when running the script.
user: string
Optional. Explicitly set the user to run as. If not specified, this defaults to the user configured by the task's image. If not specified there, it's up to the Garden backend, and may be e.g. root
on Linux.
Required. The command to execute in the container.
Note that this is not provided as a script blob, but explicit path
and args
values; this allows fly
to forward arguments to the script, and forces your config .yml
to stay fairly small.
params: {string: string}
Optional. A key-value mapping of values that are exposed to the task via environment variables.
Use this to provide things like credentials, not to set up the task's Bash environment (they do not support interpolation).
Examples
Ruby App
This configuration specifies that the task must run with the ruby:2.1
Docker image with a my-app
input, and when the task is executed it will run the scripts/test
script in the same repo.
---
platform: linux
image_resource:
type: docker-image
source:
repository: ruby
tag: '2.1'
inputs:
- name: my-app
run:
path: my-app/scripts/test
Producing Outputs
A task can configure outputs
to produce artifacts that can then be propagated to a put
step or another task
step in the same plan. They can also be downloaded with fly execute
by passing -o
.
---
platform: linux
image_resource: # ...
inputs:
- name: project-src
outputs:
- name: built-project
run:
path: project-src/ci/build
...assuming project-src/ci/build
looks something like:
#!/bin/bash
set -e -u -x
export GOPATH=$PWD/project-src
go build -o built-project/my-project \
github.com/concourse/my-project
...this task could then be used in a build plan like so:
plan:
- get: project-src
- task: build-bin
file: project-src/ci/build.yml
- put: project-bin
params: file: built-project/my-project
Caching Ephemeral State
The following task and script could be used by a Node project to cache the node_modules
directory:
---
platform: linux
image_resource: # ...
inputs:
- name: project-src
caches:
- path: project-src/node_modules
run:
path: project-src/ci/build
...assuming project-src/ci/build
looks something like:
#!/bin/bash
set -e -u -x
cd project-src
npm install
# ...
...this task would cache the contents of project-src/node_modules
between runs of this task on the same worker.
Private Docker Registry
The following task uses an image from a private registry. Assuming the CA is configured properly on the workers, SSL should Just Workâ˘.
Note that, at present, credentials must be inlined when using image_resource
. We recognize that this kind of sucks, since tasks are often checked in. Until we fix this, you can use image
in your pipeline instead. (However this will prevent fly execute
from having an image available. Bummer.)
---
platform: linux
image_resource:
type: docker-image
source:
repository: my.local.registry:8080/my/image
username: myuser
password: mypass
email: x@x.com
inputs:
- name: my-app
run:
path: my-app/scripts/test