Define deployments with YAML
Use YAML to schedule and trigger flow runs and manage your code and deployments.
The prefect.yaml
file is a YAML file describing base settings for your deployments, procedural steps for preparing deployments,
and instructions for preparing the execution environment for a deployment run.
Initialize your deployment configuration, which creates the prefect.yaml
file, with the CLI command prefect init
in any directory or repository that stores your flow code.
Deployment configuration recipes
Prefect ships with many off-the-shelf “recipes” that allow you to get started with more structure within
your prefect.yaml
file. Run prefect init
to be prompted with available recipes in your installation.
You can provide a recipe name in your initialization command with the --recipe
flag, otherwise Prefect will attempt to
guess an appropriate recipe based on the structure of your working directory (for example if you initialize within a git
repository, Prefect will use the git
recipe).
The prefect.yaml
file contains:
- deployment configuration for deployments created from this file
- default instructions for how to build and push any necessary code artifacts (such as Docker images)
- default instructions for pulling a deployment in remote execution environments (for example, cloning a GitHub repository).
You can override any deployment configuration through options available on the prefect deploy
CLI command when creating a deployment.
prefect.yaml
file flexibility
In older versions of Prefect, this file must be in the root of your repository or project directory and named prefect.yaml
.
With Prefect 3, this file can be located in a directory outside the project or a subdirectory inside the project.
It can be named differently if the filename ends in .yaml
. You can have multiple prefect.yaml
files with the same name
in different directories.
By default, prefect deploy
uses a prefect.yaml
file in the project’s root directory. To use a custom deployment
configuration file, supply the new --prefect-file
CLI argument when running the deploy
command from the root of your
project directory:
prefect deploy --prefect-file path/to/my_file.yaml
The base structure for prefect.yaml
looks like this:
# generic metadata
prefect-version: null
name: null
# preparation steps
build: null
push: null
# runtime steps
pull: null
# deployment configurations
deployments:
- # base metadata
name: null
version: null
tags: []
description: null
schedule: null
# flow-specific fields
entrypoint: null
parameters: {}
# infra-specific fields
work_pool:
name: null
work_queue_name: null
job_variables: {}
The metadata fields are always pre-populated for you. These fields are for bookkeeping purposes only. The other sections are pre-populated based on recipe; if no recipe is provided, Prefect attempts to guess an appropriate one based on local configuration.
You can create deployments with the CLI command prefect deploy
without altering the deployments
section of your
prefect.yaml
file. The prefect deploy
command helps in deployment creation through interactive prompts. The prefect.yaml
file facilitates version-controlling your deployment configuration and managing multiple deployments.
Deployment actions
Deployment actions defined in your prefect.yaml
file control the lifecycle of the creation and execution of your deployments.
The three actions available are build
, push
, and pull
.
pull
is the only required deployment action. It defines how Prefect pulls your deployment in remote execution
environments.
Each action is defined as a list of steps executed in sequence. Each step has the following format:
section:
- prefect_package.path.to.importable.step:
id: "step-id" # optional
requires: "pip-installable-package-spec" # optional
kwarg1: value
kwarg2: more-values
Every step optionally provides a requires
field. Prefect uses this to auto-install if the step is not
found in the current environment. Each step can specify an id
to reference outputs in
future steps. The additional fields map directly to Python keyword arguments to the step function. Within a given section,
steps always run in their order within the prefect.yaml
file.
Deployment instruction overrides
You can override build
, push
, and pull
sections on a per-deployment basis; define build
, push
, and pull
fields within a deployment definition in the prefect.yaml
file.
The prefect deploy
command uses any build
, push
, or pull
instructions from the deployment’s definition in the
prefect.yaml
file.
This capability is useful for multiple deployments that require different deployment instructions.
The build action
Use the build section of prefect.yaml
to specify setup steps or dependencies,
(like creating a Docker image), required to run your deployments.
If you initialize with the Docker recipe, you are prompted to provide required information, such as image name and tag:
prefect init --recipe docker
>> image_name: < insert image name here >
>> tag: < insert image tag here >
Use --field
to avoid the interactive experience
We recommend that you only initialize a recipe when first creating your deployment structure. Then store
your configuration files within version control.
Sometimes you may need to initialize programmatically and avoid the interactive prompts.
To do this, provide all required fields for your recipe using the --field
flag:
prefect init --recipe docker \
--field image_name=my-repo/my-image \
--field tag=my-tag
build:
- prefect_docker.deployments.steps.build_docker_image:
requires: prefect-docker>=0.3.0
image_name: my-repo/my-image
tag: my-tag
dockerfile: auto
push: true
Once you confirm that these fields are set to their desired values, this step automatically builds a Docker image with
the provided name and tag and pushes it to the repository referenced by the image name.
As the prefect-docker
package documentation explains,
this step produces optional fields for future steps, or within prefect.yaml
as template values.
We recommend using {{ image }}
within prefect.yaml
(specifically the work pool’s job variables section). This is so
you don’t risk your build step and deployment specification get out of sync with hardcoded values.
Some steps require Prefect integrations
In the build step example above, you relied on the prefect-docker
package; in cases that deal with external services,
additional required packages are auto-installed for you.
Pass output to downstream steps
Each deployment action can be composed of multiple steps. For example, if you wanted to build a Docker image tagged with the
current commit hash, you could use the run_shell_script
step and feed the output into the build_docker_image
step:
build:
- prefect.deployments.steps.run_shell_script:
id: get-commit-hash
script: git rev-parse --short HEAD
stream_output: false
- prefect_docker.deployments.steps.build_docker_image:
requires: prefect-docker
image_name: my-image
image_tag: "{{ get-commit-hash.stdout }}"
dockerfile: auto
The id
field is used in the run_shell_script
step to reference its output in the next step.
The push action
The push section is most critical for situations where code is not stored on persistent filesystems or in version control. In this scenario, code is often pushed and pulled from a Cloud storage bucket (for example, S3, GCS, Azure Blobs). The push section allows users to specify and customize the logic for pushing this code repository to arbitrary remote locations.
For example, a user who stores their code in an S3 bucket and relies on default worker settings for its runtime environment
could use the s3
recipe:
prefect init --recipe s3
>> bucket: < insert bucket name here >
In your newly createdprefect.yaml
file, you should find that the push
and pull
sections have been templated out
as follows:
push:
- prefect_aws.deployments.steps.push_to_s3:
id: push-code
requires: prefect-aws>=0.3.0
bucket: my-bucket
folder: project-name
credentials: null
pull:
- prefect_aws.deployments.steps.pull_from_s3:
requires: prefect-aws>=0.3.0
bucket: my-bucket
folder: "{{ push-code.folder }}"
credentials: null
The bucket is populated with the provided value (which also could have been provided with the --field
flag); note that the
folder
property of the push
step is a template—the pull_from_s3
step outputs both a bucket
value as well as a folder
value for the template downstream steps. This helps you keep your steps consistent across edits.
As discussed above, if you use blocks, you can template the credentials section with a block reference for secure and dynamic credentials access:
push:
- prefect_aws.deployments.steps.push_to_s3:
requires: prefect-aws>=0.3.0
bucket: my-bucket
folder: project-name
credentials: "{{ prefect.blocks.aws-credentials.dev-credentials }}"
Anytime you run prefect deploy
, this push
section executes upon successful completion of your build
section.
The pull action
The pull section is the most important section within the prefect.yaml
file. It contains instructions for preparing your
flows for a deployment run. These instructions execute each time a deployment in this folder is run through a worker.
There are three main types of steps that typically show up in a pull
section:
set_working_directory
: this step sets the working directory for the process prior to importing your flowgit_clone
: this step clones the provided repository on the provided branchpull_from_{cloud}
: this step pulls the working directory from a Cloud storage location (for example, S3)
Use block and variable references
All block and variable references within your pull step will remain unresolved until runtime and are pulled each time your deployment runs. This avoids storing sensitive information insecurely; it also allows you to manage certain types of configuration from the API and UI without having to rebuild your deployment every time.
Below is an example of how to use an existing GitHubCredentials
block to clone a private GitHub repository:
pull:
- prefect.deployments.steps.git_clone:
repository: https://github.com/org/repo.git
credentials: "{{ prefect.blocks.github-credentials.my-credentials }}"
Alternatively, you can specify a BitBucketCredentials
or GitLabCredentials
block to clone from Bitbucket or GitLab. In
lieu of a credentials block, you can also provide a GitHub, GitLab, or Bitbucket token directly to the ‘access_token` field.
Use a Secret block to do this securely:
pull:
- prefect.deployments.steps.git_clone:
repository: https://bitbucket.org/org/repo.git
access_token: "{{ prefect.blocks.secret.bitbucket-token }}"
Utility steps
Use utility steps within a build, push, or pull action to assist in managing the deployment lifecycle:
run_shell_script
allows for the execution of one or more shell commands in a subprocess, and returns the standard output and standard error of the script. This step is useful for scripts that require execution in a specific environment, or those which have specific input and output requirements.
Here is an example of retrieving the short Git commit hash of the current repository to use as a Docker image tag:
build:
- prefect.deployments.steps.run_shell_script:
id: get-commit-hash
script: git rev-parse --short HEAD
stream_output: false
- prefect_docker.deployments.steps.build_docker_image:
requires: prefect-docker>=0.3.0
image_name: my-image
tag: "{{ get-commit-hash.stdout }}"
dockerfile: auto
Provided environment variables are not expanded by default
To expand environment variables in your shell script, set expand_env_vars: true
in your run_shell_script
step. For example:
- prefect.deployments.steps.run_shell_script:
id: get-user
script: echo $USER
stream_output: true
expand_env_vars: true
Without expand_env_vars: true
, the above step returns a literal string $USER
instead of the current user.
pip_install_requirements
installs dependencies from arequirements.txt
file within a specified directory.
Here is an example of installing dependencies from a requirements.txt
file after cloning:
pull:
- prefect.deployments.steps.git_clone:
id: clone-step # needed to be referenced in subsequent steps
repository: https://github.com/org/repo.git
- prefect.deployments.steps.pip_install_requirements:
directory: "{{ clone-step.directory }}" # `clone-step` is a user-provided `id` field
requirements_file: requirements.txt
Here is an example that retrieves an access token from a third party key vault and uses it in a private clone step:
pull:
- prefect.deployments.steps.run_shell_script:
id: get-access-token
script: az keyvault secret show --name <secret name> --vault-name <secret vault> --query "value" --output tsv
stream_output: false
- prefect.deployments.steps.git_clone:
repository: https://bitbucket.org/samples/deployments.git
branch: master
access_token: "{{ get-access-token.stdout }}"
You can also run custom steps by packaging them. In the example below, retrieve_secrets
is a custom python module packaged
into the default working directory of a Docker image (which is /opt/prefect by default).
main
is the function entry point, which returns an access token (for example, return {"access_token": access_token}
) like the
preceding example, but utilizing the Azure Python SDK for retrieval.
- retrieve_secrets.main:
id: get-access-token
- prefect.deployments.steps.git_clone:
repository: https://bitbucket.org/samples/deployments.git
branch: master
access_token: '{{ get-access-token.access_token }}'
Templating options
Values that you place within your prefect.yaml
file can reference dynamic values in several different ways:
- step outputs: every step of both
build
andpush
produce named fields such asimage_name
; you can reference these fields withinprefect.yaml
andprefect deploy
will populate them with each call. References must be enclosed in double brackets and in"{{ field_name }}"
format - blocks: you can reference Prefect blocks with the
{{ prefect.blocks.block_type.block_slug }}
syntax. It is highly recommended that you use block references for any sensitive information (such as a GitHub access token or any credentials) to avoid hardcoding these values in plaintext - variables: you can reference Prefect variables with the
{{ prefect.variables.variable_name }}
syntax. Use variables to reference non-sensitive, reusable pieces of information such as a default image name or a default work pool name. - environment variables: you can also reference environment variables with the special syntax
{{ $MY_ENV_VAR }}
. This is especially useful for referencing environment variables that are set at runtime.
Here’s a prefect.yaml
file as an example:
build:
- prefect_docker.deployments.steps.build_docker_image:
id: build-image
requires: prefect-docker>=0.3.0
image_name: my-repo/my-image
tag: my-tag
dockerfile: auto
push: true
deployments:
- # base metadata
name: null
version: "{{ build-image.tag }}"
tags:
- "{{ $my_deployment_tag }}"
- "{{ prefect.variables.some_common_tag }}"
description: null
schedule: null
concurrency_limit: null
# flow-specific fields
entrypoint: null
parameters: {}
# infra-specific fields
work_pool:
name: "my-k8s-work-pool"
work_queue_name: null
job_variables:
image: "{{ build-image.image }}"
cluster_config: "{{ prefect.blocks.kubernetes-cluster-config.my-favorite-config }}"
So long as your build
steps produce fields called image_name
and tag
, every time you deploy a new version of our deployment,
the {{ build-image.image }}
variable is dynamically populated with the relevant values.
Docker step
The most commonly used build step is prefect_docker.deployments.steps.build_docker_image
which produces both the image_name
and tag
fields.
A prefect.yaml
file can have multiple deployment configurations that control the behavior of several deployments.
You can manage these deployments independently of one another, allowing you to deploy the same flow with different
configurations in the same codebase.
Work with multiple deployments with prefect.yaml
Prefect supports multiple deployment declarations within the prefect.yaml
file. This method of declaring multiple
deployments supports version control for all deployments through a single command.
Add new deployment declarations to the prefect.yaml
file with a new entry to the deployments
list.
Each deployment declaration must have a unique name
field to select deployment declarations when using the
prefect deploy
command.
When using a prefect.yaml
file that is in another directory or differently named, the value for
the deployment entrypoint
must be relative to the root directory of the project.
For example, consider the following prefect.yaml
file:
build: ...
push: ...
pull: ...
deployments:
- name: deployment-1
entrypoint: flows/hello.py:my_flow
parameters:
number: 42,
message: Don't panic!
work_pool:
name: my-process-work-pool
work_queue_name: primary-queue
- name: deployment-2
entrypoint: flows/goodbye.py:my_other_flow
work_pool:
name: my-process-work-pool
work_queue_name: secondary-queue
- name: deployment-3
entrypoint: flows/hello.py:yet_another_flow
work_pool:
name: my-docker-work-pool
work_queue_name: tertiary-queue
This file has three deployment declarations, each referencing a different flow. Each deployment declaration has a unique name
field and can be deployed individually with the --name
flag when deploying.
For example, to deploy deployment-1
, run:
prefect deploy --name deployment-1
To deploy multiple deployments, provide multiple --name
flags:
prefect deploy --name deployment-1 --name deployment-2
To deploy multiple deployments with the same name, prefix the deployment name with its flow name:
prefect deploy --name my_flow/deployment-1 --name my_other_flow/deployment-1
To deploy all deployments, use the --all
flag:
prefect deploy --all
To deploy deployments that match a pattern, run:
prefect deploy -n my-flow/* -n *dev/my-deployment -n dep*prod
The above command deploys:
- all deployments from the flow
my-flow
- all flows ending in
dev
with a deployment namedmy-deployment
- all deployments starting with
dep
and ending inprod
.
CLI Options When deploying multiple deployments
When deploying more than one deployment with a single prefect deploy
command, any additional attributes provided are ignored.
To provide overrides to a deployment through the CLI, you must deploy that deployment individually.
Reuse configuration across deployments
Because a prefect.yaml
file is a standard YAML file, you can use YAML aliases
to reuse configuration across deployments.
This capability allows multiple deployments to share the work pool configuration, deployment actions, or other configurations.
Declare a YAML alias with the &{alias_name}
syntax and insert that alias elsewhere in the file with the *{alias_name}
syntax. When aliasing YAML maps, you can override specific fields of the aliased map with the <<: *{alias_name}
syntax and
adding additional fields below.
We recommend adding a definitions
section to your prefect.yaml
file at the same level as the deployments
section to store your
aliases.
For example:
build: ...
push: ...
pull: ...
definitions:
work_pools:
my_docker_work_pool: &my_docker_work_pool
name: my-docker-work-pool
work_queue_name: default
job_variables:
image: "{{ build-image.image }}"
schedules:
every_ten_minutes: &every_10_minutes
interval: 600
actions:
docker_build: &docker_build
- prefect_docker.deployments.steps.build_docker_image: &docker_build_config
id: build-image
requires: prefect-docker>=0.3.0
image_name: my-example-image
tag: dev
dockerfile: auto
push: true
deployments:
- name: deployment-1
entrypoint: flows/hello.py:my_flow
schedule: *every_10_minutes
parameters:
number: 42,
message: Don't panic!
work_pool: *my_docker_work_pool
build: *docker_build # Uses the full docker_build action with no overrides
- name: deployment-2
entrypoint: flows/goodbye.py:my_other_flow
work_pool: *my_docker_work_pool
build:
- prefect_docker.deployments.steps.build_docker_image:
<<: *docker_build_config # Uses the docker_build_config alias and overrides the dockerfile field
dockerfile: Dockerfile.custom
- name: deployment-3
entrypoint: flows/hello.py:yet_another_flow
schedule: *every_10_minutes
work_pool:
name: my-process-work-pool
work_queue_name: primary-queue
In the above example, YAML aliases reuse work pool, schedule, and build configuration across multiple deployments:
deployment-1
anddeployment-2
use the same work pool configurationdeployment-1
anddeployment-3
use the same scheduledeployment-1
anddeployment-2
use the same build deployment action, butdeployment-2
overrides thedockerfile
field to use a custom Dockerfile
Deployment declaration reference
Deployment fields
These are fields you can add to each deployment declaration.
Property | Description |
---|---|
name | The name to give to the created deployment. Used with the prefect deploy command to create or update specific deployments. |
version | An optional version for the deployment. |
tags | A list of strings to assign to the deployment as tags. |
description | An optional description for the deployment. |
schedule | An optional schedule to assign to the deployment. Fields for this section are documented in the Schedule Fields section. |
concurrency_limit | An optional deployment concurrency limit. Fields for this section are documented in the Concurrency Limit Fields section. |
triggers | An optional array of triggers to assign to the deployment |
entrypoint | Required path to the .py file containing the flow you want to deploy (relative to the root directory of your development folder) combined with the name of the flow function. In the format path/to/file.py:flow_function_name . |
parameters | Optional default values to provide for the parameters of the deployed flow. Should be an object with key/value pairs. |
enforce_parameter_schema | Boolean flag that determines whether the API should validate the parameters passed to a flow run against the parameter schema generated for the deployed flow. |
work_pool | Information of where to schedule flow runs for the deployment. Fields for this section are documented in the Work Pool Fields section. |
Schedule fields
These are fields you can add to a deployment declaration’s schedule
section.
Property | Description |
---|---|
interval | Number of seconds indicating the time between flow runs. Cannot use them in conjunction with cron or rrule . |
anchor_date | Datetime string indicating the starting or “anchor” date to begin the schedule. If no anchor_date is supplied, the current UTC time is used. Can only use with interval . |
timezone | String name of a time zone, used to enforce localization behaviors like DST boundaries. See the IANA Time Zone Database for valid time zones. |
cron | A valid cron string. Cannot use in conjunction with interval or rrule . |
day_or | Boolean indicating how croniter handles day and day_of_week entries. Must use with cron . Defaults to True . |
rrule | String representation of an RRule schedule. See the rrulestr examples for syntax. Cannot used them in conjunction with interval or cron . |
Concurrency limit fields
These are fields you can add to a deployment declaration’s concurrency_limit
section.
Property | Description |
---|---|
limit | The maximum number of concurrent flow runs for the deployment. |
collision_strategy | Configure the behavior for runs once the concurrency limit is reached. Options are ENQUEUE , and CANCEL_NEW . Defaults to ENQUEUE . |
Work pool fields
These are fields you can add to a deployment declaration’s work_pool
section.
Property | Description |
---|---|
name | The name of the work pool to schedule flow runs in for the deployment. |
work_queue_name | The name of the work queue within the specified work pool to schedule flow runs in for the deployment. If not provided, the default queue for the specified work pool is used. |
job_variables | Values used to override the default values in the specified work pool’s base job template. Maps directly to a created deployments infra_overrides attribute. |
Deployment mechanics
Anytime you run prefect deploy
in a directory that contains a prefect.yaml
file, the following actions take place in order:
- The
prefect.yaml
file load. First, thebuild
section loads and all variable and block references resolve. The steps then run in the order provided. - Next, the
push
section loads and all variable and block references resolve; the steps within this section then run in the order provided. - Next, the
pull
section is templated with any step outputs but is not run. Block references are not hydrated for security purposes: they are always resolved at runtime. - Next, all variable and block references resolve with the deployment declaration. All flags provided through the
prefect deploy
CLI are then overlaid on the values loaded from the file. - The final step occurs when the fully realized deployment specification is registered with the Prefect API.
Deployment instruction overrides
The build
, push
, and pull
sections in deployment definitions take precedence over the corresponding sections above them in
prefect.yaml
.
Each time a step runs, the following actions take place in order:
- The step’s inputs and block / variable references resolve.
- The step’s function is imported; if it cannot be found, the special
requires
keyword installs the necessary packages. - The step’s function is called with the resolved inputs.
- The step’s output is returned and used to resolve inputs for subsequent steps.
Next steps
Now that you are familiar with creating deployments, you can explore infrastructure options for running your deployments:
Was this page helpful?