Learn how to use deployments to trigger flow runs remotely.
Deployments allow you to run flows on a schedule and trigger runs based on events.
Deployments are server-side representations of flows. They store the crucial metadata for remote orchestration including when, where, and how a workflow should run.
In addition to manually triggering and managing flow runs, deploying a flow exposes an API and UI that allow you to:
In Prefect Cloud, deployment configuration is versioned, and a new deployment version is created each time a deployment is updated.
Work pools allow you to switch between different types of infrastructure and to create a template for deployments. Data platform teams find work pools especially useful for managing infrastructure configuration across teams of data professionals.
Common work pool types include Docker, Kubernetes, and serverless options such as AWS ECS, Azure ACI, GCP Vertex AI, or GCP Google Cloud Run.
Deployments created through the Python SDK that use a work pool require a name
.
This value becomes the deployment name.
A work_pool_name
is also required.
Your flow code location can be specified in a few ways:
image
argument in the deploy
method.from_source
on a flow and specify one of the following:
See the Retrieve code from storage docs for more information about flow code storage.
You can set a deployment to run manually, on a schedule, or in response to an event.
The deployment inherits the infrastructure configuration from the work pool, and can be overridden at deployment creation time or at runtime.
To run a deployment with a hybrid work pool type, such as Docker or Kubernetes, you must start a worker.
A Prefect worker is a client-side process that checks for scheduled flow runs in the work pool that it matches. When a scheduled run is found, the worker kicks off a flow run on the specified infrastructure and monitors the flow run until completion.
Prefect Cloud offers push work pools that run flows on Cloud provider serverless infrastructure without a worker and that can be set up quickly.
Prefect Cloud also provides the option to run work flows on Prefect’s infrastructure through a Prefect Managed work pool.
These work pool types do not require a worker to run flows. However, they do require sharing a bit more information with Prefect, which can be a challenge depending upon the security posture of your organization.
You can deploy your flows on long-lived static infrastructure or on dynamic infrastructure that is able to scale horizontally. The best choice depends on your use case.
When you have several flows running regularly, the serve
method
of the Flow
object or the serve
utility
is a great option for managing multiple flows simultaneously.
Once you have authored your flow and decided on its deployment settings, run this long-running process in a location of your choosing. The process stays in communication with the Prefect API, monitoring for work and submitting each run within an individual subprocess. Because runs are submitted to subprocesses, any external infrastructure configuration must be set up beforehand and kept associated with this process.
Benefits to this approach include:
Consider running flows on dynamically provisioned infrastructure with work pools when you have any of the following:
Work pools allow Prefect to exercise greater control of the infrastructure on which flows run. Options for serverless work pools allow you to scale to zero when workflows aren’t running. Prefect even provides you with the ability to provision cloud infrastructure via a single CLI command, if you use a Prefect Cloud push work pool option.
With work pools:
You don’t have to commit to one approach
You can mix and match approaches based on the needs of each flow. You can also change the deployment approach for a particular flow as its needs evolve. For example, you might use workers for your expensive machine learning pipelines, but use the serve mechanics for smaller, more frequent file-processing pipelines.
All methods for creating Prefect deployments are interfaces for populating this schema.
Deployments require a name
and a reference to an underlying Flow
.
The deployment name is not required to be unique across all deployments, but is required to be unique
for a given flow ID. This means you will often see references to the deployment’s unique identifying name
{FLOW_NAME}/{DEPLOYMENT_NAME}
.
For example, to trigger a deployment run from the Prefect CLI:
or from the Python SDK with run_deployment
:
The other two fields are:
path
: think of the path as the runtime working directory for the flow.
For example, if a deployment references a workflow defined within a Docker image, the path
is the
absolute path to the parent directory where that workflow will run anytime the deployment is triggered.
This interpretation is more subtle in the case of flows defined in remote filesystems.entrypoint
: the entrypoint of a deployment is a relative reference to a function decorated as a
flow that exists on some filesystem. It is always specified relative to the path
.
Entrypoints use Python’s standard path-to-object syntax
(for example, path/to/file.py:function_name
or simply path:object
).The entrypoint must reference the same flow as the flow ID.
Prefect requires that deployments reference flows defined within Python files. Flows defined within interactive REPLs or notebooks cannot currently be deployed as such. They are still valid flows that will be monitored by the API and observable in the UI whenever they are run, but Prefect cannot trigger them.
Deployments do not contain code definitions
Deployment metadata references code that exists in potentially diverse locations within your environment. This separation means that your flow code stays within your storage and execution infrastructure.
This is key to the Prefect hybrid model: there’s a boundary between your proprietary assets, such as your flow code, and the Prefect backend (including Prefect Cloud).
One of the primary motivations for creating deployments of flows is to remotely schedule and trigger them. Just as you can call flows as functions with different input values, deployments can be triggered or scheduled with different values through parameters.
These are the fields to capture the required metadata for those actions:
schedules
: a list of schedule objects.
Most of the convenient interfaces for creating deployments allow users to avoid creating this object themselves.
For example, when updating a deployment schedule in the UI
basic information such as a cron string or interval is all that’s required.parameter_openapi_schema
: an OpenAPI compatible schema that defines
the types and defaults for the flow’s parameters.
This is used by the UI and the backend to expose options for creating manual runs as well as type validation.parameters
: default values of flow parameters that this deployment will pass on each run.
These can be overwritten through a trigger or when manually creating a custom run.enforce_parameter_schema
: a boolean flag that determines whether the API should validate the parameters
passed to a flow run against the schema defined by parameter_openapi_schema
.Scheduling is asynchronous and decoupled
Pausing a schedule, updating your deployment, and other actions reset your auto-scheduled runs.
Prefect supports managing concurrency at the deployment level to enable limiting how many runs of a deployment can be active at once. To enable this behavior, deployments have the following fields:
concurrency_limit
: an integer that sets the maximum number of concurrent flow runs for the deployment.collision_strategy
: configure the behavior for runs once the concurrency limit is reached.
Falls back to ENQUEUE
if unset.
ENQUEUE
: new runs transition to AwaitingConcurrencySlot
and execute as slots become available.CANCEL_NEW
: new runs are canceled until a slot becomes available.Important information for the versions, descriptions, and tags fields:
version
: versions are always set by the client and can be any arbitrary string.
We recommend tightly coupling this field on your deployments to your software development lifecycle and choosing human-readable version strings.
If left unset, the version field will be automatically populated in one of two ways:
version
will be the first eight characters of your commit hash.version
will be your flow’s version, which if not assigned in the flow decorator (@flow(version="my-version")
) will be a hash of the file the flow is defined in.version_type
: When a deployment is created or updated, Prefect will attempt to infer version information from your environment.
Providing a version_type
instructs Prefect to only attempt version information collection from an environment of that type.
The following version types are available: vcs:github
, vcs:gitlab
, vcs:bitbucket
, vcs:azuredevops
, vcs:git
, or prefect:simple
.
vcs:git
offers similar versioning detail to officially supported version control platforms, but does not support direct linking to commits from the Prefect Cloud UI.
It is meant as a fallback option in case your version control platform is not supported.
prefect:simple
is for any deployment version created where no Git context is available.
If left unset, Prefect will automatically select the appropriate version_type
based on the detected environment.description
: provide reference material such as intended use and parameter documentation.
Markdown is accepted. The docstring of your flow function is the default value.tags
: group related work together across a diverse set of objects.
Tags set on a deployment are inherited by that deployment’s flow runs. Filter, customize views, and
searching by tag.Everything has a version
Deployments have a version attached; and flows and tasks also have versions set through their respective decorators. These versions are sent to the API anytime the flow or task runs, allowing you to audit changes.
Work pools and workers are an advanced deployment pattern that allow you to dynamically provision infrastructure for each flow run. The work pool job template interface allows users to create and govern opinionated interfaces to their workflow infrastructure. To do this, a deployment using workers needs the following fields:
work_pool_name
: the name of the work pool this deployment is associated with.
Work pool types mirror infrastructure types, which means this field impacts the options available
for the other fields.work_queue_name
: if you are using work queues to either manage priority or concurrency, you can
associate a deployment with a specific queue within a work pool using this field.job_variables
: this field allows deployment authors to customize whatever infrastructure
options have been exposed on this work pool.
This field is often used for Docker image names, Kubernetes annotations and limits,
and environment variables.pull_steps
: a JSON description of steps that retrieves flow code or
configuration, and prepares the runtime environment for workflow execution.Pull steps allow users to highly decouple their workflow architecture. For example, a common use of pull steps is to dynamically pull code from remote filesystems such as GitHub with each run of their deployment.
Learn how to use deployments to trigger flow runs remotely.
Deployments allow you to run flows on a schedule and trigger runs based on events.
Deployments are server-side representations of flows. They store the crucial metadata for remote orchestration including when, where, and how a workflow should run.
In addition to manually triggering and managing flow runs, deploying a flow exposes an API and UI that allow you to:
In Prefect Cloud, deployment configuration is versioned, and a new deployment version is created each time a deployment is updated.
Work pools allow you to switch between different types of infrastructure and to create a template for deployments. Data platform teams find work pools especially useful for managing infrastructure configuration across teams of data professionals.
Common work pool types include Docker, Kubernetes, and serverless options such as AWS ECS, Azure ACI, GCP Vertex AI, or GCP Google Cloud Run.
Deployments created through the Python SDK that use a work pool require a name
.
This value becomes the deployment name.
A work_pool_name
is also required.
Your flow code location can be specified in a few ways:
image
argument in the deploy
method.from_source
on a flow and specify one of the following:
See the Retrieve code from storage docs for more information about flow code storage.
You can set a deployment to run manually, on a schedule, or in response to an event.
The deployment inherits the infrastructure configuration from the work pool, and can be overridden at deployment creation time or at runtime.
To run a deployment with a hybrid work pool type, such as Docker or Kubernetes, you must start a worker.
A Prefect worker is a client-side process that checks for scheduled flow runs in the work pool that it matches. When a scheduled run is found, the worker kicks off a flow run on the specified infrastructure and monitors the flow run until completion.
Prefect Cloud offers push work pools that run flows on Cloud provider serverless infrastructure without a worker and that can be set up quickly.
Prefect Cloud also provides the option to run work flows on Prefect’s infrastructure through a Prefect Managed work pool.
These work pool types do not require a worker to run flows. However, they do require sharing a bit more information with Prefect, which can be a challenge depending upon the security posture of your organization.
You can deploy your flows on long-lived static infrastructure or on dynamic infrastructure that is able to scale horizontally. The best choice depends on your use case.
When you have several flows running regularly, the serve
method
of the Flow
object or the serve
utility
is a great option for managing multiple flows simultaneously.
Once you have authored your flow and decided on its deployment settings, run this long-running process in a location of your choosing. The process stays in communication with the Prefect API, monitoring for work and submitting each run within an individual subprocess. Because runs are submitted to subprocesses, any external infrastructure configuration must be set up beforehand and kept associated with this process.
Benefits to this approach include:
Consider running flows on dynamically provisioned infrastructure with work pools when you have any of the following:
Work pools allow Prefect to exercise greater control of the infrastructure on which flows run. Options for serverless work pools allow you to scale to zero when workflows aren’t running. Prefect even provides you with the ability to provision cloud infrastructure via a single CLI command, if you use a Prefect Cloud push work pool option.
With work pools:
You don’t have to commit to one approach
You can mix and match approaches based on the needs of each flow. You can also change the deployment approach for a particular flow as its needs evolve. For example, you might use workers for your expensive machine learning pipelines, but use the serve mechanics for smaller, more frequent file-processing pipelines.
All methods for creating Prefect deployments are interfaces for populating this schema.
Deployments require a name
and a reference to an underlying Flow
.
The deployment name is not required to be unique across all deployments, but is required to be unique
for a given flow ID. This means you will often see references to the deployment’s unique identifying name
{FLOW_NAME}/{DEPLOYMENT_NAME}
.
For example, to trigger a deployment run from the Prefect CLI:
or from the Python SDK with run_deployment
:
The other two fields are:
path
: think of the path as the runtime working directory for the flow.
For example, if a deployment references a workflow defined within a Docker image, the path
is the
absolute path to the parent directory where that workflow will run anytime the deployment is triggered.
This interpretation is more subtle in the case of flows defined in remote filesystems.entrypoint
: the entrypoint of a deployment is a relative reference to a function decorated as a
flow that exists on some filesystem. It is always specified relative to the path
.
Entrypoints use Python’s standard path-to-object syntax
(for example, path/to/file.py:function_name
or simply path:object
).The entrypoint must reference the same flow as the flow ID.
Prefect requires that deployments reference flows defined within Python files. Flows defined within interactive REPLs or notebooks cannot currently be deployed as such. They are still valid flows that will be monitored by the API and observable in the UI whenever they are run, but Prefect cannot trigger them.
Deployments do not contain code definitions
Deployment metadata references code that exists in potentially diverse locations within your environment. This separation means that your flow code stays within your storage and execution infrastructure.
This is key to the Prefect hybrid model: there’s a boundary between your proprietary assets, such as your flow code, and the Prefect backend (including Prefect Cloud).
One of the primary motivations for creating deployments of flows is to remotely schedule and trigger them. Just as you can call flows as functions with different input values, deployments can be triggered or scheduled with different values through parameters.
These are the fields to capture the required metadata for those actions:
schedules
: a list of schedule objects.
Most of the convenient interfaces for creating deployments allow users to avoid creating this object themselves.
For example, when updating a deployment schedule in the UI
basic information such as a cron string or interval is all that’s required.parameter_openapi_schema
: an OpenAPI compatible schema that defines
the types and defaults for the flow’s parameters.
This is used by the UI and the backend to expose options for creating manual runs as well as type validation.parameters
: default values of flow parameters that this deployment will pass on each run.
These can be overwritten through a trigger or when manually creating a custom run.enforce_parameter_schema
: a boolean flag that determines whether the API should validate the parameters
passed to a flow run against the schema defined by parameter_openapi_schema
.Scheduling is asynchronous and decoupled
Pausing a schedule, updating your deployment, and other actions reset your auto-scheduled runs.
Prefect supports managing concurrency at the deployment level to enable limiting how many runs of a deployment can be active at once. To enable this behavior, deployments have the following fields:
concurrency_limit
: an integer that sets the maximum number of concurrent flow runs for the deployment.collision_strategy
: configure the behavior for runs once the concurrency limit is reached.
Falls back to ENQUEUE
if unset.
ENQUEUE
: new runs transition to AwaitingConcurrencySlot
and execute as slots become available.CANCEL_NEW
: new runs are canceled until a slot becomes available.Important information for the versions, descriptions, and tags fields:
version
: versions are always set by the client and can be any arbitrary string.
We recommend tightly coupling this field on your deployments to your software development lifecycle and choosing human-readable version strings.
If left unset, the version field will be automatically populated in one of two ways:
version
will be the first eight characters of your commit hash.version
will be your flow’s version, which if not assigned in the flow decorator (@flow(version="my-version")
) will be a hash of the file the flow is defined in.version_type
: When a deployment is created or updated, Prefect will attempt to infer version information from your environment.
Providing a version_type
instructs Prefect to only attempt version information collection from an environment of that type.
The following version types are available: vcs:github
, vcs:gitlab
, vcs:bitbucket
, vcs:azuredevops
, vcs:git
, or prefect:simple
.
vcs:git
offers similar versioning detail to officially supported version control platforms, but does not support direct linking to commits from the Prefect Cloud UI.
It is meant as a fallback option in case your version control platform is not supported.
prefect:simple
is for any deployment version created where no Git context is available.
If left unset, Prefect will automatically select the appropriate version_type
based on the detected environment.description
: provide reference material such as intended use and parameter documentation.
Markdown is accepted. The docstring of your flow function is the default value.tags
: group related work together across a diverse set of objects.
Tags set on a deployment are inherited by that deployment’s flow runs. Filter, customize views, and
searching by tag.Everything has a version
Deployments have a version attached; and flows and tasks also have versions set through their respective decorators. These versions are sent to the API anytime the flow or task runs, allowing you to audit changes.
Work pools and workers are an advanced deployment pattern that allow you to dynamically provision infrastructure for each flow run. The work pool job template interface allows users to create and govern opinionated interfaces to their workflow infrastructure. To do this, a deployment using workers needs the following fields:
work_pool_name
: the name of the work pool this deployment is associated with.
Work pool types mirror infrastructure types, which means this field impacts the options available
for the other fields.work_queue_name
: if you are using work queues to either manage priority or concurrency, you can
associate a deployment with a specific queue within a work pool using this field.job_variables
: this field allows deployment authors to customize whatever infrastructure
options have been exposed on this work pool.
This field is often used for Docker image names, Kubernetes annotations and limits,
and environment variables.pull_steps
: a JSON description of steps that retrieves flow code or
configuration, and prepares the runtime environment for workflow execution.Pull steps allow users to highly decouple their workflow architecture. For example, a common use of pull steps is to dynamically pull code from remote filesystems such as GitHub with each run of their deployment.