Deployment overview

​

Deployments

Deployments are server-side representations of flows. They store the crucial metadata for remote orchestration including when, where, and how a workflow should run.

Attributes of a deployment include:

• Flow entrypoint: path to your flow function
• Schedule or Trigger: optional schedule or triggering rules for this deployment
• Tags: optional text labels for organizing deployments

Some of the most common reasons to use an orchestration tool like Prefect are for scheduling and event-based triggering. As opposed to manually triggering and managing flow runs, deploying a flow exposes an API and UI that allow you to:

• trigger new runs, cancel active runs, pause scheduled runs, customize parameters, and more
• remotely configure schedules and automation rules for your deployments
• dynamically provision infrastructure using workers

​

Deployment schema

Here’s the complete schema that defines a deployment:

class Deployment:
    """
    Structure of the schema defining a deployment
    """

    # required defining data
    name: str 
    flow_id: UUID
    entrypoint: str
    path: Optional[str] = None

    # workflow scheduling and parametrization
    parameters: Optional[Dict[str, Any]] = None
    parameter_openapi_schema: Optional[Dict[str, Any]] = None
    schedules: list[Schedule] = None
    paused: bool = False
    trigger: Trigger = None

    # metadata for bookkeeping
    version: Optional[str] = None
    description: Optional[str] = None
    tags: Optional[list] = None

    # worker-specific fields
    work_pool_name: Optional[str] = None
    work_queue_name: Optional[str] = None
    job_variables: Optional[Dict[str, Any]] = None
    pull_steps: Optional[Dict[str, Any]] = None

All methods for creating Prefect deployments are interfaces for populating this schema. The following sections explain each component in more detail.

​

Required defining data

Deployments require a name and a reference to an underlying Flow.

The deployment name is not required to be unique across all deployments, but is required to be unique for a given flow ID. This means you will often see references to the deployment’s unique identifying name {FLOW_NAME}/{DEPLOYMENT_NAME}.

For example, you can trigger a run of a deployment from the Prefect CLI:

prefect deployment run my-first-flow/my-first-deployment

or with the SDK using run_deployment:

from prefect.deployments import run_deployment

run_deployment(
    name="my-first-flow/my-first-deployment",
    parameters={"my_param": "42"},
    job_variables={"env": {"MY_ENV_VAR": "staging"}},
    timeout=0, # don't wait for the run to finish
)

The other two fields are:

• path: think of the path as the runtime working directory for the flow. For example, if a deployment references a workflow defined within a Docker image, the path is the
  absolute path to the parent directory where that workflow will run anytime the deployment is triggered. This interpretation is more subtle in the case of flows defined in remote filesystems.
• entrypoint: the entrypoint of a deployment is a relative reference to a function decorated as a flow that exists on some filesystem. It is always specified relative to the path. Entrypoints use Python’s standard path-to-object syntax (for example, path/to/file.py:function_name or simply path:object).

The entrypoint must reference the same flow as the flow ID.

Prefect requires that deployments reference flows defined within Python files. Flows defined within interactive REPLs or notebooks cannot currently be deployed as such. They are still valid flows that will be monitored by the API and observable in the UI whenever they are run, but Prefect cannot trigger them.

​

Workflow scheduling and parametrization

One of the primary motivations for creating deployments of flows is to remotely schedule and trigger them. Just as you can call flows as functions with different input values, deployments can be triggered or scheduled with different values through parameters.

These are the fields to capture the required metadata for those actions:

• schedules: a list of schedule objects. Most of the convenient interfaces for creating deployments allow users to avoid creating this object themselves. For example, when updating a deployment schedule in the UI basic information such as a cron string or interval is all that’s required.
• parameter_openapi_schema: an OpenAPI compatible schema that defines the types and defaults for the flow’s parameters. This is used by the UI and the backend to expose options for creating manual runs as well as type validation.
• parameters: default values of flow parameters that this deployment will pass on each run. These can be overwritten through a trigger or when manually creating a custom run.
• enforce_parameter_schema: a boolean flag that determines whether the API should validate the parameters passed to a flow run against the schema defined by parameter_openapi_schema.

​

Metadata for bookkeeping

Important information for the versions, descriptions, and tags fields:

• version: versions are always set by the client and can be any arbitrary string. We recommend tightly coupling this field on your deployments to your software development lifecycle. For example if you leverage git to manage code changes, use either a tag or commit hash in this field. If you don’t set a value for the version, Prefect will compute a hash.
• description: provide reference material such as intended use and parameter documentation. Markdown is accepted. The docstring of your flow function is the default value.
• tags: group related work together across a diverse set of objects. Tags set on a deployment are inherited by that deployment’s flow runs. Filter, customize views, and searching by tag.

​

Worker-specific fields

Work pools and workers are an advanced deployment pattern that allow you to dynamically provision infrastructure for each flow run. The work pool job template interface allows users to create and govern opinionated interfaces to their workflow infrastructure. To do this, a deployment using workers needs the following fields:

• work_pool_name: the name of the work pool this deployment is associated with. Work pool types mirror infrastructure types, which means this field impacts the options available for the other fields.
• work_queue_name: if you are using work queues to either manage priority or concurrency, you can associate a deployment with a specific queue within a work pool using this field.
• job_variables: this field allows deployment authors to customize whatever infrastructure options have been exposed on this work pool. This field is often used for Docker image names, Kubernetes annotations and limits, and environment variables.
• pull_steps: a JSON description of steps that retrieves flow code or configuration, and prepares the runtime environment for workflow execution.

Pull steps allow users to highly decouple their workflow architecture. For example, a common use of pull steps is to dynamically pull code from remote filesystems such as GitHub with each run of their deployment.

​

Static vs. dynamic infrastructure

You can deploy your flows on long-lived static infrastructure or on dynamic infrastructure that is able to scale horizontally. The right choice depends on your use case.

​

Static infrastructure

When you have several flows running regularly, the serve method of the Flow object or the serve utility is a great option for managing multiple flows simultaneously.

Once you have authored your flow and decided on its deployment settings, run this long-running process in a location of your choosing. The process stays in communication with the Prefect API, monitoring for work and submitting each run within an individual subprocess. Because runs are submitted to subprocesses, any external infrastructure configuration must be set up beforehand and kept associated with this process.

This approach has these benefits:

• Users are in complete control of their infrastructure, and anywhere the “serve” Python process can run is a suitable deployment environment.
• It is simple to reason about.
• Creating deployments requires a minimal set of decisions.
• Iteration speed is fast.

​

Dynamic infrastructure

Consider running flows on dynamically provisioned infrastructure with work pools when you have any of the following:

• Flows that require expensive infrastructure due to the long-running process.
• Flows with heterogeneous infrastructure needs across runs.
• Large volumes of deployments.
• An internal organizational structure in which deployment authors or runners are not members of the team that manages the infrastructure.

Work pools allow Prefect to exercise greater control of the infrastructure on which flows run. Options for serverless work pools allow you to scale to zero when workflows aren’t running. Prefect even provides you with the ability to provision cloud infrastructure via a single CLI command, if you use a Prefect Cloud push work pool option.

With work pools:

• You can configure and monitor infrastructure configuration within the Prefect UI.
• Infrastructure is ephemeral and dynamically provisioned.
• Prefect is more infrastructure-aware and collects more event data from your infrastructure by default.
• Highly decoupled setups are possible.