Run flows on serverless compute
Learn how to use Prefect push work pools to schedule work on serverless infrastructure without having to run a worker.
Push work pools are a special type of work pool. They allow Prefect Cloud to submit flow runs for execution to serverless computing infrastructure without requiring you to run a worker. Push work pools currently support execution in AWS ECS tasks, Azure Container Instances, Google Cloud Run jobs, Modal, and Coiled.
In this guide you will:
- Create a push work pool that sends work to Amazon Elastic Container Service (AWS ECS), Azure Container Instances (ACI), Google Cloud Run, Modal, or Coiled
- Deploy a flow to that work pool
- Execute a flow without having to run a worker process to poll for flow runs
You can automatically provision infrastructure and create your push work pool using the prefect work-pool create
CLI command with the --provision-infra
flag.
This approach greatly simplifies the setup process.
First, you will set up automatic infrastructure provisioning for push work pools. Then you will learn how to manually set up your push work pool.
Automatic infrastructure provisioning
With Perfect Cloud you can provision infrastructure for use with an AWS ECS, Google Cloud Run, ACI push work pool. Push work pools in Prefect Cloud simplify the setup and management of the infrastructure necessary to run your flows. However, setting up infrastructure on your cloud provider can still be a time-consuming process. Prefect dramatically simplifies this process by automatically provisioning the necessary infrastructure for you.
We’ll use the prefect work-pool create
CLI command with the --provision-infra
flag to automatically provision your
serverless cloud resources and set up your Prefect workspace to use a new push pool.
Prerequisites
To use automatic infrastructure provisioning, you need:
- the relevant cloud CLI library installed
- to be authenticated with your cloud provider
Install the AWS CLI, authenticate with your AWS account, and set a default region.
If you already have the AWS CLI installed, be sure to update to the latest version.
You will need the following permissions in your authenticated AWS account:
IAM Permissions:
- iam:CreatePolicy
- iam:GetPolicy
- iam:ListPolicies
- iam:CreateUser
- iam:GetUser
- iam:AttachUserPolicy
- iam:CreateRole
- iam:GetRole
- iam:AttachRolePolicy
- iam:ListRoles
- iam:PassRole
Amazon ECS Permissions:
- ecs:CreateCluster
- ecs:DescribeClusters
Amazon EC2 Permissions:
- ec2:CreateVpc
- ec2:DescribeVpcs
- ec2:CreateInternetGateway
- ec2:AttachInternetGateway
- ec2:CreateRouteTable
- ec2:CreateRoute
- ec2:CreateSecurityGroup
- ec2:DescribeSubnets
- ec2:CreateSubnet
- ec2:DescribeAvailabilityZones
- ec2:AuthorizeSecurityGroupIngress
- ec2:AuthorizeSecurityGroupEgress
Amazon ECR Permissions:
- ecr:CreateRepository
- ecr:DescribeRepositories
- ecr:GetAuthorizationToken
If you want to use AWS managed policies, you can use the following:
- AmazonECS_FullAccess
- AmazonEC2FullAccess
- IAMFullAccess
- AmazonEC2ContainerRegistryFullAccess
The above policies give you all the permissions needed, but are more permissive than necessary.
Docker is also required to build and push images to your registry.
Install the AWS CLI, authenticate with your AWS account, and set a default region.
If you already have the AWS CLI installed, be sure to update to the latest version.
You will need the following permissions in your authenticated AWS account:
IAM Permissions:
- iam:CreatePolicy
- iam:GetPolicy
- iam:ListPolicies
- iam:CreateUser
- iam:GetUser
- iam:AttachUserPolicy
- iam:CreateRole
- iam:GetRole
- iam:AttachRolePolicy
- iam:ListRoles
- iam:PassRole
Amazon ECS Permissions:
- ecs:CreateCluster
- ecs:DescribeClusters
Amazon EC2 Permissions:
- ec2:CreateVpc
- ec2:DescribeVpcs
- ec2:CreateInternetGateway
- ec2:AttachInternetGateway
- ec2:CreateRouteTable
- ec2:CreateRoute
- ec2:CreateSecurityGroup
- ec2:DescribeSubnets
- ec2:CreateSubnet
- ec2:DescribeAvailabilityZones
- ec2:AuthorizeSecurityGroupIngress
- ec2:AuthorizeSecurityGroupEgress
Amazon ECR Permissions:
- ecr:CreateRepository
- ecr:DescribeRepositories
- ecr:GetAuthorizationToken
If you want to use AWS managed policies, you can use the following:
- AmazonECS_FullAccess
- AmazonEC2FullAccess
- IAMFullAccess
- AmazonEC2ContainerRegistryFullAccess
The above policies give you all the permissions needed, but are more permissive than necessary.
Docker is also required to build and push images to your registry.
Install the Azure CLI and authenticate with your Azure account.
If you already have the Azure CLI installed, be sure to update to the latest version with az upgrade
.
You will also need the following roles in your Azure subscription:
- Contributor
- User Access Administrator
- Application Administrator
- Managed Identity Operator
- Azure Container Registry Contributor
Docker is also required to build and push images to your registry. You can install Docker here.
Install the gcloud CLI and authenticate with your GCP project.
If you already have the gcloud CLI installed, be sure to update to the latest version with gcloud components update
.
You will also need the following permissions in your GCP project:
- resourcemanager.projects.list
- serviceusage.services.enable
- iam.serviceAccounts.create
- iam.serviceAccountKeys.create
- resourcemanager.projects.setIamPolicy
- artifactregistry.repositories.create
Docker is also required to build and push images to your registry. You can install Docker here.
Install modal
by running:
Create a Modal API token by running:
See Run flows on Modal for more details.
Install coiled
by running:
Create a Coiled API token by running:
Connect Coiled to your cloud account by running
or by navigating to cloud.coiled.io/get-started
Automatically create a new push work pool and provision infrastructure
To create a new push work pool and configure the necessary infrastructure, run this command for your particular cloud provider:
The --provision-infra
flag automatically sets up your default AWS account to execute
flows with ECS tasks.
In your AWS account, this command creates a new IAM user, IAM policy, and
ECS cluster that uses AWS Fargate, VPC, and ECR repository (if they don’t already exist).
In your Prefect workspace, this command creates an
AWSCredentials
block for storing the generated credentials.
Here’s an abbreviated example output from running the command:
Default Docker build namespace
After infrastructure provisioning completes, you will be logged into your new ECR repository and the default Docker build namespace will be set to the URL of the registry.
While the default namespace is set, you do not need to provide the registry URL when building images as part of your deployment process.
To take advantage of this, you can write your deploy scripts like this:
This builds an image with the tag <ecr-registry-url>/my-image:latest
and push it to the registry.
Your image name needs to match the name of the repository created with your work pool. You can create new repositories in the ECR console.
The --provision-infra
flag automatically sets up your default AWS account to execute
flows with ECS tasks.
In your AWS account, this command creates a new IAM user, IAM policy, and
ECS cluster that uses AWS Fargate, VPC, and ECR repository (if they don’t already exist).
In your Prefect workspace, this command creates an
AWSCredentials
block for storing the generated credentials.
Here’s an abbreviated example output from running the command:
Default Docker build namespace
After infrastructure provisioning completes, you will be logged into your new ECR repository and the default Docker build namespace will be set to the URL of the registry.
While the default namespace is set, you do not need to provide the registry URL when building images as part of your deployment process.
To take advantage of this, you can write your deploy scripts like this:
This builds an image with the tag <ecr-registry-url>/my-image:latest
and push it to the registry.
Your image name needs to match the name of the repository created with your work pool. You can create new repositories in the ECR console.
The --provision-infra
flag automatically sets up your default Azure account to execute
flows through Azure Container Instances.
In your Azure account, this command creates a resource group, app registration, service account with necessary permission,
generates a secret for the app registration, and creates an Azure Container Registry, (if they don’t already exist).
In your Prefect workspace, this command creates an
AzureContainerInstanceCredentials
block
to store the client secret value from the generated secret.
Here’s an abbreviated example output from running the command:
Default Docker build namespace After infrastructure provisioning completes, you are logged into your new Azure Container Registry and the default Docker build namespace is set to the URL of the registry.
While the default namespace is set, any images you build without specifying a registry or username/organization are pushed to the registry.
To use this capability, write your deploy scripts like this:
This builds an image with the tag <acr-registry-url>/my-image:latest
and pushes it to the registry.
The --provision-infra
flag allows you to select a GCP project to use for your work pool and automatically
configures it to execute flows through Cloud Run.
In your GCP project, this command activates the Cloud Run API, creates a service account, and creates a key for the
service account, (if they don’t already exist).
In your Prefect workspace, this command creates a
GCPCredentials
block to store the service account key.
Here’s an abbreviated example output from running the command:
Default Docker build namespace
After infrastructure provisioning completes, you are logged into your new Artifact Registry repository and the default Docker build namespace is set to the URL of the repository.
While the default namespace is set, any images you build without specifying a registry or username/organization are pushed to the repository.
To use this capability, write your deploy scripts like this:
This builds an image with the tag <region>-docker.pkg.dev/<project>/<repository-name>/my-image:latest
and pushes it to the repository.
Using the --provision-infra
flag triggers the creation of a ModalCredentials
block in your Prefect Cloud workspace.
This block stores your Modal API token, which authenticates with Modal’s API. By default, the token for your
current Modal profile is used for the new ModalCredentials
block. If Prefect is unable to discover a Modal
API token for your current profile, you will be prompted to create a new one.
Using the --provision-infra
flag triggers the creation of a CoiledCredentials
block in your Prefect Cloud workspace.
This block stores your Coiled API token, which authenticates with Coiled’s API.
You’re ready to create and schedule deployments that use your new push work pool. Reminder that no worker is required to run flows with a push work pool.
Use existing resources with automatic infrastructure provisioning
If you already have the necessary infrastructure set up, Prefect detects that at work pool creation and the infrastructure provisioning for that resource is skipped.
For example, here’s how prefect work-pool create my-work-pool --provision-infra
looks when existing Azure resources are detected:
Provision infrastructure for an existing push work pool
If you already have a push work pool set up, but haven’t configured the necessary infrastructure, you can use the
provision-infra
sub-command to provision the infrastructure for that work pool.
For example, you can run the following command if you have a work pool named “my-work-pool”.
Prefect creates the necessary infrastructure for the my-work-pool
work pool and provides you with a summary
of the changes:
This command speeds up your infrastructure setup process.
As with the examples above, you need to have the related cloud CLI library installed and to be authenticated with your cloud provider.
Manual infrastructure provisioning
If you prefer to set up your infrastructure manually, exclude the --provision-infra
flag in the CLI command.
In the examples below, you’ll create a push work pool through the Prefect Cloud UI.
To push work to ECS, AWS credentials are required.
Create a user and attach the AmazonECS_FullAccess permissions.
From that user’s page, create credentials and store them somewhere safe for use in the next section.
To push work to ECS, AWS credentials are required.
Create a user and attach the AmazonECS_FullAccess permissions.
From that user’s page, create credentials and store them somewhere safe for use in the next section.
To push work to Azure, you need an Azure subscription, resource group, and tenant secret.
Create Subscription and Resource Group
- In the Azure portal, create a subscription.
- Create a resource group within your subscription.
Create App Registration
- In the Azure portal, create an app registration.
- In the app registration, create a client secret. Copy the value and store it somewhere safe.
Add App Registration to Resource Group
- Navigate to the resource group you created earlier.
- Choose the “Access control (IAM)” blade in the left-hand side menu. Click ”+ Add” button at the top, then “Add role assignment”.
- Go to the “Privileged administrator roles” tab, click on “Contributor”, then click “Next” at the bottom of the page.
- Click on ”+ Select members”. Type the name of the app registration (otherwise it may not autopopulate) and click to add it. Then hit “Select” and click “Next”. The default permissions associated with a role like “Contributor” may not be sufficient for all operations related to Azure Container Instances (ACI). The specific permissions required depend on the operations you need to perform (like creating, running, and deleting ACI container groups) and your organization’s security policies. In some cases, additional permissions or custom roles are necessary.
- Click “Review + assign” to finish.
You need a GCP service account and an API Key to push work to Cloud Run.
Create a service account by navigating to the service accounts page and clicking Create. Name and describe your service account, and click continue to configure permissions.
The service account must have two roles at a minimum: Cloud Run Developer and Service Account User.
Once you create the Service account, navigate to its Keys page to add an API key. Create a JSON type key, download it, and store it somewhere safe for use in the next section.
You need a Modal API token to push work to Modal.
Navigate to Settings in the Modal UI to create a Modal API token. In the API Tokens section of the Settings page, click New Token.
Copy the token ID and token secret and store them somewhere safe for use in the next section.
You need a Coiled API token to push work to Coiled.
In the Coiled UI click on your avatar and select Profile. Then press the Create API Token button and copy the result. You will need it in the next section.
Work pool configuration
The push work pool stores information about what type of infrastructure the flow will run on, what default values to provide to compute jobs, and other important execution environment parameters. Because the push work pool needs to integrate securely with your serverless infrastructure, you need to store your credentials in Prefect Cloud by making a block.
Create a Credentials block
Navigate to the blocks page, click create new block, and select AWS Credentials for the type.
For use in a push work pool, set the region, access key, and access key secret.
Provide any other optional information and create your block.
Navigate to the blocks page, click create new block, and select AWS Credentials for the type.
For use in a push work pool, set the region, access key, and access key secret.
Provide any other optional information and create your block.
Navigate to the blocks page and click the ”+” at the top to create a new block. Find the Azure Container Instance Credentials block and click “Add +”.
Locate the client ID and tenant ID on your app registration and use the client secret you saved earlier. Make sure the value of the secret, not the secret ID.
Provide any other optional information and click “Create”.
Navigate to the blocks page, click create new block, and select GCP Credentials for the type.
For use in a push work pool, this block must have the contents of the JSON key stored in the Service Account Info field, as such:
Provide any other optional information and create your block.
Navigate to the blocks page, click create new block, and select Modal Credentials for the type.
For use in a push work pool, this block must have the token ID and token secret stored in the Token ID and Token Secret fields, respectively.
Navigate to the blocks page, click create new block, and select Coiled Credentials for the type. Provide the API token from the previous section.
Create a push work pool
Now navigate to the work pools page. Click Create to configure your push work pool by selecting a push option in the infrastructure type step.
Each step has several optional fields that are detailed in the work pools documentation. Select the block you created under the AWS Credentials field. This allows Prefect Cloud to securely interact with your ECS cluster.
Each step has several optional fields that are detailed in the work pools documentation. Select the block you created under the AWS Credentials field. This allows Prefect Cloud to securely interact with your ECS cluster.
Fill in the subscription ID and resource group name from the resource group you created.
Add the Azure Container Instance Credentials block you created in the step above.
Each step has several optional fields that are detailed in Manage infrastructure with work pools. Select the block you created under the GCP Credentials field. This allows Prefect Cloud to securely interact with your GCP project.
Each step has several optional fields that are detailed in the work pools documentation. Select the block you created under the Modal Credentials field. This allows Prefect Cloud to securely interact with your Modal account.
Each step has several optional fields that are detailed in the work pools documentation. Select the block you created under the Coiled Credentials field. This allows Prefect Cloud to securely interact with your Coiled account.
Create your pool to be ready to deploy flows to your Push work pool.
Deployment
You need to configure your deployment to send flow runs to your push work pool.
For example, if you create a deployment through the interactive command line experience,
choose the work pool you just created. If you are deploying an existing prefect.yaml
file, the deployment would contain:
Deploying your flow to the my-push-pool
work pool ensures that runs that are ready for execution are
submitted immediately—without the need for a worker to poll for them.
Serverless infrastructure may require a certain image architecture
Serverless infrastructure may assume a certain Docker image architecture; for example,
Google Cloud Run will fail to run images built with linux/arm64
architecture. If using Prefect to build your image,
you can change the image architecture through the platform
keyword (for example, platform="linux/amd64"
).
Putting it all together
With your deployment created, navigate to its detail page and create a new flow run. You’ll see the flow start running without polling the work pool, because Prefect Cloud securely connected to your serverless infrastructure, created a job, ran the job, and reported on its execution.
Usage Limits
Push work pool usage is unlimited. However push work pools limit flow runs to 24 hours.
Next steps
Learn more about work pools and workers.
Learn about installing dependencies at runtime or baking them into your Docker image in the Deploy to Docker guide.
Was this page helpful?