This guide is valid for users of self-hosted Prefect server or Prefect Cloud users with a tier that allows hybrid work pools.

​
Why use ECS for flow run execution?

ECS (Elastic Container Service) tasks are a good option for executing Prefect flow runs for several reasons:
  1. Scalability: ECS scales your infrastructure in response to demand, effectively managing Prefect flow runs. ECS automatically administers container distribution across multiple instances based on demand.
  2. Flexibility: ECS lets you choose between AWS Fargate and Amazon EC2 for container operation. Fargate abstracts the underlying infrastructure, while EC2 has faster job start times and offers additional control over instance management and configuration. Fargate also unlocks financial flexibility by allowing you to pay only for the resources you use, while EC2 allows you to reserve instances for cost savings with predictable workloads.
  3. AWS Integration: Easily connect with other AWS services, such as AWS IAM and CloudWatch.
  4. Containerization: ECS supports Docker containers and offers managed execution. Containerization encourages reproducible deployments.

​
ECS flow run execution

Prefect enables remote flow execution via workers and work pools. To learn more about these concepts please see our deployment docs. For details on how workers and work pools are implemented for ECS, see the diagram below.

​
ECS and Prefect

ECS tasks != Prefect tasksAn ECS task is not the same thing as a Prefect task.ECS tasks are groupings of containers that run within an ECS Cluster. An ECS task’s behavior is determined by its task definition.
An ECS task definition is the blueprint for the ECS task. It describes which Docker containers to run and what you want to have happen inside these containers. ECS tasks are instances of a task definition. A Task Execution launches container(s) as defined in the task definition until they are stopped or exit on their own. This setup is ideal for ephemeral processes such as a Prefect flow run. The ECS task running the Prefect worker should be an ECS Service, given its long-running nature and need for auto-recovery in case of failure. An ECS service automatically replaces any task that fails, which is ideal for managing a long-running process such as a Prefect worker. When a Prefect flow is scheduled to run it is assigned to the work pool specified in the flow’s deployment. Work pools are typed according to the infrastructure the flow will run on. Flow runs scheduled in an ecs typed work pool are executed as ECS tasks. Only Prefect ECS workers can poll an ecs typed work pool. When the ECS worker receives a scheduled flow run from the ECS work pool it is polling, it spins up the specified infrastructure on AWS ECS. The worker knows to build an ECS task definition for each flow run based on the configuration specified in the work pool. Once the flow run completes, the ECS containers of the cluster are spun down to a single container that continues to run the Prefect worker. This worker continues polling for work from the Prefect work pool. If you specify a task definition ARN (Amazon Resource Name) in the work pool, the worker will use that ARN when spinning up the ECS Task, rather than creating a task definition from the fields supplied in the work pool configuration. You can use either EC2 or Fargate as the capacity provider. Fargate simplifies initiation, but lengthens infrastructure setup time for each flow run. Using EC2 for the ECS cluster can reduce setup time. In this example, we will show how to use Fargate.
If you prefer infrastructure as code check out the Prefect ECS worker Terraform module to provision an ECS cluster with a worker.

​
Prerequisites

You will need the following to successfully complete this guide:
For production deployments, it is recommended that you create your own VPC with appropriate security policies based on your organization’s recommendations.If you want to create a new VPC for this guide, follow the VPC creation guide.

​
Create the Prefect ECS work pool

First, create an ECS work pool for your deployments to use. You can do this either from the CLI or the Prefect UI. If doing so from the CLI, be sure to authenticate with Prefect Cloud or run a local Prefect server instance.
Run the following command to create a new ECS work pool named my-ecs-pool:
prefect work-pool create --type ecs my-ecs-pool
Because this guide uses Fargate as the capacity provider, this step requires no further action.

​
Create a Secret for the Prefect API key

If you are using a Prefect self-hosted server and have authentication disabled, you can skip this step.
The Prefect worker needs to authenticate with your Prefect server to poll the work pool for flow runs. For authentication, you must provide a Bearer token (PREFECT_API_KEY) or Basic Auth string (PREFECT_API_AUTH_STRING) to the Prefect API. As a security best practice, we recommend you store your Prefect API key in AWS Secrets Manager or Systems Manager Parameter Store.
1

Find your secret

You can find your Prefect API key several ways:
2

Create a secret

Choose between AWS Secrets Manager or Systems Manager Parameter Store to store your Prefect API key. Both services allow you to securely store and manage sensitive information such as API keys, passwords, and other secrets.
To create a Secret in AWS Secrets Manager, use the aws secretsmanager create-secret command:
aws secretsmanager create-secret --name PrefectECSWorkerAPIKey --secret-string '<your-prefect-api-key>'
Make a note of the Amazon Resource Name (ARN) of the secret that is returned in the command output. You will need it later when configuring the ECS worker task definition.

​
Create the AWS IAM resources

We will create two IAM roles:
  1. ecsTaskExecutionRole: This role will be used by ECS to start ECS tasks.
  2. ecsTaskRole: This role will contain the permissions required by Prefect ECS worker in order to run your flows as ECS tasks.
The role permissions are based on the principle of least-privilege, meaning that each role will only have the permissions it needs to perform its job.

​
Create a trust policy

The trust policy will allow the ECS service containing the Prefect worker to assume the role required for calling other AWS services. This is called a service-linked role. The trust policy is a JSON document that specifies which AWS service can assume the role. Save this policy to a file, such as trust-policy.json: 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
Alternately, you can download this file using the following command: 
curl -O https://raw.githubusercontent.com/PrefectHQ/prefect/refs/heads/main/docs/integrations/prefect-aws/ecs/iam/trust-policy.json

​
Create the IAM roles

Now, we will create the IAM roles that will be used by the ECS worker.

​
Create the ECS task execution role

The ECS task execution role will be used to start the ECS worker task. We will assign it a minimal set of permissions to allow the worker to pull images from ECR and publish logs to CloudWatch.
1

Create the role

Create the role using the aws iam create-role command:
aws iam create-role --role-name ecsTaskExecutionRole --assume-role-policy-document file://trust-policy.json
Make a note of the ARN (Amazon Resource Name) of the role that is returned in the command output. You will need it later when creating the ECS task definition.
2

Create the Secret Policy

The following is a minimal policy that grants the necessary permissions for ECS to obtain the current value of the secret and inject it into the ECS task. Save this policy to a file, such as secret-policy.json:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "secretsmanager:GetSecretValue",
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:secretsmanager:<region>:<account-id>:secret:PrefectECSWorkerAPIKey"
        }
    ]
}
Alternately, you can download this file using the following command:
curl -O https://raw.githubusercontent.com/PrefectHQ/prefect/refs/heads/main/docs/integrations/prefect-aws/ecs/iam/secrets/secrets-manager/secret-policy.json
3

Register the policy

Create a new IAM policy named ecsTaskExecutionPolicy using the policy document you just created.
aws iam create-policy --policy-name ecsTaskExecutionPolicy --policy-document file://secret-policy.json
4

Attach the Policies

The AmazonECSTaskExecutionRolePolicy managed policy grants the minimum permissions necessary for starting ECS tasks. See here for other common execution role permissions.Attach this policy to your task execution role using the aws iam attach-role-policy:
aws iam attach-role-policy --role-name ecsTaskExecutionRole --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
Attach the custom policy you created in the previous step so that the ECS task can access the Prefect API key stored in AWS Secrets Manager or Systems Manager Parameter Store:
aws iam put-role-policy --role-name ecsTaskExecutionRole --policy-name PrefectECSWorkerSecretPolicy --policy-document file://secret-policy.json

​
Create the worker ECS task role

The worker ECS task role will be used by the Prefect worker to interact with the AWS API to run flows as ECS containers. This role will require the ability to describe, register, and deregister ECS task definitions, as well as the ability to start and stop ECS tasks.
1

Create the role

Use the following command to create the role. The same trust policy is also used for this role.
aws iam create-role --role-name ecsTaskRole --assume-role-policy-document file://trust-policy.json
2

Create the task policy

The following is a minimal policy that grants the necessary permissions for the Prefect ECS worker to run your flows as ECS tasks. Save this policy to a file, such as worker-policy.json:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs",
        "ecs:DeregisterTaskDefinition",
        "ecs:DescribeTaskDefinition",
        "ecs:DescribeTasks",
        "ecs:RegisterTaskDefinition",
        "ecs:RunTask",
        "ecs:StopTask",
        "ecs:TagResource",
        "iam:PassRole",
        "logs:GetLogEvents",
        "logs:PutLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
Alternately, you can download this file using the following command:
curl -O https://raw.githubusercontent.com/PrefectHQ/prefect/refs/heads/main/docs/integrations/prefect-aws/ecs/iam/worker-policy.json
3

Register the policy

Create a new IAM policy named ecsTaskPolicy using the policy document you just created.
aws iam create-policy --policy-name ecsTaskPolicy --policy-document file://worker-policy.json
4

Attach policy to the role

Attach the custom ecsTaskPolicy to the ecsTaskRole so that the Prefect worker can dispatch flows to ECS:
aws iam attach-role-policy --role-name ecsTaskRole --policy-arn arn:aws:iam::<your-account-id>:policy/ecsTaskPolicy
Replace <your-account-id> with your AWS account ID.

​
Create an ECS task role for Prefect flows

This step is optional, but recommended if your flows require access to other AWS services.
Depending on the requirements of your flows, it is advised to create a separate role for your ECS tasks. This role will contain the permissions required by the ECS tasks in which your flows will run. For example, if your workflow loads data into an S3 bucket, you would need a role with additional permissions to access S3.

​
Creating the ECS worker service

Now that all the AWS IAM roles have been created, we can deploy the Prefect worker to the ECS cluster.
1

Create the task definition

This task definition will be used to run the Prefect worker in an ECS task.Ensure you replace the placeholders for:
  • <ecs-task-execution-role-arn> with the ARN of the ecsTaskExecutionRole you created in Step 2.
  • <ecs-task-role-arn> with the ARN of the ecsTaskRole you created in Step 2.
  • <prefect-api-url> with the URL of your Prefect Server.
  • <aws-arn-of-secret> with the ARN of the resource from Secrets Manager or Systems Manager Parameter Store.
Save the following JSON to a file named task-definition.json:
{
    "family": "prefect-worker-task",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "512",
    "memory": "1024",
    "executionRoleArn": "<ecs-task-execution-role-arn>",
    "taskRoleArn": "<ecs-task-role-arn>",
    "containerDefinitions": [
        {
            "name": "prefect-worker",
            "image": "prefecthq/prefect:3-latest",
            "cpu": 512,
            "memory": 1024,
            "essential": true,
            "command": [
                "/bin/sh",
                "-c",
                "pip install prefect-aws && prefect worker start --pool my-ecs-pool --type ecs"
            ],
            "environment": [
                {
                    "name": "PREFECT_API_URL",
                    "value": "<prefect-api-url>"
                }
            ],
            "secrets": [
                {
                    "name": "PREFECT_API_KEY",
                    "valueFrom": "<aws-arn-of-secret>"
                }
            ]
        }
    ]
}
Alternately, you can download this file using the following command:
curl -O https://raw.githubusercontent.com/PrefectHQ/prefect/refs/heads/main/docs/integrations/prefect-aws/ecs/iam/task-definition.json
Notice that the CPU and Memory allocations are relatively small. The worker’s main responsibility is to submit work through API calls to AWS, not to execute your Prefect flow code.
To avoid hardcoding your API key into the task definition JSON see how to add sensitive data using AWS secrets manager to the container definition.
2

Register task definition

Before creating a service, you first need to register a task definition. You can do that using the register-task-definition command:
aws ecs register-task-definition --cli-input-json file://task-definition.json
Replace task-definition.json with the name of your task definition file.
3

Create the ECS service

Finally, create a service that will manage your Prefect worker.Ensure you replace the placeholders for:
  • <ecs-cluster> with the name of your ECS cluster.
  • <task-definition-arn> with the ARN of the task definition you just registered.
  • <subnet-ids> with a comma-separated list of your VPC subnet IDs.
  • Replace <security-group-ids> with a comma-separated list of your VPC security group IDs.
Use the aws ecs create-service command to create an ECS service running on Fargate for the Prefect worker:
aws ecs create-service --service-name prefect-worker-service --cluster <ecs-cluster> --task-definition <task-definition-arn> --launch-type FARGATE --desired-count 1 --network-configuration "awsvpcConfiguration={subnets=[<subnet-ids>],securityGroups=[<security-group-ids>],assignPublicIp='ENABLED'}"
4

Verify the Prefect worker is running

The work pool page in the Prefect UI allows you to check the health of your workers - make sure your new worker is live!
It may take a few minutes for the worker to come online after creating the service.
Refer to the troubleshooting section for further assistance if the worker isn’t online.

​
Deploy a flow run to your ECS work pool

This guide uses the AWS Elastic Container Registry (ECR) to store a Docker image containing your flow code. To do this, we will write a flow, then deploy it using build and push steps that copy flow code into a Docker image and push that image to an ECR repository.
1

Write a simple test flow

my_flow.py
from prefect import flow
from prefect.logging import get_run_logger

@flow
def my_flow():
    logger = get_run_logger()
    logger.info("Hello from ECS!!")

if __name__ == "__main__":
    my_flow()
2

Create an ECR repository

Use the aws ecr create-repository command to create an ECR repository. The name you choose for your repository will be reused in the next step when defining your Prefect deployment.
aws ecr create-repository --repository-name <my-ecr-repo>
3

Create a `prefect.yaml` file

To have Prefect build your image when deploying your flow create a prefect.yaml file with the following specification:
prefect.yaml
name: ecs-worker-guide

pull:
- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect/ecs-worker-guide

# build section allows you to manage and build docker images
build:
- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
    requires: prefect-docker>=0.3.1
    image_name: <my-ecr-repo>
    tag: latest
    dockerfile: auto

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_docker.deployments.steps.push_docker_image:
    requires: prefect-docker>=0.3.1
    image_name: '{{ build_image.image_name }}'
    tag: '{{ build_image.tag }}'

 # the deployments section allows you to provide configuration for deploying flows
deployments:
- name: my_ecs_deployment
    version:
    tags: []
    description:
    entrypoint: flow.py:my_flow
    parameters: {}
    work_pool:
      name: my-ecs-pool
      work_queue_name:
      job_variables:
        image: '{{ build_image.image }}'
    schedules: []
4

Deploy the flow

Deploy the flow to the Prefect Cloud or your self-managed server instance.
prefect deploy my_flow.py:my_ecs_deployment
5

Run!

Find the deployment in the UI and click the Quick Run button!

​
Troubleshooting

If your worker does not appear in the Prefect UI, check the following:
  • Ensure that the ECS service is running and that the task definition is registered correctly.
  • Check the ECS service logs in CloudWatch to see if there are any errors.
  • Verify that the IAM roles have the correct permissions.
  • Ensure that the PREFECT_API_URL and PREFECT_API_KEY environment variables are set correctly in the task definition.
  • For self-hosted Prefect servers, ensure that you replaced PREFECT_API_KEY from the example with PREFECT_API_AUTH_STRING in the task definition.
  • Ensure your Prefect ECS worker has network connectivity to the Prefect API. If you are using a private VPC, ensure that there is a NAT gateway or internet gateway configured to allow outbound traffic to the Prefect API.

​
Next steps

Now that you are confident your ECS worker is healthy, you can experiment with different work pool configurations.
  • Do your flow runs require higher CPU?
  • Would an EC2 Launch Type speed up your flow run execution?
These infrastructure configuration values can be set on your ECS work pool or they can be overridden on the deployment level through job_variables if desired.