How to scale self-hosted Prefect

Running multiple Prefect server instances enables high availability and distributes load across your infrastructure. This guide covers configuration and deployment patterns for scaling self-hosted Prefect.

​

Requirements

Multi-server deployments require:

• PostgreSQL database (SQLite does not support multi-server synchronization)
• Redis for event messaging
• Load balancer for API traffic distribution

​

Architecture

A scaled Prefect deployment typically includes:

• Multiple API server instances - Handle UI and API requests
• Background services - Runs the scheduler, automation triggers, and other loop services
• PostgreSQL database - Stores all persistent data and synchronizes state across servers
• Redis - Distributes events between services
• Load balancer - Routes traffic to healthy API instances (e.g. NGINX or Traefik)

​

Configuration

​

Database setup

Configure PostgreSQL as your database backend:

export PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://user:password@host:5432/prefect"

​

Redis setup

Configure Redis as your message broker:

export PREFECT_MESSAGING_BROKER="prefect_redis.messaging"
export PREFECT_MESSAGING_CACHE="prefect_redis.messaging"
export PREFECT_REDIS_MESSAGING_HOST="redis-host"
export PREFECT_REDIS_MESSAGING_PORT="6379"
export PREFECT_REDIS_MESSAGING_DB="0"

Each server instance automatically generates a unique consumer name to prevent message delivery conflicts.

​

Service separation

For optimal performance, run API servers and background services separately:

API servers (multiple instances):

prefect server start --host 0.0.0.0 --port 4200 --no-services

Background services:

prefect server services start

​

Database migrations

Disable automatic migrations in multi-server deployments:

export PREFECT_API_DATABASE_MIGRATE_ON_START="false"

Run migrations separately before deployment:

prefect server database upgrade -y

​

Load balancer configuration

Configure health checks for your load balancer:

• Health endpoint: /api/health
• Expected response: HTTP 200 with JSON {"status": "healthy"}
• Check interval: 5-10 seconds

Example NGINX configuration:

upstream prefect_api {
    least_conn;
    server prefect-api-1:4200 max_fails=3 fail_timeout=30s;
    server prefect-api-2:4200 max_fails=3 fail_timeout=30s;
    server prefect-api-3:4200 max_fails=3 fail_timeout=30s;
}

server {
    listen 4200;
    
    location /api/health {
        proxy_pass http://prefect_api;
        proxy_connect_timeout 1s;
        proxy_read_timeout 1s;
    }
    
    location / {
        proxy_pass http://prefect_api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

​

Reverse proxy configuration

When hosting Prefect behind a reverse proxy, ensure proper header forwarding:

server {
    listen 80;
    server_name prefect.example.com;
    
    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name prefect.example.com;

    ssl_certificate /path/to/ssl/certificate.pem;
    ssl_certificate_key /path/to/ssl/certificate_key.pem;

    location /api {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        
        # WebSocket support
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        # Authentication headers
        proxy_set_header Authorization $http_authorization;
        proxy_pass_header Authorization;
        
        proxy_pass http://prefect_api;
    }

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://prefect_api;
    }
}

​

UI proxy settings

When self-hosting the UI behind a proxy:

• PREFECT_UI_API_URL: Connection URL from UI to API
• PREFECT_UI_SERVE_BASE: Base URL path to serve the UI
• PREFECT_UI_URL: URL for clients to access the UI

​

SSL certificates

For self-signed certificates:

1. Add certificate to system bundle and set:

   Copy

   export SSL_CERT_FILE=/path/to/certificate.pem
   

2. Or disable verification (testing only):

   Copy

   export PREFECT_API_TLS_INSECURE_SKIP_VERIFY=True
   

​

Environment proxy settings

Prefect respects standard proxy environment variables:

export HTTPS_PROXY=http://proxy.example.com:8080
export HTTP_PROXY=http://proxy.example.com:8080
export NO_PROXY=localhost,127.0.0.1,.internal

​

Deployment examples

​

Docker Compose

​

Operations

​

Migration considerations

​

Handling large databases

When running migrations on large database instances (especially where tables like events, flow_runs, or task_runs can reach millions of rows), the default database timeout of 10 seconds may not be sufficient for creating indexes.

If you encounter a TimeoutError during migrations, increase the database timeout:

# Set timeout to 10 minutes (adjust based on your database size)
export PREFECT_API_DATABASE_TIMEOUT=600

# Then run the migration
prefect server database upgrade -y

For Docker deployments:

docker run -e PREFECT_API_DATABASE_TIMEOUT=600 prefecthq/prefect:latest prefect server database upgrade -y

​

Recovering from failed migrations

If a migration times out while creating indexes, you may need to manually complete it. For example, if migration 7a73514ca2d6 fails:

1. First, check which indexes were partially created:

   Copy

   SELECT indexname FROM pg_indexes WHERE tablename = 'events' AND indexname LIKE 'ix_events%';
   

2. Manually create the missing indexes using CONCURRENTLY to avoid blocking:

   Copy

   -- Drop any partial indexes from the failed migration
   DROP INDEX IF EXISTS ix_events__event_related_occurred;
   DROP INDEX IF EXISTS ix_events__related_resource_ids;
   
   -- Create the new indexes
   CREATE INDEX CONCURRENTLY ix_events__related_gin ON events USING gin(related);
   CREATE INDEX CONCURRENTLY ix_events__event_occurred ON events (event, occurred);
   CREATE INDEX CONCURRENTLY ix_events__related_resource_ids_gin ON events USING gin(related_resource_ids);
   

3. Mark the migration as complete:

   Copy

   UPDATE alembic_version SET version_num = '7a73514ca2d6';
   

​

Monitoring

Monitor your multi-server deployment:

• Database connections: Watch for connection pool exhaustion
• Redis memory: Ensure adequate memory for message queues
• API response times: Track latency across different endpoints
• Background service lag: Monitor time between event creation and processing

​

Best practices

1. Start with 2-3 API instances and scale based on load
2. Use connection pooling to manage database connections efficiently
3. Monitor extensively before scaling further (e.g. Prometheus + Grafana or Logfire)
4. Test failover scenarios regularly

​

Further reading

• Server concepts
• Deploy Helm charts for Kubernetes