prefect.server.orchestration.core_policy
Orchestration logic that fires on state transitions.
CoreFlowPolicy and CoreTaskPolicy contain all default orchestration rules that
Prefect enforces on a state transition.
Classes
CoreFlowPolicy
Orchestration rules that run against flow-run-state transitions in priority order.
Methods:
priority
CoreTaskPolicy
Orchestration rules that run against task-run-state transitions in priority order.
Methods:
priority
ClientSideTaskOrchestrationPolicy
Orchestration rules that run against task-run-state transitions in priority order,
specifically for clients doing client-side orchestration.
Methods:
priority
BackgroundTaskPolicy
Orchestration rules that run against task-run-state transitions in priority order.
Methods:
priority
MinimalFlowPolicy
Methods:
priority
MarkLateRunsPolicy
Methods:
priority
MinimalTaskPolicy
Methods:
priority
SecureTaskConcurrencySlots
Checks relevant concurrency slots are available before entering a Running state.
This rule checks if concurrency limits have been set on the tags associated with a
TaskRun. If so, a concurrency slot will be secured against each concurrency limit
before being allowed to transition into a running state. If a concurrency limit has
been reached, the client will be instructed to delay the transition for the duration
specified by the “PREFECT_TASK_RUN_TAG_CONCURRENCY_SLOT_WAIT_SECONDS” setting
before trying again. If the concurrency limit set on a tag is 0, the transition will
be aborted to prevent deadlocks.
Methods:
before_transition
cleanup
ReleaseTaskConcurrencySlots
Releases any concurrency slots held by a run upon exiting a Running or
Cancelling state.
Methods:
after_transition
SecureFlowConcurrencySlots
Enforce deployment concurrency limits.
This rule enforces concurrency limits on deployments. If a deployment has a concurrency limit,
this rule will prevent more than that number of flow runs from being submitted concurrently
based on the concurrency limit behavior configured for the deployment.
We use the PENDING state as the target transition because this allows workers to secure a slot
before provisioning dynamic infrastructure to run a flow. If a slot isn’t available, the worker
won’t provision infrastructure.
A lease is created for the concurrency limit. The client will be responsible for maintaining the lease.
Methods:
before_transition
cleanup
CopyDeploymentConcurrencyLeaseID
Copies the deployment concurrency lease ID to the proposed state.
Methods:
before_transition
RemoveDeploymentConcurrencyLeaseForOldClientVersions
Removes a deployment concurrency lease if the client version is less than the minimum version for leasing.
Methods:
after_transition
ReleaseFlowConcurrencySlots
Releases deployment concurrency slots held by a flow run.
This rule releases a concurrency slot for a deployment when a flow run
transitions out of the Running or Cancelling state.
Methods:
after_transition
CacheInsertion
Caches completed states with cache keys after they are validated.
Methods:
after_transition
before_transition
CacheRetrieval
Rejects running states if a completed state has been cached.
This rule rejects transitions into a running state with a cache key if the key
has already been associated with a completed state in the cache table. The client
will be instructed to transition into the cached completed state instead.
Methods:
before_transition
RetryFailedFlows
Rejects failed states and schedules a retry if the retry limit has not been reached.
This rule rejects transitions into a failed state if retries has been
set and the run count has not reached the specified limit. The client will be
instructed to transition into a scheduled state to retry flow execution.
Methods:
before_transition
RetryFailedTasks
Rejects failed states and schedules a retry if the retry limit has not been reached.
This rule rejects transitions into a failed state if retries has been
set, the run count has not reached the specified limit, and the client
asserts it is a retriable task run. The client will be instructed to
transition into a scheduled state to retry task execution.
Methods:
before_transition
EnqueueScheduledTasks
Enqueues background task runs when they are scheduled
Methods:
after_transition
RenameReruns
Name the states if they have run more than once.
In the special case where the initial state is an “AwaitingRetry” scheduled state,
the proposed state will be renamed to “Retrying” instead.
Methods:
before_transition
CopyScheduledTime
Ensures scheduled time is copied from scheduled states to pending states.
If a new scheduled time has been proposed on the pending state, the scheduled time
on the scheduled state will be ignored.
Methods:
before_transition
WaitForScheduledTime
Prevents transitions to running states from happening too early.
This rule enforces that all scheduled states will only start with the machine clock
used by the Prefect REST API instance. This rule will identify transitions from scheduled
states that are too early and nullify them. Instead, no state will be written to the
database and the client will be sent an instruction to wait for delay_seconds
before attempting the transition again.
Methods:
before_transition
CopyTaskParametersID
Ensures a task’s parameters ID is copied from Scheduled to Pending and from
Pending to Running states.
If a parameters ID has been included on the proposed state, the parameters ID
on the initial state will be ignored.
Methods:
before_transition
HandlePausingFlows
Governs runs attempting to enter a Paused/Suspended state
Methods:
after_transition
before_transition
HandleResumingPausedFlows
Governs runs attempting to leave a Paused state
Methods:
after_transition
before_transition
UpdateFlowRunTrackerOnTasks
Tracks the flow run attempt a task run state is associated with.
Methods:
after_transition
HandleTaskTerminalStateTransitions
We do not allow tasks to leave terminal states if:
- The task is completed and has a persisted result
- The task is going to CANCELLING / PAUSED / CRASHED
before_transition
cleanup
HandleFlowTerminalStateTransitions
We do not allow flows to leave terminal states if:
- The flow is completed and has a persisted result
- The flow is going to CANCELLING / PAUSED / CRASHED
- The flow is going to scheduled and has no deployment
before_transition
cleanup
PreventPendingTransitions
Prevents transitions to PENDING.
This rule is only used for flow runs.
This is intended to prevent race conditions during duplicate submissions of runs.
Before a run is submitted to its execution environment, it should be placed in a
PENDING state. If two workers attempt to submit the same run, one of them should
encounter a PENDING -> PENDING transition and abort orchestration of the run.
Similarly, if the execution environment starts quickly the run may be in a RUNNING
state when the second worker attempts the PENDING transition. We deny these state
changes as well to prevent duplicate submission. If a run has transitioned to a
RUNNING state a worker should not attempt to submit it again unless it has moved
into a terminal state.
CANCELLING and CANCELLED runs should not be allowed to transition to PENDING.
For re-runs of deployed runs, they should transition to SCHEDULED first.
For re-runs of ad-hoc runs, they should transition directly to RUNNING.
Methods:
before_transition
EnsureOnlyScheduledFlowsMarkedLate
Methods:
before_transition
PreventRunningTasksFromStoppedFlows
Prevents running tasks from stopped flows.
A running state implies execution, but also the converse. This rule ensures that a
flow’s tasks cannot be run unless the flow is also running.
Methods:
before_transition
EnforceCancellingToCancelledTransition
Rejects transitions from Cancelling to any terminal state except for Cancelled.
Methods:
before_transition
BypassCancellingFlowRunsWithNoInfra
Rejects transitions from Scheduled to Cancelling, and instead sets the state to Cancelled,
if the flow run has no associated infrastructure process ID. Also Rejects transitions from
Paused to Cancelling if the Paused state’s details indicates the flow run has been suspended,
exiting the flow and tearing down infra.
The Cancelling state is used to clean up infrastructure. If there is not infrastructure
to clean up, we can transition directly to Cancelled. Runs that are Resuming are in a
Scheduled state that were previously Suspended and do not yet have infrastructure.
Runs that are AwaitingRetry are a Scheduled state that may have associated infrastructure.
Methods:
before_transition
PreventDuplicateTransitions
Prevent duplicate transitions from being made right after one another.
This rule allows for clients to set an optional transition_id on a state. If the
run’s next transition has the same transition_id, the transition will be
rejected and the existing state will be returned.
This allows for clients to make state transition requests without worrying about
the following case:
- A client making a state transition request
- The server accepts transition and commits the transition
- The client is unable to receive the response and retries the request