rules
prefect.server.orchestration.rules
Prefect’s flow and task-run orchestration machinery.
This module contains all the core concepts necessary to implement Prefect’s state orchestration engine. These states correspond to intuitive descriptions of all the points that a Prefect flow or task can observe executing user code and intervene, if necessary. A detailed description of states can be found in our concept documentation.
Prefect’s orchestration engine operates under the assumption that no governed user code will execute without first requesting Prefect REST API validate a change in state and record metadata about the run. With all attempts to run user code being checked against a Prefect instance, the Prefect REST API database becomes the unambiguous source of truth for managing the execution of complex interacting workflows. Orchestration rules can be implemented as discrete units of logic that operate against each state transition and can be fully observable, extensible, and customizable — all without needing to store or parse a single line of user code.
Classes
OrchestrationContext
A container for a state transition, governed by orchestration rules.
When a flow- or task- run attempts to change state, Prefect REST API has an opportunity
to decide whether this transition can proceed. All the relevant information
associated with the state transition is stored in an OrchestrationContext,
which is subsequently governed by nested orchestration rules implemented using
the BaseOrchestrationRule ABC.
OrchestrationContext introduces the concept of a state being None in the
context of an intended state transition. An initial state can be None if a run
is is attempting to set a state for the first time. The proposed state might be
None if a rule governing the transition determines that no state change
should occur at all and nothing is written to the database.
Args:
session: a SQLAlchemy database sessioninitial_state: the initial state of a runproposed_state: the proposed state a run is transitioning into
Methods:
initial_state_type
The state type of self.initial_state if it exists.
proposed_state_type
The state type of self.proposed_state if it exists.
validated_state_type
The state type of self.validated_state if it exists.
run_settings
Run-level settings used to orchestrate the state transition.
safe_copy
Creates a mostly-mutation-safe copy for use in orchestration rules.
Orchestration rules govern state transitions using information stored in
an OrchestrationContext. However, mutating objects stored on the context
directly can have unintended side-effects. To guard against this,
self.safe_copy can be used to pass information to orchestration rules
without risking mutation.
Returns:
- A mutation-safe copy of the
OrchestrationContext
entry_context
A convenience method that generates input parameters for orchestration rules.
An OrchestrationContext defines a state transition that is managed by
orchestration rules which can fire hooks before a transition has been committed
to the database. These hooks have a consistent interface which can be generated
with this method.
exit_context
A convenience method that generates input parameters for orchestration rules.
An OrchestrationContext defines a state transition that is managed by
orchestration rules which can fire hooks after a transition has been committed
to the database. These hooks have a consistent interface which can be generated
with this method.
FlowOrchestrationContext
A container for a flow run state transition, governed by orchestration rules.
When a flow- run attempts to change state, Prefect REST API has an opportunity
to decide whether this transition can proceed. All the relevant information
associated with the state transition is stored in an OrchestrationContext,
which is subsequently governed by nested orchestration rules implemented using
the BaseOrchestrationRule ABC.
FlowOrchestrationContext introduces the concept of a state being None in the
context of an intended state transition. An initial state can be None if a run
is is attempting to set a state for the first time. The proposed state might be
None if a rule governing the transition determines that no state change
should occur at all and nothing is written to the database.
Args:
session: a SQLAlchemy database sessionrun: the flow run attempting to change stateinitial_state: the initial state of a runproposed_state: the proposed state a run is transitioning into
Methods:
safe_copy
Creates a mostly-mutation-safe copy for use in orchestration rules.
Orchestration rules govern state transitions using information stored in
an OrchestrationContext. However, mutating objects stored on the context
directly can have unintended side-effects. To guard against this,
self.safe_copy can be used to pass information to orchestration rules
without risking mutation.
Returns:
- A mutation-safe copy of
FlowOrchestrationContext
run_settings
Run-level settings used to orchestrate the state transition.
TaskOrchestrationContext
A container for a task run state transition, governed by orchestration rules.
When a task- run attempts to change state, Prefect REST API has an opportunity
to decide whether this transition can proceed. All the relevant information
associated with the state transition is stored in an OrchestrationContext,
which is subsequently governed by nested orchestration rules implemented using
the BaseOrchestrationRule ABC.
TaskOrchestrationContext introduces the concept of a state being None in the
context of an intended state transition. An initial state can be None if a run
is is attempting to set a state for the first time. The proposed state might be
None if a rule governing the transition determines that no state change
should occur at all and nothing is written to the database.
Args:
session: a SQLAlchemy database sessionrun: the task run attempting to change stateinitial_state: the initial state of a runproposed_state: the proposed state a run is transitioning into
Methods:
safe_copy
Creates a mostly-mutation-safe copy for use in orchestration rules.
Orchestration rules govern state transitions using information stored in
an OrchestrationContext. However, mutating objects stored on the context
directly can have unintended side-effects. To guard against this,
self.safe_copy can be used to pass information to orchestration rules
without risking mutation.
Returns:
- A mutation-safe copy of
TaskOrchestrationContext
run_settings
Run-level settings used to orchestrate the state transition.
BaseOrchestrationRule
An abstract base class used to implement a discrete piece of orchestration logic.
An OrchestrationRule is a stateful context manager that directly governs a state
transition. Complex orchestration is achieved by nesting multiple rules.
Each rule runs against an OrchestrationContext that contains the transition
details; this context is then passed to subsequent rules. The context can be
modified by hooks that fire before and after a new state is validated and committed
to the database. These hooks will fire as long as the state transition is
considered “valid” and govern a transition by either modifying the proposed state
before it is validated or by producing a side-effect.
A state transition occurs whenever a flow- or task- run changes state, prompting
Prefect REST API to decide whether or not this transition can proceed. The current state of
the run is referred to as the “initial state”, and the state a run is
attempting to transition into is the “proposed state”. Together, the initial state
transitioning into the proposed state is the intended transition that is governed
by these orchestration rules. After using rules to enter a runtime context, the
OrchestrationContext will contain a proposed state that has been governed by
each rule, and at that point can validate the proposed state and commit it to
the database. The validated state will be set on the context as
context.validated_state, and rules will call the self.after_transition hook
upon exiting the managed context.
Examples:
Create a rule:
class BasicRule(BaseOrchestrationRule):
allowed initial state types
FROM_STATES = [StateType.RUNNING]
allowed proposed state types
TO_STATES = [StateType.COMPLETED, StateType.FAILED]
async def before_transition(initial_state, proposed_state, ctx):
side effects and proposed state mutation can happen here
…
async def after_transition(initial_state, validated_state, ctx):
operations on states that have been validated can happen here
…
async def cleanup(intitial_state, validated_state, ctx):
reverts side effects generated by
before_transitionif necessary…
Use a rule:
intended_transition = (StateType.RUNNING, StateType.COMPLETED) async with BasicRule(context, *intended_transition):
context.proposed_state has been governed by BasicRule
…
Use multiple rules:
rules = [BasicRule, BasicRule] intended_transition = (StateType.RUNNING, StateType.COMPLETED) async with contextlib.AsyncExitStack() as stack: for rule in rules: stack.enter_async_context(rule(context, *intended_transition))
context.proposed_state has been governed by all rules
…
Args:
context: AFlowOrchestrationContextorTaskOrchestrationContextthat is passed between rulesfrom_state_type: The state type of the initial state of a run, if this state type is not contained inFROM_STATES, no hooks will fireto_state_type: The state type of the proposed state before orchestration, if this state type is not contained inTO_STATES, no hooks will fire
FlowRunOrchestrationRule
TaskRunOrchestrationRule
GenericOrchestrationRule
BaseUniversalTransform
An abstract base class used to implement privileged bookkeeping logic.
Beyond the orchestration rules implemented with the BaseOrchestrationRule ABC,
Universal transforms are not stateful, and fire their before- and after- transition
hooks on every state transition unless the proposed state is None, indicating that
no state should be written to the database. Because there are no guardrails in place
to prevent directly mutating state or other parts of the orchestration context,
universal transforms should only be used with care.
Args:
context: AFlowOrchestrationContextorTaskOrchestrationContextthat is passed between transforms
Methods:
nullified_transition
Determines if the transition has been nullified.
Transitions are nullified if the proposed state is None, indicating that
nothing should be written to the database.
Returns:
- True if the transition is nullified, False otherwise.
exception_in_transition
Determines if the transition has encountered an exception.
Returns:
- True if the transition is encountered an exception, False otherwise.