prefect.task_runners
Classes
TaskRunner
Abstract base class for task runners.
A task runner is responsible for submitting tasks to the task run engine running
in an execution environment. Submitted tasks are non-blocking and return a future
object that can be used to wait for the task to complete and retrieve the result.
Task runners are context managers and should be used in a with
block to ensure
proper cleanup of resources.
Methods:
name
The name of this task runner
duplicate
Return a new instance of this task runner with the same configuration.
submit
submit(self, task: 'Task[P, Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> F
submit
submit(self, task: 'Task[Any, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> F
submit
submit(self, task: 'Task[P, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> F
map
map(self, task: 'Task[P, R]', parameters: dict[str, Any | unmapped[Any] | allow_failure[Any]], wait_for: Iterable[PrefectFuture[R]] | None = None) -> PrefectFutureList[F]
Submit multiple tasks to the task run engine.
Args:
task
: The task to submit.
parameters
: The parameters to use when running the task.
wait_for
: A list of futures that the task depends on.
Returns:
- An iterable of future objects that can be used to wait for the tasks to
- complete and retrieve the results.
ThreadPoolTaskRunner
A task runner that executes tasks in a separate thread pool.
Methods:
duplicate
duplicate(self) -> 'ThreadPoolTaskRunner[R]'
submit
submit(self, task: 'Task[P, Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectConcurrentFuture[R]
submit
submit(self, task: 'Task[Any, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectConcurrentFuture[R]
submit
submit(self, task: 'Task[P, R | Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectConcurrentFuture[R]
Submit a task to the task run engine running in a separate thread.
Args:
task
: The task to submit.
parameters
: The parameters to use when running the task.
wait_for
: A list of futures that the task depends on.
Returns:
- A future object that can be used to wait for the task to complete and
- retrieve the result.
map
map(self, task: 'Task[P, Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectConcurrentFuture[R]]
map
map(self, task: 'Task[Any, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectConcurrentFuture[R]]
map
map(self, task: 'Task[P, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectConcurrentFuture[R]]
cancel_all
PrefectTaskRunner
Methods:
duplicate
duplicate(self) -> 'PrefectTaskRunner[R]'
submit
submit(self, task: 'Task[P, Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectDistributedFuture[R]
submit
submit(self, task: 'Task[Any, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectDistributedFuture[R]
submit
submit(self, task: 'Task[P, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectDistributedFuture[R]
Submit a task to the task run engine running in a separate thread.
Args:
task
: The task to submit.
parameters
: The parameters to use when running the task.
wait_for
: A list of futures that the task depends on.
Returns:
- A future object that can be used to wait for the task to complete and
- retrieve the result.
map
map(self, task: 'Task[P, Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectDistributedFuture[R]]
map
map(self, task: 'Task[Any, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectDistributedFuture[R]]
map
map(self, task: 'Task[P, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectDistributedFuture[R]]