prefect.task_runners

Classes

TaskRunner

Abstract base class for task runners.

A task runner is responsible for submitting tasks to the task run engine running in an execution environment. Submitted tasks are non-blocking and return a future object that can be used to wait for the task to complete and retrieve the result.

Task runners are context managers and should be used in a with block to ensure proper cleanup of resources.

Methods:

name

name(self) -> str

The name of this task runner

duplicate

duplicate(self) -> Self

Return a new instance of this task runner with the same configuration.

submit

submit(self, task: 'Task[P, Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> F

submit

submit(self, task: 'Task[Any, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> F

submit

submit(self, task: 'Task[P, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> F

map

map(self, task: 'Task[P, R]', parameters: dict[str, Any | unmapped[Any] | allow_failure[Any]], wait_for: Iterable[PrefectFuture[R]] | None = None) -> PrefectFutureList[F]

Submit multiple tasks to the task run engine.

Args:

  • task: The task to submit.
  • parameters: The parameters to use when running the task.
  • wait_for: A list of futures that the task depends on.

Returns:

  • An iterable of future objects that can be used to wait for the tasks to
  • complete and retrieve the results.

ThreadPoolTaskRunner

A task runner that executes tasks in a separate thread pool.

Methods:

duplicate

duplicate(self) -> 'ThreadPoolTaskRunner[R]'

submit

submit(self, task: 'Task[P, Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectConcurrentFuture[R]

submit

submit(self, task: 'Task[Any, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectConcurrentFuture[R]

submit

submit(self, task: 'Task[P, R | Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectConcurrentFuture[R]

Submit a task to the task run engine running in a separate thread.

Args:

  • task: The task to submit.
  • parameters: The parameters to use when running the task.
  • wait_for: A list of futures that the task depends on.

Returns:

  • A future object that can be used to wait for the task to complete and
  • retrieve the result.

map

map(self, task: 'Task[P, Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectConcurrentFuture[R]]

map

map(self, task: 'Task[Any, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectConcurrentFuture[R]]

map

map(self, task: 'Task[P, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectConcurrentFuture[R]]

cancel_all

cancel_all(self) -> None

PrefectTaskRunner

Methods:

duplicate

duplicate(self) -> 'PrefectTaskRunner[R]'

submit

submit(self, task: 'Task[P, Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectDistributedFuture[R]

submit

submit(self, task: 'Task[Any, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectDistributedFuture[R]

submit

submit(self, task: 'Task[P, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None, dependencies: dict[str, set[TaskRunInput]] | None = None) -> PrefectDistributedFuture[R]

Submit a task to the task run engine running in a separate thread.

Args:

  • task: The task to submit.
  • parameters: The parameters to use when running the task.
  • wait_for: A list of futures that the task depends on.

Returns:

  • A future object that can be used to wait for the task to complete and
  • retrieve the result.

map

map(self, task: 'Task[P, Coroutine[Any, Any, R]]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectDistributedFuture[R]]

map

map(self, task: 'Task[Any, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectDistributedFuture[R]]

map

map(self, task: 'Task[P, R]', parameters: dict[str, Any], wait_for: Iterable[PrefectFuture[Any]] | None = None) -> PrefectFutureList[PrefectDistributedFuture[R]]