Accelerate your workflows by running tasks in parallel with Ray
prefect-ray
integration makes it easy to accelerate your flow runs with Ray.
prefect-ray
prefect-ray
compatible with your installed version of prefect
.
If you don’t already have prefect
installed, it will install the newest version of prefect
as well.
prefect
and prefect-ray
:
pip
alone and will be skipped during installation of Prefect. It is possible to manually install the blocking component with conda
. See the Ray documentation for instructions.RayTaskRunner
is a Prefect task runner that submits tasks to Ray for parallel execution.
By default, a temporary Ray instance is created for the duration of the flow run.
For example, this flow counts to three in parallel:
address
argument.
To configure your flow to use the RayTaskRunner
:
prefect-ray
collection is installed as described earlier: pip install prefect-ray
.RayTaskRunner
from prefect_ray.task_runners
.task_runner=RayTaskRunner
argument.RayTaskRunner
with a local, temporary Ray instance created by Prefect at flow run time.
RayTaskRunner
configured to access an existing Ray instance at ray://<head_node_host>:10001
.
RayTaskRunner
accepts the following optional parameters:
Parameter | Description |
---|---|
address | Address of a currently running Ray instance, starting with the ray:// URI. |
init_kwargs | Additional kwargs to use when calling ray.init . |
address
of a Ray instance, Prefect creates a temporary instance automatically.
RayTaskRunner
with a remote Ray cluster, you may run into issues that are not seen when using a local Ray instance.
To resolve these issues, we recommend taking the following steps when working with a remote Ray cluster:
prefect
is installed on the remote cluster, with:RayTaskRunner
. The address should look similar to: address='ray://<head_node_ip_address>:10001'
:remote_options
context can be used to control the task’s remote options. For example, we can set the number of CPUs and GPUs to use for the process
task:
prefect-ray
SDK documentation to explore all the capabilities of the prefect-ray
library.
For further assistance using Ray, consult the Ray documentation.