Getting started
Prerequisites
- A Databricks account and the necessary permissions to access desired services.
Install prefect-databricks
The following command will install a version of prefect-databricks
compatible with your installed version of prefect
.
If you don’t already have prefect
installed, it will install the newest version of prefect
as well.
prefect
and prefect-databricks
:
List jobs on the Databricks instance
Use with_options
to customize options on any existing task or flow
Launch a new cluster and run a Databricks notebook
Notebook namedexample.ipynb
on Databricks which accepts a name parameter:
example.ipynb
:
AutoScale(min_workers=1, max_workers=2)
is equivalent to {"min_workers": 1, "max_workers": 2}
.
Resources
For assistance using Databricks, consult the Databricks documentation. Refer to theprefect-databricks
SDK documentation to explore all the capabilities of the prefect-databricks
library.