prefect-snowflake
integration makes it easy to connect to Snowflake in your Prefect flows. You can run queries both synchronously and asynchronously as Prefect flows and tasks.
Getting started
Prerequisites
- A Snowflake account and the necessary connection information.
Installation
Installprefect-snowflake
as a dependency of Prefect.
If you don’t already have Prefect installed, it will install the newest version of prefect
as well.
prefect
and prefect-snowflake
:
Blocks setup
Theprefect-snowflake
integration has two blocks: one for storing credentials and one for storing connection information. Register blocks in this module to view and edit them on Prefect Cloud:
Create the credentials block
Below is a walkthrough on saving aSnowflakeCredentials
block through code. Log into your Snowflake account to find your credentials.
The example below uses a user and password combination, but refer to the SDK documentation for a full list of authentication and connection options.
Create the connection block
Then, to create aSnowflakeConnector
block:
- After logging in, click on any worksheet.
- On the left side, select a database and schema.
- On the top right, select a warehouse.
- Create a short script, replacing the placeholders below.
Examples
To set up a table, use theexecute
and execute_many
methods. Then, use the fetch_all
method. If the results are too large to fit into memory, use the fetch_many
method to retrieve data in chunks.
By using the SnowflakeConnector
as a context manager, you can make sure that the Snowflake connection and cursors are closed properly after you’re done with them.
Resources
Refer to theprefect-snowflake
SDK documentation to explore other capabilities of the prefect-snowflake
library, such as async methods.
For further assistance using Snowflake, consult the Snowflake documentation or the Snowflake Python Connector documentation.