Retrieve code from storage
Learn about where to store your flow code.
When a deployment runs, the execution environment needs access to the flow code. Flow code is not stored in Prefect server or Prefect Cloud.
You have several flow code storage options for remote execution:
- Git-based storage (GitHub, GitLab, Bitbucket)
- Docker image-based storage
- Cloud-provider storage (AWS S3, Azure Blob Storage, GCP GCS)
Each of these options is popular - your choice will depend upon your team’s needs and tools.
Local storage is also an option for deployments that run locally.
In the examples below, we show how to create a work pool-based deployment for each of these storage options.
Deployment creation options
You can create a deployment through Python code with the flow.deploy
method or through a YAML specification defined in a prefect.yaml
file.
When using the Python deploy
method specify a flow storage location other than a Docker image requires the flow.from_source
method. The source
and entrypoint
arguments are required.
The entrypoint
is the path to the file the flow is located in and the function name, separated by a colon.
The source
is either a URL to a git repository or a storage object.
To create a prefect.yaml
file interactively, run prefect deploy
from the CLI and select the appropriate storage option.
Git-based storage
Git-based version control platforms provide redundancy, version control, and collaboration capabilities. Prefect supports:
For a public repository, you can use the repository URL directly.
If you are using a private repository and are authenticated in your environment at deployment creation and deployment execution, you can use the repository URL directly.
Alternatively, for a private repository, you can create a Secret
block or create a credentials block specific to your git-based version control platform to store your credentials.
Then you can reference the block in the Python deploy
method or the prefect.yaml
file pull step.
If using the Python deploy
method with a private repository that references a block, provide a GitRepository
object instead of a URL, as shown below.
For accessing a private repository, we suggest creating a personal access token. We recommend using HTTPS with fine-grained Personal Access Tokens to limit access by repository.
See the GitHub docs for Personal Access Tokens (PATs).
Under Your Profile -> Developer Settings -> Personal access tokens -> Fine-grained token choose Generate New Token and fill in the required fields.
Under Repository access choose Only select repositories and grant the token permissions for Contents.
If using a Secret block, you can create it through code or the UI ahead of time and reference it at deployment creation as shown above.
If using a credentials block, you can create it ahead of time and reference it at deployment creation.
- Install prefect-github with
pip install -U prefect-github
- Register the blocks in that library to make them available on the server with
prefect block register -m prefect_github
- Create a GitHub Credentials block through code or the Prefect UI and reference it at deployment creation as shown above.
Note that you can specify a branch
if creating a GitRepository
object.
The default is "main"
.
Push your code
When you make a change to your code, Prefect does not push your code to your git-based version control platform. You need to push your code manually or as part of your CI/CD pipeline. This is intentional to avoid confusion about the git history and push process.
Docker-based storage
Another popular flow code storage option is to include it in a Docker image. All work pool options except Process and Prefect Managed allow you to bake your code into a Docker image.
To create a deployment with Docker-based flow code storage use the Python deploy
method or create a prefect.yaml
file.
If you use the Python deploy
method to store the flow code in a Docker image, you don’t need to use the from_source
method.
The prefect.yaml
file below was generated by running prefect deploy
from the CLI (a few lines of metadata were excluded from the top of the file output for brevity).
Note that the build
section is necessary if baking your flow code into a Docker image.
Any pip packages specified in a requirements.txt
file will be included in the image.
In the examples above, we elected not to push the image to a remote registry.
To push the image to a remote registry, pass push=True
in the Python deploy
method or change push: true
in the prefect.yaml
file.
Custom Docker image
By default, your deployment will use the base Prefect image when creating your image.
Alternatively, you can create a custom Docker image outside of Prefect.
If doing this with and you don’t need push or pull steps in the prefect.yaml
file.
Instead, the work pool can reference the image directly.
For more information, see this discussion of custom Docker images.
Cloud-provider storage
Another option for flow code storage is any fsspec-supported storage location, such as AWS S3, Azure Blob Storage, or GCP GCS.
If the storage location is publicly available, or if you are authenticated in the environment where you are creating and running your deployment, you can reference the storage location directly. You don’t need to pass credentials explicitly.
To pass credentials explicitly to authenticate to your storage location, you can use either of the following block types:
- Prefect integration library storage blocks, such as the
prefect-aws
library’sS3Bucket
block, which can use aAWSCredentials
block when it is created. - Secret blocks
If you use a storage block such as the S3Bucket
block, you need to have the prefect-aws
library available in the environment where your flow code runs.
You can do any of the following to make the library available:
- Install the library into the execution environment directly
- Specify the library in the work pool’s Base Job Template in the Environment Variables section like this:
{"EXTRA_PIP_PACKAGES":"prefect-aws"}
- Specify the library in the environment variables of the
deploy
method as shown in the examples below - Specify the library in a
requirements.txt
file and reference the file in thepull
step of theprefect.yaml
file like this:
- prefect.deployments.steps.pip_install_requirements:
directory: "{{ pull_code.directory }}"
requirements_file: requirements.txt
The examples below show how to create a deployment with flow code in a cloud provider storage location.
For each example, we show how to access code that is publicly available.
The prefect.yaml
example includes an additional line to reference a credentials block if authenticating to a private storage location through that option.
We also include Python code that shows how to use an existing storage block and an example of that creates, but doesn’t save, a storage block that references an existing nested credentials block.
To create an AwsCredentials
block:
- Install the prefect-aws library with
pip install -U prefect-aws
- Register the blocks in prefect-aws with
prefect block register -m prefect_aws
- Create a user with a role with read and write permissions to access the bucket. If using the UI, create an access key pair with IAM -> Users -> Security credentials -> Access keys -> Create access key. Choose Use case -> Other and then copy the Access key and Secret access key values.
- Create an
AWSCredentials
block in code or the Prefect UI. In addition to the block name, most users will fill in the AWS Access Key ID and AWS Access Key Secret fields. - Reference the block as shown above.
Another authentication option is to give the worker access to the storage location at runtime through SSH keys.
Store code locally
If using a Process work pool, you can use one of the remote code storage options shown above, or you can store your flow code in a local folder.
Here is an example of how to create a deployment with flow code stored locally:
Include or exclude files from storage
By default, Prefect includes all files in the current folder when you create a deployment.
When using a git repository, Docker image, or cloud-provider storage location, you may want to exclude certain files or directories.
If you are familiar with Docker you are likely familiar with the .dockerignore
file.
For remote storage, the .prefectignore
file serves the same purpose and follows a similar syntax.
For example, an entry of *.pyc
will exclude all .pyc
files from upload.
Update flow code
After creating a deployment, you may need to change your flow code. If baking your flow code into a Docker image, you will need to rebuild your image. If storing your flow code in a git-based version control platform or a cloud-based storage location, often you can update your flow code without rebuilding your deployment.
The exception is when something the server needs to know about has changed, such as the flow entrypoint parameters.
Rerun the Python script with deploy
or run prefect deploy
from the CLI for YAML-based deployments to update your deployment with the new flow code.
Flow code storage for deployments created with serve
The Python serve
method creates a deployment and a local long-running process to poll for flow runs at the same time.
The deployment creation mechanics for serve
are similar to deploy
.
deploy
just requires a work pool name and has a number of parameters dealing with flow code storage for Docker images.
Unlike serve
, if you don’t specify an image to use for your flow, you must specify where to pull the flow code from at runtime with the from_source
method; from_source
is optional with serve
.
Read more about when to consider using serve
here.
Was this page helpful?