Learn how to identify and resolve common issues with Prefect Cloud.
Proxies handle network requests between a server and a client.
To communicate with Prefect Cloud, the Prefect client library makes HTTPS requests.
These requests are made with the httpx
Python library.
httpx
respects accepted proxy environment variables, so the Prefect client can communicate through proxies.
To enable communication through proxies, set the HTTPS_PROXY
and SSL_CERT_FILE
environment
variables in your execution environment.
See the Using Prefect Cloud with proxies GitHub Discussion for more information.
You should whitelist the URLs for the UI, API, Authentication, and the current OCSP server for outbound-communication in a secure environment:
If the Prefect Cloud API key, environment variable settings, or account login for your execution environment are not configured correctly, you may experience errors or unexpected flow run results when using Prefect CLI commands, running flows, or observing flow run results in Prefect Cloud.
Use the prefect config view
CLI command to make sure your execution environment is correctly configured
to access Prefect Cloud:
Make sure PREFECT_API_URL
is configured to use https://api.prefect.cloud/api/...
.
Make sure PREFECT_API_KEY
is configured to use a valid API key.
You can use the prefect cloud workspace ls
CLI command to view or set the active workspace.
You can also check that the account and workspace IDs specified in the URL for PREFECT_API_URL
match those shown in the URL for your Prefect Cloud workspace.
If you’re having difficulty logging in to Prefect Cloud, the following troubleshooting steps may resolve the issue, or will provide more information when sharing your case to the support channel.
Other tips to help with login difficulties:
None of this worked?
Email us at <help@prefect.io> and provide answers to the questions above in your email to make it faster to troubleshoot and unblock you. Make sure you add the email address you used to attempt to log in, your Prefect Cloud account name, and, if applicable, the organization it belongs to.
The first troubleshooting step is to confirm that you are running the latest version of Prefect. If you are not, upgrade (see below) to the latest version, since the issue may have already been fixed. Beyond that, there are several categories of errors:
Prefect is constantly evolving by adding new features and fixing bugs. A patch may have already been identified and released. Search existing issues for similar reports and check out the Release Notes.
Upgrade to the newest version with the following command:
Different components may use different versions of Prefect:
Integration Versions
Integrations are versioned and released independently of the core Prefect library. They should be upgraded simultaneously with the core library, using the same method.
In many cases, there is an informative stack trace in Prefect’s logs. Read it carefully, locate the source of the error, and try to identify the cause.
There are two types of logs:
If your flow and task logs are empty, there may have been an infrastructure issue that prevented your flow from starting. Check your worker logs for more details.
If there is no clear indication of what went wrong, try updating the logging level from the default INFO
level
to the DEBUG
level. Settings such as the logging level are propagated
from the worker environment to the flow run environment. Set them with environment variables or the prefect config set
CLI:
The DEBUG
logging level produces a high volume of logs, so consider setting it back to INFO
once any issues are resolved.
When using Prefect Cloud, there are the additional concerns of authentication and authorization.
The Prefect API authenticates users and service accounts, collectively known as actors, with API keys.
Missing, incorrect, or expired API keys result in a 401 response with detail Invalid authentication credentials
.
Use the following command to check your authentication, replacing $PREFECT_API_KEY
with your API key:
Users vs Service Accounts
Service accounts (sometimes referred to as bots),
represent non-human actors that interact with Prefect such as workers and CI/CD systems.
Each human that interacts with Prefect should be represented as a user.
User API keys start with pnu_
and service account API keys start with pnb_
.
Actors can be members of workspaces. An actor attempting an action in a workspace they are not a member of results in a 404 response. Use the following command to check your actor’s workspace memberships:
Formatting JSON
Python comes with a helpful tool for formatting JSON.
Append the following to the end of the command above to make the output more readable: | python -m json.tool
Make sure your actor is a member of the workspace you are working in. Within a workspace, an actor has a role which grants them certain permissions. Insufficient permissions result in an error. For example, starting a worker with the Viewer role results in errors.
The user can execute flows locally, or remotely by a worker. Local execution generally means that you, the user,
run your flow directly with a command like python flow.py
. Remote execution generally means that a worker runs your flow
through a deployment (optionally on different infrastructure).
With remote execution, the creation of your flow run happens separately from its execution.
Flow runs are assigned to a work pool and a work queue. For flow runs to execute, a worker must be subscribed
to the work pool and work queue, or the flow runs will go from Scheduled
to Late
.
Ensure that your work pool and work queue have a subscribed worker.
Local and remote execution can also differ in their treatment of relative imports.
If switching from local to remote execution results in local import errors, try replicating the
behavior by executing the flow locally with the -m
flag (For example, python -m flow
instead of python flow.py
).
Read more about -m
in this Stack Overflow post.
Summary: requests require a trailing /
in the request URL.
If you write a test that does not include a trailing /
when making a request to a specific endpoint:
You’ll see a failure like:
To resolve this, include the trailing /
:
Note: requests to nested URLs may exhibit the opposite behavior and require no trailing slash:
Reference: “HTTPX disabled redirect following by default” in
0.22.0
.
pytest.PytestUnraisableExceptionWarning
or ResourceWarning
As you’re working with one of the FlowRunner
implementations, you may get an
error like this:
This error states that your test suite (or the prefect
library code) opened a
connection to something (like a Docker daemon or a Kubernetes cluster) and didn’t close
it.
It may help to re-run the specific test with PYTHONTRACEMALLOC=25 pytest ...
so that
Python can display more of the stack trace where the connection was opened.