Scheduled Jobs and Prefect

How to use Prefect and/or Prefect Cloud to manage scheduled jobs

Prefect is an open source workflow orchestration tool made for data-intensive workloads. This allows you to schedule and organize jobs to run any time, in your chosen order. It can accommodate dependencies between jobs, and is very useful for data pipelines and machine learning workflows.

Saturn Cloud supports two different tools in the Prefect ecosystem:

  • Prefect Core, which is an open source library users can install, and
  • Prefect Cloud, which is a fully hosted cloud service which you can purchase. The same team that maintains the prefect core library runs Prefect Cloud. Prefect Cloud is a hosted, high-availability, fault-tolerant service that handles all the orchestration responsibilities for running data pipelines.

If you’re already feeling confused, or would like an introduction to the concepts behind job scheduling with Prefect, you can start with our Introduction to Prefect Concepts.

Prefect Concepts

Get familiar with the concepts around scheduled jobs and managing tasks with Prefect

Develop a Scheduled Data Pipeline

Create scheduled deployments with Prefect

Using Prefect Cloud with Saturn Cloud

Learn to use fully hosted Prefect Cloud with your Saturn Cloud projects

Prefect Cloud Concepts

Learn more about the available orchestration options with Prefect Cloud