Tools to make things happen at the right time in the right order

Making things happen at the right time in the right order is much harder than it ought to be. Dagster is the latest attempt to make it easier.

A DAG is a Directed Acyclic Graph, a fancy name for a sequence of tasks that depend on each other. These types of jobs are the bread and butter of MLOps, controlling data pipelines, transformations and prediction jobs.

The current go-to tool for this kind of thing, Airflow, was first developed by Airbnb. It is a bit tricky to set up and use. Dagster is a replacement. Dagster recently wrote an article about why it’s way better than Airflow. Another alternative is Prefect, which also claims to be better than Airflow. I’ve used Prefect and Airflow, and I preferred Prefect. Maybe Dagster is even better?

πŸ›ŽοΈ Why this matters: Better than Airflow. Remember cron? How many task schedulers do we need?

πŸ“– Read more (4116 words) πŸ“–