Airflow DAG dependencies: The Datasets, TriggerDAGRunOperator, and ExternalTaskSensor
Airflow DAG dependencies: The Datasets, TriggerDAGRunOperator and ExternalTaskSensor
A DAG dependency in Apache Airflow is a link between two or multiple data pipelines.
There are three ways of creating DAG Dependencies in Apache Airflow:
👉 Datasets
👉 TriggerDagRunOperator
👉 ExternalTaskSensor
In this video, you will discover an overview of those 3 ways and how to use them.
Materials: robust-dinosaur-2ef.notion.si...
🏆 BECOME A PRO: www.udemy.com/course/the-comp...
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🚨 My Patreon: / marclamberti
Enjoy ❤️
Пікірлер: 15
Thank you for the great tutorials. Could you please consider touching email-sending methods? There are multiple ways to send an email and it will be interested, which one do you prefer.
Hi, I have a question. If I have a DAG which is scheduled to run on daily basis. Can I keep a dependency on the previous run of the same dag?
How can I use ontology with Graph to extract dependency 😖 ???
Great!!!
@MarcLamberti
10 ай бұрын
Thank you
Hey...I have a question. If 2 DAGs remain within 2 different airflow clusters.....which of these methods can be used for managing cross DAG dependency?
@MarcLamberti
10 ай бұрын
Check out the REST API
Hi Marc, I love you Airflow videos. Do you have a solution for file watching files matching pattern like filename_YYYYMMdd.csv (eg: filename_20240501.csv) in S3 and also onprem? Thanks
Why we don't use API to trigger another DAG instead of use the datasets?
@MarcLamberti
Жыл бұрын
Because datasets are easier and more flexible to use than the API. The API is useful if you have a dependency between two DAGs that are in two DIFFERENT Airflow instances
Max i pay u train udemy.. my case i wanna copy data base from postgres server to local server .. and update every day.. please give me example...i dont understand..
ok
@MarcLamberti
Жыл бұрын
ok
@reviewmaster9435
Жыл бұрын
@@MarcLamberti i really like your videos and hard work, you are awesome, love from india Marc😇
@viniciuskreischer1868
Жыл бұрын
Hi, Marc. Is it posdible to trigger specific indtances of a DAG using datasets?