Airflow DAG dependencies: The Datasets, TriggerDAGRunOperator, and ExternalTaskSensor

Airflow DAG dependencies: The Datasets, TriggerDAGRunOperator and ExternalTaskSensor
A DAG dependency in Apache Airflow is a link between two or multiple data pipelines.
There are three ways of creating DAG Dependencies in Apache Airflow:
👉 Datasets
👉 TriggerDagRunOperator
👉 ExternalTaskSensor
In this video, you will discover an overview of those 3 ways and how to use them.
Materials: robust-dinosaur-2ef.notion.si...
🏆 BECOME A PRO: www.udemy.com/course/the-comp...
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🚨 My Patreon: / marclamberti
Enjoy ❤️

Пікірлер: 15

  • @revazg4071
    @revazg4071 Жыл бұрын

    Thank you for the great tutorials. Could you please consider touching email-sending methods? There are multiple ways to send an email and it will be interested, which one do you prefer.

  • @akshitapanyam
    @akshitapanyam Жыл бұрын

    Hi, I have a question. If I have a DAG which is scheduled to run on daily basis. Can I keep a dependency on the previous run of the same dag?

  • @SfZq
    @SfZq25 күн бұрын

    How can I use ontology with Graph to extract dependency 😖 ???

  • @claudiodeluca2357
    @claudiodeluca235710 ай бұрын

    Great!!!

  • @MarcLamberti

    @MarcLamberti

    10 ай бұрын

    Thank you

  • @user-gg9nz5ns1k
    @user-gg9nz5ns1k10 ай бұрын

    Hey...I have a question. If 2 DAGs remain within 2 different airflow clusters.....which of these methods can be used for managing cross DAG dependency?

  • @MarcLamberti

    @MarcLamberti

    10 ай бұрын

    Check out the REST API

  • @Sir_Dave_Kennedy
    @Sir_Dave_KennedyАй бұрын

    Hi Marc, I love you Airflow videos. Do you have a solution for file watching files matching pattern like filename_YYYYMMdd.csv (eg: filename_20240501.csv) in S3 and also onprem? Thanks

  • @luongky06
    @luongky06 Жыл бұрын

    Why we don't use API to trigger another DAG instead of use the datasets?

  • @MarcLamberti

    @MarcLamberti

    Жыл бұрын

    Because datasets are easier and more flexible to use than the API. The API is useful if you have a dependency between two DAGs that are in two DIFFERENT Airflow instances

  • @TheMisterDonald
    @TheMisterDonald Жыл бұрын

    Max i pay u train udemy.. my case i wanna copy data base from postgres server to local server .. and update every day.. please give me example...i dont understand..

  • @reviewmaster9435
    @reviewmaster9435 Жыл бұрын

    ok

  • @MarcLamberti

    @MarcLamberti

    Жыл бұрын

    ok

  • @reviewmaster9435

    @reviewmaster9435

    Жыл бұрын

    @@MarcLamberti i really like your videos and hard work, you are awesome, love from india Marc😇

  • @viniciuskreischer1868

    @viniciuskreischer1868

    Жыл бұрын

    Hi, Marc. Is it posdible to trigger specific indtances of a DAG using datasets?