Focus On: The Apache Airflow PythonOperator, all you need in 20 mins!

Ойын-сауық

Focus On: The Apache Airflow PythonOperator, all you need in 20 mins!
In this video you are going to discover everything about the PythonOperator in Airflow 2.0
At the end of the video you will be able to:
⏺ Execute a Python function (Amazing isn't it?)
⏺ Pass positional arguments to your Python function
⏺ Pass keyword arguments to your Python function
⏺ Evaluate your arguments at runtime and pass variables
⏺ Reduce the number of calls to your database
⏺ Make your DAG cleaner
⏺ Get the current execution date in Airflow 2.0
⏺ Leverage the TaskFlow API to transform your PythonOperator in a mind-blowing way!
Materials:
www.notion.so/The-PythonOpera...
The NEW WAY OF CREATING DAGS:
• TaskFlow API in Airflo...
Enjoy ❤️

Пікірлер: 29

  • @chris0628
    @chris0628 Жыл бұрын

    To the point, no fluff 👍🏽

  • @averychen4633
    @averychen46333 ай бұрын

    you are the best

  • @steliosioannides7128
    @steliosioannides71282 жыл бұрын

    Thanks Marc for this very useful video! High appreciated ! Keep it up !!

  • @Leonardo-jv1ls
    @Leonardo-jv1ls2 жыл бұрын

    Thank you. You've helped me a lot.

  • @andrikramer686
    @andrikramer6863 жыл бұрын

    Awesome. Thanks a lot, Marc.

  • @amitjain000
    @amitjain0002 жыл бұрын

    Awesome.. thanks for such a beautiful knowledge share.

  • @JosePla
    @JosePla2 жыл бұрын

    This is exactly what I was looking for

  • @AbhishekAmeria
    @AbhishekAmeria3 жыл бұрын

    That's was very useful. Thanks :)

  • @FranVarVar
    @FranVarVar Жыл бұрын

    Amazing. Thanks Marc!

  • @Astronomer

    @Astronomer

    Жыл бұрын

    My pleasure!

  • @vladimirobellini6128
    @vladimirobellini61283 жыл бұрын

    this is great. thanks!

  • @SpiritOfIndiaaa
    @SpiritOfIndiaaa2 жыл бұрын

    Excellent thanks a lot...I have to check oracle table if a record exists for current date ,if exists then only call downdown stream task ,how to achieve this ?

  • @shivangitomar5557
    @shivangitomar55572 жыл бұрын

    Very good vidoe!!!

  • @bullandrooster
    @bullandrooster2 жыл бұрын

    Can I run 2 dags with different python version and different pandas version? For now I see airflow offers only one python version at the time, and only predefined required dependencies. And all my dags must match that specs.

  • @manjunathmani3678
    @manjunathmani36782 жыл бұрын

    nice man

  • @sridhar43299
    @sridhar432992 жыл бұрын

    How to send object from gcs to email bu airflow. If possible send me code

  • @brendoaraujo9110
    @brendoaraujo91102 жыл бұрын

    instead of calling the script function, is there an operator that I can call the script completely to be executed?

  • @valetta6789
    @valetta6789 Жыл бұрын

    I am trying to pull the value from another task in a different folder, however, I see only the current task instance. Is this new behaviour, or am I unable to pass between different directories?

  • @Astronomer

    @Astronomer

    Жыл бұрын

    So your DAG's are in different folders?

  • @valetta6789

    @valetta6789

    Жыл бұрын

    @@Astronomer I have no idea what I was talking about😄

  • @eunheechoi3745
    @eunheechoi37457 ай бұрын

    whenever I first run the airflow test command, it throws an error saying 'sqlalchemy.exc.NoReferenceTableError: Foreign key associated with column 'dag_run_note.user_id' could not find table 'ab_user' with which to generate a foreign key to target column 'id' do you what it is about? and how to fix it? Second run it doesn't occur....

  • @Astronomer

    @Astronomer

    6 ай бұрын

    Hmmmmm interesting, if it works on the second run maybe it needs to generate the table first?

  • @GeandersonLenz
    @GeandersonLenz3 жыл бұрын

    How to turn ON this module includes?

  • @leomax87

    @leomax87

    2 жыл бұрын

    did you resolve it?

  • @BalvanshHeerekar
    @BalvanshHeerekar2 жыл бұрын

    Hey, while using includes I get the error - ModuleNotFoundError: No module named 'includes'

  • @leomax87

    @leomax87

    2 жыл бұрын

    I'm facing the same error. How Can I resolve it?

  • @eldrigeampong8573

    @eldrigeampong8573

    Жыл бұрын

    @@leomax87 you can import the sys module in python after which you will navigate a number of levels either up or down into the directory your file is located. import sys sys.path.append(".") # that's move a level up into the dags folder from includes.my_dags.functions import process

  • @archanam4224
    @archanam42242 жыл бұрын

    import airflow cannot be resolved

Келесі