Getting Started with Airflow for Beginners
Getting Started with Airflow for Beginners.
At the end of this video, you will be able to:
✅ Identify the different ways of installing and running Airflow in local
✅ Create a local dev environment with Airflow following best practices
✅ Create data pipelines using Variables, XComs, and the Taskflow API
✅ Share data between tasks
✅ Trigger and monitor data pipelines
and more.
🏆 BECOME A PRO WITH AIRFLOW: www.udemy.com/course/the-comp...
If you're entirely new to Apache Airflow, take a look at those videos first:
1️⃣ Why Airflow? The Top 5 reasons to use it: • Why Airflow? The Top 5...
2️⃣ What is Apache Airflow?: • What is Apache Airflow...
00:00 Introduction
00:28 Install and setup Airflow in local
00:38 Pip install
00:57 The docker-compose file
01:51 The Astro CLI
03:46 Airflowctl
04:17 Reminder of the basics
04:30 The DAG you will create
05:00 Create the DAG
07:08 Create the first task
08:55 Create the second task
09:24 Airflow variables
10:55 Create the last task
11:15 Define task dependencies
12:35 The final DAG
13:05 Run it!
13:33 Use the UI to monitor and manage your DAGs
With that, you have all the keys to getting started with Airflow!
Пікірлер: 15
Today I watched the entire playlist without a break, you have a talent for explaining! Thank you. Also, perhaps the last two videos would be better placed at the beginning of the playlist.
@MarcLamberti
8 ай бұрын
woah! Thank you so much ❤️
Thank you the video is really help full
Many thanks for the video ; great tutorial ! I have a question regarding Airflow though ; I keep seeing DAG file where task functions are defined directly into the DAG file. What if I want to import functions from python scripts from other Repositories ? How can I call them in my DAG file ? I plan to use Airflow to run many scripts I have deployed on various Git repositories. I don't want to recreate those already existing functions into my DAG file but just call those scripts with their functions in my Airflow repository. Any idea how can I achieve that ? Many thanks and keep up making great videos ! ;)
@joshualobo220
4 ай бұрын
I think you would need to import the functions (defined in the external repositories )and decorate them with the "@task" decorator to make them tasks within your dag
Great video Marc! But why doesn't Visual Studio Code recognize the Airflow library even though I'm running my containers correctly? I've noticed that you're experiencing the same issue because the 'airflow.decorators' library is underlined in yellow. Is there a way to fix this?
@MarcLamberti
8 ай бұрын
Yup, will make a video about that but you need to connect VS to your containers running Airflow 🥹
i have setup airflow with docker using astro cli. how to make code suggestion work in vscode?
@suchitranair683
3 ай бұрын
I am facing the same issue since a week!
Using tempfile module from python, can I create a temporay directory associated with a DAG RUN that will be automatically deleted at the end of a DAG RUN regardless if the DAG RUN was successful or failure or otherwise?
@MarcLamberti
8 ай бұрын
Yes, two options: - trigger rules - setup and teardown tasks (this one better)
@0kazaki
8 ай бұрын
@@MarcLamberti can I set teardown to only run on dag failure?
why the video resolution looks messed up lol
Please speak clearly!
@MarcLamberti
12 күн бұрын
What’s not clear? 🥹