What's new in Apache Airflow 2.7?
Ғылым және технология
What's new in Apache Airflow 2.7?
🏆 BECOME A PRO WITH AIRFLOW: www.udemy.com/course/the-comp...
Here are the big changes:
➡️ Initialize and delete resources tasks need at ease!
💻 With the setup and teardown tasks (clusters, compute resources, etc)
➡️ Monitor Airflow with the new Cluster Activity page!
💻 Track the most important metrics of your Airflow instance in one click
➡️ New Gantt view!
💻 To better spot task bottlenecks in your DAGs
➡️ New Code view!
💻 Easier to read code changes and verify they've been applied
➡️ New Running and Failed filters!
💻 Click on a button to only see DAG in running or failed states
➡️ New BranchExternalPythonOperator
💻 Choose between tasks while avoiding dependency conflicts
➡️ New cache when calling Variables and Connections
💻 Better optimize resources by reducing the number of calls to the DB
➡️ Limit the concurrency of deferred tasks with Pools
💻 Set include_deferred=True to a pool
➡️ New Graph View!
💻 Navigating your DAGs has never been easier
➡️ db init and db upgrade are deprecated
💻 Use db migrate to create or upgrade the database
➡️ The button for testing connections is disabled
💻 For security reasons. Enable it with AIRFLOW__CORE__TEST_CONNECTION="Enabled"
And more!
🏆 BECOME A PRO: www.udemy.com/course/the-comp...
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🚨 My Patreon: / marclamberti
Ready?
Let's go!
Пікірлер: 17
Way to less likes! Your videos are top! Thanks
Please, do some "deep dive" tutorial on managing multiple airflow instances in a cluster like dev/prod. Never used git branching with airflow btw. What are best practices?
@MarcLamberti
10 ай бұрын
Will do 🫶
@JeanBzh
10 ай бұрын
good point, indeed
I'd like to see more about setup and teardown tasks. I'm using airflow intensively but we create our pipelines based on a Yaml file. (custom framework) so our dependecies are taken from the structure of the yaml
@MarcLamberti
7 ай бұрын
Will do
how to create dags or jobs using this latest ui i dont see any option to do so
Please do make a video about setup and teardown
@MarcLamberti
9 ай бұрын
Will do 🤓
setup is just the first task, and teardown is the last task with a "all_done" trigger rule, what do these new labels bring ?
@MarcLamberti
9 ай бұрын
Great question. I think the best answer is to look at the following tutorial: docs.astronomer.io/learn/airflow-setup-teardown setup and teardown tasks have specificities that make them much more convenient to manage resources than a task with all_done. I will make a video about it very soon.
@JeanBzh
9 ай бұрын
@@MarcLamberti I'll read it right away, thank you Marc
i am currently have an issue with subtasks in apache airflow. EXAMPLE: let say there are 3 subtasks (in parallerl) which are already completed and now two new tasks are added.. HOw do i make sure that only these two new subtasks are run? (instead of the entire 5 subtasks running) Attempts: from GUI perspective we have option called "queue up new tasks" that does the job. I am coming from the script point of view , about how to activate this button/option Could you please provide me with a hint/idea/suggestions (literally anything )
@JeanBzh
10 ай бұрын
I suggest you open a stack overflow topic, I'm not convinced this comment has anything to do with "what's new in Airflow 2.7"
Where do i find task logs ?
@MarcLamberti
8 ай бұрын
On the UI
@user-xy7ed4xq5k
8 ай бұрын
@@MarcLamberti earlier there was an option where we can click on task ( task box in the DAg visualization) and go to the logs of particular task. With the UI upgrade how can I navigate to the logs ?