Hey folk 👋 Here is the updated video with the sound fixed: Dynamic DAGs in Apache Airflow for beginners kzread.info/dash/bejne/aJl817GdaZnbnrQ.html Enjoy ❤

thank you for the great tutorial again. I love the airflow series.

I believe, this method is much more reliable if you are going to generate multiple dags with different schedule intervals. I was using dynamic dag generation with globals()[dag_id]=dag, but if you are working with APIs or clients, scheduler & worker don't work as expected. Also in the documentation, isolated DAG files are suggested, therefore Airflow community must be using your methods as well. Generator.py can be scheduled to generate the files using cron job, and it will be good to go :) Thank you for the content you have shared and informed Airflow users. It was extremely helpful for me :)

awesome! that's what i was looking for)

Awesome content, thank you.

Hi Marc, thank you very much for sharing great information! one quick question though, this could have also done by using normal replace method as well where it will replace the values it receives from yaml in dag template and generate new dag for each yaml config files. Any specific advantage of using jinja2 templates instead?

Informative ❤️

Hi Marc, can you please elaborate on the risks of using config with a loop and globals? I would agree that having one file per DAG is more ideal but you also linearly increase your codebase for each config so that con needs to outweigh the risks/downsides of the loop/global approach

Hi! Thanks for your video. Very helpful. Just a comment, you're recording your voice through just one channel, and music in the other, which makes it a bit quieter than normal.

This is informative but I think the piece that I always struggle with is whats the best way to keep the load on the scheduler down if I need to pull data to build the dag based on whats in a database.

Lovely!

Hey. Can I access the airflow instance which is deployed on astronomer using kubernetes?? I mainly want meta data to access

Hi Marc, first of all, thanks a lot for teaching us airflow! I also got your course on Udemy but I had a question, there you ask us to use virtual box but I didn't and I used your KZread video to setup airflow using docker in 5 minutes. That shouldn't be a problem for the rest of the course, right? Edit: fixed tracing to teaching

@MarcLamberti

2 жыл бұрын

Not at all and I'm actually thinking of removing this VM

@97Arshan

2 жыл бұрын

@@MarcLamberti that's great, thank you very much!

Hi Mark, I am new to airflow and I was hoping you’d be able to help provide me guidance on this: I want to create a job that retrieves 1.api key based on params (ie params=api id) 2. make api call 3. load data However I need to do this for many different Apikey (ie do this for 100+ different api keys, but function is the same across all, only difference for each job is params of step1) how would you approach this? Would you dynamically create individual dags based on params for each api key? Would you have a parent dag to generate subdags for each apikey? Or any other ideas? Thanks a lot in advance!

@mukundreddy9374

Жыл бұрын

Hi did you find a way to do it?

Do you have any git repo this example?

Can we use JSON also instead of YAML?

@MarcLamberti

2 жыл бұрын

of course :)

Hmm.. why can I only hear music but not the voice from my right earbud. great video nonetheless thank you !!

@MarcLamberti

2 жыл бұрын

My mic :'( Sorry about that

may you re-record this?

@MarcLamberti

Жыл бұрын

Why? 👀

I never managed to install airflow to start learning…the local host 8080 never opens to me

Audio is not good