Airflow Tutorial: Running Data Quality Checks with Snowflake and Soda
Ғылым және технология
Airflow Tutorial: Running Data Quality Checks with Snowflake and Soda
🏆 BECOME A PRO WITH AIRFLOW: www.udemy.com/course/the-comp...
In this project, you will learn:
✅ How to set up an Airflow environment with the Astro CLI
✅ How to set up and configure Snowflake with Airflow
✅ How to load data from an HTTP endpoint into a Snowflake table
✅ How to use python virtual environments to avoid dependency conflicts
✅ How to run data quality checks with Soda and the ExternalPythonOperator
Enjoy ❤️
Пікірлер: 15
I just added this video to my weekend KZread playlist, but thanks, Mark, for consistently creating great tutorials and sharing tips for data people in simple human language.
@MarcLamberti
9 ай бұрын
Thank you ❤️
How to enable Test connection in Airflow.
@MarcLamberti can we do soda integration wthout storing Snowflake password in code?
Great video. This complete environment that you used can I to use in prod? (astro, dbt soda, etc) And eveything is free?
@MarcLamberti
8 ай бұрын
Yes except snowflake
Do we need Docker Desktop to installed in the machine?
@MarcLamberti
5 күн бұрын
Yes
Great video Marc, Very well put together. 👏👏👏👏👏👏 PS: It would been nice if you add the link to the source movie file in the comment or description for easier follow up.
@MarcLamberti
9 ай бұрын
Thank you so much! will do
I have a problem with @task.external_python decorator. It seems that it also requires airflow package in soda_venv virtual environment which is kinda odd..
@MarcLamberti
8 ай бұрын
You can find the code in the link in description from this video kzread.info/dash/bejne/dq6s1qWxmZe_kaQ.html :)
@O9W5I7Q
8 ай бұрын
@@MarcLamberti Particularly, I am getting an error "ModuleNotFoundError: No module named 'airflow'" and "ModuleNotFoundError: No module named 'pendulum'" when running "astro@f867522c2cb4:/usr/local/airflow$ airflow tasks test movie check_movie" (the same error is obtained also if I step into the virtual environment using "astro@f867522c2cb4:/usr/local/airflow$ source soda_venv/bin/activate")
Is this tutorial anywhere on Git?
*PromoSM*