Airflow Tutorial: Running Data Quality Checks with Snowflake and Soda

Ғылым және технология

Airflow Tutorial: Running Data Quality Checks with Snowflake and Soda
🏆 BECOME A PRO WITH AIRFLOW: www.udemy.com/course/the-comp...
In this project, you will learn:
✅ How to set up an Airflow environment with the Astro CLI
✅ How to set up and configure Snowflake with Airflow
✅ How to load data from an HTTP endpoint into a Snowflake table
✅ How to use python virtual environments to avoid dependency conflicts
✅ How to run data quality checks with Soda and the ExternalPythonOperator
Enjoy ❤️

Пікірлер: 15

  • @hizokadarkwolf
    @hizokadarkwolf9 ай бұрын

    I just added this video to my weekend KZread playlist, but thanks, Mark, for consistently creating great tutorials and sharing tips for data people in simple human language.

  • @MarcLamberti

    @MarcLamberti

    9 ай бұрын

    Thank you ❤️

  • @JoseR-ui9vn
    @JoseR-ui9vn4 күн бұрын

    How to enable Test connection in Airflow.

  • @rishirajtandon3849
    @rishirajtandon38493 ай бұрын

    @MarcLamberti can we do soda integration wthout storing Snowflake password in code?

  • @rafaelg8238
    @rafaelg82388 ай бұрын

    Great video. This complete environment that you used can I to use in prod? (astro, dbt soda, etc) And eveything is free?

  • @MarcLamberti

    @MarcLamberti

    8 ай бұрын

    Yes except snowflake

  • @JoseR-ui9vn
    @JoseR-ui9vn5 күн бұрын

    Do we need Docker Desktop to installed in the machine?

  • @MarcLamberti

    @MarcLamberti

    5 күн бұрын

    Yes

  • @mikekenneth2339
    @mikekenneth23399 ай бұрын

    Great video Marc, Very well put together. 👏👏👏👏👏👏 PS: It would been nice if you add the link to the source movie file in the comment or description for easier follow up.

  • @MarcLamberti

    @MarcLamberti

    9 ай бұрын

    Thank you so much! will do

  • @O9W5I7Q
    @O9W5I7Q8 ай бұрын

    I have a problem with @task.external_python decorator. It seems that it also requires airflow package in soda_venv virtual environment which is kinda odd..

  • @MarcLamberti

    @MarcLamberti

    8 ай бұрын

    You can find the code in the link in description from this video kzread.info/dash/bejne/dq6s1qWxmZe_kaQ.html :)

  • @O9W5I7Q

    @O9W5I7Q

    8 ай бұрын

    @@MarcLamberti Particularly, I am getting an error "ModuleNotFoundError: No module named 'airflow'" and "ModuleNotFoundError: No module named 'pendulum'" when running "astro@f867522c2cb4:/usr/local/airflow$ airflow tasks test movie check_movie" (the same error is obtained also if I step into the virtual environment using "astro@f867522c2cb4:/usr/local/airflow$ source soda_venv/bin/activate")

  • @O9W5I7Q
    @O9W5I7Q8 ай бұрын

    Is this tutorial anywhere on Git?

  • @kaycullen8707
    @kaycullen87078 ай бұрын

    *PromoSM*

Келесі