In this video, I'll be going through how you can run and monitor Databricks workflows using Airflow so you can have a full picture of every step of your data pipeline in one place! This is a great tool if you have steps in your data pipeline before or after your Databricks notebooks, so that you can monitor it all from a single UI, rather than constantly switching between management UI's. Check out the code and follow along at the link below:
docs.astronomer.io/learn/airf...