In this video I focus on a different side of monitoring:
What do the Databricks system tables offer me for monitoring?
How much does this overlap with the application logs and Spark metrics?
Databricks System Tables are a public preview feature that can be enabled if you have Unity Catalog on your workspace. I introduce the concept in the first 3 minutes then summarize where this is most helpful in the last 3 minutes. In between are some example queries and table explanations.
* All thoughts and opinions are my own *
Blog post with more detail:
dustinvannoy.com/2024/02/22/d...
Options to enable system tables:
Curl - learn.microsoft.com/en-us/azu...
dbdemos notebook - www.databricks.com/resources/...
Useful blogs:
/ databricks-system-tabl...
www.databricks.com/blog/impro...
Useful docs:
learn.microsoft.com/en-us/azu...
learn.microsoft.com/en-us/azu...
More from Dustin:
Website: dustinvannoy.com
LinkedIn: / dustinvannoy
Github: github.com/datakickstart
CHAPTERS
0:00 Intro
1:15 Enable system tables
3:28 Billing usage query
6:55 Docs highlights
7:57 Lineage query
9:02 Audit query
11:17 Additional tables
13:20 Compare to Log4J/Metrics
14:08 Summary