Databricks Asset Bundles: Advanced Examples
Ғылым және технология
Databricks Asset Bundles is now GA (Generally Available). As more Databricks users start to rely on Databricks Asset Bundles (DABs) for their development and deployment workflows, let's look at some advanced patterns people have been asking for examples to help them get started.
Blog post with these examples: dustinvannoy.com/2024/06/25/d...
Intro post: dustinvannoy.com/2023/10/03/d...
* All thoughts and opinions are my own *
References:
Datakickstart DABs repo: github.com/datakickstart/data...
Data & AI Summit Presentation: www.databricks.com/dataaisumm...
Data & AI Summit Repo: github.com/databricks/databri...
More from Dustin:
Website: dustinvannoy.com
LinkedIn: / dustinvannoy
Github: github.com/datakickstart
CHAPTERS
0:00 Intro
1:52 Monorepo project setup
3:09 DABs basics (review)
6:32 Deploy and View in UI
11:28 Config Re-use (YAML Anchor)
16:09 Shared Python Package (Wheel)
19:04 Serverless Compute
20:54 Modify in UI
23:10 Create in UI
25:37 Extra tips
27:38 Outro
Пікірлер: 11
Exciting stuff! Will definitely be trying to implement this in my future work!
Great video ! Is there a way to overide variables defined in the databricks.yml in each of the job yml definition so that the variable has a different value for that job only ?
Thanks a lot, @DustinVannoy for this great presentation! I have a question: which is the better approach for project structuration: one bundle yml config file for all my sub-projects or each sub-project have its own Databricks and bundle yml file? Thanks again :)
Loving bundles so far. Only issue so far I've had is the databricks vscode extension seems to be modifying my bundles yml file behind the scenes. For example when I attach to a cluster in the extension it will override my job cluster to use that attached cluster when I deploy to the dev target in development mode.
@DustinVannoy
10 күн бұрын
Which version of the extension are you on, 1.3.0?
@gardnmi
9 күн бұрын
@@DustinVannoyYup, I did have it on a pre release which I thought was the issue but switched back to 1.3.0 and the "feature" persisted.
Can we integrate Azure pipelines + DAB for ci cd implementation?
@DustinVannoy
9 күн бұрын
Are you referring to Azure DevOps CI pipelines? You can do that and I am considering a video on that since it has been requested a few times.
@fortheknowledge145
9 күн бұрын
@@DustinVannoy yes, thank you!
Once the code is deployed it gets uploaded in the shared folder can't we store that some where else like an artifact or storage account because there are chances that someone may deleted that bundle from shared folder. It is always like with databricks deployment before and after asset bundles.
@DustinVannoy
8 күн бұрын
You can set permissions on the workspace folder and I recommend also having it all checked into version control such as GitHub in case you ever need to recover an older version.