Databricks Asset Bundles: Advanced Examples

Ғылым және технология

Databricks Asset Bundles is now GA (Generally Available). As more Databricks users start to rely on Databricks Asset Bundles (DABs) for their development and deployment workflows, let's look at some advanced patterns people have been asking for examples to help them get started.
Blog post with these examples: dustinvannoy.com/2024/06/25/d...
Intro post: dustinvannoy.com/2023/10/03/d...
* All thoughts and opinions are my own *
References:
Datakickstart DABs repo: github.com/datakickstart/data...
Data & AI Summit Presentation: www.databricks.com/dataaisumm...
Data & AI Summit Repo: github.com/databricks/databri...
More from Dustin:
Website: dustinvannoy.com
LinkedIn: / dustinvannoy
Github: github.com/datakickstart
CHAPTERS
0:00 Intro
1:52 Monorepo project setup
3:09 DABs basics (review)
6:32 Deploy and View in UI
11:28 Config Re-use (YAML Anchor)
16:09 Shared Python Package (Wheel)
19:04 Serverless Compute
20:54 Modify in UI
23:10 Create in UI
25:37 Extra tips
27:38 Outro

Пікірлер: 11

  • @NoahPitts713
    @NoahPitts7133 күн бұрын

    Exciting stuff! Will definitely be trying to implement this in my future work!

  • @etiennerigaud7066
    @etiennerigaud70667 күн бұрын

    Great video ! Is there a way to overide variables defined in the databricks.yml in each of the job yml definition so that the variable has a different value for that job only ?

  • @ameliemedem1918
    @ameliemedem19188 күн бұрын

    Thanks a lot, @DustinVannoy for this great presentation! I have a question: which is the better approach for project structuration: one bundle yml config file for all my sub-projects or each sub-project have its own Databricks and bundle yml file? Thanks again :)

  • @gardnmi
    @gardnmi10 күн бұрын

    Loving bundles so far. Only issue so far I've had is the databricks vscode extension seems to be modifying my bundles yml file behind the scenes. For example when I attach to a cluster in the extension it will override my job cluster to use that attached cluster when I deploy to the dev target in development mode.

  • @DustinVannoy

    @DustinVannoy

    10 күн бұрын

    Which version of the extension are you on, 1.3.0?

  • @gardnmi

    @gardnmi

    9 күн бұрын

    ​@@DustinVannoyYup, I did have it on a pre release which I thought was the issue but switched back to 1.3.0 and the "feature" persisted.

  • @fortheknowledge145
    @fortheknowledge1459 күн бұрын

    Can we integrate Azure pipelines + DAB for ci cd implementation?

  • @DustinVannoy

    @DustinVannoy

    9 күн бұрын

    Are you referring to Azure DevOps CI pipelines? You can do that and I am considering a video on that since it has been requested a few times.

  • @fortheknowledge145

    @fortheknowledge145

    9 күн бұрын

    @@DustinVannoy yes, thank you!

  • @9829912595
    @98299125958 күн бұрын

    Once the code is deployed it gets uploaded in the shared folder can't we store that some where else like an artifact or storage account because there are chances that someone may deleted that bundle from shared folder. It is always like with databricks deployment before and after asset bundles.

  • @DustinVannoy

    @DustinVannoy

    8 күн бұрын

    You can set permissions on the workspace folder and I recommend also having it all checked into version control such as GitHub in case you ever need to recover an older version.

Келесі