Great job as always Bryan, keep it up, you are helping us all!

@BryanCafferky

Жыл бұрын

Thanks Verone.

Wonderful demo. Thanks

@BryanCafferky

Жыл бұрын

You're welcome.

Another awesome tutorial, thank you Bryan.

@BryanCafferky

Жыл бұрын

You're Welcome!

Really useful

Really great content to understand in detail about how DLT works. Thanks @Bryan for your effort in making this video.

@BryanCafferky

11 ай бұрын

You're welcome!

Thanks. This was useful.

Great stuff!

Great one Bryan. Super Video

@BryanCafferky

11 ай бұрын

Thanks

Great explanation,Thank you

@BryanCafferky

5 ай бұрын

You're Welcome!

2:40 It seems like Premium is required for most features now, as everything is based on Unity Catalog which in turn is a premium feature.

Hello Bryan Sir, Thanks for your amazing videos.

@BryanCafferky

Жыл бұрын

HI Ibrahim, Thanks. Did you watch the video? I explain about that.

I love your video consistent

@BryanCafferky

Жыл бұрын

Thank You!

Great video. Like how you dive into other topics like should we use it? What does it cost? It's running extra nodes in the background....etc. Lot of useful info in your explanations. Just wanted to mention on the expectations not having a splitter to an error table, we had a demo from Databricks recently and their approach was to create a copy of the function with the expectation, but pointed at the error table and with the inverse expectation of the main function. I mentioned this wasn't ideal since you would have to run the full job twice and they didn't have much to say. We have a different approach to dealing with errors so not a huge deal from our standpoint, but still not great in general.

@BryanCafferky

11 ай бұрын

Thanks for the feedback and your experience with expectations.

Thanks for this video Bryan. 13:27 if you want to quarantine some data based on a given rule, the workaround is to create another table and put an expectation to drop all the good records and keep only the bad one

Hi. Just wanted to make sure something. I am using Azure databricks where I already have two clusters in production. Now, if I want to create a DLT pipeline (assuming that's the only way to use Delta live tables ), would that create a new cluster/compute resource ?

Would it be possible to create unmanaged tables with a location in datalake using DLT pipelines ?

Thanks for the awesome video! A question if you could help: How to do CI/CD with delta live tables?

@BryanCafferky

Жыл бұрын

This blog explains it www.databricks.com/blog/applying-software-development-devops-best-practices-delta-live-table-pipelines

@krishnakoirala2088

Жыл бұрын

@@BryanCafferky Thank you!

Hey Bryan, Thanks For the video. Just curious, do we know the list of decorators which we can use in DLT pipelines. I looked into the documentation but was unable to find it

@BryanCafferky

3 ай бұрын

Since you have the dlt package, you have the code so you should be able to inspect the modules using Python functions like dir() or even view the code, see stackoverflow.com/questions/48983597/how-to-print-source-code-of-a-builtin-module-in-python DLT doc is here docs.databricks.com/en/delta-live-tables/python-ref.html#:~:text=In%20Python%2C%20Delta%20Live%20Tables,materialized%20views%20and%20streaming%20tables. I've not tried these things on dlt so let me know how it goes please.

hey bryan, great video, I have a quick quesiton, when you create a DLT for RAW, PREPARED and the last layer, that tables are created in the lakehous into BRONZE< SILVER AND GOLD?

@BryanCafferky

Ай бұрын

Yes, if I understand you. You can direct the tables to fit into the medallion architecture. See www.databricks.com/glossary/medallion-architecture

Hi Bryan, Is it possible to use Standard cluster to create Delta live tables instead of creating new cluster every time ?

@BryanCafferky

11 ай бұрын

I don't see coverage of that in the docs but here's the link to check yourself. learn.microsoft.com/en-us/azure/databricks/delta-live-tables/settings You may be able to create a workflow with your own cluster and call a DLT pipeline. Not sure if that will still create a separate cluster.

what I have observed, the materialized view is recomputing everything from scratch, what can we do to do incremental ingestion into the materialized view based on the group by clause if we provide.

Nice info! Is is a bad design to have bronze, silver and gold layer in the same schema. I believe DLT doesn’t work with multiple schemas

Really confused if i use DLT's for my project or old way of doing it for Medallion architecture. Now i watching your video, that DLT's cost alot more than normal ingestion pyspark pipelines? :(

@BryanCafferky

6 ай бұрын

Right. Best use case is for streaming and it has some nice features but it's not for everyone nor is it free. 🙂

The worst thing about DLT is you cannot run it cell by cell and check what you are doing.

@BryanCafferky

10 ай бұрын

Check this out. An opensource project that lets you test DLT interactively. I have not tried it. github.com/souvik-databricks/dlt-with-debug

Hi Bryan, I'm unable to import dlt module using import command I also used magic command and other solutions from stackoverflow too Can you help me to import dlt module

@BryanCafferky

11 ай бұрын

Please watch the video. I explain that.

How to create dlt pipeline using json ?( No option is coming to load json)

I couldn't create the pipeline because it says "The Delta Pipelines feature is not enabled in your workspace." So far I searched for few hours, couldn't find where to set this up. Quite disappointed that your video misses this vital feature.

@BryanCafferky

4 ай бұрын

Actually, I do talk about that. See 5:07 where I talk about the Databricks Services. You need to have the Premium service. I did a quick Google search and found this blog to help you stackoverflow.com/questions/71784405/delta-live-tables-feature-missing