i have came across many video for catalyst optimizer. i found this is the best and well explained:)

You are precisely choosing the topics and very very accurately explaining them. Please keep it up.

@DataSavvy

3 жыл бұрын

Thanks Vishwanath

Clearly explained thanks

@DataSavvy

Жыл бұрын

Glad it helped

Your video is very nice and specific. Apart from this sir, i want to add some thing... We will get many physical plan only when cost based optimization is enabled.

@DataSavvy

3 жыл бұрын

You are Right... Thanks for adding information

1. Parsing - create an abstract syntax tree. 2. Analysis - Catalyst analyzer performs semantic analysis on tree. This includes resolving references, type checking, and creating a logical plan. The analyzer also infers data types. 3. Logical optimisation - Rewrite the plan into a more efficient form. This includes predicate pushdown, constant folding. 4. Physical planning - Spark stages and tasks created. 5. Physical optimisation - optimized further by considering factors like data partitioning, join order, and choosing the most efficient physical operators 6.Code Generation - generates Java bytecode for the optimized physical plan

Thanks for the detailed explanation. However I am slightly confused now after watching previous video. There you mentioned logical plan , dag , execution plan is the pattern. Could you please connect that in this detailed context. Is DAG is part of this catalyst optimizer?

Hi sir, great video! Could you please let us know what is whole stage code generation ? Is it the RDD code which is generated after picking up the most optimized plan ?

Thanks for your spark explanation. Can you please make a video on serialization, deserialization? thanku

Good explaining of optimizer

Please try keeping volume at a higher pitch, ur videos are very educative elaborative and helpful. Please try improving the sound as well. Sometimes it is very difficult to understand and I close the video.

very good explanation

Good info. Can you publish a video showing dataframe vs dataset difference with an example.

@DataSavvy

5 жыл бұрын

kzread.info/dash/bejne/jJ2mxKtqY5ibcps.html

great

Hi Sir, Thanks for sharing valuable spark interview questions with us. could you please tell us the difference between Tungsten and Catalyst optimizer? can we create more than one spark context for an application, I have confusion with the allowMultipleContext property while creating a spark context? Kindly share any information with us on this. Thanks

@rajeshwarreddyracha4655

3 жыл бұрын

Multiple spark contexts by setting up, Spark.driver.allowMultipleContexts to TRUE. Multiple spark contexts for single JVM is not recommended, since crashing of one spark context will affect other. Spark Context contains same ContextId, But Spark Session contains different Session id’s while creating new ones and all Spark sessions will share the same Context id.

Add scenario based questions from Spark (Core , SQL , Streaming) . .. also add Questions for Scala

@DataSavvy

6 жыл бұрын

Sure Hemanshu... Do u have any examples of scenario based questions? I will create video for that

@himanshusekharpaul476

6 жыл бұрын

I don't have complet list of scenario . But it can be created ..Like . Let's say you got a file of 8 GB . How can you copy it to each executer memory . What it the meaning of add jar parameter in Spark- submit? What each parameter in Spark submit do internally? How you can do some customization with those parameter list ?? Etc

Great contents :)

@DataSavvy

4 жыл бұрын

Thanks Kaushik

also pls make a video on spark RDD vs df vs sparksql performance and which one outperforms other and in which case.

@DataSavvy

3 жыл бұрын

This video will be helpful kzread.info/dash/bejne/jJ2mxKtqY5ibcps.html

Hi. Can you explain about Case classes

Regardless of using any join in my code does optimizer converts it in (terms of physical plan) into most efficient join like you said map side/broadcast/hash join??

@DataSavvy

4 жыл бұрын

yes... it does... wherever optimization is possible

Very nicely explained 👍

@DataSavvy

4 жыл бұрын

Thank Ankita... I am happy that you liked it... Please share your suggestions, if any to improve content on this channel.

@DataSavvy

4 жыл бұрын

Please subscribe to channel. It motivates to create more useful content for everyone.. Thanks :)

Great and helpful video but voice is low

REAL TIME SCENARIOSSS PLS

Is there any video on Spark optimization techniques ? I did not found so please help me with this. Thanks in advance.

@DataSavvy

3 жыл бұрын

Are your looking for act specific technique?

Could you please suggest a good spark tutorial?

@DataSavvy

5 жыл бұрын

Bro, I thought my channel has good tutorial. :) Can you suggest what is missing here

@dilsha795

5 жыл бұрын

@@DataSavvy Your is channel is excellent on an interview point of view. I couldn't find proper tutorial that explains from basic level

@DataSavvy

5 жыл бұрын

@@dilsha795 got it... :) Will start creating videos for tutorial point of view also

would suggest before publishing it .plz check if it is audible or not

@DataSavvy

6 жыл бұрын

Thanks Santosh for suggestion... I have been doing that... However as soon as I upload video on KZread, KZread decreases voice quality after processing video... In New videos I have used new microphone and changed format of video... There is some improvement in voice quality... Apologies for inconvenience

@DataSavvy

6 жыл бұрын

Low audio is issue on mobiles majorly, I tested on laptop, it looks fine... If that helps

logical plan is lineage and physical plan is DAG, pls confirm?

@DataSavvy

3 жыл бұрын

Not Really...

sound aa thaaan pesaan da

Volume very low sir

cost based optimizer and rule based optimizer eliminated catalyst from spark 2??

@DataSavvy

4 жыл бұрын

That's a news... Let me check and get back

@rakeshdey1702

4 жыл бұрын

@@DataSavvy Actually CBO is used to select most optimized execution plan.. so catalyst optimizer actually does from logical to execution plan. Before converting RDD, CBO actually selects most optimized execution plan. Let me know if you conclude same.. CBO comes in picture from spark 2.3 I think

Apki awaz bahut dheere hai.. Saabhi vedios mai.. Please make louder vedios. Content is awesum...

@DataSavvy

3 жыл бұрын

Thanks Heena... I was very new jab maine ye sari video banai.. It was microphone issue.. unfortunately youtube does not give option to edit already uploaded videos... I have improved this in latest videos...

@heenasaxena6118

3 жыл бұрын

@@DataSavvy I see all your vedios. Major problem I face in interview is in explaining project flow from end to end. Can you please make some vedio which teaches me how to explain project to interviewers.

please increase the audio.

from your next video kindly speak louder

please improve audio..

@DataSavvy

3 жыл бұрын

Hi Bhavana... I have improved this in New videos... Excuse me for inconvenience

Would you speak a little louder, please.

Pathetic sound quality