In this era of paid courses...i found gem on KZread who is teaching concepts in depth...❤️

@ishaangupta4941

11 ай бұрын

agreeeeeeed!!!!

@DpIndia

10 ай бұрын

same @@ishaangupta4941

"ye to humko bhi nahi maloom hai" ... "nahi dikh raha hai chhoro" :) bahut sahi...unnecessary cheezon me time waste nahi karna chahiye

Hi Manish I saw your channel recently and I found it very insightful. You are explaining the spark core concepts nicely. Keep continue ❤ You have that caliber to grow on KZread.

both the videos on join strategy are awesome...explained in deep...thanks Manish

Gem of video in today's world where everyone is selling something.. please do a video for local setup, really struggling

well explained Manish Kumar Thank you for the lectures..

Excellent teaching skills u have bro ....very clearly explained..Thank u

Thank you. This was a deep explanation.

Amazing video. I have scored the depth of the internet nobody is able to clarify it. All are just copy pasting from each other.

Nice explanation!.Use of aliases also resolves the ambiguity error

Hi Manish . I have got job but not clear broadcast join .Today clear .Thank you . Please continue .

@rajamaurya4098

4 ай бұрын

hey brother you are fresher or experienced

Thanks bro for this series. It has given a huge boost to my DE preparation !!

Bohut baria parate ho bhai. Keep up the good work. Har ek video zabardast 👏

Great content...keep it up brother

bro u are teaching very well.

greate details provided

@17:46 you mentioned that wide dep trasformation creates 200 partition. But you said 11/11 as 1 partition? also why 4/4 1/1 4/4 1/1 were not counted?

it was really helpful ! keep up the good work

Awesome explanation

Mast. Manish bhai🎉🎉🎉❤❤

Request you to please make a video for local set of pyspark and please alos guide how to use pyspark in Jupiter notebook Thanks in advance 🙏

Please sir local machine me Spark setup karne ka video banado na practice keliye asan ho jaye ga. Thank you

Hi Manish, Can you pls make a video on local setup to practice PySpark and Python? If already made, can you pls share the link ? Much Appreciated. Thanks :)

Could you please make a video on Spark Web UI. I see you've already explained the UI partly in the stages, jobs and tasks video but a dedicated and detailed video would be very useful. Thank you!

@manish_kumar_1

11 ай бұрын

Sure

Thanks for the detailed session. it would be nice to have a local pyspark local setup.

@manish_kumar_1

3 ай бұрын

Already video is there

thank you sir

Sir please make a video on how to setup spark in local machine

Thanks a lot! Love your simplicity...Local setup bhi krvado plzzzzz :)

@manish_kumar_1

3 ай бұрын

Already karwa diya hai

@akashprabhakar6353

3 ай бұрын

@@manish_kumar_1 thanks bro

Plz also make a video on cache and persist

day 5 done👍

Hi manish pls provide a video on how to do local set up as well.

@manish_kumar_1

Жыл бұрын

Sure 👍

Hi Manish, Could you Please do a video on , How to do an local setup...

@manish_kumar_1

Жыл бұрын

Sure

Hi Manish Can you please explain one important topic that sort merge bucket join because I faced this question in interview and it is very important

Make Video on :- Locally setting up Spark environment

Hi, How to handle situation in broadcast hash join where we have OOM error in executor level or let's say executor is out of memory because of broadcast table ?

traditional drivers and executors aren't available in local environment because a single JVM is present, and processes are executed in parallel across these threads.

At,18:54 When we are doing shuffle partition=5 4/4 it’s ok. What is 11/11 ,and why we are counting it as 1 Partition?

hi manish, can you please explain why so many jobs are creating..there is only one action so job have to be only one?

Local Setup Batado Manish sir

Make video for setting local spark

Hello Manish, you are just awesome and I hardly found one other than you who teaches the in-depth. I am from Bangladesh, and my Hindi is not that good so can you please add English subtitles to your video?

@manish_kumar_1

Жыл бұрын

I will try

@mohamedmeeransubairs7204

7 ай бұрын

Please put videos on English as well👍

dear sir, You are really a great teacher. Kindly make a video of local spark setup. if you already done then please provide me video link.

Hi Manish Thank you very much for sharing great knowledge . Currently I have 10.5 Year Experience in IT including SQL,PLSQL(7 Year), SQL Server T-SQL (1.5 Year) and Snowflake Query Optimization 6 Month . When I was joined before 2 Year as Data Engineer (Spark with Scala) in one MNC company but He was given project on T-SQL . I was only taken trainings and search interview question and clear interview . At time I on bench what should be we take decision Please suggest me?.

@manish_kumar_1

11 ай бұрын

Chat me kaise batau. Aap ek session book Kar sakte hai topmate par if you are confused. Waise to main yaha par padha hi rha hu DE. To aap isko follow karte jaiye aapko sab idea lagne lagega

manish bhai spark streaming bhi padaoge kya ??

broadcast variable and broadcast join me kya difference hai ?

Hlw sir I have some doubts can you please help

Bhaiya please help in configure Spark ui with pycharm

can you make one video on how to negotiate notice period

@manish_kumar_1

Жыл бұрын

Baat kijiye apne HR se, Maan gaye to thik warna serve karna parega

I have a doubt. In shuffle sort merge join and shuffle hash join, is it correct that sorting and hashing are performed first before the join? Furthermore, does the join process occur the same way as you taught in the previous video?

@manish_kumar_1

11 ай бұрын

Yes for both of the questions

@aryankhandelwal8517

11 ай бұрын

@@manish_kumar_1 thank you so much

Why @18:53, 11/11 is considered as 1 partition whereas 4/4 is considered as 4 partitions?

please make video on how to understand jobs on spark UI seperately

@manish_kumar_1

11 ай бұрын

Watch one video where I have talked how many jobs, stages and task will be created

Hi Manish, shuffling takes place when we are joining two tables then how in broadcast we are saying that we are not doing shuffling and due which performance is good as we are using broadcast. As broadcast mein bhi toh join toh lag raha hai na toh shuffling toh hogi phir kaise it is different from shuffle sort or hash

@manish_kumar_1

9 ай бұрын

Aapne smjha hi nahi fir. Wapas video dekhiye join ka and broadcast ka dono hi

Hello Sir, Please help in creating a local setup for running Pyspark on Databricks.

@manish_kumar_1

2 ай бұрын

Local setup video already Bana diya hai

If I have a spark job running perfectly fine in prod someday got crashed, how to check in prod env? As not everyone directly gets spark prod access. Please answer.

@manish_kumar_1

Жыл бұрын

You will have to ask infra team to extract the logs from spark history server or you can store your error logs somewhere in DB. And ask the table read permission from the infra team

@saumyasingh9620

Жыл бұрын

@@manish_kumar_1 How to store error in db?

@manish_kumar_1

Жыл бұрын

@@saumyasingh9620 google kar lijiye. Solution mil jayega

Manish one video on local setup.

@manish_kumar_1

Жыл бұрын

Sure

@ashutoshkumarsingh3337

Жыл бұрын

@@manish_kumar_1 yes plzz i have pycharm but trying to integrate the pyspark , its not happening

Please make video for local setup

@manish_kumar_1

11 ай бұрын

Already did

@aryankhandelwal8517

11 ай бұрын

OK Thanks@@manish_kumar_1

The code is written in pycharm ? Can we do in databricks

@manish_kumar_1

Жыл бұрын

Yes absolutely

60 tb datarame and 20 tb Dataframe what is the optimised way to join these ?

@manish_kumar_1

Жыл бұрын

Let spark decide. In most of the cases it will be pick sort merge join if you are using equi join. Keep AQE enabled for better performance

Manish bhai, data science aur data engineer kya fark hota hai

@manish_kumar_1

Жыл бұрын

Google kar lijiye answer mil jayega aapko

Manish bhai mujhe anpa assisant bana lo, mai bina paise ke aap ke liye kam karunga. mai pichle 2 sal se spark ki practice kar raha hu.

@manish_kumar_1

Жыл бұрын

Av to Jo aap bol rhe wo possible nahi hai

in this total partitions got created was 210 but u said its 200?

@manish_kumar_1

Жыл бұрын

Where in my video? Or in your process

@user-px2pz3ec1x

9 ай бұрын

​@@manish_kumar_1at 17:45

@akhiladevangamath1277

2 ай бұрын

yes even I have this doubt

Thanks Sir