How spark works internally | what are the different components and how do they interact
Ғылым және технология
This video talks about how spark components work together and what is a Job, Stage and task. We will walk through a spark program to see how stages and tasks are created.
0:00 Intro
0:36 A spark program walk through
1:25 creation of RDD
2:37 Execution
4:51 Spark Stages
6:55 Spark Tasks
9:00 Executor and task parallelism
12:10 spark component hierarchy
Пікірлер: 40
Just like "Spark" your explanation is very powerful and professional. Your topics help very much to revise before attending interviews.
@BigDataThoughts
2 жыл бұрын
Thanks harish
Seriously, well presented - Simple and sufficient.
@BigDataThoughts
Жыл бұрын
Thanks Vignesh
Hi Shreya, the way you explain things is very clear and to the point. Thank you very much for such a nice explanation. Please make some videos on real time big data project working on cloud.
Thank you for sharing your expertise @Shreya Pal, appreciate!
@BigDataThoughts
2 жыл бұрын
Thanks jai
Insightful
Thank you for this awesome !!! content
@BigDataThoughts
2 жыл бұрын
Thanks vishwas
Nice and simplistic approach to explain something which is very confusing and complex at first , you can also consider adding other things like, how spark engine perceive our code e.g Logical to Physical to cost effective plan. And How exactly no. Of shuffle operations, wide transformation has relation with No. of spark jobs and stages.
@BigDataThoughts
2 жыл бұрын
Thanks Nikhil
Thanks so much for sharing, I got this from a fb AI Community. thanks
@BigDataThoughts
2 жыл бұрын
Thanks Raj
Awaited topic ✅✅ Thank you 👏👏
@BigDataThoughts
2 жыл бұрын
Thanks chanakya
Really really helpful.. I am following your videos regularly. Pls keep doing the brilliant job and Thanks alot 🙏
@BigDataThoughts
2 жыл бұрын
Thanks
Very nice and compact explanation. Thank you.
@BigDataThoughts
Жыл бұрын
Thanks sunip
Best explanation till date ❤️
@BigDataThoughts
2 жыл бұрын
Thanks Subhashree
Nice
Can Node/Thread have more partition than no of executors, if yes where the no of partition information will be stored.
best before interviews🙂...thanks a lot
@BigDataThoughts
Жыл бұрын
Thanks Abhishek
superb explanation.
@BigDataThoughts
Жыл бұрын
Thanks yaswanth
Super Explanation 👌👌👌
@BigDataThoughts
2 жыл бұрын
Thanks Mohammed
Well done
@BigDataThoughts
6 ай бұрын
Thanks
very nice explanation
@BigDataThoughts
Жыл бұрын
Thanks sandip
Great explanation ma'am!!
@BigDataThoughts
Жыл бұрын
Thanks
Hi , I would like to know, do you offer any course on spark ? what if someone wants to learn from you online , please let me know , thanks!
Job is a sequence of stages, jobs are divied into tasks
Mam please change your mic, your content is very gud
@BigDataThoughts
Жыл бұрын
Thanks gaurav. This is a old video. New ones are with better audio quality