How spark works internally | what are the different components and how do they interact

Ғылым және технология

This video talks about how spark components work together and what is a Job, Stage and task. We will walk through a spark program to see how stages and tasks are created.
0:00 Intro
0:36 A spark program walk through
1:25 creation of RDD
2:37 Execution
4:51 Spark Stages
6:55 Spark Tasks
9:00 Executor and task parallelism
12:10 spark component hierarchy

Пікірлер: 40

  • @HARISHKUMAR-vo2cp
    @HARISHKUMAR-vo2cp2 жыл бұрын

    Just like "Spark" your explanation is very powerful and professional. Your topics help very much to revise before attending interviews.

  • @BigDataThoughts

    @BigDataThoughts

    2 жыл бұрын

    Thanks harish

  • @vigneshkumar3076
    @vigneshkumar3076 Жыл бұрын

    Seriously, well presented - Simple and sufficient.

  • @BigDataThoughts

    @BigDataThoughts

    Жыл бұрын

    Thanks Vignesh

  • @salmansayyad4522
    @salmansayyad4522 Жыл бұрын

    Hi Shreya, the way you explain things is very clear and to the point. Thank you very much for such a nice explanation. Please make some videos on real time big data project working on cloud.

  • @ramsiddu007
    @ramsiddu0072 жыл бұрын

    Thank you for sharing your expertise @Shreya Pal, appreciate!

  • @BigDataThoughts

    @BigDataThoughts

    2 жыл бұрын

    Thanks jai

  • @machhindrasmathematics2551
    @machhindrasmathematics2551 Жыл бұрын

    Insightful

  • @vishwassrinivas474
    @vishwassrinivas4742 жыл бұрын

    Thank you for this awesome !!! content

  • @BigDataThoughts

    @BigDataThoughts

    2 жыл бұрын

    Thanks vishwas

  • @nikhilgupta110
    @nikhilgupta1102 жыл бұрын

    Nice and simplistic approach to explain something which is very confusing and complex at first , you can also consider adding other things like, how spark engine perceive our code e.g Logical to Physical to cost effective plan. And How exactly no. Of shuffle operations, wide transformation has relation with No. of spark jobs and stages.

  • @BigDataThoughts

    @BigDataThoughts

    2 жыл бұрын

    Thanks Nikhil

  • @rajdas3889
    @rajdas38892 жыл бұрын

    Thanks so much for sharing, I got this from a fb AI Community. thanks

  • @BigDataThoughts

    @BigDataThoughts

    2 жыл бұрын

    Thanks Raj

  • @chanakyamishra4795
    @chanakyamishra47952 жыл бұрын

    Awaited topic ✅✅ Thank you 👏👏

  • @BigDataThoughts

    @BigDataThoughts

    2 жыл бұрын

    Thanks chanakya

  • @sanyoge389
    @sanyoge3892 жыл бұрын

    Really really helpful.. I am following your videos regularly. Pls keep doing the brilliant job and Thanks alot 🙏

  • @BigDataThoughts

    @BigDataThoughts

    2 жыл бұрын

    Thanks

  • @suniguha
    @suniguha Жыл бұрын

    Very nice and compact explanation. Thank you.

  • @BigDataThoughts

    @BigDataThoughts

    Жыл бұрын

    Thanks sunip

  • @subhashreebehera2994
    @subhashreebehera29942 жыл бұрын

    Best explanation till date ❤️

  • @BigDataThoughts

    @BigDataThoughts

    2 жыл бұрын

    Thanks Subhashree

  • @guptaashok121
    @guptaashok1212 жыл бұрын

    Nice

  • @yashawanthraj8872
    @yashawanthraj88724 күн бұрын

    Can Node/Thread have more partition than no of executors, if yes where the no of partition information will be stored.

  • @abhisekhmishra4029
    @abhisekhmishra4029 Жыл бұрын

    best before interviews🙂...thanks a lot

  • @BigDataThoughts

    @BigDataThoughts

    Жыл бұрын

    Thanks Abhishek

  • @yaswanthkumar9960
    @yaswanthkumar9960 Жыл бұрын

    superb explanation.

  • @BigDataThoughts

    @BigDataThoughts

    Жыл бұрын

    Thanks yaswanth

  • @mohammadrizwan830
    @mohammadrizwan8302 жыл бұрын

    Super Explanation 👌👌👌

  • @BigDataThoughts

    @BigDataThoughts

    2 жыл бұрын

    Thanks Mohammed

  • @GuruVideoAC
    @GuruVideoAC6 ай бұрын

    Well done

  • @BigDataThoughts

    @BigDataThoughts

    6 ай бұрын

    Thanks

  • @sandippatel6999
    @sandippatel6999 Жыл бұрын

    very nice explanation

  • @BigDataThoughts

    @BigDataThoughts

    Жыл бұрын

    Thanks sandip

  • @abhishektambewagh6156
    @abhishektambewagh6156 Жыл бұрын

    Great explanation ma'am!!

  • @BigDataThoughts

    @BigDataThoughts

    Жыл бұрын

    Thanks

  • @samk_jg
    @samk_jg11 ай бұрын

    Hi , I would like to know, do you offer any course on spark ? what if someone wants to learn from you online , please let me know , thanks!

  • @thelifehackerpro9943
    @thelifehackerpro9943 Жыл бұрын

    Job is a sequence of stages, jobs are divied into tasks

  • @gauravk4050
    @gauravk4050 Жыл бұрын

    Mam please change your mic, your content is very gud

  • @BigDataThoughts

    @BigDataThoughts

    Жыл бұрын

    Thanks gaurav. This is a old video. New ones are with better audio quality

Келесі