Apache Spark - 03 - Architecture - Part 1

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
forms.gle/Nxk8dQUPq4o4XsA47
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.
Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.
If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.
ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.
The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews
Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.
forms.gle/Nxk8dQUPq4o4XsA47
--------------------------------------------------------------------------
Learn more at www.scholarnest.com/
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================
SPARK COURSES
-----------------------------
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/d...
KAFKA COURSES
--------------------------------
www.scholarnest.com/courses/a...
www.scholarnest.com/courses/k...
www.scholarnest.com/courses/s...
AWS CLOUD
------------------------
www.scholarnest.com/courses/a...
www.scholarnest.com/courses/a...
PYTHON
------------------
www.scholarnest.com/courses/p...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjournal.guru/cour...
=======================================
You can also find us on Oreilly Learning
www.oreilly.com/library/view/...
www.oreilly.com/videos/apache...
www.oreilly.com/videos/kafka-...
www.oreilly.com/videos/spark-...
www.oreilly.com/videos/spark-...
www.oreilly.com/videos/apache...
www.oreilly.com/videos/real-t...
www.oreilly.com/videos/real-t...
=========================================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/ScholarNest
github.com/learningJournal/
========================================

Пікірлер: 141

  • @ScholarNest
    @ScholarNest3 жыл бұрын

    Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/

  • @jayeshpatiljp

    @jayeshpatiljp

    2 жыл бұрын

    atleast show courtesy to share link of 2nd part in description.

  • @wiwauhichale
    @wiwauhichale4 жыл бұрын

    I love the concise language and diagrams you use to explain each concept. Makes it easy to understand for new learners like me. Many manuals and books are overcomplicating things by jargons and not goes straight to the point like you do.

  • @thedude6341
    @thedude63416 жыл бұрын

    You are pure gold. Content is amazing, structured and well deductively explained, thank you

  • @4ukcs2004
    @4ukcs20046 жыл бұрын

    No words to express..simply awesome video... :) thank a lot Prashant sir

  • @jayanthsn4581
    @jayanthsn45815 жыл бұрын

    one of the best spark arch explanation ever seen on youtube,,, wondeful!!! Thank u so much sir

  • @harishurgent
    @harishurgent6 жыл бұрын

    Wonderful. We are lucky to have your materials. Simple and powerful.

  • @chetansp912
    @chetansp9126 жыл бұрын

    One of the best videos of Spark i have come across untill now.. Thank you for explaining it so beautifully.. :)

  • @JawDroppingFacts
    @JawDroppingFacts6 жыл бұрын

    WOW. This video was so helpful!! Thank you

  • @hamidkhan0590
    @hamidkhan05905 жыл бұрын

    I could never imagine I can learn Apache Spark without attending the physical sessions. You not only saved my time but give me confidence in all senses. Thanks for such a honest efforts !

  • @shivam.shakya
    @shivam.shakya2 жыл бұрын

    Wow! This was fun to watch. Great demo. Thanks.

  • @anuradha8700
    @anuradha87005 жыл бұрын

    I have been following your videos for quiet a long time.I found each and every videos of yours to be so informative and easy.Thanks Prashant for all of these videos

  • @althafmohammed5285
    @althafmohammed52856 жыл бұрын

    Yu ll become trend setter the way ur showing slides the explanation evry thng peeks

  • @dibyajyotisarkar5288
    @dibyajyotisarkar52885 жыл бұрын

    OMG! This video is so amazing! Thank you so much for the amazing session.

  • @abhi-wanderlust
    @abhi-wanderlust5 жыл бұрын

    These videos are simply amazing. Thanks a lot. Hope to see more tutorials from you.

  • @juniorfarmer1351
    @juniorfarmer13514 жыл бұрын

    sir multiple people saying u became your self like a god...........then i did not belive but...after when i watched the video then i fixed s u bacame ur self like a god.....yes its true.....and one more thing i have seen multiple videos regarding spark architecture but i didnt get this much of clarity.....so tnx a lot.....its amazing video

  • @hariparuchuri8406
    @hariparuchuri84064 жыл бұрын

    the quality of your content is amazing.

  • @payalbhatia6927
    @payalbhatia69274 ай бұрын

    Great Video .. I got this much information by reading thousand articles and getting confused. Worse , still in doubt. your one video in 2024 made it all clear.. Thanks

  • @rameshthamizhselvan2458
    @rameshthamizhselvan24586 жыл бұрын

    for all other videos i have to watch 5 to 6 times then only i can able to grasp few things but your videos are quite impressive in the single shot itself i can able to grasp more things ... thanks a lot.... i'm willing to donate you tooo

  • @raminology
    @raminology6 жыл бұрын

    Thank you, for the very practical and compact introduction to Spark.

  • @MR-vm7jp
    @MR-vm7jp4 жыл бұрын

    Dude I am falling in love with you..amazing teaching content and style.

  • @temp9483
    @temp94836 жыл бұрын

    One more gem of a video from Mr. Pandey. Awesome work !

  • @ashishlimaye2408
    @ashishlimaye24085 жыл бұрын

    Very well explained the concepts in the video. Great work Sir!!!

  • @samhusksr
    @samhusksr6 жыл бұрын

    Much appreciate the standard of the content.It's simple,precise and context.Thank you

  • @robind999
    @robind9996 жыл бұрын

    Very good demo, learned more today, thank you.

  • @mrcool4uall
    @mrcool4uall5 жыл бұрын

    Thank you very much.. This is one of the best tutorial I have come across.Appreciate your efforts.

  • @palashwarghane1420
    @palashwarghane14205 жыл бұрын

    Sir its avery informative video series..pls keep uploading..big fan after watching first few videos..thanks

  • @mohindersingh0711
    @mohindersingh07112 жыл бұрын

    You are one of the best teachers on youtube. Your style of teaching with simple examples is really helpful. thank you so much for making these videos. you are awesome sir.

  • @josedejesuslopezdiaz
    @josedejesuslopezdiaz5 жыл бұрын

    Thank you sir, the quality of your content is amazing.

  • @Bikerbug
    @Bikerbug2 жыл бұрын

    Best video to understand Spark Architecture

  • @ranjandas1797
    @ranjandas17975 жыл бұрын

    I am unable to stop myself to subscribe and like this video. its clean and fantastic explanation... making things clear.

  • @hamidkhan0590
    @hamidkhan05904 жыл бұрын

    Everything is very nicely explained. Thank you.

  • @sudheerthulluri
    @sudheerthulluri6 жыл бұрын

    Sir ur explanation was very good. waiting for the next video

  • @sundarrajkumaresan8045
    @sundarrajkumaresan80454 жыл бұрын

    Wow, vera level. Thanks for the course. im one of the follower in Udemy & Linkedin. Once again thanks for this low hanging fruitful tutorial :) :) :) I got it what i was looking for :)

  • @piyushdoifode8066
    @piyushdoifode80666 жыл бұрын

    Great Video sir!!!.. Really clears my concepts Thank you

  • @ateeqtalks
    @ateeqtalks5 жыл бұрын

    Loved it. Thank you. Subscribed

  • @ab-zo4kw
    @ab-zo4kw4 жыл бұрын

    Wow, best lecture by far. Well done

  • @sarfarazhussain6883
    @sarfarazhussain68835 жыл бұрын

    Great video.. Came to know many unknown things from this video... :)

  • @ganeshsundarachary8882
    @ganeshsundarachary88826 жыл бұрын

    Sir, i seriously suggest you to post your videos in UDEMY, to reach max and all. This is excellent and better than many paid videos. Excellent ,pls do share more such videos. we are thankful to you.

  • @ScholarNest

    @ScholarNest

    6 жыл бұрын

    +ganesh sundarachary thanks for the suggestion. I will think about it. However, I know my viewers are good at sharing, and sooner or later, all those udemy viewers that are looking for Spark will reach to this place as well.

  • @chsrinivas2426
    @chsrinivas24264 жыл бұрын

    Million Thanks Sir, Course of the content amazing sir.

  • @balabhaskararaoguntupalli1275
    @balabhaskararaoguntupalli12755 жыл бұрын

    thank you, wonderful connect.

  • @aneksingh4496
    @aneksingh44965 жыл бұрын

    Fantastic teaching style...please continue to do so

  • @bhavaniv1721
    @bhavaniv17214 жыл бұрын

    Thanks sir for sharing your knowledge, expecting more videos like this

  • @pariksheetde4573
    @pariksheetde45733 жыл бұрын

    Hi Sir...I really like and admire your efforts that you put for us. Thanks once again Pariksheet

  • @ryas8360
    @ryas83605 жыл бұрын

    This is very good explanation.. this is first video on KZread where I am commenting..

  • @papunisahu2411
    @papunisahu24114 жыл бұрын

    Superb explanation.Expecting more spark video from you.Thanks for sharing this video

  • @sairajaramandoju3025
    @sairajaramandoju30255 жыл бұрын

    The pace at which you are explaining and the way is excellent. Please make videos on Hadoop on AWS

  • @tejasnandola9375
    @tejasnandola93753 жыл бұрын

    This is mind blowing .. Fan of you

  • @amardhillon314
    @amardhillon3146 жыл бұрын

    You are the most awesome teacher. hats off

  • @gcpdev123
    @gcpdev1235 жыл бұрын

    Prashant Sir, I don't know how to thank you for such a wonderful content. I had to revise the spark and hadoop concepts, and what a wonderful way to do this. Thank you Prashant.

  • @poonamraykar4426
    @poonamraykar44265 жыл бұрын

    Excellent explanation. Do you have videos on spark streaming ?

  • @mohitbansal6360
    @mohitbansal63606 жыл бұрын

    Very useful Sir.. In depth knowledge you are providing with proper examples. Thanks a lot..!!! Looking forward to watch your more videos. And Sir, My jupyter notebook is not working..I have specified my problem in that video. Look into that too.

  • @sudhanshuguru209
    @sudhanshuguru2095 жыл бұрын

    Pandey Sir... Teacher ho to aap sa.. warna na ho :). What a clear way of teaching.

  • @duongpham9861
    @duongpham98613 жыл бұрын

    This video is so amazing! It's very very clear. Thank you so much

  • @bharathjii
    @bharathjii6 жыл бұрын

    Excellent !

  • @VikashSingh-cj1qw
    @VikashSingh-cj1qw4 жыл бұрын

    This video is pure gold.

  • @vickgoldgmailcom
    @vickgoldgmailcom4 жыл бұрын

    Best tutorial so far on youtube

  • @rahuljaiswal9379
    @rahuljaiswal93795 жыл бұрын

    Supperb explaination sir.... Thanks 😊

  • @zp4288
    @zp42883 жыл бұрын

    wow Excellent !

  • @sushmashamsundar8219
    @sushmashamsundar82194 жыл бұрын

    sir.. great work.. very nice explanation. i have seen lot of links, gone thru many videos, but didn't get clarity. your way of explaining is unique.. thank you soo much sir..

  • @yitbarekmirete6098
    @yitbarekmirete60984 жыл бұрын

    you are a awesome instructor. good job

  • @vladvlad2798
    @vladvlad27982 жыл бұрын

    AMAZING VIDEO AND EXPLANATIONS!

  • @codewithrajranjan
    @codewithrajranjan6 жыл бұрын

    very nice explanation

  • @rohangupta4290
    @rohangupta42902 жыл бұрын

    Loved the content.

  • @chetanbonde2256
    @chetanbonde22566 жыл бұрын

    Good Explanation sir

  • @haneefsk2845
    @haneefsk28455 жыл бұрын

    awesome video very useful thanks

  • @surabhidixit293
    @surabhidixit2936 жыл бұрын

    thank you so much!

  • @svdfxd
    @svdfxd6 жыл бұрын

    Awesome explanation sir

  • @pc0riginal870
    @pc0riginal8705 жыл бұрын

    thank you so so so much Prashant sir

  • @parthdayala7519
    @parthdayala75195 жыл бұрын

    god bless you all your life sir

  • @gabancariaok
    @gabancariaok3 ай бұрын

    Thank you sooo much for the great explanation!!

  • @sbylk99
    @sbylk995 жыл бұрын

    So much appreciated bow

  • @cherrejim
    @cherrejim4 жыл бұрын

    Thanks 🙌🙌🙌

  • @SurendraKapkoti
    @SurendraKapkoti Жыл бұрын

    Great learning!!!!

  • @Jainadarsh3092
    @Jainadarsh30926 жыл бұрын

    sir, the way you explain very use and simple.could you please any for spark reference.Thanks in advance.

  • @AudioVoiceStories
    @AudioVoiceStories2 жыл бұрын

    Sir I'm very thankful for your video. If I stuck in any point I come here to clear my doubts. Just a small request what ever you explain on CLI please increase the font size to make it visible. Thanks.

  • @mamtajain1882
    @mamtajain18825 жыл бұрын

    When we start spark-shell through edge node of cluster,it is which mode(client/cluster),because driver in running on one node of cluster right.Could you please explain.

  • @mthatipamula
    @mthatipamula4 жыл бұрын

    Excellent explanation.

  • @mohammadhaque1873
    @mohammadhaque1873 Жыл бұрын

    Great explanation.

  • @csharma8732
    @csharma87325 жыл бұрын

    Nice. Thank you

  • @creativethoughts2720
    @creativethoughts27206 жыл бұрын

    Hi sir, can you explain about how to define executor memory and cores while spark submit time

  • @tamilarasan948
    @tamilarasan9485 жыл бұрын

    Really nice... 👏

  • @ReinePoukou
    @ReinePoukou4 жыл бұрын

    Thank you billion times…..

  • @lovetoka
    @lovetoka2 жыл бұрын

    Best in the internet...

  • @AwaraGhumakkad
    @AwaraGhumakkad4 жыл бұрын

    At -19:43 when you did a ssh to the master node and started spark-shell with 3 executors then why the driver will start from your machine ? It shd start from one of the nodes

  • @anshulgautam2344
    @anshulgautam23442 жыл бұрын

    One word - Best

  • @pallavigupta321
    @pallavigupta3215 жыл бұрын

    Sir can you please upload videoes regarding the project in spark or Hadoop. If not please suggest me some project related to spark and Hadoop. Bcoz your way of explanation and clearness in voice is great

  • @balajicherukuri
    @balajicherukuri4 жыл бұрын

    Amazing content, do you have slides for these.

  • @murifedontrun3363
    @murifedontrun33635 жыл бұрын

    Hi sir, your videos are great. I have a request. Will it be okay if you make a video on real time Spark applications getting slow or getting failed. The causes of such failures and slowness and what could be done to prevent it. Thanks in advance :)

  • @pratheek5096
    @pratheek50962 жыл бұрын

    You rock !!

  • @rmaleshri
    @rmaleshri5 жыл бұрын

    good explanation

  • @bentyryan
    @bentyryan6 жыл бұрын

    Thank you for the video, it was very informative. When can we expect the sequel?

  • @ScholarNest

    @ScholarNest

    6 жыл бұрын

    Next week for sure. It's another 20-minute long video and hence taking some time.

  • @gopinathGopiRebel
    @gopinathGopiRebel5 жыл бұрын

    Sir what are slices you mentioned ? how is it related to cluster resources.

  • @niharranjanbiswas3812
    @niharranjanbiswas38124 жыл бұрын

    suitable

  • @deepakgupta-hk9ig
    @deepakgupta-hk9ig6 жыл бұрын

    Sir, It means that as a developer we worked on local and standalone mode and never worked on cluster using yarn or other cluster manager?

  • @mdtausif3480
    @mdtausif34806 жыл бұрын

    Great explanation sir. Thank you so much!! I have one doubt. In the example you are using 1000 slice. What is the background concept of it and how is it helping to make the process run longer?

  • @ScholarNest

    @ScholarNest

    6 жыл бұрын

    Keep watching, I don't remember exactly, but it is covered in some video.

  • @sudhakark3244
    @sudhakark32446 жыл бұрын

    Excellent demo video... Thank you..I feel that i couldn't see spark shell content and spark summit command...It would be better if video quality is more on clarity

  • @ScholarNest

    @ScholarNest

    6 жыл бұрын

    +sudhakar k, video is full HD 1920x1080. Check your internet speed. However, all the commands and the code is also available at www.learningjournal.guru for copy/paste.

  • @ravikantsharma6976
    @ravikantsharma69766 жыл бұрын

    when will u upload the next video..waiting

  • @ramkishoregandi
    @ramkishoregandi5 жыл бұрын

    Sir, by any chance can we get the slides what you're using for this presentation?

  • @AwaraGhumakkad
    @AwaraGhumakkad4 жыл бұрын

    Very informative sir thanks 🙏. I have few questions- 1. Let’s suppose we have 3 nodes + 1 master - while staring the executors does that mean - each node will have 1 executor . 2. Do we need to keep the spark binaries in executor as well . Or that will be taken by spark master node

  • @deepakgupta-hk9ig
    @deepakgupta-hk9ig6 жыл бұрын

    Very good vedio . Sir,I have a question, pratically in industries in which mode we do the development and which cluster manger we used. And how we do the unit testing.

  • @ScholarNest

    @ScholarNest

    6 жыл бұрын

    Development is widely done in standalone mode on your local machine or a vm. Some people setup dev clusters as well and access it from a local or SSH client. Unit test code is mostly written using local mode. Local mode is a kind of embedded Spark in your JVM. Production environment is normally a Yarn cluster. However, there are many variations and we have Cloud DevOps implementation depending on preference of different teams, size of project, other required tech stack and target environment.

  • @deepakgupta-hk9ig

    @deepakgupta-hk9ig

    6 жыл бұрын

    Sir it means that we do development on single machine without yarn or mesos and do the unit testing on small data . After that we prepare the jar an give it to test team for testing. Then how can we make sure that jar is correct. Many people ask this question in interview.