Apache Spark - 03 - Architecture - Part 1
Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
forms.gle/Nxk8dQUPq4o4XsA47
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.
Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.
If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.
ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.
The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews
Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.
forms.gle/Nxk8dQUPq4o4XsA47
--------------------------------------------------------------------------
Learn more at www.scholarnest.com/
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================
SPARK COURSES
-----------------------------
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/d...
KAFKA COURSES
--------------------------------
www.scholarnest.com/courses/a...
www.scholarnest.com/courses/k...
www.scholarnest.com/courses/s...
AWS CLOUD
------------------------
www.scholarnest.com/courses/a...
www.scholarnest.com/courses/a...
PYTHON
------------------
www.scholarnest.com/courses/p...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjournal.guru/cour...
=======================================
You can also find us on Oreilly Learning
www.oreilly.com/library/view/...
www.oreilly.com/videos/apache...
www.oreilly.com/videos/kafka-...
www.oreilly.com/videos/spark-...
www.oreilly.com/videos/spark-...
www.oreilly.com/videos/apache...
www.oreilly.com/videos/real-t...
www.oreilly.com/videos/real-t...
=========================================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/ScholarNest
github.com/learningJournal/
========================================
Пікірлер: 141
Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/
@jayeshpatiljp
2 жыл бұрын
atleast show courtesy to share link of 2nd part in description.
I love the concise language and diagrams you use to explain each concept. Makes it easy to understand for new learners like me. Many manuals and books are overcomplicating things by jargons and not goes straight to the point like you do.
You are pure gold. Content is amazing, structured and well deductively explained, thank you
No words to express..simply awesome video... :) thank a lot Prashant sir
one of the best spark arch explanation ever seen on youtube,,, wondeful!!! Thank u so much sir
Wonderful. We are lucky to have your materials. Simple and powerful.
One of the best videos of Spark i have come across untill now.. Thank you for explaining it so beautifully.. :)
WOW. This video was so helpful!! Thank you
I could never imagine I can learn Apache Spark without attending the physical sessions. You not only saved my time but give me confidence in all senses. Thanks for such a honest efforts !
Wow! This was fun to watch. Great demo. Thanks.
I have been following your videos for quiet a long time.I found each and every videos of yours to be so informative and easy.Thanks Prashant for all of these videos
Yu ll become trend setter the way ur showing slides the explanation evry thng peeks
OMG! This video is so amazing! Thank you so much for the amazing session.
These videos are simply amazing. Thanks a lot. Hope to see more tutorials from you.
sir multiple people saying u became your self like a god...........then i did not belive but...after when i watched the video then i fixed s u bacame ur self like a god.....yes its true.....and one more thing i have seen multiple videos regarding spark architecture but i didnt get this much of clarity.....so tnx a lot.....its amazing video
the quality of your content is amazing.
Great Video .. I got this much information by reading thousand articles and getting confused. Worse , still in doubt. your one video in 2024 made it all clear.. Thanks
for all other videos i have to watch 5 to 6 times then only i can able to grasp few things but your videos are quite impressive in the single shot itself i can able to grasp more things ... thanks a lot.... i'm willing to donate you tooo
Thank you, for the very practical and compact introduction to Spark.
Dude I am falling in love with you..amazing teaching content and style.
One more gem of a video from Mr. Pandey. Awesome work !
Very well explained the concepts in the video. Great work Sir!!!
Much appreciate the standard of the content.It's simple,precise and context.Thank you
Very good demo, learned more today, thank you.
Thank you very much.. This is one of the best tutorial I have come across.Appreciate your efforts.
Sir its avery informative video series..pls keep uploading..big fan after watching first few videos..thanks
You are one of the best teachers on youtube. Your style of teaching with simple examples is really helpful. thank you so much for making these videos. you are awesome sir.
Thank you sir, the quality of your content is amazing.
Best video to understand Spark Architecture
I am unable to stop myself to subscribe and like this video. its clean and fantastic explanation... making things clear.
Everything is very nicely explained. Thank you.
Sir ur explanation was very good. waiting for the next video
Wow, vera level. Thanks for the course. im one of the follower in Udemy & Linkedin. Once again thanks for this low hanging fruitful tutorial :) :) :) I got it what i was looking for :)
Great Video sir!!!.. Really clears my concepts Thank you
Loved it. Thank you. Subscribed
Wow, best lecture by far. Well done
Great video.. Came to know many unknown things from this video... :)
Sir, i seriously suggest you to post your videos in UDEMY, to reach max and all. This is excellent and better than many paid videos. Excellent ,pls do share more such videos. we are thankful to you.
@ScholarNest
6 жыл бұрын
+ganesh sundarachary thanks for the suggestion. I will think about it. However, I know my viewers are good at sharing, and sooner or later, all those udemy viewers that are looking for Spark will reach to this place as well.
Million Thanks Sir, Course of the content amazing sir.
thank you, wonderful connect.
Fantastic teaching style...please continue to do so
Thanks sir for sharing your knowledge, expecting more videos like this
Hi Sir...I really like and admire your efforts that you put for us. Thanks once again Pariksheet
This is very good explanation.. this is first video on KZread where I am commenting..
Superb explanation.Expecting more spark video from you.Thanks for sharing this video
The pace at which you are explaining and the way is excellent. Please make videos on Hadoop on AWS
This is mind blowing .. Fan of you
You are the most awesome teacher. hats off
Prashant Sir, I don't know how to thank you for such a wonderful content. I had to revise the spark and hadoop concepts, and what a wonderful way to do this. Thank you Prashant.
Excellent explanation. Do you have videos on spark streaming ?
Very useful Sir.. In depth knowledge you are providing with proper examples. Thanks a lot..!!! Looking forward to watch your more videos. And Sir, My jupyter notebook is not working..I have specified my problem in that video. Look into that too.
Pandey Sir... Teacher ho to aap sa.. warna na ho :). What a clear way of teaching.
This video is so amazing! It's very very clear. Thank you so much
Excellent !
This video is pure gold.
Best tutorial so far on youtube
Supperb explaination sir.... Thanks 😊
wow Excellent !
sir.. great work.. very nice explanation. i have seen lot of links, gone thru many videos, but didn't get clarity. your way of explaining is unique.. thank you soo much sir..
you are a awesome instructor. good job
AMAZING VIDEO AND EXPLANATIONS!
very nice explanation
Loved the content.
Good Explanation sir
awesome video very useful thanks
thank you so much!
Awesome explanation sir
thank you so so so much Prashant sir
god bless you all your life sir
Thank you sooo much for the great explanation!!
So much appreciated bow
Thanks 🙌🙌🙌
Great learning!!!!
sir, the way you explain very use and simple.could you please any for spark reference.Thanks in advance.
Sir I'm very thankful for your video. If I stuck in any point I come here to clear my doubts. Just a small request what ever you explain on CLI please increase the font size to make it visible. Thanks.
When we start spark-shell through edge node of cluster,it is which mode(client/cluster),because driver in running on one node of cluster right.Could you please explain.
Excellent explanation.
Great explanation.
Nice. Thank you
Hi sir, can you explain about how to define executor memory and cores while spark submit time
Really nice... 👏
Thank you billion times…..
Best in the internet...
At -19:43 when you did a ssh to the master node and started spark-shell with 3 executors then why the driver will start from your machine ? It shd start from one of the nodes
One word - Best
Sir can you please upload videoes regarding the project in spark or Hadoop. If not please suggest me some project related to spark and Hadoop. Bcoz your way of explanation and clearness in voice is great
Amazing content, do you have slides for these.
Hi sir, your videos are great. I have a request. Will it be okay if you make a video on real time Spark applications getting slow or getting failed. The causes of such failures and slowness and what could be done to prevent it. Thanks in advance :)
You rock !!
good explanation
Thank you for the video, it was very informative. When can we expect the sequel?
@ScholarNest
6 жыл бұрын
Next week for sure. It's another 20-minute long video and hence taking some time.
Sir what are slices you mentioned ? how is it related to cluster resources.
suitable
Sir, It means that as a developer we worked on local and standalone mode and never worked on cluster using yarn or other cluster manager?
Great explanation sir. Thank you so much!! I have one doubt. In the example you are using 1000 slice. What is the background concept of it and how is it helping to make the process run longer?
@ScholarNest
6 жыл бұрын
Keep watching, I don't remember exactly, but it is covered in some video.
Excellent demo video... Thank you..I feel that i couldn't see spark shell content and spark summit command...It would be better if video quality is more on clarity
@ScholarNest
6 жыл бұрын
+sudhakar k, video is full HD 1920x1080. Check your internet speed. However, all the commands and the code is also available at www.learningjournal.guru for copy/paste.
when will u upload the next video..waiting
Sir, by any chance can we get the slides what you're using for this presentation?
Very informative sir thanks 🙏. I have few questions- 1. Let’s suppose we have 3 nodes + 1 master - while staring the executors does that mean - each node will have 1 executor . 2. Do we need to keep the spark binaries in executor as well . Or that will be taken by spark master node
Very good vedio . Sir,I have a question, pratically in industries in which mode we do the development and which cluster manger we used. And how we do the unit testing.
@ScholarNest
6 жыл бұрын
Development is widely done in standalone mode on your local machine or a vm. Some people setup dev clusters as well and access it from a local or SSH client. Unit test code is mostly written using local mode. Local mode is a kind of embedded Spark in your JVM. Production environment is normally a Yarn cluster. However, there are many variations and we have Cloud DevOps implementation depending on preference of different teams, size of project, other required tech stack and target environment.
@deepakgupta-hk9ig
6 жыл бұрын
Sir it means that we do development on single machine without yarn or mesos and do the unit testing on small data . After that we prepare the jar an give it to test team for testing. Then how can we make sure that jar is correct. Many people ask this question in interview.