very informative one. Thanks Buddy.

Very useful video. I have been working with spark for more than two years now but never really bothered about SparkSession vs SparkContext. For me its just the entry point and you go from there. But the idea of having multiple sparkSessions with a single underlying SparkContext makes great sense and was an eye opener. Thanks

I would say that this is by far the best explanation I have found after hours of search on the topic. Congrats!!!

Detailed, Clear and straightforward, all at the same time. Superb..!

Today i understood exact meaning of sparkContext and sparkSession. Thanks a lot, your video helped!!!

@DataSavvy

5 жыл бұрын

Thanks... I am happy that our is useful... Please provide your feedback on other videos of this channel

very clear and simple explanation. Thanks :)

Short and to the point. I like your explanation.

Nicely explained. Thank you!!

Very useful stuff thank you so much

Nice ans it was very clear explaination, thanku sir 🙏

still relevent as of today and frequently asked. The practical on databricks made things crystal clear.

In the older version of spark like spark 1.6 we had the entry point in the spark application created using spark context sc ,but the later version like spark 2.0 spark context has been deprecated and added all the context in one level above the abstraction and added them in spark session which contains all the spark context,spark-sql context,hive context etc..

@shayshaswishes373

5 жыл бұрын

can you please post the vedio , how to add SparkSession.builder in existing code.

Great details

This is what exactly I am looking for. Niw I got to know exact difference between Context and session. Thank you dude. Do you know which is the best certification on spark as a Spark developer?

thanks, clearly explained

Thx a lot, really clear.

Hi Harjeet thanks for the clear and simple explanations of all your videos. Can you upload videos serial wise pyspark tutorial if you have because in most of the tutorials around its starts with creation of spark dataframe using Sparksession and operations on dataframe. You can also suggest any tutorial/blog to read regarding pyspark. Thanks Man....your explanation are great

Nice Explanation.

The best explanation. Congrats

@amitSingh-je8vh

4 жыл бұрын

Congrats or Thanks 😅

Under which scenarios it will be meaningful to have separate sparkContext for each user?

Nice explanation.. Thank you

@DataSavvy

5 жыл бұрын

Thanks for appreciation :)

excellent content in a simple and easy format. Are you providing any trainings on databricks? if so, how do I contact you

Very good information, can you please help in clarifying this doubts: 1. What are included in configurations and properties of different spark sessions of the spark context, and it's effect on cluster 2. What is purpose of spark context and for what spark context is responsible for can you make a video to understand the spark context in full fledge?

Hi , Thanks for nice explanation, Scala works with datasets and python with dataframes and they both generate RDDs as end Results, is my uinderstanding correct

What is the tool or software that is used in this demo of creating sessions. . is it python based or scala ?

Best explanation till date 👍

@DataSavvy

3 жыл бұрын

Thanks Kushagra :)

Thank You

Neat and clean presentation... 😊

@DataSavvy

6 ай бұрын

Thanks a lot 😊

Superb

Great video sir. Just one question. In which node does the spark context and spark session run?

What if same table was being updated by two users at a time..which one would be updated,let's say if we change the datatype of column rename it to same as previous column and store it to table..back again..and by table I mean global table

Thanks for the video bro . I have doubt suppose user 1 is sharing table 1 and user 2 is updating a value for the column in the table 1 will the change also got update user 1 shared table too.

@srikanthchillapalli1037

5 жыл бұрын

It wont happen as user1 and user2 will have isolated sessions from one another and so one user operation doesnt have any impact on other user table. Actually u can have different data for both these users though the table name is same.

What is the advantage of creating multiple spark sessions instead of having multiple spark contexts.

@meenakshisundaraar7267

6 ай бұрын

Think spark context as server And spark session as client

can we call stop on spark session what will happen if we call.

I have a doubt,Can we apply actions directly on RDD with out transformations?

@DataSavvy

5 жыл бұрын

Loading of a file and creating a rdd is also a transformation... So logically you cannot run action without transformation... If you you don't count creating an rdd as transformation, then you can say that you run action action without transformation

In one of my interviews I faced this question. What happens if the executor got crashed unexpectedly which has already processed 50 records. Will it continues from 51 or from 0? Do we have any service that tracks the execution status of a executor?

@rahuldey1182

4 жыл бұрын

yes by creating checkpoint and mentioning the checkpoint folder location in ur program

@indiannewyorkbabies6872

2 жыл бұрын

Doesn’t rdds store those lineage information and when does the executor fails, the rdds gives that info to another new executor and starts the execution…!! Thtsy rdds ade fault tolerant

I had an interview and he asked me on spark process. Could you please explain what happens when the spark job is stopped in midway of execution? Will it start from the beginning or from where it left off?

@DataSavvy

5 жыл бұрын

It depends on how the job was stopped... Do you mean that you killed spark context and stopped or only the job running action had failed... Recovery will depend on this...

@DataSavvy

5 жыл бұрын

It will also depend on if you have any checkpoints in your job

Thanks, Sir for a wonderful video explaining the differences. one qq, when we close/stop a sparkSession which is created from a sparkContext, then this makes other sparkSessions as well get stopped which are created from the same sparkContext?

@rvalusa

3 жыл бұрын

found this, which is weird implementation and apparently a bug in spark - apache-spark-developers-list.1001551.n3.nabble.com/Closing-a-SparkSession-stops-the-SparkContext-td26932.html

By executor do u mean node manager?

Can we have multiple contexts. Could you show with some examples

Nice

so what happens when different users create their own spark context. ( say before spark session was introduced) ? are multiple spark contexts created in such cases ? if yes, what are we gaining by moving the abstraction away from spark context to spark session ?

@ajithkannan522

Жыл бұрын

only one spark context avaialable. You can create multiple sparkSessions under the spark context

We can something SparkSession is similar to sqlContext in spark 1.6

Hi harjeet, can you make a video for how to read hbase table data into spark dataframe, and how to insert spark dataframe into hbase table. is there any spark-hbase connector available for cloudera?

@DataSavvy

5 жыл бұрын

Sure Vamsi... I will add this in my to-do list... Thanks for suggestion :)

Hi harjeet, when this type of use case come any example bcz in batch processing there will one spark session is enough.

@DataSavvy

5 жыл бұрын

When u want your users to have live connection for data analysis etc

Sorry, but I am little confused here. What do you mean when you say every spark context represents one application? When I submit a spark application aren't I am the only user who is attached to that application. How do multiple users make configuration changes to my spark application? Don't they have to submit their own copy of spark application again with config they wish to set? Thank you!

@DataSavvy

4 жыл бұрын

Imagine where u have already running app on cluster. Whatever code needs to be run you are getting at run time... That will be good use case for multiple spark sessions... Drop me an email at aforalgo@gmail.com. will share more content to read on this

@N12SR48SLC

3 жыл бұрын

stackoverflow.com/questions/52410267/how-many-spark-session-to-create#:~:text=4%20Answers&text=No%2C%20you%20don't%20create,in%20the%20same%20spark%20job. Why is it saying 1 SS per application then?

sir can you tell me some about housekeeping executive spark deta. i dont understand spark word. facility company JLL requird he have spark exprience

Hi harjeet..why we are using multiple sparksession instead of multiple sparkcontext....any advantage is there???

@DataSavvy

5 жыл бұрын

It makes it easier to share tables and share cluster resources among your users... As you well know starting different application for each user usually cause cluster contention

I have a doubt, in this scenario if we have 4 spark sessions for a single spark context, when spark context goes down will all 4 spark sessions killed? Please confirm.

@DataSavvy

4 жыл бұрын

Yes Vijay...all spark session will be killed

Very informative content... I've a doubt.... I opened 2 separate spark2-shell using 2 different Ids... When I hit spark.sparkContext in two different terminals, the reference numbers were different. Shouldn't they be same as you explained at the beginning of this video where multiple users shared the same sparkContext object?

@atnafudargaso8374

4 жыл бұрын

same here

@yashdeepkumar2495

2 жыл бұрын

He is talking when working in a clustered environment with more than one worker node I think...usually that will be the scenario. If you open 2 spark shells and check it will create two seperate contexts.I am new to this and pls let me know if you found the correct ans to your question after two years.

@sandeshhegde2847

Жыл бұрын

If you open 2 shells, they're 2 different applications. this video talks about having multiple spark sessions within a single application

you said some thing at 2:39.. i did not get that word.. the sentence is "I can ___ a spark context per user" what is that missing word?

@srikd9829

4 жыл бұрын

I think the missing word is: Spun. its the past tense of the word: Spin. generally the word is used as 'spun a server' means different meanings like "introducing a new server or node, or starting or booting the server or node". This is because, starting or booting the server, spins the hard-disk to load the OS. This is how the word came into practise. hope this helps.

Hi Sir... Very useful topic and very well explained... Thank you, Sir... 1. "Each user can have different spark session... " -- Does this mean -- Different jobs submission...? That means only one Spark Context for the entire cluster which handles many jobs... Right...? 2. Then what about Driver... Is it similar to Spark Context... Only one Driver for all jobs...? 3. In the demo you showed creating many spark sessions in the same job... Each sessions are different within the same job itself... Am I right...? But why creating different sessions in the same code / job...? Thank you, Sir...

Yes it’s nice

I don't think we can create multiple spark context in spark 1.x as well. There is a parameter spark.driver.allowMultipleContexts=true, but this is only used on test scripts but cannot be used to create multiple contexts when coding in IDE. And in spark 2.x we will create multiple spark sessions. Please let me know if I'm wrong.

@murifedontrun3363

5 жыл бұрын

There can be only one SparkContext per JVM process. If there would have been multiple SCs running in the same JVM then it would be very difficult to handle GC tuning up, communication overhead among the executors etc.

@srikanthchillapalli1037

5 жыл бұрын

@@murifedontrun3363 Yes, but here in video the tutor explained that in old versions multiple spark contexts were created. So, I got a doubt how it is possible.

What will happen if driver program fails in Spark. And how to recover it?

@DataSavvy

4 жыл бұрын

It depends on what settings you have for that job. If you have checkpoints and retry enabled. spark will start to recreate those objects... otherwise the job will fail..

@datasavvy thanks for the video. Could you please make a video on where we can practice production level scenarios in pyspark .

@DataSavvy

3 жыл бұрын

Sure Saurav... Let me know if you have list of scenario which u want me to cover. Drop me a email at aforalgo@gmail.com

Please post the python code sheet..

What are the users here ???

Plz

I think you missed many points

@DataSavvy

5 жыл бұрын

Please suggest what are you pointing Towards... Will cover in another video

@dalwindersingh9282

5 жыл бұрын

please suggest, raise few of them.

3 ads in 5 minutes. did not finish.

@DataSavvy

3 жыл бұрын

KZread has increased the ads based on user stats. I unfortunately don't have method to decrease that. Any suggestion is welcomed (except a complete switch off)