Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/

@rishigc

3 жыл бұрын

Hi, your videos are very interesting. Could you please provide me the URL of the video where you discuss Spark UI ?

I have to say that your explanations are better than the actual trainings provided by Databricks/Partner Academy. Thank you for your work!

one of the best if not the best video I've seen explaining joins in spark. Thank you!

Finally a clear explanation to shuffle in Spark

Very neat and clear demo, thanks.

Short, informative and easy to understand. Thanks.

BEST EXPLANATION EVER!!! THANK YOU!!!!

When we'll see the next part of this video on Tuning the join operations ? Eagerly waiting for that.

Thanks for sharing the information, very few people knows the internals of the spark

Love you sir . I joined the premium

The clarity is exceptional

Very clear and crisp..

Amazing sir!!!!!

Awesome easy explanation thanks!

Thanksssss!!!!

Concept clear👍

really good one, thanks!!

Wow what an explanation ✌️✌️

@sudeeprawat5792

3 жыл бұрын

One question i have while reading the data in dataframe. Data is distributed across the executor on the basis of algorithm or randomly distributed across executor??

I am still confused about what happens in the map phase.Can you explain this "Each executor will map based on the join key and send it to an exchange. "?

Thanks for the nice video. QQ: When I read from S3 with a bunch of filters on (partitioned and non-partitioned) columns, how many Spark RDD partitions should I expect to get? Would that be different if I use DataFrames? Effectively, All I need to achieve is to read from a massive dataset (TB+), perform some filtering, and writing the results back to S3. I'm trying to optimize the cluster size and number of partitions. Thank you.

one point you mentioned that if the partitions from both the dataframe is present in the same Executor then shuffling doesn't happen. but as per the other sources one task work on single partition hence even if we have required partition on the single executor still they are many partitions of the dataframe which contains the required join key data e.g. ID=100. Then how join is performed in this case.

can you please share video on Chained Transformations?

do you have a video for sort merge join?

Nice

What if the number of shuffle partitions is too much bigger than the number of nodes ? In the company I've just joined, they run the spark-submit in the developer cluster using 1 node, 30 partitions, 8GB each and shuffle partitions = 200. Maybe this 200 partitions can slow everything. The datasets are by the order of hundreds of GB

I'm not quite sure if I understood something: an exchange / shuffling in Spark is always basically a map-reduce operation ? ( so it uses the HDFS ?) Am I mixing things or am I right ? Thank you so much!

The courses are quite interesting. Can I get the order in which I an take Apache Spark courses with my monthly subscription.

@ScholarNest

3 жыл бұрын

Follow the playlist. I have four Spark playlists. 1. Spark programming using Scala. 2. Spark programming using Python. Finish one or both depending on your language preference. Then start one or both of the next. 1. Spark Streaming in Scala 2. Spark Streaming in Python. I am hoping to get some more playlists in near future.

What would happen if the shuffle.partition is set to > 3 but we have only 3 unique keys for join operation? please help.

Keeps repeating himself it’s annoying