Faadu explanation! Finally someone explained why Lazy evaluation actually works betters for Big Data processing. Amazing examples, very nice code! Loved the way you explained each line and each job step by step

wow! very nicely explained. Thank you! :)

Wow! Thank you for such great explanation. ❤

You are an excellent teacher, you make lectures so interesting! ye answer dekar to interview ko sikha denge :D

Very good Bhaiya. Aaj bhaut kuch isme indepth sikhne ko Mila jo phele pata hi nahi tha. Keep it up 👍

Truly impressed Manish bhai. Great explanation as you mentioned already "Itna detail mein kahin nhi milega"

Hi Manish Thanks for explaining the concept in detail by running the code. I have understood the concept of dag ,lazy evaluation and optimization.

Awesome bhaiyaji!!!!!

Thank you Thank you Thank you Manish for this video✨✨✨

Thank you so much for clarifying my doubts 🙏

very well explained...thanks Manish

great video Manish bro

amazing content

@wow...!..v.nice bro

Awesome lecture...thanks a lot!

bohot sahi explanation !

Great explanation sir, really helpful!

very well explained Sir..Thank You.

Maza aa gya lekin video dekh ke... Wahhh❤

thank you ! great teaching style

He has mastered and crunched the spark.

Thanks for the session and Please make video on Spark Lineage

Detailed Explanation. Better than paid lectures.

Very well explained.

I think spark considers 'sum' as action. hence 4 jobs. Greatt series !!!

"Job kaise create hota hai?" ye question pehle kabi clear nai hua tha but thanks Manish for clearing this out and add on was lazy evaluation understanding. 👍

@vsbnr5992

Жыл бұрын

NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

I have confusion? read and inferSchema are typically used with Spark's DataFrame API, which is part of Spark SQL. They are not transformations or actions ,part of the logical and physical planning phase of Spark, which occurs before any actions are executed

bhai gajabe padate ho yar maja hi a gya yar

Sir ur teaching is good

one Action one job is true for rdd api only. one action in dataframe or dataset can lead to multiple actions being generated internally. or sue to adaptive query executions as well multiple jobs are created in databricks which you can see using describe method

Awesome. By the way, do we know why its creating 4 Spark Jobs instead of 3 ?

Do you have theory notes for spark?

Is there any cheat sheet to know what all are transformations and actions, like read is a action whereas filter is a transformation?

where is the practical session link ?

Hello Manish. When you mentioned the sql query gets optimized by spark. Just wanted to know will it help improve the execution time if we use the optimized query in our code itself so that spark need not do it ?

@manish_kumar_1

7 ай бұрын

Spark optimization is very limited. So as a developer we should write optimized code to run our process faster

Bro Plese add summery at the end.

any idea now, where the 4th job came from?

Databricks community ko kaise use karege, Spark kaise setup karege databricks ke sath. Please ye bata dijiye so that code write kr sake

Hi Sir..jab main yeh code run kar raha hoon I am getting error as--name 'col' is not defined isko kaise solve kare?

Sir Read an inferschema to Transformations hai na and not actions? to inki alag jobs kyu bani hai?

Sir please ye series thodi jaldi finish karo taki ham interview de sake mene apki puri channel dekhli hai Thank you

where to find the files ? I just have started the course

is lazy evaluation consider the partition also like after we have applied orderby on salary col and now we want to show only first two highest salary so will lazy evaluation also works here spark will process only that partition which has these two salary records or it will process all partitions and then extract first two highest salary record for us

@manish_kumar_1

Жыл бұрын

Yes until you write .head(2) for 2 highest record your process will not start although in backend it will create DAG.

Directly connect with me on:- topmate.io/manish_kumar25

I too got confused as to why sometimes number of jobs as more or less than Actions. Try clearing the state using menu option run ->clear state and option and then run the cell again which has code from reading of file till all the things you want to perform . I think Data bricks intelligently stores state of system and later when you run same read command the Jobs count might not match I tried this and it seems to be working

@jatinyadav6158

5 ай бұрын

Jobs count is right it is 4 only because sum() function is an action, which I guess Manish missed by mistake. Btw @Manish thank you so much for the amazing course.

@deepanshuaggarwal7042

2 ай бұрын

@@jatinyadav6158 If 'sum' is an action then why it didn't create a job before adding 'show' codeline ?

@jatinyadav6158

2 ай бұрын

@deepanshuaggarwal7042 yes sum is an action, I am not sure why it didn't show a job earlier

Sir apko samjh aya ki ek action hit karne par 2 jobs kaise create hui?

I am trying to find , why 4 jobs are generating here although we have provided only 3 actions

Hi Manish Bhaiya, in the code snippet you told there are three actions in this applications(read, infer schema and show) but in spark ui there are 4 jobs created ,can you please explain this.

@manish_kumar_1

10 ай бұрын

1 job skip hua hoga. Agar data Kam hai to explain karke dekhiye 3 aana chahiye

Where he explained about fligh data download

Why do we have 4 jobs when we are using only 3 actions in the application??

well explained.....Sir could you please clarify, 3 actions and 4 jobs created?

@manish_kumar_1

11 ай бұрын

Aapke 3 actions me 4 jobs create hue hai? Aapne show use kara hai? And aap apna code v paste kar dijiye comment section me

@hazard-le7ij123

9 ай бұрын

@@manish_kumar_1 Aapne jo code likha hai usme bhi 4 jobs create hue hain. Can you explain that? Below is my code and same thing is happening. 4 Jobs are getting created. Stage is getting skipped but why do we have an extra job with 4 diff Job Ids? from pyspark.sql import SparkSession from pyspark.sql.functions import * spark = SparkSession.builder.master('local[5]') \ .appName("Lazy Evaluation internal working") \ .getOrCreate() flight_data = spark.read.format("csv")\ .option("header","true")\ .option("inferSchema","true")\ .load("D:\\Spark\\flight_data.csv") flight_data_repartition = flight_data.repartition(3) us_flight_data = flight_data.filter(col("DEST_COUNTRY_NAME")=='United States') us_india_data = us_flight_data.filter((col("ORIGIN_COUNTRY_NAME")=='India') | (col("ORIGIN_COUNTRY_NAME")=='Singapore')) total_flight_ind_sing = us_india_data.groupby("DEST_COUNTRY_NAME").sum("count") total_flight_ind_sing.show() input("Enter to terminate")

In my case same code has created 5 jobs?I have import col so it will also be treated as action and new job will be created is it so?

@manish_kumar_1

Жыл бұрын

Can you please paste your code in comment section

@avanibafna6207

Жыл бұрын

@@manish_kumar_1 from pyspark.sql.functions import col flight_data=spark.read.format("csv")\ .option("header","true")\ .option("inferSchema","true")\ .load("dbfs:/FileStore/tables/flight_data.csv") flight_data_reparition=flight_data.repartition(3) us_flight_data=flight_data_reparition.filter("DEST_COUNTRY_NAME='United States'") us_india_data=us_flight_data.filter((col("ORIGIN_COUNTRY_NAME")=='India')|(col("ORIGIN_COUNTRY_NAME")=='Singapore')) total_flight_ind_sing=us_india_data.groupby("DEST_COUNTRY_NAME").sum("count") total_flight_ind_sing.show() (5) Spark Jobs Job 22 View(Stages: 1/1) Job 23 View(Stages: 1/1) Job 24 View(Stages: 1/1) Job 25 View(Stages: 1/1, 1 skipped) Job 26 View(Stages: 1/1, 2 skipped) flight_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] flight_data_reparition:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] us_flight_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] us_india_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] total_flight_ind_sing:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, sum(count): long] +-----------------+----------+ |DEST_COUNTRY_NAME|sum(count)| +-----------------+----------+ | United States| 100| +-----------------+----------+

Where I get CSV file

waiting for ur next vidio...

@AmitSharma-ow8wm

Жыл бұрын

@@rampal4570 is it true bro

@manish_kumar_1

Жыл бұрын

Aaj aa jayega

@vsbnr5992

Жыл бұрын

@@AmitSharma-ow8wm NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

Bhai iske phle wala video dedo, mujhe mil nhi rha hai..

Bro, you are explaining superbly but why don't you explain in English. Everyone doesn't know Hindi. I don't know Hindi but watching your videos to understand the concepts but not getting it fully because it was explained in Hindi.

could you provide notebook link?. It will good for hands on

@manish_kumar_1

Жыл бұрын

I don't provide notebook or pdf. Take notes and type every line of code by yourself. This will give you confidence

@sankuM

Жыл бұрын

@@manish_kumar_1 this is indeed really great point! However, if possible, do share your own reference material for our benefit! Thanks! This series is really helpful, I've 4+ YoE in DE but never tried to go into spark internals, now while interviewing for switch, I'm definitely going to utilize all this! Keep 'em coming!! 🙌🏻👏🏻

ek code likha hai using spark .usse dekhkar kese bataya ja sakta hai ki wo code 'action' hai ya 'transformation' hai.

@manish_kumar_1

4 ай бұрын

Aapko google karke pata karna chahiye ki kon kon se actions hai. Rest are transformation

How many more videos to come on Apache spark??

@manish_kumar_1

Жыл бұрын

Around 20-30. It's just beginning of spark

Aj apne interview questions bataya Nehi.

@manish_kumar_1

Жыл бұрын

Basic questions would be there. Like what is DAG and what is edges and vertices in it.

Because of group by jobs is created

@manish_kumar_1

2 ай бұрын

Nope

Why it is called acyclic?

@manish_kumar_1

11 ай бұрын

Because it doesn't make cycle. If it's get into cycle or you can consider it as a circle then it will run endlessly

Sir csv read karne pr two jobs kaise create ho raha hai, read only one action call kr rahe hain. If possible please clearify

@manish_kumar_1

7 ай бұрын

Aur inferschema v use kiye honge. Isliye aa rha hoga

@khurshidhasankhan4700

7 ай бұрын

@@manish_kumar_1 thank you sir, can you please share the action list how many action hai spark me, if possible please share sir

Hi manish Bhai, Could u provide the flight data file.

@manish_kumar_1

Жыл бұрын

Kisi ek video me bataya tha. Please follow all videos in sequence

please make a video in Windows function

@manish_kumar_1

Жыл бұрын

Sure

@ajaysinghjadoun9799

Жыл бұрын

Sir also considers spark 5s problems.spill, shuffle, storage, etc

Where can I get the flight_data.csv file? Please help.

@shivakrishna1743

Жыл бұрын

Got the file, thanks

bhai iski file de do iss lecture ki

bhai please vo extra job kese create hua ye doubt clear krdo

@manish_kumar_1

Жыл бұрын

Aap sql tab me jaake dekho. Kitne jobs skip hue hai. And share me your code and screenshot of the sql tab on LinkedIn or Instagram.

@rohitgade2382

10 ай бұрын

@@manish_kumar_1 abe chutiya tere video ka bol Raha he wo 😂

Lec 7 Tak view kr liya Jo aap code dika rahe hai usko kise databricks and pyspark

@manish_kumar_1

9 ай бұрын

Aapko practical and fundamentals sath me dekhne hai. First video me hi bataya tha

NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

@manish_kumar_1

Жыл бұрын

Seems like your df is not defined

@vsbnr5992

Жыл бұрын

@@manish_kumar_1 ok working now thanks

csv kaha milegi sir ?

@manish_kumar_1

3 ай бұрын

2 playlist hai. Parallely dekhiye. Practical wale me data milega description me usko copy karke save Kar lijiye as csv

us_india_data = us_flight_data.filter((col("ORIGIN_COUNTRY_NAME") == 'India') | (col("ORIGIN_COUNTRY_NAME") == 'Singapore')) Ismein error bata raha hai : NameError: name 'col' is not defined Isko kaise define kare?

@AliKhanLuckky

Жыл бұрын

Col ko import karna padenga voh ek function hai toh import functions karo I think so

@manish_kumar_1

Жыл бұрын

Correct "from pyspark.sql.functtions import *"

@vsbnr5992

Жыл бұрын

@@AliKhanLuckky NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

@vsbnr5992

Жыл бұрын

@@manish_kumar_1 NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

Guru ibutton khi nhi aaya na idhr na udhr😂😂😂

@manish_kumar_1

9 ай бұрын

Bhaari mistake ho gaya 😂

Hi sir.. In which video series u have explained about schema @manish_kumar_1