i was worked from last 3 years as a data engineer but now at this time i know when we have to use windows function..... thank you Manish Sir....

Sir please discuss some examples for more clarification . Nice explanation sir 😍😍😍

great explaination.

Tum bohot Accha Kam Karta Hai Maqsud Bhai

Amazing. Title should say lecture- 17

NICE

Thank you so much for all the videos including Pyspark theory & practical videos -- Please upload some videos on Python Question for data engineering real use case ke upor , Pyspark ka aur series age ayega kya Manish Bhai-- After SCD 2 Videos -

Will you please teach various optimization techniques and when we use them in real-time?

@manish_kumar_1

Жыл бұрын

Batate rahta hu bahut saare techniques. Aur v aane wale hai abhi alag alag videos me

Df.groupBy("dept","country").agg(sum("salary")).show()

Thanks Sir Ji

Hi Sir, Can you please tell how many lectures still left for spark and when we will be making a project in spark .

@manish_kumar_1

Жыл бұрын

Av to bahut lecture bache hue hai.

Sir, you didn't teach Windows function. directly jumped on joins. Please upload a video of the Windows function too.

@manish_kumar_1

Жыл бұрын

Windows join ke baad aayega don't worry

@KaranSingh-hx8dh

Жыл бұрын

@@manish_kumar_1 Thank you. I didn't got so good explanation like you any where.

emt.groupBy("dept","country")\ .agg(sum("salary").alias("total_salary")).show()

emp_df.groupBy("dept")\ .agg(sum("salary")).show() TypeError: unsupported operand type(s) for +: 'int' and 'str'. How to resolve this error?

@siladityasarkar5143

5 ай бұрын

Please execute this command before executing the mentioned pyspark command: from pyspark.sql.functions import * from pyspark.sql.types import *

a_df = emp_data_countrywise_df.groupby(col("job"),col("country")).agg(sum(col("salary"))) a_df.show()

spark.sql(""" SELECT *, SUM(salary) OVER (PARTITION BY dept) AS New_col, ROUND((salary/New_col)*100,2) AS percentage_of_empSalary FROM employee_dff """).show()

@adarsharora6097

6 ай бұрын

emp_df.withColumn('New_col',sum('salary').over(Window.partitionBy('dept')))\ .withColumn('percentage_of_empSalary', round(col('salary')/col('New_col')*100,2)).select('*').show()

df.groupBy("Dept")\ .agg(sum("salry").alias("Sum_Salary"),\ avg("salry").alias("avg_Salry")\ ) \ .show(truncate=False)

Solution for the last question: emp_df.groupby("emp_dept", "emp_country").agg(sum("emp_salary")).where((col("emp_country")== "india") & (col("emp_dept") == "IT")).display()

@13:33 # Q. group by dept and country columns with sum of salary emp_country_data_df.groupBy("dept","country")\ .agg(sum("salary")).show() +---------+-------+-----------+ | dept|country|sum(salary)| +---------+-------+-----------+ | IT| india| 115000| | sales| us| 60000| |marketing| india| 125000| | sales| india| 90000| | IT| us| 180000| |marketing| us| 95000| +---------+-------+-----------+

emp_df.groupBy("country", "dept")\ .agg(sum("salary").alias("Total_Salary"), count("dept").alias("Total_Count"))\ .show() emp_sql = spark.sql(""" select country, dept, sum(salary) as Total_salary, count(dept) as Total_count from emp_tbl group by country,dept """) emp_sql.show()

emp_df1.groupBy('dept', 'country')\ .agg(sum("salary").alias("Total_Salary")).show()

I know you've already covered it, but @manish_kumar_1 while learning about group by, it would've been more helpful to know why group by is a wide dependency transformation. thanks for the content you put out & congrats for 5.03K...! 🙌🙌🙌

@manish_kumar_1

Жыл бұрын

Did I not mention it?

@sankuM

Жыл бұрын

@@manish_kumar_1 I guess not in this one, I meant ki if you showed Spark UI while applying any join to showcase the shuffle, it would've helped..!

emp_df1.groupBy("dept" , "country")\ .agg(sum("salary")).show()

emp_df.groupBy("country","dept")\ .agg(sum("salary") .alias("Total_Salary"))\ .sort(col("country")).show()

office_df.groupBy("dept", "country").agg(avg("salary")).show()

employee_df.groupBy("dept","country")\ .agg(sum("salary")).show()