Haven't seen better videos for databricks on youtube, your dedication towards teaching each topic in deep is commendable brother, God bless

This DE playlist is one of the best videos I have ever seen. Keep up the good work, Manish. Thank you for this 🙌

i have been following your course since last 10 days ..It was osm.

Great.. please continue

Tq so much Manish❤, clear cut explanation

Excellent..... please continue...

Best video on partition on KZread

very nice explanation sir....and thank you so much for taking out your precious time to make this video for us :)

greatbro acaha samjha rahe ho keep on

Very nice. Mai apka playlist use kr k apne next interview ke liye prepare kar rha hoon

Hey Manish, thanks a lot for these tutorials, I have seen almost all data engineer play list but the way you explained each topic so deep, I really appreciate and because of your video I was able to understand and ultimately cracked interviews. Thanks a lot and looking forward for more tutorials on this.

@manish_kumar_1

Жыл бұрын

Congratulations. BTW which company did you crack?

@ChandanKumar-xj3md

Жыл бұрын

@@manish_kumar_1 in medical domain.

@SHUBHAMKUMAR-zv8ru

Жыл бұрын

@@ChandanKumar-xj3md Congratulations Chandan, btw can you please tell the package how many years of experience do you have in Data Engineering domain

Good job buddy .. even the minute information performance impact also you are giving in details that's really great.. if you can do series in English version also that would be really great...

sir please partition pruning and bucket pruning me confussion ho raha hai.kindly clear kijiye

Hi Manish, Could you please make a video on an introduction speech where 3.5 years of data engineering exp is involved with AWS cloud technology and give one or two examples of data Quality pipeline from IOT to s3 also include day 2-day activity. Thanks

Spark series pura khatam kar lijiye fir design interview ka thora overview aur tips and tricks dena... Ya fir soch lijiye Walmart ya Product me jo managerial ya fir dusra round hota hai design wala, uska kuch question, resource, tips and tricks bataiyega to kaafi help ho jayega. Aur haan thank you so much spark series ke liye kaafi acha kaam kar rahe hai aap, dil se shukriya

@manish_kumar_1

Жыл бұрын

Sure

Hi manish how can we filter if we have two delimiter like | pipe and tab \t delimiter in our source file please explain this topic also thanks

Hi Manish. For incremental load how partition and bucketing will work ? Will it recreate partition or buckets depend on incoming data ? Like partition = India already created and now new data for India is coming incrementally then what will happened to the partition ?

Thanks for this awesome explanation, this make senses to me now, however I am still figuring out if we could use partition and bucket on same data frame. As the data frame I have has low cardinality thus used partition, however for one of the key-value the count of records are too high. For example, using the same employee data set from the video... partition is on Address /Country and for INDIA the record count is in Millions where as for other Country says its in thousands... now because one of the partition data got skewed. So should I use Bucketing or what approach should I use. Please suggest. Above said question was asked to me in one of the interview.

@soumyaranjanrout2843

7 ай бұрын

@mayanktripathi4u I think bucketing might be better approach because as we know if specific partition will be extremely large then it will reduce the query performance and it will expensive operation. Otherwise if we know about the data clearly then we should look for other column or combination of column with low cardinality and then apply partitionby. And we can also apply both partitionBy on "address" and bucketingBy on "gender" which may reduce the data size for each partition(but I am not sure whether this approach will be better or not) Correct me if I am wrong because I am a beginner.

bhai tumne 2 videos mai yeh aadhar card data waala bataya hai, but dono video mai concept video se cut ho gya hai aadhe mai se, pls bataiye ki hum aadhar ko 10000 se kyu divide kr rhe hai. each bucket mai kitna data hoga and kitne no. of bucket lenge in case of bucketing on aadhar data.

kya isi ko small file issue kehte hai ? wo jo 200 *5 = 1000 files create ho jata hai ?

The dataframe in a partition doesn't have column that the data is partitioned on. Is this normal?

do you have the English version of lessons

bhai agar ham partiton kartai hai from original file in HDFS, then we have 1 full file and other 3 files due to partition. means same data 2 times in HDFS. Yahi concept hai na?

Directly connect with me on:- topmate.io/manish_kumar25

Hi Manish, I did not understand the point where you said repartition(5) and bucketBy(5) will produce 5 files However in prev examples you said that it will generate 200 * 5 files ? Can you please explain this ??

22:50 what if we repartition it to 5 and we are not able to store all the data on those executors

Manish Bhaiya aapne repartition(5).bucketBy(5, id) ki isse sirf 5 hi bucket banenge? I know repartition is a runtime re partitioning but agar repartition kar k kuch store kar rhe hai toh will it affect anything? Please repond. 200 * 5 = 1000 5 * 5 = 25 buckets banenge right?

One interview question, demonstrate how we can perform "spark-broadcast join" ? Not sure how to do this

@manish_kumar_1

Жыл бұрын

I will teach while learning join

sir if dataframe is only for structure and semistructure .so what should we use in case of unstructure data.

@manish_kumar_1

Жыл бұрын

Either through rdd or convert your unstructured into semi structured or structured by finding some pattern in your data

sir ek live doubt session rakhiye

@manish_kumar_1

Жыл бұрын

Bilkul

Bhai, How many remaining topics/videos are there in this spark series to complete it?

@manish_kumar_1

Жыл бұрын

20

After repartition(5), 5*5=25 buckets should get created right?

@manish_kumar_1

11 ай бұрын

Do you have 5 executor?

sir japan mein female nahi thi samjh gaya lekin japan male kyun nahi show ho rahe

Hi Manish, I couldn't understand how bucket avoided the shuffling here?

@manish_kumar_1

10 ай бұрын

Agar same id's same bucket me honge from both the df then you won't need to shuffle the data.

bhai i'm getting this error "AnalysisException: Partition column `address` not found in schema struct" even though i tried to load the file again... however bucketby simply works but not partitionby

@manish_kumar_1

Жыл бұрын

Share me the code and schema in comment section or on LinkedIn

if you want to decrease the number of partition then u told should use colacese but while explaning you mentioned repartition ? and you mentioned that repartition to 5 and buckets are also 5 is it a good practice to have num of partitions = num of buckets??

@manish_kumar_1

9 ай бұрын

Repatriation can increase or decrease number of partitions but coalesce can only decrease. Coalesce doesn't make sure you will have evenly distributed data.

@khaderafghan1085

9 ай бұрын

@@manish_kumar_1 and if we have number of partition != number of bucket is it fine?

sir what is difference between pyspark and pandas in apache spark?

@manish_kumar_1

Жыл бұрын

Pandas doesn't work in distributed manner where as pyspark does

@SHUBHAMKUMAR-zv8ru

Жыл бұрын

If you are dealing with large dataset, pyspark is better

bhai bihar me kaha se ho aap?

@manish_kumar_1

Жыл бұрын

Patna

What is repartition? I have seen you using repartition(3).

@manish_kumar_1

Жыл бұрын

Repartition 3 means your data will be divided in to 3 part. Earlier it may have 20 part but Repartition 3 will make sure that you have 3 partition and also it will be of almost same size. No skewness in data

same location pe partition karke file daalenge to error kyun throw karta hai?

@manish_kumar_1

Жыл бұрын

Overwrite karne par v aa rha hai?

bro, i am confused about 200 tasks and 5 bucket causing 1000 buckets , what do you mean by 200 tasks here?

@manish_kumar_1

Жыл бұрын

Watch stages and task wala video

bhaia agli video kb aaegi?

@manish_kumar_1

Жыл бұрын

It will take sometime

@Watson22j

Жыл бұрын

@@manish_kumar_1 ye spark series kab tak khatam krne ka plan hai aapka??

Sir ek doubt h ..ki ye data write kha pr ho rha h ......driver pr ya executer pr 😢

@manish_kumar_1

8 ай бұрын

🤔 spark to data bas process karta hai. Write to koi storage system like s3, hdfs, local, any server etc par hoga

@akashprajapati6635

8 ай бұрын

@@manish_kumar_1 sir aap abhi kha ki jio Branch me working ho 🙄

@manish_kumar_1

8 ай бұрын

@@akashprajapati6635 Gurgaon

Bhayya..u said for 200 tasks..if we want 5 buckets it will be 200×5 =1000. My doubt here is won't it take 40 records in each bucket?

@manish_kumar_1

Жыл бұрын

Number of record can or can't be same in each bucket. Based on pmod hash it send to respective bucket.

@rameshbayanavenkata1305

10 ай бұрын

@@manish_kumar_1 sir , please explain with an example where no. of records can't be same in each bucket.

AttributeError: 'NoneType' object has no attribute 'Write' How I can solve this error df.write.format("csv")\ .option("header","true")\ .option("mode","overwrite")\ .option("path","/FileStore/tables/partition_by_address/")\ .partitionBy("address")\ .save()

@manish_kumar_1

3 ай бұрын

Kyunki aapka df jo hai usko aapne .show karke rakha hua hai. .show hone ke baad aapka df None ho jayega and NoneType error aayega

Bar bar data write karte waqt object has no attribute 'write' q aa ja raha

@manish_kumar_1

11 ай бұрын

Kuch code snippet bhejiye. Aisa error kv aaya nahi hai mere ko

bhai ya 200 task wala concept nahu samagh aaya, ya 200 task ka kya matlab hua, in which sense?

@manish_kumar_1

10 ай бұрын

By default spark creates 200 partition when there is data shuffling involved like join or repartitioning or group by etc. And all the data is moved into either of partition based on pmod and murmur3 hashing

@praveenkumarrai101

10 ай бұрын

@@manish_kumar_1 ok bro, thanks

bhai aur details me samjhao.samajh me nahin aa raha hai. bhale hi apne code likh ke samjhaya hai. lekin samajh me nahin aya. Pehle to aapne Optimization technique kyun chahiye , wo nahin bataya. Jab hum terabytes/Petabytes of data ki baat karte hain. To us samay optimize kyun karna chahiye . Databricks toh computing resources provide kar raha hai. Aur jab ki spark large data handle karne ke liye banaya gaya hai. Phir optimization kyun? Aap ne video me kahin show nahin kiya ki agar hum x/y Terabytes of data read karenge without partitioning/bucketing kya problem ayega aur hum partitioning/bucketing concept use karenge to kya changes ayega. Mene apna feedback dia. Mene medium pe articles padhe, linkedin pe articles padhe, Guide to spark book bhi padha. lekin samajh me nahin aya. Jab me youtube me transformations & actions in spark search kia to bahut channels aya including apka channel . lekin jitne bhi channel aya wo mene ek ek karke dekha including yours. parantu spark ka ye jatil concept koi bhi aasan tarike se aur details me nahin bata saka. aap bhi nahin. Ek aur video ho sake to banaiye. Lage 1 hr/2hr/3hr lekin action/transformation pe jo cheez itna details me aap bataiye ki koi youtuber aaj tak nahin banaya hoga.

Hi Manish Ji, I have executed below code reading_file_for_write_df.repartition(3).write.format("csv").mode("overwrite").option("header","true").option("path","/FileStore/tables/bucketby_id/").bucketBy(3,"id").saveAsTable("bucket_by_id") I am expecting it should part data into 3 part files, but I get 7 part files. Ypu said when we have multiple taskid it create multiple part files in that case. Without giving repartition value it gives 3 bucket files but with repartition(3) it give 7, why that sir? Could you please explain more on this. Also on .mode("overwrite") is it the correct way to pass ? I think by mistake you have given it wrong in video. Could you please confirm. Thanks O/p : FileInfo(path='dbfs:/FileStore/tables/bucketby_id/part-00000-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-52-1_00000.c000.csv', name='part-00000-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-52-1_00000.c000.csv', size=89, modificationTime=1710875137000), FileInfo(path='dbfs:/FileStore/tables/bucketby_id/part-00000-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-52-2_00001.c000.csv', name='part-00000-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-52-2_00001.c000.csv', size=87, modificationTime=1710875137000), FileInfo(path='dbfs:/FileStore/tables/bucketby_id/part-00000-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-52-3_00002.c000.csv', name='part-00000-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-52-3_00002.c000.csv', size=59, modificationTime=1710875137000), FileInfo(path='dbfs:/FileStore/tables/bucketby_id/part-00001-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-53-1_00000.c000.csv', name='part-00001-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-53-1_00000.c000.csv', size=106, modificationTime=1710875137000), FileInfo(path='dbfs:/FileStore/tables/bucketby_id/part-00001-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-53-2_00002.c000.csv', name='part-00001-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-53-2_00002.c000.csv', size=90, modificationTime=1710875137000), FileInfo(path='dbfs:/FileStore/tables/bucketby_id/part-00002-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-54-1_00000.c000.csv', name='part-00002-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-54-1_00000.c000.csv', size=143, modificationTime=1710875137000), FileInfo(path='dbfs:/FileStore/tables/bucketby_id/part-00002-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-54-2_00001.c000.csv', name='part-00002-tid-2977452082670177534-05219f91-66cc-4958-bbcc-b89d4d251389-54-2_00001.c000.csv', size=60, modificationTime=1710875137000)]

Directly connect with me on:- topmate.io/manish_kumar25