Don't forget to hit that Subscribe Button for more amazing content :)

@0to1_war

10 ай бұрын

Get ready with project.

@ashishkumar-ns3sg

10 ай бұрын

Please also upload GCP data engineering End-to-End project

@cs_soldier5292

10 ай бұрын

You deserve much more than 1000 buddy. I learn so much from your channel

@BabaiChakraborty-ss8pt

10 ай бұрын

lets get that.

@user-lp3qe9jj3m

10 ай бұрын

Are you from Gujarat?

Thanks for actually explaining spark, instead of making general comments or assuming we know the basics. Great video. Thumbs up, subscribed.

That was an extremely good explanation. Not only explained the theory but also practical examples.

best explanation on spark in 10 minutes. its like feynman explaining physics. excellent job!

Such a nice content! What a man you are! You have covered everything in spark in just 10 mins. I wonder how you made this video and the effort u put in to make this video is wonderful. Thank you for sharing nice content in such a simple manner!!

Apache Spark -- explained core concept in such a simple language.. Wonderful job 👍👍👍

You are doing a fabulous job of making Data analytics so easy for everyone. Thank you so very much. God bless you!

I usually be off from content titled learn/master/excel X in Y minutes. would have definitely done the same had I came accross this by myself. Watched it only because my frd shared to me. Now I feel that I am lucky after watching this as I could wrap my head around SPARK. Subscribed.

I never knew I could recall so much in just under 10min... Wonderful content and well explained keeping it simple...

@DarshilParmar

10 ай бұрын

Glad you liked it

A excellent video on Apache Spark. Covered almost everything. Very helpful video to the beginners like me.

Thank you very much and it's a very nice primer to refresh once the concepts. Thank you for your contributions 👍

I really understand the software really quickly, thanks man

You explained the content simple and clear. Thank you for this video.

I was waiting for this. Please share an end to end project using Spark.

@DarshilParmar

10 ай бұрын

Yes

@ImranKhan-jn6zh

10 ай бұрын

Waiting for the same...right from spark installation on local as well as on cloud platform

@sumant542

10 ай бұрын

Please upload ASAP.

@nirakarsahu4844

9 ай бұрын

Yes, if possible can you please also share using pyspark as well..

Very well explained! Thank you!

Great introduction. Thank you so much.

Wonderfully explained in just 10 mins.

Thank you for this video, I liked it: simple, clear, and short! Perfect :)

Excellent video Darshil. Clear and concise! Subscribed!

Amazing content, keep up the good work, and thank you for the brilliant presentation. You really present topics precisely, simple-to-understand.

Amazing explanation!! Thank you!

Awesome video mate! well done.

Just Amazing😇Thank you

Very well explained , thank you very much

Nice Explanation, Thank you

Great video! Thank you

Thanks alot.

Wonderful video you explained everything perfectly

A very very good video. Thanks, you are doing a really great job!

Wonderful summarize!

THanks. very helpful.

I didn't understand apache spark since my undergraduate until I found this gem.

Great explaination

Thanks for this.Currently reading spark definitive guide.Looking forward to full tutorial

@DarshilParmar

10 ай бұрын

Coming soon!

Nicely presented and explained.

superb man.. didn't waste the time.. great explaination..

Great video, thanks :)

The best Spark tutorial I have ever gone through. Thanks a lot Darshil.

@DarshilParmar

3 ай бұрын

Wow, thanks!

You nailed it Bro in just 10 mins 😊

It was really helpful. Thanks.

Best tutorial ❤❤all in one

Excellent Explanation...

Thanks Darshil

Super explanation bro, I got many answers in one vedio 🥳🥳

Nice one 👍

Alright, but need a full tutorial on this topic, if you can.

@DarshilParmar

10 ай бұрын

Working on it!

@anasshaikh3885

10 ай бұрын

​@@DarshilParmarthank you please upload it ASAP

@ajinkyagaikwad4181

8 ай бұрын

​@@DarshilParmarplease upload

Very brief and informative video

Great content buddy 💯💯 any specific resources to go with spark as I am reading the definive guide i find it bit overwhelming any course??

Explained well

Really productive video.

This is a great explanation

Very well explained😊

Hi Darshil your videos are very informative. I have one request to make please if possible can you upload course on end to end project using databricks snowflake informatica and airflow or can you please make data engineering course on these technologies as it is in demand skill now a days. It will be helping a lot of us who are aspiring to become data engineer.

Fantastic explanation… 👏👏 the way you take your audience through the flow of explaining these concepts is very effective👌

@DarshilParmar

2 ай бұрын

Thanks a lot 😊

Awesome ❤

Thanks for the content

Understood video very well. Without any prior knowledge of apache spark

@DarshilParmar

10 ай бұрын

Glad it was helpful

Thanks Sir!

Waiting for full course from you apache spark

Impressive explanation of spark. Making it easy for every beginner to understand.

@DarshilParmar

2 ай бұрын

Glad it was helpful!

Hyy darshil, I've sentiment analysis code that I'm running in dataproc of gcp. Dataset is large enough so I first store it in df, process with our code, then store the results in the df. So I reduced the processing time drastically. But after that when I want to store that results in a file so that we can use it. It takes a lot of time. We tried saving the file but it writes row by row, takes huge amount of time, tried storing with converting df into pandas df, tried storing df directly into cloud sql database still it takes large amount of time. So how do I save the results df into any file which I could access then. Please share the solution with details as possible. Thanks!

such a clear and crisp video Thanks a lot Darshil for this Please share an end to end project using Spark.

@DarshilParmar

10 ай бұрын

Thank you, I will

Thanks Darshil for this great video, in the video you mentioned a concept "spark dataframe", does it euqal to the "RDD" that you talked about?

As simple as that.. Liked

Thank😊

Good knowledge sharing skills

I learned a lot from the video. It was really helpful and interesting.

@DarshilParmar

10 ай бұрын

Glad it was helpful!

plz make video on unit testing in pyspark

that was just wow

Amazing video. Please share the project doc😊

excellent video

Please bring more videos on spark

you can run spark on databricks as a single computer (still hosted on cloud) right

Plz upload video on debugging in pyspark

ok, thank you very much! where's the code pyspark ?

Pls start taking classes for data engineer i am ready to enroll 😅 or pls suggest some best slass or program to learn

Super🎉 Waiting for full tutorial

@DarshilParmar

10 ай бұрын

Very soon

when is the apace spark learning series course coming?

Thank you Sir !

@DarshilParmar

10 ай бұрын

Most welcome!

Very Good

Please provide a data engineering full stack course on your website

Darshil Sir, I had a query regarding Memory Management concept of Spark. As per my understanding, Spark uses it Execution memory to store intermediate data in execution memory which it shares with storage memory too, if needed. It can also utilize the off-heap memory for storing extra data. 1) Does it access the off heap memory after filling up storage memory? 2) What if it fills up Off heap memory too? Does it wait till GC clears up on-heap part or spills the extra data to disc? Now, in a wide transformation, Spark either sends the data back to disc or transfer it over the network, say for a join operation. Is the part of data sending data back to disc same as above where Spark has the option to spill data to disc on filling up on-heap memory? Please do clarify my above queries, sir. I feel like breaking my head as I couldn't make a headway through it yet even after referring few materials.

@DarshilParmar

10 ай бұрын

In Spark, memory management involves both on-heap memory and off-heap memory. Let me address your queries regarding Spark's memory management: 1. Off-heap memory usage: By default, Spark primarily uses on-heap memory for storing data and execution metadata. However, Spark can also utilize off-heap memory for certain purposes, such as caching and data serialization. Off-heap memory is typically used when the data size exceeds the available on-heap memory or when explicit off-heap memory is configured. It is not used as an overflow for storage memory. 2. Filling up off-heap memory: If off-heap memory fills up, Spark does not automatically spill the data to disk. Instead, it relies on garbage collection (GC) to free up memory. Spark's memory management relies on the JVM's garbage collector to reclaim memory when it becomes necessary. When off-heap memory is full, Spark waits for the JVM's garbage collector to reclaim memory by cleaning up unused objects. Therefore, if off-heap memory fills up, Spark may experience performance degradation or even out-of-memory errors if the garbage collector cannot free enough memory. Thanks, ChatGPT

When is the azure data engineering project is coming Really waiting for that one❤

Bro,that DE course link you have provided is not working.Unable to open that

@Darshil Parmer - Great content... What are the pre-requisites for the paid courses that you have? I would like to enroll for "Python + SQL + Data warehouse (Snowflake) + Apache Spark". Does this have engaging content and also whats the duration?

@DarshilParmar

Ай бұрын

Hi, Combo pack starts from basic so nothing required as such You get lifetime access with around 45-50 hours of content Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b93447 USE CODE: COMBO50 for 50% off

awesome :) ,

Hi Darshil kindly help me on this I am getting the below error after installation of "> pip install databricks-cli" > databricks --help > 'databricks' is not recognized as an internal or external command, operable program or batch file.

Hi I'm fairly new to spark!Question: as you explained spark is faster for processing! Does it utilize hadoop as storage/source or entirely replace hadoop and access data from source directly to process and feed data into something like a data warehouse?

@soumyaranjanrout2843

5 ай бұрын

Spark does not have it's own storage layer. But it can integrate with multitude of data storage layers like Database(RDBMS and NoSQL) or any file system(Flat file or Distributed file system)including Hadoop's file system also i.e. HDFS. Upon processing the data in Spark you can feed into Datawarehouse or any other Data storage system for your use cases.

bhai shaandar

Very helpful..

@DarshilParmar

10 ай бұрын

Glad it was helpful!

Can you please make a tutorial on data engineering usingg python and sql

Terrific explanation. Just one feedback which is not related to your tech knowledge. You need to learn when to use the word “The” and when not to use it.

@DarshilParmar

17 күн бұрын

Thank you for the feedback, most of the time when recoding video I lose track of grammar and focus on conveying information

Hi Drashil, which IDE you used for processing spark code in batch mode and which is suitable for reading or writing data from various sources?

@thedataguyfromB

5 ай бұрын

Python + Java + spark + PySpark + PyCharm Installation Step by step kzread.info/dash/bejne/nINt2byAdda2gtI.htmlsi=aEZ-AM-pGUmaEEVF

Hi, Can you suggest any udemy course to learn pyspark ?

I am absolute noob to this, but how is it any different than writing to distributed databases? From what I understand , is it because of the coordination required across different cluster nodes

make a video on API vs SDK.

very, very good ;)

@DarshilParmar Do you offer consultation? i need help with a project to convert full load pipelines into incremental?

Superb one! Can we expect full tutorial over spark!?

@DarshilParmar

10 ай бұрын

Yes, coming soon!

@hritikapal683

10 ай бұрын

@@DarshilParmar Ah nice then 😍

crazy bhai pura unit 4 10 min me hogaya