I hope you loved the video. if you do, do share a word on social media. It would me the world to me ❤

I can't express my gratitude adequately for creating such excellent content

Great explanation. Thank you! Of the multiple times I felt "wow", two places are 1. How heartbeat is efficiently used and avoided persistence of chunk to location mapping. 2. How the write failures are handled with 2 step acks from primary replica.

It's 4 in the morning and just completed watching this amazing amazing video on GFS, really amazing decisions they took in 2003. Got few questions but first I will try to look-up on my own. Hatsoff to your teaching skills 🤩

@AsliEngineering

4 ай бұрын

Thank you Pranjal :)

More paper reviews! I like how you explain

Great video. The Primary replica concept reminds me of the leader replica concept of a Kafka topic across brokers.

Amazing explanation!! Loved it!!

Nice explanation! This was covered in the fourth year and Hadoop is inspired by GFS and that's when yahoo open sourced and big data reveloution started

Nailed it.. Thank you very much for the detailed video Arpit. Stay curious :)

Wow a great way to explain .. Thanks ..

when you dropped this gem! I was also reading GFS white paper and came here to watch video after completing the white paper . Even though i have understood the paper , watching your video made me fall in love with GFS even more 😊

@AsliEngineering

4 ай бұрын

Thanks man! means a ton ✨

Great video loved it

I want someone to say to me the way Arpit says "such a beautiful software" 😊 Jokes apart, what a video! I'm really glad that i stumbled on this. Thank you

@AsliEngineering

10 күн бұрын

Hahhaha 😅

This has been awesome so far, I am half way through the video. If this is not too much to ask for, it would be great if the notes that is presented can be shared 🙏

Just wow ❤

this is great would love to see paper on big table as well

Very well explained and rly helped me understand… please do Big Table next😃

Loved this video and also implemented a hello world of GFS, can you also make a similar video of Kafka paper..

Good video. Not really into file management systems or computer science (I'm a 3d artist) but still watched.

Dhanyavaad ji.

Would love to have a similar walkthrough for aurora db

As a sequel, can you please add another video on the whitepaper MapReduce: Simplified Data Processing on Large Clusters? Also will it be possible to have a video covering Colossus? I know there are not enough resources available for that.

Arpit, as you mentioned that the GFS Client will directly talk with the Chunk Servers to get particular chunk data and the ACL is maintained on the master server. Does it means that there is no access rights checking on the chunk servers? Can anyone with the details (meta data obtained from master server) of the chunk server access the data in that particular chunk server?

Your teaching skills are great👍

@AsliEngineering

2 ай бұрын

Thanks Ganesh!

Do HDFS has same primary replication scenario and how to set the primary replication for a file ?

Wat so special about doesn't torrent work same way? It's just looks like google renaming bit client to Google file system.

Great video! Really made me want to go back and read the source paper 😃 Around 1:13 :00, I was slightly confused as to how the master detects corruption of chunks on a specific chunk server (doesn’t seem scalable for the master to keep checking the checksums regularly). After skimming the paper it seems the master is informed about potential corruption by the chunkserver with the corrupted chunk rather than the master detecting it proactively.

@AsliEngineering

4 ай бұрын

Thank you 🙌

Where can i find the notes thats being displayed in the video?

Hey sir, in case of hot spots, won't writes be further affected?

👍👍

💯💯👍🏻👍🏻

Thanks for making such amazing content big fan of your work ❤, just a small doubt if i am not missing 1:01 :19 you said that client have to write to each replica itself , don't you think it would be better if client only write to primary replica and from there its primary replica duty to write it to all secondary replicas, it will help in 2 ways 1) client don't have to send the confirmation that write to all replicas is completed , all these responsibility can be encapsulated to the primary replica 2) network call overhead to write from client to each replica can be reduce by primary server as they can use very high bandwidth connectivity of the Google also if replicas are near or in same physical zone than we can save network overhead to pass the data across hundreds of miles waiting for your response

@koteshwarraomaripudi1080

3 ай бұрын

Kafka does the same thing what u are suggesting. My best guess for gfs to take this call would be based on the network bandwidth availability in data centers in 2003.

How do we handle atomic or contentious operation here. ? two client writing same chunks are they going to acquire lock on LRU ?

@AsliEngineering

4 ай бұрын

this is not a transactional system like a database. so contentions are not common. But in case there are then pessimistic locking is a simple solution.

This was a paper I read earlier then you hehe 😁

Doesn't torrent work the same way, I didn't find any difference other than renaming bit clients to Google file system

Gold Content , so nice presentation , handwriting ,Thanks for all the efforts 🫶🫡

Writes: 48:05

Hi Arpit, would it be feasible to implement this in Golang? What are your thoughts?

@AsliEngineering

4 ай бұрын

Yes. totally. pretty easy to implement a quick prototype.

can you share these notes?

Great discussion Arpit, please make more such long videos. I have one question from the discussion. Why is the client sending the same write to all the chunk servers ? And why not just to the primary replica and then the primary replica sending the write to other replicas ? (Similar to Kafka, where the leader broker send the writes to other brokers and waits for the ack from the replica brokers)

@kuldeepranjan8703

4 ай бұрын

Exactly, Why client is waiting acknowledgement from the majority, isn't primary acknowledgement is enough, system will take care of the replication then.

@shankarbanerjee8746

4 ай бұрын

Maybe to protect the primary replica's bandwidth. Primary replica sending the writes to the other 2 will eat up its bandwidth, instead they chose to preserve it and use up the client's bandwidth. That's my hunch

@SanjayJayasankar

4 ай бұрын

Same thought.. The bandwidth would often be faster between chunk servers.

@hiteshbitscs

4 ай бұрын

That's great question. My take is client writes to primary & then primary writes to at least half of the replica. Let say if Replication factor = 3 then it waits for at least one more replica to give ACK. Hence total 2 writes are confimed. Then they update master and return to the client. so this ensures at least N / 2 replicas are consistent in the system. Also my take is rather than saving chunk server's bandwidth as writes to client to server is very unreliable and need to cross entire internet. Hence it is slow. mostly I believe chunk server would internally copy the data and ensures consistency. Client -> broadcast to all replica .. I AM NOT IN FAVOUR.

How do you make the diagrams?

@AsliEngineering

4 ай бұрын

they are all hand drawn n my GoodNotes app.

What font are you using in the video?

@AsliEngineering

4 ай бұрын

These are handwritten notes. But glad you thought it was a font.

@linc008

4 ай бұрын

You have very good looking hand writing!@@AsliEngineering

Will watch it again after completing my ongoing assignment 🫶🫡

I am in final year of undergrad, have never read anything in system design, will i be able to understand this, please be honest sir

@AsliEngineering

4 ай бұрын

Go through it and find out. If I were you and in college, I would have gone through such videos more often than not. Do not doubt your abilities before even trying. To be honest, being a student you will be able to learn quicker than the experienced folks. Do not let others tell you what you can learn and what you cannot.

@GauravSinghvi-yu8jd

4 ай бұрын

@@AsliEngineering Thanks a lot sir for your answer, would surely go by your advice

Thank you for this awesome video. Here master is working more like a coordinator node, isn't it? the zookeeper also does the same thing, right? Handle the information about which replica is residing on which node. I think we are using here multi-master replication, every replica is acting like a master for one chunk and others acting like a follower.

Don't know man.. Google drive kinda sucks