Hey, in the commit phase, what's if I am able to successfully "assign the order to that reserved food", but because of network failure my request to "assign the order to that delivery partner" fails twice because of timeout. After sometime, the "reserved delivery partner" will be freed (as the timer runs out), and the "store is already heating the food" for an order which doesn't have a delivery agent. It seems two-phase commit isn’t fully atomic after all. 🤔

@AsliEngineering

2 жыл бұрын

A delivery partner is reserved. now the only thing that remains is assigning it to an order. this is only possible when the delivery service is facing an outage. If the outage exceeds the timeout then the reservation of the delivery agent will be freed. Which anyway would happen in a distributed setup plus given a major incident like ouutage of delivery service. The timeouts are typically in the range of 2 to 5 minutes giving you enough time to reboot. Because the delivery agent is reserved, we are ensuring that it is not assigned to any other order. So, as soon as the service comes back up the agent will be assigned to the order. Giving you atomicity. You rightly mentioned that we cannot get atomicity when the outage is longer than the timeout. Hope I made sense :)

@subh_8208

2 жыл бұрын

​@@AsliEngineering Thank you bhaiya for the reply. Incase of severe network failure I guess this problem of "weak atomicity" will always be there (as you rightly mentioned). Similarly if the "order service" fails in between, then too it's a problem. I thought maybe three phase commit might solve this issue (in our case, food reserve -> delivery agent reserve -> food commit -> delivery agent commit -> ack to food service that delivery agent is found). But I was wrong. We are experiencing the famous two generals problem here. :( Is there any other way to solve it, with an assumption that network may fail anytime? ---Footer--- To get more scale, and have decoupled services, we are now stuck with new problems. Maybe microservices doesn't make sense always (at least when atomicity, and consistency is mission critical). Or we can be optimistic and neglect this rare event. :)

@sahinsarkar7293

Жыл бұрын

Doesn't it make sense to "revert the booking of the food" (by deleting the association of the food to the order), when we detect that there has been a failure in the delivery service during the "booking of the agent"?

@insane2539

11 ай бұрын

@@sahinsarkar7293 yes that will be the case the order will remain in pending state till commit success from both store and delivery service is received. if the network call from order to delivery service gets timeout or throws an exception then another service (order rollback service) would be called to roll back the already committed transactions and change the state of order from pending to failed. This service can be called asynchronously by pushing message to a queue.

@SaketAnandPage

11 ай бұрын

@@sahinsarkar7293You can’t do it.

Hi Arpit! your depth of knowledge is commendable. I have been searching for similar content where examples are actually from real life scenarios. I really appreciate the fact that you have put all this content for free. I hope this channel reaches more engineers who are passionate about distributed systems.

Watched 2 videos of yours, and subscribed. Great content bro this is what exactly I needed, Please keep making such valuable content

What a dhasu lesson. Seriously #AsliEngineering. True to its name. Hats off.

This is pure engineering video, thanks Arpit for such top-notch content...

Awesome man keep making such videos ❤️

This is #AsliEngineering. Kudos for sharing such a great content.

Now here I am starting with the new beginnings 😍😎😎 with distributed transactions

Great Explanation!!. SAGA Pattern also tries to solve the same problem but it is async. TBH I feel a 2-phase commit might complicate the code with a lot if's and we might miss some edge cases. Can you please throw some light on when to choose what ? (2phase vs SAGA).

Your explaination is awesome sir

You should definitely make a course on databases.

how deadlock will occur? I am assuming stores and delivery is separate service. We are not calling any delivery-service api from store-service api and vice versa.

Are they all synchronous API calls or something else? If it fails to place order and both the food and delivery is assigned. What will happen then ?

Thanks for the wonderful video Arpit! Really great lessons. Can you recommend some resources to read through on microservice communication, as in how microservices communicate with each other and the possible ways?

@AsliEngineering

2 жыл бұрын

Have written a few articles about it you can find them on my site arpitbhayani.me/blogs. Not a direct answer but you will get an idea. But yes. Thanks for suggesting, something I should be making a video about. Also, a great resource could be microservices.io

Hey Arpit, The demo was really amazing, but I have a doubt about executing and rolling back transactions across two different services, running in two separate ports.

What if while we're booking the food, the timer goes off in the delivery partner service? We'll again have a case of having food ready but no delivery partner assigned. I guess one can keep the timers apart by a few minutes to reduce these cases.

@AsliEngineering

2 жыл бұрын

Yes. The timeout is longer, typically 2-5 minutes giving you enough time for retries. You can refer to comment by @subh_ below and my reply to it. You will see how we are navigating the situation.

liked a lot.. thanks bro

Amazing sir

Amazing❤️

Can you please explain how a deadlock might occur in such a scenario. I mean firstly in the reservation phase you have put timer on locks so they'll get free no matter if it succeeds or not and in the commit phase both are booked, so say a service for booking fails how does that create a deadlock since although realistically 10 minute guarantee cannot be fulfilled but still the order will get cancelled totally. What part am I getting wrong where the deadlock can occur?

@mohammadkaif8143

3 ай бұрын

Deadlock can occur but once a timer is finished, it will release the lock anyway. Within that timer deadlock can happen

DRY: Don't (fking) repeat yourself !

Hey Arpit , will it be possible for you to explain SAGA pattern implementation ? The thing is 2PC doesn't scale. It will be a great help if you can make a series on SAGA.

@AsliEngineering

2 жыл бұрын

I have that in the plan. About to complete Hash Table Internals and then Microservices would resume.

Locks are acquired per table right? There can be only 1 writer to the table. Sorry I am still confused. Can you please share a small demo or code example. It would be very helpful 🙂

@AsliEngineering

2 жыл бұрын

Locks are not shared or exclusive that you get out of SQL db. These are explicit lock taken on Redis or a remote locking service. A demonstration of this is dropping on Wednesday 10 am. I am mimicking the entire distributed transaction. Do watch it.

@nikhiltaneja6673

2 жыл бұрын

@@AsliEngineering ah that's why i got confused. Thanks for the reply. We have to be careful about Redis locks if we are using clustered nodes.

Aren't Saga pattern used to propagate distributed transaction in microservices? Two phase commit may hold good in monolith applications

@saurabhthube3748

2 жыл бұрын

Yes, saga pattern is used over 2 phase commits in micro-service architecture

@saurabhthube3748

2 жыл бұрын

Video does solves the problem though and gives a clear context about how to approach such problem statements

@subh_8208

2 жыл бұрын

@@saurabhthube3748 Actually both have their own advantages and disadvantages. 2 phase commit aim is more towards consistency, and atomicity. It's synchronous (it might be essential in some cases). The disadvantage which I am able to think is that "things are tightly coupled" in 2 phase commit. On the other hand Saga is an async way of doing things, and it doesn't guarantee consistency (but ensures atomicity). In our case, it will be a very bad design to show the user "PENDING ORDER", and then show "DELIVERY AGENT NOT AVAILABLE" or "FOOD NOT AVAILABLE" (using saga). Instead waiting for a seconds or two, and directly showing "ORDER PLACED" or "ORDER FAILED" is a better experience (using 2 PC). Do share your thoughts on it. (correct me if you find anything wrong)

is it good idea here to retry or pooling when timer out for specific time period in between any service outage ?

@AsliEngineering

Жыл бұрын

Yes

@Arpit : At start of video, you said that zomato for 10 minute delivery, they intially put the food the store in zomato store. Let's have store having 10 same burger. So in db we store name and count Later in two phase you said that we put the lock on the specific row item. Suppose if 10 person booking the same item at time other 9 should wait? Until 2 phase is complete ? If 2 phase commit or fail don't you think that other request have to wait for long. Can you please make me more clearer, I got confuse

@AsliEngineering

Жыл бұрын

9 would not have to wait they will move forward and book other item.

@amitranjan6998

Жыл бұрын

@arpit , thanks for reply but if all 10 person booking only one item in my case same burger, then they have to wait, right.

In case of seat booking systems, the lock on seats is required because the user takes some time for the payment, right? In case of Food delivery systems, do we need to lock the food item because it takes some time to get a delivery partner? is my understanding correct? if both the requirements, ie the food item and the delivery partner were immediately available, then we wouldn't have needed to lock/reserve them, right?

@AsliEngineering

7 ай бұрын

Not just the only reason. Locks are required even while blocking the seats

@ayushjindal4981

7 ай бұрын

@@AsliEngineering do you mean that we need to lock the seat because the DB will take some time to commit the booking of the seat...and till then we dont want any other person to select that seat? Is it?

Thanks nice explanation. I had a query. The timer you showed in the reserve phase for food and agent, will that timer countinue for the commit phase or its just for reserve phase. Like if reserve phase succeeds, will that timer thing end there and the commit phase then gets a new timer or will it continue over the total transaction?

@AsliEngineering

2 жыл бұрын

Only for reserve phase.

@sahinsarkar7293

Жыл бұрын

I think the timer for reserve would end only if either the timeout is hit, or when the commit is successful.

Can you add the notes in your video description as well for our revision ?

@AsliEngineering

2 жыл бұрын

Soon. I am already putting them on LinkedIn and Twitter. I recommend you to go through them by the time I automate the process.

Want 3 Phase commit protocol too

Hi, I have one qs, (assume a general scenario with multiple nodes and a co-coordinator). Assume 1st phase is done and three nodes are ready. Then co-coordinator starts 2nd phase but some node could not perform the action and returns error. The coordinator sees the error from that node and sends abort message to all the three nodes to rollback the 2nd phase. (Correct me if there is anything wrong here) Now, what if some abort message get lost in the network and some node could not rollback even though the transaction has failed overall. My question is, is there is importance of 1st phase to this abort loss problem ? If not could you comment on what if we do not have 1st phase and just perform the 2nd phase with abort ?

@debashishdeka7698

Жыл бұрын

Is 1st phase is helping us with isolation ?

@debashishdeka7698

Жыл бұрын

I think I got it. 2nd phase is must to tell participating nodes about consensus info collected in the 1st phase. The data write can happen in the 1st phase however it can be removed on 2nd phase.

Subscribed.

what tool are you using for these notes?

@AsliEngineering

2 жыл бұрын

IPad

@agarwalr5205

7 ай бұрын

GoodNotes is the name of app on which notes are being written.

superrrrr

Now I will have to order a burger from Zomato

Hello, one small feedback, should you first describe in detail what the protocol is: en.m.wikipedia.org/wiki/Two-phase_commit_protocol , and then dive into the zomato example and apply it there, instead of just directly starting from the example and explaining only this particular example?