Thank you Arpit it's very clear and simply explained 👍

Thanks a lot Arpit. U explained it really well

Thanks Arpit Very Good Information !!

One thought Arpit.. What if we use circuit breaker.. in case of many failures it goes to OPEN state(means don't call downstream) and in configurable time it goes to HALF_OPEN state. If calls are still failing it goes to OPEN and won't call downstream. If downstream recovered then it goes to CLOSE state which means allow call to happen. Do you think this is also one of the solution?

This is also called circuit breaker design pattern in Micro services. We need to handle the API failures by using some retries and jitter until the service is up. Meanwhile return the service unavailable response to the client. Am I correct Arpit ?

Hey Arpit, this is great, thanks. But what about from server's perspective? Suppose its your server who is facing the thundering herd problem, how would you handle it in such a case? We don't have control over the clients, they may or may not retry with backoff or just keep bombarding our server with requests. I felt this was missing in this video. Few things come in my mind, do let me know what else I'm missing: rate-limiting(start dropping reqs all together), tweak the LB to distribute the load to different servers, power of two choices algorithm(?), caching and auto-scaling your replicas could also help.

Awesome 👍❤ bro

My organization doesn't follow what mentioned in this video but it handles in much better way and we call that as "Activating/Deactivating Emergency mode" in much simpler way.

@jilvasheth7127

Жыл бұрын

Can you please explain that in brief? Would like to know the other approach as well.

@jaisamtani303

Жыл бұрын

@@jilvasheth7127 sorry bro I can not tell. If my company shares a technical blog then I will share the link with you

@ajnabee01

Жыл бұрын

@@jaisamtani303 why not? That is not confidential.

bhai ap great ho 🙏

Hi Arpit, thanks for the amazing video. I have a question : shouldn't the range of value of random jitter that you mention also increase exponentially as the number of retry increase? Otherwise the range of time when spike happen will be in a certain small range, right?

@AsliEngineering

Жыл бұрын

Not necessarily. But implementation can vary.

kya gajab padhate ho bhai

Now I know why the retries are scheduled like this. Your videos are amazing as always.

Nice

I believe we can use queue/pipeline based approach in that case (to avoid such bombarding), please correct me if am wrong...great short video though !!

@AsliEngineering

Жыл бұрын

Not everything can be modelled in a queue.

@gauravaws20

Жыл бұрын

This is what you do on the client side. Ofc you still have to do something on the server side to avoid this. Using queue could be on of the approaches but then you’re getting into a totally different kind of architecture. Load shedding, throttling and server side circuit breaking could be other approaches too.

I guess you would have learned this first at Amazon? We use exponential backoffs + jitter for any retry at Amazon and even AWS SDK directly uses it in built.

@AsliEngineering

Жыл бұрын

No. Learnt it as a fresher when working with a latency sensitive service.

Isn't there some amount of jitter already with different clients having different network speeds, machine specs etc?

@AsliEngineering

Жыл бұрын

Yes but still that's not enough in real world scenarios

@a..b1343

Жыл бұрын

Oh, got it. Thanks for clarifying!

Even if we retry with Jitter but what about new requests that will come to our APIs, our API server will still be flooded with requests that are waiting to be served, right? (Just trying to understand how we can solve that problem?)

@AsliEngineering

Жыл бұрын

Yes. But not all retries will come at the same time giving server the necessary breathing space.

@asifmohammed5436

Жыл бұрын

do the following. at the server, if you receive more requests from clients within x period, send service unavailable as the response. and the client end, if you receive this status code, use exp back off. this will optimize both ends. also make sure to have auto-scale deployment configuration with min and max number of servers, if the number of connections increase by some x percentage.