I'm keeping the moustache sorry not sorry. Stay posted for this Saturday, ultimate video incoming.
Жүктеу.....
Пікірлер: 53
@VijayJain4 ай бұрын
The key thing to mention is that If sensor metrics are spread across multiple partitions, they can still be read out of order, so in-order is only guaranteed within the partition.
@jordanhasnolife5163
4 ай бұрын
Yep!!
@pankajjahagirdar1278
25 күн бұрын
partitioning can be on deviceId, so readings of same device can be in same parititon
@Aditigoyal19973 ай бұрын
Insightful and crisp video. Thanks.!
@rhodyborn4 ай бұрын
Throwing together a design doc and this video was quite helpful, thanks. Also, P!
@jordanhasnolife5163
4 ай бұрын
Great! I should make the caveat though that you should definitely check the official docs of both technologies at this point, because it is very possible that they have evolved significantly!
@quirkyquesterАй бұрын
amazing topics, and video been thinking about learning more about rabbit mq and kafka for awhile. wanted to know the diff. this answers my question in mind.
@abdelrahmanemam647821 күн бұрын
Great explanation!
@kinshaabid30635 күн бұрын
Very clear and easy explanation.❤
@indraneelghosh66074 ай бұрын
This was a really helpful overview of the messaging systems. Can you do a video that goes more into the implementation details of the message brokers you discussed? For instance, answer to questions like how and under what conditions does Kafka guarantee exactly once message delivery and what sort of guarantees these message brokers give in case of failures etc.
@jordanhasnolife5163
4 ай бұрын
The brokers themselves don't really give any guarantees I'd say, but rather the broker in combination with a stream processing framework is where you start seeing those. Hopefully the stream processing video clears that one up a bit.
@htm3329 ай бұрын
Gangster stuff
@jordanhasnolife5163
9 ай бұрын
I'm certainly trapping upon these streets
@benagarr4 күн бұрын
Great video
@AlexBowman-19995 ай бұрын
Im building out a game with a microservice architecture and an eventbus system. I want to know some thoughts on the idea of using both kafka AND rabbitmq. I would use kafka for the user auth service where all user data will be saved to disk to maintain reliable syncing with other services at all costs. My thought is that the auth service is by far the most critical service and needs a robust system in place. I am ok with the processing time taking a bit longer as this is just for the signing up/registering a user process(this only happens one time per user). Then for all other game related events, it will be handled with rabbitmq to utilize the in memory speeds. I understand that setting up/maintaining these two systems will be more difficult, but overall do you think this idea has some merit to it?
@jordanhasnolife5163
5 ай бұрын
Well, you may not love my answer, but considering this is a personal project, I think that you're building for scale that you don't need :) I'd probably just use some existing OAuth service and for the game related events you could always just have the server connect to all clients via websockets.
@2005kpboy9 ай бұрын
Low latency backbone used by trading firms beat these two by wide margins. I guess these are designed for high throughput whereas LLBs are designed for low latency.
@jordanhasnolife5163
9 ай бұрын
I'll be honest I've never heard of these and I work in trading I'll give it a look
@yashkhd1100
9 ай бұрын
I guess u r talking about ultra low latency full stack systems which uses combination of HW and SW to get really low latencies. Solace is one of such vendor. These systems have very limited use cases and mostly they are for trading. Kafka and RabbitMQ are all about scale and provides good enough latencies sufficient for wide range of use cases.
@ROFEL
9 ай бұрын
@@jordanhasnolife5163 you a quant?
@david68519 ай бұрын
Wait I think RabbitMQ has some failover strategies? I think they also have quorum queues and classic queues where quorum queues actually have a persist log on disk? Not 100% sure about what I am saying though, maybe a bit too specific to RMQ. someone correct me if I am wrong.
@jordanhasnolife5163
9 ай бұрын
You may be right - when I talk about these technologies I try to just focus on "in memory" versus "log based" because specific implementations can change all the time. It's very possible that depending on how you configure RMQ you can do any of those things.
@varshard0
3 ай бұрын
You are correct. RabbitMQ persist messages if both queue and message are marked as durable.
@goldmund67
2 ай бұрын
Wanted to clarify the same thing. RMQ supports durable queues but just to be clear that means saving messages in the queue not yet ACK'd when the system goes down or memory is unavailable. Doesn't mean full replay.
@prashantshubham9 ай бұрын
Audio and video sync issue? 🤔
@jordanhasnolife5163
9 ай бұрын
It's possible, I fully edited this thing with my sound off
@smoran02
Ай бұрын
@@jordanhasnolife5163 I have this problem on all your videos FWIW. Love the content though.
@Bruh4685 ай бұрын
Great videos + How someone so young can have so much knowledge .. but bro why do you have named it -> Systems Design Interview 0 to 1 ? Systems Design 0 to 1 could have been an apt name .. don't you think? its more useful than just for interviews..
@jordanhasnolife5163
5 ай бұрын
I think it's harder to speak authoritatively about actually systems because there's a lot more "it depends" and existing company considerations going into making those choices :). Plus, these things change so fast that for the purposes of accuracy about the technologies my videos are good for interviews, but could be incorrect IRL
@vadimc48124 ай бұрын
RabbitMQ has a different storage options that are log-based. Calling it an "in memory" doesn't sound right. Not sure about others in a list but it could be the same case with them.
@jordanhasnolife5163
4 ай бұрын
Yeah I always make this videos with the caveat that they could very well be incorrect. When a technology is open source, it can adapt significantly over time, and additionally people like to throw in a lot of features so that they have more to advertise for the product. A better title for the video would be "in memory vs. log based message brokers", but I just like to clickbait because I'm a narcissist.
@medievalogic
2 ай бұрын
are you talking about Durable queues?
@MrWayneStephen2 күн бұрын
You don’t actually mention if Kafka is the log or message type vs rabbit
@jordanhasnolife5163
2 күн бұрын
Kafka is the log
@joshidev.dev888 ай бұрын
You can have round robin in kafka using null partition key
@jordanhasnolife5163
8 ай бұрын
I'll have to look into that, afaik every consumer still sees every message but I'm sure there's some way to do it
@firezdogАй бұрын
I don’t see anything inherent in the log based queue that would limit throughput - except the slower reads from the disk. I feel like the semantics of the queue should be independent of the way it is stored - if that makes sense.
@jordanhasnolife5163
Ай бұрын
Makes sense to me, but at least for jms vs Kafka they're inherently different paradigms.
@firezdog
Ай бұрын
I was thinking about this more and I suppose the limits on memory could be one reason why the semantics are different. When you’re processing from memory you need to clear items out or the queue gets too big and you’ll get thrashing / crashing. I’ve seen this happen with RabbitMQ at work.
@jordanhasnolife5163
Ай бұрын
@@firezdog Absolutely - if you buffer too much you'll bring down the broker
@phsaurav3 ай бұрын
Great video. Clear and on point. Thank you. 🫡
@fallencheeto47629 ай бұрын
Nice mustache
@jordanhasnolife5163
9 ай бұрын
Thanks Mr. Cheeto, unfortunately I can't actually grow facial hair so now I'm left with this abomination
@rodfrazier11253 ай бұрын
For a visual learner like me, I understand but the way you scribbled all over the board. It can throw people off. Just a bit of advice maybe you can take some goodness from. Great video though man!
@jordanhasnolife5163
3 ай бұрын
Appreciate it, thanks Rod!
@sarudon5615Ай бұрын
6:16 if its a message queue and last message processed by consumer B was M2 then according to Queue next message to process should be M1 but you are saying next message to be processed is M3. This is confusing explain / correct me if I'm wrong
@jordanhasnolife5163
Ай бұрын
probably just a poor explanation on my part, messages are processed FIFO
@medievalogic2 ай бұрын
gold
@cloud_architector4 ай бұрын
Kafka has much better performance than in memory rabbit mq
@jordanhasnolife5163
4 ай бұрын
In practice this may be true, I'd have to look at benchmarks. I think that from an interviewing perspective though, knowing the architectural differences between them and what that means (at least in theory) is important.
Пікірлер: 53
The key thing to mention is that If sensor metrics are spread across multiple partitions, they can still be read out of order, so in-order is only guaranteed within the partition.
@jordanhasnolife5163
4 ай бұрын
Yep!!
@pankajjahagirdar1278
25 күн бұрын
partitioning can be on deviceId, so readings of same device can be in same parititon
Insightful and crisp video. Thanks.!
Throwing together a design doc and this video was quite helpful, thanks. Also, P!
@jordanhasnolife5163
4 ай бұрын
Great! I should make the caveat though that you should definitely check the official docs of both technologies at this point, because it is very possible that they have evolved significantly!
amazing topics, and video been thinking about learning more about rabbit mq and kafka for awhile. wanted to know the diff. this answers my question in mind.
Great explanation!
Very clear and easy explanation.❤
This was a really helpful overview of the messaging systems. Can you do a video that goes more into the implementation details of the message brokers you discussed? For instance, answer to questions like how and under what conditions does Kafka guarantee exactly once message delivery and what sort of guarantees these message brokers give in case of failures etc.
@jordanhasnolife5163
4 ай бұрын
The brokers themselves don't really give any guarantees I'd say, but rather the broker in combination with a stream processing framework is where you start seeing those. Hopefully the stream processing video clears that one up a bit.
Gangster stuff
@jordanhasnolife5163
9 ай бұрын
I'm certainly trapping upon these streets
Great video
Im building out a game with a microservice architecture and an eventbus system. I want to know some thoughts on the idea of using both kafka AND rabbitmq. I would use kafka for the user auth service where all user data will be saved to disk to maintain reliable syncing with other services at all costs. My thought is that the auth service is by far the most critical service and needs a robust system in place. I am ok with the processing time taking a bit longer as this is just for the signing up/registering a user process(this only happens one time per user). Then for all other game related events, it will be handled with rabbitmq to utilize the in memory speeds. I understand that setting up/maintaining these two systems will be more difficult, but overall do you think this idea has some merit to it?
@jordanhasnolife5163
5 ай бұрын
Well, you may not love my answer, but considering this is a personal project, I think that you're building for scale that you don't need :) I'd probably just use some existing OAuth service and for the game related events you could always just have the server connect to all clients via websockets.
Low latency backbone used by trading firms beat these two by wide margins. I guess these are designed for high throughput whereas LLBs are designed for low latency.
@jordanhasnolife5163
9 ай бұрын
I'll be honest I've never heard of these and I work in trading I'll give it a look
@yashkhd1100
9 ай бұрын
I guess u r talking about ultra low latency full stack systems which uses combination of HW and SW to get really low latencies. Solace is one of such vendor. These systems have very limited use cases and mostly they are for trading. Kafka and RabbitMQ are all about scale and provides good enough latencies sufficient for wide range of use cases.
@ROFEL
9 ай бұрын
@@jordanhasnolife5163 you a quant?
Wait I think RabbitMQ has some failover strategies? I think they also have quorum queues and classic queues where quorum queues actually have a persist log on disk? Not 100% sure about what I am saying though, maybe a bit too specific to RMQ. someone correct me if I am wrong.
@jordanhasnolife5163
9 ай бұрын
You may be right - when I talk about these technologies I try to just focus on "in memory" versus "log based" because specific implementations can change all the time. It's very possible that depending on how you configure RMQ you can do any of those things.
@varshard0
3 ай бұрын
You are correct. RabbitMQ persist messages if both queue and message are marked as durable.
@goldmund67
2 ай бұрын
Wanted to clarify the same thing. RMQ supports durable queues but just to be clear that means saving messages in the queue not yet ACK'd when the system goes down or memory is unavailable. Doesn't mean full replay.
Audio and video sync issue? 🤔
@jordanhasnolife5163
9 ай бұрын
It's possible, I fully edited this thing with my sound off
@smoran02
Ай бұрын
@@jordanhasnolife5163 I have this problem on all your videos FWIW. Love the content though.
Great videos + How someone so young can have so much knowledge .. but bro why do you have named it -> Systems Design Interview 0 to 1 ? Systems Design 0 to 1 could have been an apt name .. don't you think? its more useful than just for interviews..
@jordanhasnolife5163
5 ай бұрын
I think it's harder to speak authoritatively about actually systems because there's a lot more "it depends" and existing company considerations going into making those choices :). Plus, these things change so fast that for the purposes of accuracy about the technologies my videos are good for interviews, but could be incorrect IRL
RabbitMQ has a different storage options that are log-based. Calling it an "in memory" doesn't sound right. Not sure about others in a list but it could be the same case with them.
@jordanhasnolife5163
4 ай бұрын
Yeah I always make this videos with the caveat that they could very well be incorrect. When a technology is open source, it can adapt significantly over time, and additionally people like to throw in a lot of features so that they have more to advertise for the product. A better title for the video would be "in memory vs. log based message brokers", but I just like to clickbait because I'm a narcissist.
@medievalogic
2 ай бұрын
are you talking about Durable queues?
You don’t actually mention if Kafka is the log or message type vs rabbit
@jordanhasnolife5163
2 күн бұрын
Kafka is the log
You can have round robin in kafka using null partition key
@jordanhasnolife5163
8 ай бұрын
I'll have to look into that, afaik every consumer still sees every message but I'm sure there's some way to do it
I don’t see anything inherent in the log based queue that would limit throughput - except the slower reads from the disk. I feel like the semantics of the queue should be independent of the way it is stored - if that makes sense.
@jordanhasnolife5163
Ай бұрын
Makes sense to me, but at least for jms vs Kafka they're inherently different paradigms.
@firezdog
Ай бұрын
I was thinking about this more and I suppose the limits on memory could be one reason why the semantics are different. When you’re processing from memory you need to clear items out or the queue gets too big and you’ll get thrashing / crashing. I’ve seen this happen with RabbitMQ at work.
@jordanhasnolife5163
Ай бұрын
@@firezdog Absolutely - if you buffer too much you'll bring down the broker
Great video. Clear and on point. Thank you. 🫡
Nice mustache
@jordanhasnolife5163
9 ай бұрын
Thanks Mr. Cheeto, unfortunately I can't actually grow facial hair so now I'm left with this abomination
For a visual learner like me, I understand but the way you scribbled all over the board. It can throw people off. Just a bit of advice maybe you can take some goodness from. Great video though man!
@jordanhasnolife5163
3 ай бұрын
Appreciate it, thanks Rod!
6:16 if its a message queue and last message processed by consumer B was M2 then according to Queue next message to process should be M1 but you are saying next message to be processed is M3. This is confusing explain / correct me if I'm wrong
@jordanhasnolife5163
Ай бұрын
probably just a poor explanation on my part, messages are processed FIFO
gold
Kafka has much better performance than in memory rabbit mq
@jordanhasnolife5163
4 ай бұрын
In practice this may be true, I'd have to look at benchmarks. I think that from an interviewing perspective though, knowing the architectural differences between them and what that means (at least in theory) is important.
Thanks Jordan. Great video. Please get a life. 😄😄
@jordanhasnolife5163
13 күн бұрын
Never!
Great explaination!