Hey - EventBridge product manager here! A quick clarification: you're able to create multiple rules on EventBridge that each match against the same event. Each of those rules can also have 5 targets. The typical approach is to use a separate rule per consumer. So you might have 20 different services in your application that each need to receive the same event - each of those services can have a separate rule that each matches against that event. You're not charged any additional amount for rule matches, and there is a soft limit of 300 rules per event bus.

@BeABetterDev

2 жыл бұрын

Thanks for clarifying this Nick. I've pinned this to the top so others can see. Is this approach present in the EventBridge documentation? If not it may be useful to add. Cheers

@kwiknikk

2 жыл бұрын

@@BeABetterDev I just read this on a blog and made me think of this video: "You can define an SNS topic as the EventBridge rule target, and then fan out from SNS to much larger number of subscribers."

@cemsezer4152

2 жыл бұрын

Then the only justification to have event bridge in such a scenario is that you need an integration with a third party sas app. Otherwise SNS alone would be sufficient.

@subhankarhotta7094

2 жыл бұрын

In my org, we are facing certain blockages due to the 300 rules per event bus limitation. Any idea on how to circumvent the same?

@nicksmit959

2 жыл бұрын

@@subhankarhotta7094 you can request a quota increase for your rules by visiting the Service Quotas console. There is currently a hard limit of 2,000 rules per bus. Alternatively, if you need high fan-out then using SNS as a target of EventBridge is a good pattern.

Such a great explanation. I searched many documentation links for the differences and this is clear cut explanation. Thank you

This video is very easy to follow and understand for someone who isn't knees deep in this stuff all the time but needs to get across it for work. Many thanks!

The best and simplest explanation. Thank you so much!

its good one, could have added the Kinesis in this comparison as well

Amazing video! Packed with information and so clear!

Great video mate! Really nice, understandable, and concise way of touching on all of these systems. It's incredible how much value the SNS, SQS, Lambda combo brings to the table on an auto scaled and managed infrastructure. Not to mention auto retries built in, message filtering in SNS, and DLQs. Message processing volumes are also ludicrous, especially on non FIFO queues.

I can't tell you how much I like your lecture, simple, easy to understand and the comparison is really helping.

I love the summary at the end. So many videos don't have this essential piece! Ty for the great vid

@BeABetterDev

2 жыл бұрын

Glad it was helpful!

This was well explained and insightfull. Thanks for sharing!

Hi sir you are doing great work for the aws beginners as well as for experienced 🙌🏻🙌🏻🙌🏻🙌🏻

@BeABetterDev

2 жыл бұрын

Thanks so much Himanshu!

Well articulated, concise, and to the point. Nice work.

@BeABetterDev

2 жыл бұрын

Thanks ash!

You're doing one heck of a job man! Kudos to the good work :)

Thank you so much for these incredible explanations.

Very helpful comparison, thanks so much!

Very well explained. Great work.

this is real gem...nobody could explain better than this. please make a playlist on aws with concept plus hands on.pla. may god bless u with a lot of money and health

Thank you for breaking these down for us in a simple format.

@BeABetterDev

Жыл бұрын

You're very welcome Mehdi!

Holy cow, this video answered all the doubts I had for production level best practices. I had all these doubts, especially the case @14:31. This is definitely the best video which talks about the specifics and best use cases of SQS and SNS , thanks!!

@santoshlml

Жыл бұрын

I agree with @14:31. Very useful information.

Good job on describing SQS, SNS, and Event Bridge in a visualized format.

@BeABetterDev

Жыл бұрын

Thank you!

This is what I was looking for. Thank you!

@BeABetterDev

2 жыл бұрын

Glad I could help!

Another feature of event bridge for which it can be choosen instead of Sns: don't need to configure dedicated queue in front of target ,rather we can configure a DLQ for all failed events which could not be delivered bcoz of target unavailability, and can be replayed later.

thanks. very detailed explanation with usecases

Best lecture on SNS, SQS and EventBridge that I have listened to.

Thank you very much for this. Awesome content!!

Thanks for this video!

thanks very concise particularly for whens & wheres to use

@BeABetterDev

2 жыл бұрын

Glad you enjoyed Bernard!

So nicely explained. Thanks

@BeABetterDev

10 ай бұрын

Glad it was helpful!

Thanks. Content is really good!

Thank for this video, was to helpful for me!

@BeABetterDev

2 жыл бұрын

You're very welcome!

THIS IS JUST INCREDIBLE. THANK YOU SO MUCH.

Brilliant video

Thank you for the video it was helpful.

@BeABetterDev

2 жыл бұрын

You're very welcome!

what a fantastic video!

Amazing explanation man 🎉🎉🎉

@BeABetterDev

Жыл бұрын

Glad you liked it!

Excellent

Dude, great video!

@BeABetterDev

2 жыл бұрын

Thanks so much Eduardo!

I know Nick already commented, but I just want to add that EventBridge rules really look like they should be 1-1 with their pattern, but in reality they should be 1-1 with consumers. Just like each consumer of SNS can choose to subscribe to a topic, each consumer of EB can choose to "subscribe" to events by defining a rule to capture all the events they care about. If 20 services all care about the same kind of event, then you will have 20 rules with the same pattern. It might seem redundant, but if ones of them decides they care about other things, they own their rule and can update that pattern independent of the other 19. They can also own the *other* targets, for example they can send captured events to a CloudWatch log group or own their own DLQ for failed events. If it truly is a pattern that a lot of consumer are matching one or more "default" patterns, you can set SNS as the target to get the 3rd party input support you want with the mass fan-out of SNS. The limitations of EB in terms of number of targets and number of rules push you in the direction of these patterns; if you're hitting the limits of the system, you may just be using it wrong!

thank you so much for the video

@BeABetterDev

2 жыл бұрын

You're very welcome!

Hey Daniel - Great vid and thanks for publishing. Excellent summary. I think it's worth pointing out that as SNS is one of over two dozen currently supported EB Targets, would it not make sense to just pass the message to an SNS Target and manage all subscriber scalability needs there instead? Seems to me that EventBridge might easily lend itself to such a use case. Either way keep up the great work.

@BeABetterDev

2 жыл бұрын

Hey Brandon, thanks for pointing this out. This would indeed address the scalability concerns with using EB. Although this would work, I feel like it requires developers to jump through a lot of hoops to get functionality that should arguably be provided by EB itself. Hopefully soon they can support more targets, but we'll see. Thanks for the comment and support! Daniel

@BeABetterDev

2 жыл бұрын

Thanks Andrew! This indeed may be part of their plan - maybe they can add this to their documentation to make it an explicit suggestion. Daniel

@brandonhunter3036

2 жыл бұрын

@@BeABetterDev I can appreciate that sentiment. However it might also be worth acknowledging that part of the entire AWS platform architecture is providing for dozens of discreet, rather minimalist functional services and providing integration points or permitting the user to stitch together these services to solve for their own architectural requirements. Not too dissimilar to the SQS queues inserted before the SNS topics you described in the video as well. I say this as someone who works for an AWS premier partner and meets weekly with an AWS team to provide and discuss new feature requests. When the problem is more or less already solved, convincing them to dedicate resources to duplicate existing functionality is not usually result with a lot of success.

bro you're a GOD, thank you so much 🙌🏻🙌🏻🙌🏻🙌🏻.

@BeABetterDev

Жыл бұрын

Happy to help

Thanks!

Good explanation, It is clear the diference! I saw that in Event Bridge you can scheduller a event if you need to trigger batch jobs, is that another way to do that or event bridge is the only choice?

Best tutorial

What do you recommend for fan-in? Say you have an ETL pipeline made up of SNS and SQS (as you suggested) the workers consume until nothing is left and then, when all workers are done, fire some event to consolidate all the results?

Great! I just needed a tool such as Event Bridge! Just now! But I think I also need by AWS an implementation like native Server Sent Events, so a socket communication from server to client

@gaurravprakash

2 жыл бұрын

You should checkout API Gateway web socket APIs.

@BeABetterDev

2 жыл бұрын

This is the right answer. Video on websockets coming soon!

I successfully completed the SAA C03 on 12th March🎉🎉🎉

@BeABetterDev

Жыл бұрын

Congratulations!!!!

Great info! Question - for the SNS example (and SQS) the message data is shown as containing structured json data (customerId, etc.). Is this a common and valid usage? The reason I ask is the AWS docs don't include an example like that. Their examples show a single message field containing just a string. Would that structured data go in the "message" field of the SNS message?

Amazing video! Do you mean SNS and event bridge @18:50?

Amazing video man. You have a very deep understanding and it shows. You gotta make an AWS course

@BeABetterDev

2 жыл бұрын

Thanks so much Jagveer! I'm currently working on an AWS Lambda course - look for it soon ! :)

Thanks for the video and great explanation on the differences. at 15:30 you're showing a SNS implementation tied to SQS queues for each service. I'm wondering how this is different from the previously mentioned approach of having separate queues for each service? You mentioned it does not scale well, but it seems like with this architecture, you are having to do the same thing - create a queue for each service - is this just necessary to make things fault tolerant or is there something I'm missing?

@brucewayne5916

Жыл бұрын

well you can only add Queue to Receiver service, whereever you think the service might go down or must not go down. (Payments, billing makes sense). you are correct unnecessarily using Q along with SNS makes things more complicated.

@brentmarquez9057

Жыл бұрын

@@brucewayne5916 thanks for clarifying.

Perfect

Thanks for the great video. For SNS part, is this mean publish message , SQS -> SNS -> SQS (different subscriber) is the good architecture?

@mrmorcs

2 жыл бұрын

I'm not sure why you would need the initial SQS? It's usually (Publisher -> SNS) -> (SQS -> Subscriber)

@takahako5948

2 жыл бұрын

@@mrmorcs thanks! interesting. After I published this, I had a same thought as you so I deleted but somehow the comment stayed 😧

@BeABetterDev

2 жыл бұрын

Hi Taka, Looks like James beat me to it (and you figured it out too!). Thanks for watching Daniel

You are doing god's work :) !

@BeABetterDev

2 жыл бұрын

Thanks Abishek!

Can you elaborate or give me the key term for what you meant when you don’t want other services to hit the database? Segregation of tier one services.

Azure service bus = aws sqs (Queue) + aws sns(Topic)

@BeABetterDev

Жыл бұрын

Well put

For EventBridge, could you just create more rules to get passed the 5 limit? Also I usually create a rule with just one target for separation of responsibility.

@joek4563

2 жыл бұрын

Yeah exactly, you can just easily create more rules. Also you can contact AWS support to get your limit raised if you really need it.

@brucewayne5916

Жыл бұрын

you can also create matching rules, as per top pinned comment, and can have different targets to pass to

@alxjones

Жыл бұрын

Yes, you can create multiple rules per pattern. The idea is that rules are 1-1 with consumers, not patterns or targets. The difference between consumers and targets is that a consumer might have multiple things it wants to do with an event, e.g. send to a CloudWatch log group in addition to being consumed by their Lambda. This separation also lets each consumer have a separate DLQ set up for failed event processing.

Hi. What is the TTL of the message in Message Bus. What happens if the target downstream App is down?

thanks for great presentation !! Btw, you mentioned SQS typically to be owned by consuming team - I would like to know the thought process behind them ?

@julianfranco7689

Жыл бұрын

The consumer polls the queue and triggers the "visibility timeout" which is the time where the message is still in the queue but being consumed/processed. This state makes the message inaccessible to consumers. Once the timeout is done, the message will be deleted/retried/sent to a dead letter queue based on how the consumer processed the message. Basically, what happens to the message in the queue is dictated by the consumer. If no one is consuming from the queue, SQS just sits there doing nothing.

@alxjones

Жыл бұрын

I have recently been in a situation where the producer owned the queue, and we switched off of that system because of the many issues we ran into. The main problem is that the producer doesn't care about the settings on the queue, but the consumer does; on the other hand, the owner of the queue is the one that can change that. If you want to change the configuration, you now need to make a cross-team request (if you're lucky, for us it was a cross-company request). If you set up a DLQ on the queue, then who owns it? Consumer wants to own the queue filled with their failed messages so they can retry on their own time, but producer owns the main queue and so owns the redrive policy. The whole situation is a mess, but maybe there's a good reason you can't do it the other way? Well, no, there's not. Because every queue has exactly one consumer, there is no risk of stepping on toes. Everyone that can make any changes to the way that queue works agrees on those changes, because it's all one guy (read: team). By contrast, it would be an *awful* idea to let consumers own SNS topics, because there can be hundreds that all want different things. So, producer-owned SNS is the only thing that makes sense, but you still run into all the issues above if you consume directly from SNS. This is where putting a queue in-between SNS and your consuming service comes in, so producer owns the process of sending their messages out, and each consumer owns capturing, storing, processing, redrive, etc. for their own use-case.

Could you compare msk also?

what about AWS MQ?

In the video it mentions SNS to Lambda is bad form. Is it okay to talk between Lambdas with SNS if both Lambdas are released at the same time?

Does EventBridge negate the need to add an SQS queue in front of all the subscribers, like you usually would for SNS?

@BeABetterDev

2 жыл бұрын

Hey James, I think it is still a general best practice to put a SQS queue in front of any message processing application. I'm sure EventBridge has some kind of retry mechanism to deliver messages to the endpoint, but I would question its reliability. Hope this helps

Messages in SQS won't stay in the queue indefinitely. Message retention is between 1 minute and 14 days, after which messages are deleted.

Why do we need an sns then, since we are again creating 3 queues?

Actually 5 target limitation can easily be bypassed by creating multiple rules

thank you video? Do you have an Udemy course?

@BeABetterDev

2 ай бұрын

Check out courses.beabetterdev.com/

what is the slide software you use plz?

@BeABetterDev

Жыл бұрын

This is all done in powerpoint.

summary 20:55 thx

👍🏿

sns + sqs == kafka. One thing I am curious kafka has this feature of committing after reading, which ensures that the message has been read and removing the dependency of additional sqs in front of sns, why sns does not have it?

3000 Trnx per sec in sqs fifo queue

@BeABetterDev

2 жыл бұрын

Thanks Nishant!

so funny how you say we need sns instead of sqs to fix the inconsistency problem. Dude, that will not solve the problem. you can not guarantee consistency if you are working with any kind of queues: sns, sqs, kafka, redis....does not matter. also why you do not mention "exactly once" problem which is the first and most important problem to deal when we work with queue. sigh! so many misleading information in your videos.

This was well explained and insightfull. Thanks for sharing!

@BeABetterDev

2 жыл бұрын

You're very welcome Paul!