Adam here. Event-driven architecture (EDA) can mean different things to different people. They can be internal to a single application, a distributed network of asynchronous applications, or a queue of tasks for a worker to process. But these all fall under the event-driven umbrella. With this video, I wanted to highlight four main types of EDA to illustrate how each type plays a role in building resilient, scalable, and performant architectures. Let me know what you think of this video in the comments below. Also, would love to hear what topics on event-driven architecture you would like to see covered next!

Super helpful!!

Congratulations on the content Adam, this area of ​​technology of ours is very complicated, with many of the same terms having different meanings depending on the context, I'm studying postgraduate studies in software architecture and it drives me crazy. Taking advantage, when we talk about an ephemeral form without retention, we are referring to a pattern where events are processed in real time and are not stored for later consultation, is that it?

@ConfluentDevXTeam

9 күн бұрын

Ephemeral means that there is no indefinite persistence of the events. It can vary from "all events are only in memory" (and therefore vulnerable to deletion upon power loss) to "events are persisted to disk, but are deleted after a short period of time "(seconds, minutes, or perhaps after consumption). For example, a consumer may only receive the messages if it is online at the time the producer writes it to the broker. A consumer coming online later is unable to access the historic events because they are no longer stored anywhere.

cant you use kafka for app internal communication or queues aswell? or is this not recommended?

@yuriir3219

3 ай бұрын

Nothing stops you from doing it, however it all depends on the use case. For example, if your app (assuming you’re referring to a ‘service’) is a monolith running as a single instance (no horizontal scaling anticipated or supported), then in-memory queue would be a first option to consider - in general it’s simpler and cheaper. However if a service runs in multiple instances and you already have Kafka as part of your infrastructure - it’s perfectly fine to set up inter-instance communication via Kafka. Here’s a couple of examples I’ve seen in real production: - Task breakdown: receive an external event, split it into N atomic independent pieces and sent to all service instances to parallelize work (shared consumer group) - State propagation (local cache): instance receiving a write forwards it into an internal topic, all instances of the service use unique consumer groups to receive copy of this message (fan-out) - analogue of sticky sessions: relying on partitioning algorithm (ex: userId) to make sure all events belonging to the same logical entity (here - user) are getting processed by the same service instance (be careful with rebalancing and choosing right partition key)

@ConfluentDevXTeam

3 ай бұрын

Wade from Confluent here. You definitely can use Kafka for internal communication. That's the foundation of a pattern known as the "Listen to Yourself" pattern (more information here: kzread.info/dash/bejne/e5pmuZitnafUaJM.html). Furthermore, when using Event Sourcing and CQRS, it's not unreasonable to have the read models for your service built using events that come from the write model after they are published to Kafka. So, yes, there are plenty of use cases where you can do this. In this video, Adam is isolating the technologies that best represent the specific patterns. So he's not saying you can't use the technologies in other scenarios. You definitely can. Rather, he's trying to say that Kafka is a typical example of Pub/Sub, ActiveMQ is a typical example of Queuing, Akka is a typical example of internal, etc. Each of these technologies can do more, but their most common usage fits into one category or another.

Bro how do you write mirrored like that on glass?

@ConfluentDevXTeam

3 ай бұрын

Wade here, from Confluent. We actually had to scour the world for someone who could write backward and forward so that we could make this video. It's a very rare talent practiced only by a small community in Canada...Ok, actually, he just writes normally on something called a "Light Board" and then they flip the image in post-production.

Good explanation - not quite correct examples. ESB is not about ephemeral messaging (if you worked with it you would know not from theory - I did for years). Ephemeral is topic related. However, Apache Camel ESB with Apache MQ GUARANTEES delivery even on topic provided that subscriber subscribed once (it is lost in two cases: not subscribed first time or you exceed parameters of how for long it is stored).

@ConfluentDevXTeam

Ай бұрын

Adam here. I admit I waffled on putting "ESB" into that category. You're quite right that an Apache Camel ESB / MQ guarantees delivery aside from those two cases you mention. The thing that makes ESBs difficult to talk about is that there isn't a global standard, so what you described (to me) is an ESB with an enhanced persistence layer. Using my 4 types of EDA framework, I would say it's ephemeral in the sense that the record is lost if you aren't subscribed when it happens, or it times out (as you put it). But it's also queue-based as you're using durable queues to store the messages. I guess that's the hard part about trying to make 4 neat categories - real life systems and use cases rarely fall nicely into just one :)

@maciekskontakt

Ай бұрын

@@ConfluentDevXTeam Hi Adam. I liked your explanation very much. I was reading and listening to Confluent and watching and learning Kafka for last years (wondering if you should somehow join Confluent perhaps ;) I even found one problem on state race condition in microservices if someone is not careful enough in designing and implementation with Kafka events (related to Saga pattern). But Kafka for many is the way to go and I was on architecture while in Fiserv that uses Kafka in AWS for credit cards (very nice design, but not mine). So Kafka and Confluent works are very close to my heart, but I hate to use my prior intimate knowledge with other systems to point some gaps. Thanks for correction.

@ConfluentDevXTeam

Ай бұрын

@@maciekskontakt Thank you for your insights too! I appreciate your kind words as well.