Wow - a totally perfect video, clearing most of my doubts in less than 7 minutes! Thank You!

Tim has explained kafka `simplified` exceptionally well.

Best Kafka educational videos out there. thank you

I like the way Tim explains , straight to the brains

Each partition ensures the order of the messages contained therein. The producer can target messages to be produced onto a specific partition by the message key. Consumers can target a specific partition by specifying the key of the messages they're interested in. If the message order doesn't matter as in when transactions are idempotent(i.e. Order has shipped) then you can produce those messages in a round robin fashion to multiple partitions and have each partition assigned a consumer where each consumer belongs to the same consumer group, consume those messages at scale. 10 partitions then define 10 consumers within the same consumer group so that messages are consumed at scale of 10. Got it. The MOST important thing to know is if msg Order IS important for message consumption(i.e. CRUD event notification). If it is then you can only have 1 partition and have multiple consumer groups each with 1 consumer reading from that partition. If msg order IS NOT important then you can scale.

@razvanandrei2671

Жыл бұрын

Very important aspect indeed. Consumer scalability is not elastic when message ordering matters. I think though in some cases one might sacrifice consistency for performance and scalability by using compensation transactions after processing instead at the Kafka consumer groups level in order to account for order, but it’s not always possible.

@chandraballa4860

8 ай бұрын

What is the importance of key schema from consumer perspective? Is key important for the consumer within the application?

Great videos!

We run Kafka consumer on Kubernetes. For one use case we use pod-name as the groupId to filter out specific events. Is having 1000s of groupId for a topic an issue? will the Kafka cluster cleanup unused groupid safely in the background (metadata and offsets)?

where i can bought that t-shirt! it's so cool!

Great video! Any tips of: 1) how to define and monitor a good number of partitions? 2) how to handle consumers failures? For example, how to not get stucked because of a miss-formed message?

@pavelkouzmitcheu7965

6 ай бұрын

I guess you already figured everything out as you asked two years ago, but if you need a web console for managing/monitoring topics/brokers/clusters then provectuslabs/kafka-ui is a very simple. open source and easy to install solution. With docker it will take few minutes to install and run.

Very nice

How do we scale(on what criteria) consumers in consumer group(considering we have 3 partition with 1 consumer to start with )? is it message count in topic and auto scaler decides to add in new consumer service ?

Where can I get hands on api examples

Hello, great video. I'm trying to find a strategy to resolve transactions of microservices communicated to each other by Kafka. Do you think that is possible? Thanks for your help.

Hi I have a doubt in this partition

I have a question, because of two instances in the same consuming apply (group) will be assigned to differences of partition right ? so could i it miss messages from partition it not assigned ?

@nhatpham2069

3 ай бұрын

A consumer in a consumer group is definitely assigned to a particular partition. The ideal scenario is that the number of consumers in a consumer group is less or equal to the number of partitions so that there are no consumers being in idle state.

@SahilShaikh-mk8tz

2 ай бұрын

Even I have same doubt, if multiple instances are consuming, and getting messages from different partitions, won't that miss the messages from the partition from which it is not distributed?

Can I have the consumer act as a producer(publisher) as well? I have a user case where a consumer (C1) polls a topic and pulls messages. after processing the message and performing a commit, it needs to notify another process to carry on remaining work. Given this use case is it a valid design for Consumer (C1) to publish a message to a different topic? i.e. C1 is also acting as a producer

@jorickcaberio1865

Жыл бұрын

use kafka streams

Number of consumers is restricted to number of partitions?

@mosespeter9711

Жыл бұрын

Number of consumers in a consumer group is limited to number of partition. You can have multiple consumer group consuming from one partition

Complicated!!!