Apache Kafka Key Terminologies and Architecture Internal Working | Features | In Depth Explanation
In this video of code decode we have kafka architecture which is very important in terms of kafka interview questions and answers
Udemy Course of Code Decode on Microservice k8s AWS CICD link:
openinapp.co/udemycourse
Course Description Video :
yt.openinapp.co/dmjvd
Introduction to kafka : kzread.info/dash/bejne/pKBpstCdd6TXZ9Y.html
Kafka implementation with producer and consumer :
kzread.info/dash/bejne/iYOEsqlupNynoMo.html
Topic - A bucket of messages (or events in case of kafka) where services may place or read messages from. Messages are organized and durably stored in topics. A topic is similar to a folder, and these messages are the files in that folder. Topics are multi-producer and multi-subscriber. Messages can be read as many times as needed, unlike traditional messaging systems, msgs / events in kafka are not deleted after consumption. Instead, you can define for how long Kafka should retain those events/msgs.
Kafka topics are divided into a number of partitions, which contain msgs/events in an unchangeable sequence. Each msg in a partition is assigned and identified by its unique offset. A topic can also have multiple partition. This allows multiple consumers to read from a topic in parallel.
In Kafka, replication is implemented at the partition level. The redundant unit of a partition is called a replica. Each partition usually has one or more replicas over a few Kafka brokers in the cluster.
Every partition (replica) has one server acting as a leader and the rest of them as followers. The leader replica handles all read-write requests for the specific partition and the followers replicate the leader. If the lead server fails, one of the follower servers becomes the leader by default.
When a producer publishes a msg to a topic, it is published to its leader. The leader appends the msg to its offset and increments its msg offset.
Summary
Kafka has a sized component called a cluster.
Inside a cluster, we’ve got several servers, also known as brokers (usually at least three brokers to provide enough redundancy.)
when a message gets sent to a broker, it gets sent to a particular topic.Topics allow us to categorize data. The data can eventually get broken into several partitions.
By splitting the topic into multiple partitions, we’re facilitating scalability, as each partition can be read by a separate consumer.
Each broker is responsible for receiving messages from producers and committing those messages to disk. The broker is also responsible for answering consumers’ fetch requests and serving them.
While designing a Kafka system, it’s always a wise decision to factor in topic replication. As a result, its topics’ replicas from another broker can solve the crisis, if a broker goes down. Broker1 has Topic 1 and 10 partitions , its replica is in Broker2, so on and so forth. It has got a replication factor of 2; it means it will have one additional copy other than the primary one. It is not possible to have the number of replication factor more than the number of available brokers.
Moreover, exactly one consumer instance reads the data from one partition in one consumer group, at the time of reading.
Since, there is more than one consumer group, in that case, one instance from each of these groups can read from one single partition.
However, there will be some inactive consumers, if the number of consumers exceeds the number of partitions. Let’s understand it with an example if there are 8 consumers and 6 partitions in a single consumer group, that means there will be 2 inactive consumers.
Kafka Cluster: A Kafka cluster is a system that comprises of different brokers, topics, and their respective partitions.
Most Asked Core Java Interview Questions and Answers : kzread.info/head/PLyHJZXNdCXscoyL5XEZoHHZ86_6h3GWE1
Advance Java Interview Questions and Answers : kzread.info/head/PLyHJZXNdCXsexOO1VQ4vs-BM2-8CKrixd
Java 8 Interview Questions and Answers : kzread.info/head/PLyHJZXNdCXsdeusn4OM33415DCMQ6sUKy
Hibernate Interview Questions and Answers : kzread.info/head/PLyHJZXNdCXsdC-p2186C6NO4FpadnCC_q
Spring Boot Interview Questions and Answers : kzread.info/head/PLyHJZXNdCXsexOO1VQ4vs-BM2-8CKrixd
Angular Playlist : kzread.info/dash/bejne/dXWgmbSKg8fXYrQ.html
SQL Playlist : kzread.info/head/PLyHJZXNdCXse86eLuwy5uZohd_bddE9Ni
GIT : kzread.info/head/PLyHJZXNdCXscpl6pxOnL2lRWJlzvzjyZE
Subscriber and Follow Code Decode
Subscriber Code Decode : kzread.info
LinkedIn : www.linkedin.com/in/codedecodeyoutube/
Instagram : codedecode25
#kafka #codedecode #kakfainterviewquestions
Пікірлер: 126
You are doing a Spectacular job by providing these video tutorials. I wish you may achieve greater heights. May God bless you always!! Looking forward to more practical sessions on Kafka series, Kafka Stream APIs and so on.
@CodeDecode
Жыл бұрын
Thanks a lot Sudhir 👍👍
I just love the way you teach, very well explained, a series of great tutorials. Thanks a lot.
@CodeDecode
Жыл бұрын
🙂🙂Thanks a lot
I really like the way you teach keeping it simple but effective. Keep up the good work and continue doing the same. I would like to ask you to add a playlist with interview questions for Microservices. It will greatly benefit large number of people
Its a great learning again with code Decode, Thank you so much for your effort🙏
@CodeDecode
Жыл бұрын
Thanks Ashish 🙂👍
I have an interview for Kafka, Springboot based role. After seeing this video I got the confidence to crack the interview.
Speechless... Superb Explaination for this tough part
@CodeDecode
10 ай бұрын
Thanks ramesh
Yes interview questions 🧑💻 and some water lol just kidding. This a big topic to go through wow. Thanks for the big effort 🤗 and the videos.
@CodeDecode
Жыл бұрын
😂😂thanks 🙂
Please cover micro services design patterns completely.. humble request from loyal fans😀
@CodeDecode
Жыл бұрын
sure we will create it soon
way of explanation is short and simple...voice is unique and cool... I want to thank you so so much.... But please cover full cover java, Springboot,spring core,full length videos like 2 hour or 3 hour deeply, one of the underrated channel
@CodeDecode
Жыл бұрын
Thanks 🙂🙂sure 👍
Best explanation as always .. 👌👌👌 Thanks a lot for creating such a great video. It’s very informative… 🙌
@CodeDecode
Жыл бұрын
Thanks Dipali 🙂🙂
thank you so much for this wonderful video
Thanks for all your efforts. Please create video on kafka just for interview purpose, it's a much needed one .
Hi.... Its was a great learning with code Docode, Thank you so much for your effort
@CodeDecode
Жыл бұрын
Thanks Purnendu🙂🙂👍👍
Your teaching is too good please cover more on Kafka, spring boot , microservices interview related senario based questions for experienced it's a humble request 😊 , keep up the good job I am totally enjoying your video's 😊
@CodeDecode
Жыл бұрын
Thanks Harshini 🙂. Sure 👍👍
Thanks very well explained. You are doing a great job 👍
@CodeDecode
Жыл бұрын
Thanks Nova ❤️❤️
Thanks for explaining very nicely..
Simple and effective 👌 explanation
@CodeDecode
16 күн бұрын
Thanks 😊👍
Nice one .. Can you please make next video of Kafka current Challenges like if Cosumer throws exception how it can restart from particular offset where it lefts or skip it ..configuration changes reqd
A very good tutorial , thanks a lot for the clear explanation
@CodeDecode
Жыл бұрын
Thanks Sandeep 🙂👍
thanks for the detailed explanation. but i have few doubts coming like what if i have 2 brokers but replication factor is 3. then 1 partition will be on 1 broker and other 2 partitions will be on another broker ? Please cover these combinations in next video on kafka from interview perspective.
Hi Ma'am, please make video on the implementation of Active MQ and Rabbit MQ also. Thanks for these videos and please continue adding videos in this microservices series. :)
@CodeDecode
Жыл бұрын
Sure Karan 🙂👍
Please create a separate video for Kafka Interview Questions. Thank you in advance. Your videos are helping a lot in cracking the interview
@CodeDecode
7 ай бұрын
sure we are planning for it and we will upload it soon
Nice explanation ....thanks for uploading
@CodeDecode
Жыл бұрын
Thanks Tanveer
Thank you for Uploaded Videos , I have Subscribed your channel recently going though all topics really helpful and appreciate your effort. Need video on ( Microservices Design Patterns, Testing Debugging Micro Services, Event Driven approach, Monitor microservices , Deploy, Integration part, MS communication with same & Different databases , ACID properties achieve with MS)
@CodeDecode
Жыл бұрын
Awesome topics Iskala 🙂👍sure
Please make videos on confluent kafka too !!Thanks😊 How to publish messages from springboot application to control centre starting with avro schema.
Very well explained, it would be helpful if you can create video only for interview perspective. Thanks in advance.
@CodeDecode
Жыл бұрын
Sure Deval. It's in pipeline👍🙂
Thanku mam for this imformative vedio.. If possible please create the Kafka stream vedio too.. Keep up the good work..
@CodeDecode
Жыл бұрын
Thanks Sandeep 🙂🙂
Great work. Thanks a lot
@CodeDecode
Жыл бұрын
you're welcome
Thanks you for the video.👌 Please cover interview questions as well.
@CodeDecode
Жыл бұрын
Sure Rahul 👍
yes please explain kafka streams , kafka connect confluent kafka, its memory managemnt,
Thanks for your effort for explaining the things so easily and efficiently. Need your help regarding a task which was assigned to me. The task is as follows, 1. I have to read the data from DB and push it to kafka. 2. The DB table is updated after 24 hours with new record. 3. In the next iteration only the new records should be pushed to kafka. As the example you have explained was with a string but I have to pass objects to kafka, please help me with this.
@CodeDecode
Жыл бұрын
Make sure object is serializable. Rest won't be impacted much. Make a scheduler that reads data from db. Make a boolean variable named as read and mark all records true which are picked by scheduler and next time pick only unread ones. Design must be around this
@mrengineertrader
Жыл бұрын
Pass ur JSON object who will be a class in value of kafka template and in application.yml in value deserializer pass a jsonserializer except string serializer
Thanks for your efforts. Please create Kafka video for interview purpose and kafka streams tutorial.
@CodeDecode
Жыл бұрын
Sure Sona 👍
Excellent explanation maam😄
@CodeDecode
Жыл бұрын
Thanks akash 🙂👍
Thanks for creating detailed videos, Please create on Kafka stream api.
@CodeDecode
Жыл бұрын
Yes Dilip it's in pipeline🙂👍
please create a video on Kafka Streams. Thanks for all the wonderful videos
@CodeDecode
Жыл бұрын
Sure we will create it soon
Thanks for the video.
@CodeDecode
Жыл бұрын
You're welcome shubham
Hi, Thanks for the video, kindly make a video on kafka streams and kafka inteview questions
@CodeDecode
Жыл бұрын
Sure Ankit 👍👍
Around 13:50 you mention something about a book/pdf. Where can I get a link to it? Also, thanks a lot for this series of videos. Really useful :)
tq madam please do as many as possible sessions in a week
@UmmakaJagadish
Жыл бұрын
please do one real time project
@CodeDecode
Жыл бұрын
Sure Jagadish👍🙂
Mam can you please make a video on How To Deal With The Sum Of Rounded Percentage Not Being 100
Can you please post the Kafka streams video.thank you.
great effort Madam thank you so much
@CodeDecode
6 ай бұрын
you're welcome
Pls explain on kafka streams also. I am waiting for ur reach learning video. Thankq team
@CodeDecode
Жыл бұрын
Sure Thanks Padma 🙂👍
Awesome 😍
@CodeDecode
10 ай бұрын
Thanks 🙂🙂👍
Very well explained
@CodeDecode
Жыл бұрын
Thanks
please continue with the series
@CodeDecode
Жыл бұрын
Sure 👍🙂
Please make separate video about kafka Listener internal working and consumer Lag in spring boot application and advance kafka configurations
@CodeDecode
Жыл бұрын
Sure 👍🙂
You are explanation is good i would like to see kafka streams lecture can you please make it
@CodeDecode
10 ай бұрын
Sure 😊👍👍
great explanation
@CodeDecode
Жыл бұрын
Thanks Nirmal
can you please share the pdf or book you are talking about in video, regarding how consumer consumes message from broker?
Thank you.Please make video regarding kafka streams
@CodeDecode
Жыл бұрын
sure we will create it soon
Please make video on Kafka streams and Kafka connect
Tq
Kafka Streams Please, and Interview Questins Please
HI can you please explain Aggregator design pattern.
Mam... please make a video on Kafka streams as well.
@CodeDecode
3 ай бұрын
sure we will create it soon
Mam please create a video with kalfka stream and also for kalfka interview questions. Thanks in advanced 🙂
@CodeDecode
Жыл бұрын
Sure mayur we will create it soon
You are god!
@CodeDecode
Жыл бұрын
I have no words Raghu. So happy to have this much love from community. But yeah, m far from being god🙏🙏 just a teacher❤. Glad to have you❤❤
Please add it in the playlist for microservices.
@CodeDecode
Жыл бұрын
Done. And added to new kafka specific playlist too. Thanks Sayan 👍🙂
@sayanbiswas8847
Жыл бұрын
@@CodeDecode Your channel is a gold mine for cracking interviews. Keep up the good work.
Hi ,please cover the interview question on Kafka.
@CodeDecode
Жыл бұрын
Sure Mahesh 👍👍
can you please make the interview questions video for Kafka, this will be really helpful for candidates preparing for interview.
@CodeDecode
Жыл бұрын
Sure we will plan it👍👍
@divyareddy8747
Жыл бұрын
Can ypu please share the pdf for detail kafka thing or video
Please do a video on kstream
@CodeDecode
Жыл бұрын
Sure 👍👍
can same message be consumed by two subscriber?
Hi, Thanks for the video, is partitions behaving like hashMap? also hash collision condition? Also please continue Kafka series, waiting for Kafka streams.
@surpriser2292
5 ай бұрын
Producers are aware of which partition they have to write data to. This knowledge is known when the data is sent. The key and data will be serialized and converted into bytes. After the bytes conversion we run a algorithm called murmur2 on the key(generated byte code from serializer) which decides the partition number that we have to write to. If collision occurs that means that the key is given same for two different sets of data, that mean the serialization of key would result in same bytes and murmur2 would spit the same partition. You will end up with wrong data for same key.
please share the links of slides as well :)
could you please share the ppt that you are using for presentations to my gmail, I'm preparing for interviews, it will helpful to me go through slides instantly whenever I want
Mam , where can we get this pdf ?
Can you please share pdf mentioned at 13:52 ?
how can i get code decode pdfs
Kafka streams please
@CodeDecode
Жыл бұрын
Sure 👍👍
How can I can contact i need support for kafka
@CodeDecode
Жыл бұрын
Please contact us at codedecodebusiness@gmail.com
Hi ,please cover the interview question on Kafka.
@CodeDecode
Жыл бұрын
sure saurabh we will create it soon
Can you please share pdf mentioned at 13:52 ?