No video

Kafka Interview questions and answers for 2024 for Experienced | Code Decode [ MOST ASKED ] | Part-1

For java developers top kafka interview questions for freshers and experienced we have covered most asked kafka interview questions
Udemy Course of Code Decode on Microservice k8s AWS CICD link:
openinapp.co/u...
Course Description Video :
yt.openinapp.c...
Vamsi’s LinkedIn : / vamsi-karuturi
What is Apache Kafka?
Its a publish-subscribe based durable messaging system, exchanging data between processes, applications, and servers.
Apache Kafka, created by former LinkedIn data engineers and written in Scala and Java, became an open-source technology in 2011. It started as a highly scalable messaging system and is now a crucial part of the Confluent Stream Platform, managing trillions of events daily. Many well-known companies trust and use Apache Kafka.
In real time, managing data and logs is a big task. This involves processing, reprocessing, analyzing, and handling information, often in real-time. This is where Apache Kafka shines. It plays a significant role in handling messages efficiently in a streaming environment. The principles behind Kafka's design focus on meeting the increasing demand for systems that can handle a large volume of data
Architecture of kafka
The architecture of Apache Kafka is designed for distributed, fault-tolerant, and scalable handling of streaming data.
Let's explore the key components and their interactions in detail:
Producer
Producers are applications that publish messages to Kafka topics. They create ProducerRecord like this public ProducerRecord
Topic:
Topics are logical channels or categories to which messages are published. Topics can be divided into partitions for scalability and parallelism.
Partition:
Each topic is divided into partitions, which are the basic units of parallelism. Partitions allow Kafka to distribute and parallelize the processing of messages.
Broker:
Responsibility: Kafka brokers are individual servers within the Kafka cluster. They store and manage data, handle producer and consumer requests, and participate in the replication and distribution of data.
ZooKeeper:
Responsibility: ZooKeeper is used for distributed coordination and management of the Kafka cluster. It maintains metadata, leader election, and helps manage the overall state of the cluster.
Topic Partitions Replication:
Responsibility: Each partition has multiple replicas for fault tolerance. Replicas are distributed across different brokers. One replica is designated as the leader, and the others are followers.
Consumer:
Responsibility: Consumers subscribe to one or more topics and consume messages from partitions. Each consumer group can have multiple consumers, and each consumer within a group processes a specific subset of partitions.
Why did you use Kafka?
Scalability
Fault Tolerance
Durability
Real-time Stream Processing
Partitioning
High Throughput
Data Retention Policies
Dynamic Reconfiguration
Community and Support
What is the role of the ZooKeeper in Kafka
ZooKeeper acts as a centralized and reliable coordination service for Kafka, ensuring that the distributed components of a Kafka cluster can work together seamlessly.
It helps in managing the dynamic nature of Kafka clusters, providing fault tolerance, and enabling coordination among different components.
It's worth noting that starting with Kafka version 2.8.0, there have been efforts to reduce Kafka's dependency on ZooKeeper, with ongoing work to replace it with a self-managed metadata store. Its said that with kafka 4.x it will be completely removed while with 3.x its removed but still its not prod ready.
Most Asked Core Java Interview Questions and Answers: • Core Java frequently a...
Advance Java Interview Questions and Answers: • Advance Java Interview...
Java 8 Interview Questions and Answers:
• Java 8 Interview Quest...
Hibernate Interview Questions and Answers:
• Hibernate Interview Qu...
Spring Boot Interview Questions and Answers:
• Advance Java Interview...
Angular Playlist:
• Angular Course Introdu...
SQL Playlist: • SQL Interview Question...
GIT: • GIT
Subscriber and Follow Code Decode
Subscriber Code Decode: www.youtube.co...
LinkedIn : / codedecodeyoutube
Instagram: / codedecode25
#kafka #codedecode #kafkainterviewquestions

Пікірлер: 106

  • @sayanbiswas8847
    @sayanbiswas88478 ай бұрын

    Very important video. Kafka, Asynchronous messaging is forming the main part of backend Java developer interviews these days. Thanks team. Keep such content coming !

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    Thanks Sayan 😊

  • @Uniquevibers944
    @Uniquevibers9444 ай бұрын

    Working in Kafka for the past 2 years without knowing much but just configuring topics. Now u made me understand the whole. Thanks!

  • @CodeDecode

    @CodeDecode

    4 ай бұрын

    😊😊👍

  • @ravipanchal2259
    @ravipanchal22593 ай бұрын

    Awesome video. Saved my 2-3 hours of reading. Waiting for part-2

  • @CodeDecode

    @CodeDecode

    3 ай бұрын

    😊😊

  • @amansharma-ce5di
    @amansharma-ce5di4 ай бұрын

    Thank you for this video. Eagerly awaiting for the Part-2

  • @CodeDecode

    @CodeDecode

    3 ай бұрын

    Sure we will upload it soon

  • @Akashdeep-ov5kb
    @Akashdeep-ov5kb4 ай бұрын

    Thanks for great videos. Please create a video on kafka rebalancing, how to clear lag in kafka, how to decide how many partitions to chose for a kafka topic.

  • @CodeDecode

    @CodeDecode

    3 ай бұрын

    Noted👍

  • @ayafathy3232
    @ayafathy32327 күн бұрын

    great video ,waiting for part two

  • @CodeDecode

    @CodeDecode

    3 күн бұрын

    Sure, we will create it soon

  • @taranggupta9018
    @taranggupta90187 ай бұрын

    Thanks a lot for helping us to prepare for the interview.. Also the baby from the background is also trying to teach us something..it is sad we can't understand 😅

  • @CodeDecode

    @CodeDecode

    7 ай бұрын

    🙊🙊🙊🙊 sorry for the inconvenience❤️ And thanks for understanding ,😘

  • @suvarnammavadla4804
    @suvarnammavadla4804Ай бұрын

    Hi Mam, please create a video by covering all Kafka related questions for interview perpose

  • @CodeDecode

    @CodeDecode

    Ай бұрын

    Sure 👍👍

  • @ZabiVlogs3091
    @ZabiVlogs30916 ай бұрын

    I was asked recently in morgan stanley interview how consumer handles messages coming in sequence if the consumer fails. Wish i saw this video before. Useful content ❤

  • @CodeDecode

    @CodeDecode

    6 ай бұрын

    Thanks 😊👍

  • @ShivaKumarSatakuri
    @ShivaKumarSatakuri8 ай бұрын

    I'm sure this channel will grow rapidly and extensively. Content quality is too good.

  • @mohammedsufi4649
    @mohammedsufi46496 ай бұрын

    Very useful. Please upload this video continuation too. Thanks a lot!

  • @vidyakrishna901
    @vidyakrishna9017 ай бұрын

    Very informative video on kafka ..always referring these videos with brilliant teacher to have us..thanks for come up with such videos.

  • @user-ht1iv5mu3i
    @user-ht1iv5mu3i7 ай бұрын

    Hi, Thanks to Code Decode team, Can you please clarify the below points; 1. Can producer publish data on followers also or only on leader? 2. Will offset be removed after consumer consumed, if it will not remove till some time,then data will be huge data in the partition, then what is the limit of partitions? 3. Can multiple consumers consume the same offset? 4. how different consumers knows from which offset they need to start consuming? 5. Can multiple consumers from the same group or different groups consume same offset? 6. how to handle consumer lag any case of consumer fail to consume the data?

  • @JavaChunks

    @JavaChunks

    6 ай бұрын

    1. Only producer and followers will replicate data 2. Once offset is consumed, it’s consumed for particular consumer group so message can still be consumed again. There is retention policies, after that data will be deleted permanently 3. Yes, as long as consumers belong to different group 4. No, as partition data us read by specific consumer of group only 5. Fine tune buffer size, rebalance consumers or topic partition etc

  • @tlprasad
    @tlprasad3 ай бұрын

    Your presentation is very good. 👍 There are so many videos on Udemy which don’t follow minimum standards. I really liked the way you explained the concept. I am not sure if you have your training courses on udemy as well. If not you can try that and you will be a star teacher over there. Just one suggestion though, don’t repeat a point of explanation multiple times, to make your video crisp and clear.

  • @CodeDecode

    @CodeDecode

    3 ай бұрын

    Thanks man. Means a lot 😊😊

  • @user-co6wk8hi4d
    @user-co6wk8hi4d6 ай бұрын

    Wonderful video. Learn a lot in 30mins about kafka.

  • @CodeDecode

    @CodeDecode

    6 ай бұрын

    😊😊

  • @leonzakaim
    @leonzakaimАй бұрын

    Order of mesages in multiple partitions How the order of messages is kept , when we have few partitions ?

  • @gyaneshwarm5786
    @gyaneshwarm57862 ай бұрын

    Excellent Video , Thanks for the detailed notes

  • @CodeDecode

    @CodeDecode

    2 ай бұрын

    You’re welcome

  • @gurkiratkaur4087
    @gurkiratkaur40874 ай бұрын

    hi, it was a very helpful video. If you could please make more videos to make more questions on kafka, it will be extremely helpful.

  • @SuperHuman748
    @SuperHuman7488 ай бұрын

    Thanks to the Code Decide team, it's great I have been following ur classes for quite some time really appreciated. In this particular video, (1)What are the reasons that the leader broker dies ? (2) How the Leader Election does -- what's the Behind Algorithm and how this ISR works when data is produced in a broker/partition ? (3) Why Exactly Three 3 copies of Follower Brokers with the same set of data in Partitions ? (4) Do We Now have a successful Kafka running version Without ZOOKEEPER Dependency ?? which Version is it ?? How successful this new version in market is now ?? Please reply with your insights and answers. Thanks keep doing this nice work and knowledge sharing for a lot of aspirants to become successful IT careers like us. 👍🙏😊

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    1) broker is nothing but a server, which we start with a command. Now it can stop or crash any time just like our normal servers. 2) using epoch number. The most recently and most updated isr / broker is choosed az as new leader

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    3) it's a best practice to use at least 2 replica for fault tolerance. You can have 0 or you have 10

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    4) we are in 3.x version as of now. 4.x.x will have zookeeper completely removed

  • @rahulnikam777
    @rahulnikam7772 ай бұрын

    Thanks for video and explaining Kafka very well. very good content.

  • @sanketkadam4716
    @sanketkadam47163 ай бұрын

    please make more practical videos on kafka, Thank you so much

  • @CodeDecode

    @CodeDecode

    3 ай бұрын

    Sure we will create it soon

  • @kamalch8928
    @kamalch89284 ай бұрын

    Hi Mam, Thanks for an amazing content. Requesting you to create more interview videos on Kafka.

  • @ajaydangi582
    @ajaydangi5827 ай бұрын

    next level of explanation . want more videos on this👌👌

  • @CodeDecode

    @CodeDecode

    7 ай бұрын

    Sure we will create more on this

  • @devendra.sanghavi
    @devendra.sanghavi3 ай бұрын

    Very beautiful explanation.

  • @CodeDecode

    @CodeDecode

    3 ай бұрын

    Thanks

  • @vinaykumarva8538
    @vinaykumarva85383 күн бұрын

    Brocker will create automatically? Or do we need to create automatically

  • @derickdaniel6914
    @derickdaniel69144 ай бұрын

    How to handle the scenario where a consumer gets delayed reading a message, for example- Consumer B reads "booking successful" message before and a Consumer A could read "payment failure" message ? When both messages are from different topics

  • @captainsparrow101

    @captainsparrow101

    Ай бұрын

    I am not sure if I understood your questions completely how kafka will manage if data got consumed in different order like order placed is consumed without payment message is received. I will suggest there will be two different microservices one will consume data from payment topic and once confirmed payment is done it will publish data to a different topic from where booking service will book the order Now second scenario if there is single microservice consuming from a single topic then it's code responsibility to check payment status first then go for booking. Please correct me if I am wrong

  • @NareshKumarD107
    @NareshKumarD1077 ай бұрын

    Pls make part 2 and also pls explain clearly about the consumer groups in nxt video.

  • @CodeDecode

    @CodeDecode

    7 ай бұрын

    Sure we will create it soon

  • @sheikhmohduzair2886
    @sheikhmohduzair2886Ай бұрын

    kindly cover the remaining topics.

  • @CodeDecode

    @CodeDecode

    Ай бұрын

    Sure 👍

  • @sujitharavuri3572
    @sujitharavuri35722 ай бұрын

    Thanks for the video. Very informative.

  • @CodeDecode

    @CodeDecode

    2 ай бұрын

    You're welcome!

  • @venkateshthavva7499
    @venkateshthavva74994 ай бұрын

    Awesome content. Thanks for sharing knowledge.

  • @CodeDecode

    @CodeDecode

    4 ай бұрын

    You are welcome

  • @user-bm9ns6hv3i
    @user-bm9ns6hv3i5 ай бұрын

    Hi please clarify that how can u ensure that Kafka consumes the messages in the sequential? 2. Will publisher publish duplicate messages? 3. How to retailn failed messages from the dlq within 7 days?

  • @johnnybreathe
    @johnnybreathe8 ай бұрын

    I was waiting for this 😮 😊 please continue thanks for the knowledge 🙏

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    Sure thanks 😊😊

  • @susobhandas6143
    @susobhandas61432 ай бұрын

    You have made video with love, I like it

  • @CodeDecode

    @CodeDecode

    2 ай бұрын

    Thanks

  • @user-gj3nq9os7f
    @user-gj3nq9os7f6 ай бұрын

    thank u so much for making this kind of video please make second part also

  • @DhrumilShahDOTin
    @DhrumilShahDOTin17 күн бұрын

    owsm

  • @akashkharade2259
    @akashkharade22598 ай бұрын

    Waiting for this long time, Thanks. First comment

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    Thanks for your comment Akash 😊👍

  • @sreenivaskappala6266
    @sreenivaskappala62666 ай бұрын

    Great informative video, Do share the Part - 2

  • @chrislaingba2211
    @chrislaingba22118 ай бұрын

    You are the Best 🙏🙏

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    😊😊 Thanks 👍😊

  • @testtest-zb9rp
    @testtest-zb9rp8 ай бұрын

    Thanks very useful. We need more interview questions on kafka.

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    Sure 👍👍

  • @sumant2000
    @sumant20004 ай бұрын

    Awesome

  • @CodeDecode

    @CodeDecode

    4 ай бұрын

    Thanks

  • @satyasaineelapala570
    @satyasaineelapala5706 ай бұрын

    Great Video

  • @CodeDecode

    @CodeDecode

    6 ай бұрын

    Thanks 👍

  • @ayushirathore1081
    @ayushirathore10818 ай бұрын

    Please make video on grafana and kibana and Zipkin as well

  • @ektathakkar3225
    @ektathakkar32258 ай бұрын

    Pls make video on basis of cloud like clusters, pods, etc

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    Sure ekta we will create videos soon on kubernetes concept

  • @RakeshKumar-wm8pn
    @RakeshKumar-wm8pn7 ай бұрын

    Please upload more and more question

  • @CodeDecode

    @CodeDecode

    7 ай бұрын

    sure we will upload it soon

  • @karthikeyanrm3446
    @karthikeyanrm34468 ай бұрын

    Please continue interview series.

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    Sure 😊👍👍

  • @maddipatlamadhura1500
    @maddipatlamadhura1500Ай бұрын

    Thank you very much

  • @swatibhosle496
    @swatibhosle4966 күн бұрын

    When is part 2 releasing ?

  • @sandysworld7529
    @sandysworld75298 ай бұрын

    Many thanks❤

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    😊😊👍👍

  • @unemployedcse3514
    @unemployedcse35147 ай бұрын

    Awesome ❤, need other topics too

  • @CodeDecode

    @CodeDecode

    7 ай бұрын

    👍👍😊sure

  • @prashantkarpe1022
    @prashantkarpe10228 ай бұрын

    Very nice explanation can you please share the new coupon code for this course Master Spring Boot Microservice & Angular K8s CICD AWS??

  • @MohaideenA
    @MohaideenA7 ай бұрын

    Nice vedio on Kafka, but there is a small mistake. You said both Producer writes and Consumer reads from Leader partition, which is incorrect. As this would stop the concept of parallelism when there are more than one consumer, and both trys to read from leader in your case. Generally, the leader status of the partition does not have any bearing on the consumer. The leader status only applies to the producers and the broker, so the consumer is fine to read all its data from both the follower Partition and the leader Partition.

  • @CodeDecode

    @CodeDecode

    7 ай бұрын

    "The leader handles all read and write requests for the partition while the followers passively replicate the leader" As per the documentation of Kafka itself. kafka.apache.org/081/documentation.html To answer your question on parallelism, "Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster." Again as per the documentation 😊😊

  • @MohaideenA

    @MohaideenA

    7 ай бұрын

    @@CodeDecode ​​ Thanks.. I could'nt able locate the info provided in the documentation.. But for simplicity, lets say we only have one topic, three brokers and 3 consumers belongs to different consumer group. Now per the documentation only one Broker can have a leader for this one Topic and rest 2 are followers. (I assume we both agree on this) In this case, lets say reads happens only from the leader, then all three consumer would read only from the leader for this topic, which stops parallelism right. So ideally for Kafka to support parallelism here, one broker is assigned per consumer, so one of the consumer may read from leader , but the other 2 has to read from the follower. Otherwise in your case, all consumer will read from leader, which is from same broker, which distrupts parallelism. so here's a more accurate representation: "The leader handles all write requests for the partition, actively replicating the data to its followers. While the leader is responsible for coordinating writes, both leaders and followers can handle read requests. Consumers can read data from both the leader and followers, providing fault tolerance and parallelism." please correct me if my understanding is not correct .. still learning, and thanks for the reply..

  • @ShamaSh444

    @ShamaSh444

    7 ай бұрын

    @@MohaideenA you are correct , @Code decode got this one incorrect..

  • @CodeDecode

    @CodeDecode

    7 ай бұрын

    Your explanation seems reasonable to me too but am still confused as documentation is contradicting it. Give me some time. Will try to connect to open source team. Either they will revise the documentation or else our doubt will be cleared

  • @MohaideenA

    @MohaideenA

    7 ай бұрын

    @@CodeDecode Thanks for getting back to me..

  • @akashsaha9366
    @akashsaha93668 ай бұрын

    Thanks for this good video.

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    ☺️☺️

  • @sivaprasad1171
    @sivaprasad11718 ай бұрын

    Can you please cover the remaining topics of Kafka..

  • @rishiraj2548
    @rishiraj25484 ай бұрын

    👍

  • @NageshK-pv7kg
    @NageshK-pv7kg4 ай бұрын

    How much charge and duration ?

  • @CodeDecode

    @CodeDecode

    4 ай бұрын

    Sorry didn't get you question

  • @rishiraj2548
    @rishiraj25488 ай бұрын

    🙏👍

  • @CodeDecode

    @CodeDecode

    8 ай бұрын

    😊😊

  • @muhammadfahim8978
    @muhammadfahim89782 ай бұрын

    Your pronounciation of "T" alphabet is so disturbing

  • @vaibhavgarg4267
    @vaibhavgarg42676 ай бұрын

    very helpful mam. pls share contact

  • @CodeDecode

    @CodeDecode

    6 ай бұрын

    Thanks. Ucan connect with us on codedecodebusiness@gmail.com

  • @AK-tt7fc
    @AK-tt7fc3 ай бұрын

    Thank you for this video. Eagerly awaiting for the Part-2

Келесі