Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! by Robin Moffatt
Ғылым және технология
Have you ever thought that you needed to be a programmer to do stream processing and build streaming data pipelines? Think again! Apache Kafka is a distributed, scalable, and fault-tolerant streaming platform, providing low-latency pub-sub messaging coupled with native storage and stream processing capabilities. Integrating Kafka with RDBMS, NoSQL, and object stores is simple with Kafka Connect, which is part of Apache Kafka. KSQL is a SQL streaming engine for Apache Kafka, and makes it possible to build stream processing applications at scale, written using a familiar SQL interface.
In this talk we’ll explain the architectural reasoning for Apache Kafka and the benefits of real-time integration, and we’ll build a streaming data pipeline using nothing but our bare hands, Kafka Connect, and KSQL.
Gasp as we filter events in real time! Be amazed at how we can enrich streams of data with data from RDBMS! Be astonished at the power of streaming aggregates for anomaly detection!
Пікірлер: 30
Probably one of the best talk I've seen on Kafka capabilities. If you don't agree with me, please no thumb down, simply provide more relevant links 😉
Demo code: github.com/confluentinc/demo-scene/blob/master/build-a-streaming-pipeline/ Slides: rmoff.dev/ljc-kafka-01
@olehprotsenko3561
3 жыл бұрын
Thanks for the demo, would be great to have an example with KSQL (table) aggregation group by 2 or more fields as a key.
Very helpful in understanding some basics of ksql, streams, tables, topics and how are they are interconnected.
Extremely helpful and to the point. Thank you so much!
I love this! thank you, clear and simple.
Really very good presentation Robin
Great demo, Extremely useful..
Such an amazing gateway introduced to us which talks about Streaming ETL. Thank you Mam . Hope to talk to you one day !
great talk.
Great talk, and the pointer was pretty cool
Excellent Talk
awesome KSQL!! awesome Robin!!
Very impressive session
Good presentation for KSQL insights
very very good presentation, thanks very much
great demo :)
Excellent demo
awesome demo
Great explanation !
Awesome!
Very nice
17:22 LETS use CAPS FOR STREAM NAMES and TOPICS but not ALWAYS
Is there any tutorial document for a JDBC-Connector ??
@rmoff
4 жыл бұрын
Yes, check out rmoff.dev/youtube for several tutorials using this connector
Which one is best for developers career, kafka or node js
@jonnytheponny5753
4 жыл бұрын
actually at least both: they are completely orthogonal skills. node a programme language and kafka an architectural / data persistence pattern. In a real world you would always need both.
@krishnakummar16
4 жыл бұрын
for developers, solve problems, write algorithms. kafka is just a tool
all those late comers, so distracting