In this video I speak with Andrew Lamb, Staff Software Engineer @Influxdb. We discuss FDAP (Flight, DataFusion, Arrow, Parquet) stack for modern OLAP database system design. Andrew shared some insights into why the FDAP stack is so powerful in designing and implementing a modern OLAP database.
Chapters:
00:00 Introduction
01:48 Understanding Analytics: Transactional vs Analytical Databases
04:41 The Genesis and Goals of the FDAP Stack
09:31 Decoding FDAP: Flight, Data Fusion, Arrow, and Parquet
12:40 Apache Parquet: Revolutionizing Columnar Storage
17:18 Apache Arrow: The In-Memory Game Changer
23:51 Interoperability and Migration with Apache Arrow
27:10 Comparing Apache Parquet and Arrow
28:26 Exploring Data Mutability in Analytic Systems
29:19 Handling Data Updates and Deletions
29:24 The Role of Immutable Storage in Analytics
30:42 Optimizing Data Storage and Mutation Strategies
34:20 Introducing Flight: Simplifying Data Transfer
35:02 Deep Dive into Flight's Benefits and SQL Support
39:20 Unpacking Data Fusion's SQL Support and Extensibility
46:12 The Interplay of FDAP Components in Analytics
51:49 Future Directions and Innovations in Data Analytics
56:04 Concluding Thoughts on FDAP and Its Impact
FDAP Stack: www.influxdata.com/glossary/f...
FDAP Blog: www.influxdata.com/blog/fligh...
InfluxDB: www.influxdata.com/
Follow me on Linkedin and Twitter: / kaivalyaapte and / thegeeknarrator
If you like this episode, please hit the like button and share it with your network.
Also please subscribe if you haven't yet.
Database internals series: • Write-ahead-logging
Popular playlists:
Realtime streaming systems: • Realtime Streaming Sys...
Software Engineering: • Software Engineering
Distributed systems and databases: • Distributed Systems an...
Modern databases: • Modern Databases
Stay Curios! Keep Learning!
#datafusion #parquet #sql #OLAP #apachearrow #database #systemdesign