A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets - Jules Damji
Ғылым және технология
"Of all the developers' delight, none is more attractive than a set of APIs that make developers productive, that are easy to use, and that are intuitive and expressive. Apache Spark offers these APIs across components such as Spark SQL, Streaming, Machine Learning, and Graph Processing to operate on large data sets in languages such as Scala, Java, Python, and R for doing distributed big data processing at scale. In this talk, I will explore the evolution of three sets of APIs-RDDs, DataFrames, and Datasets-available in Apache Spark 2.x. In particular, I will emphasize three takeaways: 1) why and when you should use each set as best practices 2) outline its performance and optimization benefits; and 3) underscore scenarios when to use DataFrames and Datasets instead of RDDs for your big data distributed processing. Through simple notebook demonstrations with API code examples, you'll learn how to process big data using RDDs, DataFrames, and Datasets and interoperate among them. (this will be vocalization of the blog, along with the latest developments in Apache Spark 2.x Dataframe/Datasets and Spark SQL APIs: databricks.com/blog/2016/07/1... databricks.com/glossary/what-...)
Session hashtag: #EUdev12"
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: databricks.com/product/unifie...
Connect with us:
Website: databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / databricks
Instagram: / databricksinc Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. databricks.com/databricks-nam...
Пікірлер: 38
Now i know about RDDs, DataFrames and Datasets. Thanks for explaining it more precisely. Appreciated.
Such a pleasure to hear him talk!
What an amazing talk! Crisp and Clear! truly impressed.
Thanks for in-depth explaining RDD DF And DS...
this is a brilliant and fluid explanation
Amazing talk, I left off Spark to move in to ML when there was only RDD, I came back and see DataFrame in Spark and I am totally confused, your video helped a lot, Thank you
Excellent talk. Thanks Jules Damji.
it was very insightful, such talks really helps developer why/how one should use structure API
Amazing presentation. Very intuitive..Thanks Boss!
An excellent talk by a clear master.
Amazing talk! very well explained indeed.
What a brilliant talk!! Thanks
This is best and clear talk on 3 APIs
Very well explained !! Thank's
Amazing Talk. Thank you!
Thanks for the video. Very understandable!
so good!!! thanks for this
this guy is such a good speaker
I had a nice learning time thanks for the talk!
Only 300 likes for such an informative, crystal clear talk??
excellent explanation!! :D
I am wondering how the "type safe" feature combines with the "unstructured data" that is the nature of data in the systems that spark would be used in.
Very good talk!
I was trying out the example you mentioned @10:46 and as i am getting compile time error, I had to rewrite the final statement as below. parsedRdd.filter( content => content._2 == "en").map(filteredContent => filteredContent._3).reduce(_+_).take(100).foreach(reducedContent => printf(s"$reducedContent._1: $reducedContent._2")) I would really appreciate if you can review above statement
Nice and informative video
brilliant talk!
Thank you!
Amazing Talk
awwwwesome talk thanks!
Thanks! Can you attach the links here?
wow
No SS
Hibud
This was amazing! Pretty well explained! Thanks!
Great Talk