How Data Engineers Manage Data At A Hyperscale At FAANGs - The Vs of Big Data - Ft.
Ойын-сауық
A few months back I put together a conference that has been seen by well over 10,000 unique viewers!
One of the popular talks was Zach Wilson's who talked on his experience being a data engineer at companies like Facebook, Netflix and Airbnb.
If you'd like to learn more about data engineering, consider checking out Zach's Linkedin!
/ eczachly
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer In 2022
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.
Пікірлер: 17
Here is the link to Zach's presentation - docs.google.com/presentation/d/1S4tNU1jBmCoR94CBk65TJCfpzDrHVrWt9gi9rWMG2Hc/edit?usp=sharing
Both of my inspirations for Data Engineering together ❤
@SeattleDataGuy
Жыл бұрын
You're too kind!
This is a great podcast. So many things to learn about data engineering at hyperscale. Thank you Zach and Benjamin for doing this!❤
@SeattleDataGuy
11 ай бұрын
Glad you enjoyed it!
Here is the link for the little book of pipelines which Zach talks about github.com/EcZachly/little-book-of-pipelines
I need to drop from live, can you please upload the recording. Really good and helpful session. Thanks to both of you!
@SeattleDataGuy
Жыл бұрын
It'll be here!
Just to confirm that doesnt in spark 3.0 all of these smb and broadcast joins were handled auto by physical plans in spark which uses tungsten engine. And do we still need these processes to handle large data sets.
I’m not quite understanding how the variety example helps or what it does besides unions Data sets. Any clarifications would be appreciative
@DatawithZach Could you point me to a resource of how you are implementing the Scala code around time stamp 42:15? I'm not familiar with this/how this is done.
Could you share the links he mentioned? like the one from netflix and the dedup pipeline, Thanks !!!
@SeattleDataGuy
Жыл бұрын
Here is the link to his presentation docs.google.com/presentation/d/1S4tNU1jBmCoR94CBk65TJCfpzDrHVrWt9gi9rWMG2Hc/edit?usp=sharing
Data conventions or conferences worth checking out?
@SeattleDataGuy
Жыл бұрын
Putting together a list here! seattledataguy.substack.com/p/upcoming-data-events
Man makes 500k a year and still got the bunk bed 😂
@SeattleDataGuy
Жыл бұрын
I think he was at a creator house with Tina and Ken when we filmed this...also clearly you have never seen this tweet 😂 twitter.com/anothercohen/status/1412531460372795393