How NETFLIX onboards new content: Video Processing at scale 🎥
Everyday, #Netflix handles billions of requests regarding movies, trailers and other video content. Delivering at such a large scale needs an #engineering marvel. This #video will talk about how Netflix is able to onboard new video content onto their platform. We go from video chunking to collating 4 second shots into scenes.
Amazon S3 is used to store the video chunks. Netflix also provides open connect servers to internet service providers, which acts like a cache of movies. Most requests to Netflix can be served by this cache, and the remaining are sent over the network. This reduces the bandwidth and time required for Netflix to operate at scale. Synergy at it's finest.
00:00 Problem Description
00:32 Video formats and resolutions
03:18 Chunk processing
05:52 Storage
06:19 OpenConnect for video caching
10:13 Summary
Looking to ace your next interview? Try this System Design video course! 🔥
interviewready.io
With video lectures, architecture diagrams, capacity planning, API contracts and evaluation tests. It's a complete package.
References:
/ high-quality-video-enc...
/ optimized-shot-based-e...
/ video
/ stream
/ keystone-real-time-str...
media.netflix.com/en/company-...
System Design Playlist: • System Design for Begi...
You can follow me on:
LinkedIn: / gaurav-sen-56b6a941
Instagram: / applepie404
Quora: www.quora.com/profile/Gaurav-...
Twitter: / gkcs_
External content used:
Fast and Furious 7 (Car chase scene)
Boone Loves Video: • 🌎Create 3D PATH ANIMAT...
Sacred Games (Available on Netflix)
Johnny English Reborn (What's it called?)
Netflix Logo
Пікірлер: 540
I believe there is a variation to Gaurav's understanding on open-connect caching mechanism. A new uploaded video DOES NOT goes directly to the caching server, instead they are resolved to a 'a small Netflix DB replica" in that region, which in-term populates the caching layer for new videos or most frequently accessed videos. The distinction is important because otherwise the design is assumed to directly populate OC CDNs, which is not the case. So for example, if a viewer in Australia wants to stream a non-cached US content: Instead of directing all the Netflix traffic through under-sea cables, Netflix copies the video file (transcoded) from US-based storage to a storage location in Australia once during off-peak hours. Once the video has reached the continent, it’s copied to all the Open Connect servers present in the ISP networks. This is not to point out a flaw in the video, just a clarification.
@gkcs
Жыл бұрын
Thank you for this amazing clarification 😁
@adityatripathi1904
Жыл бұрын
@@gkcs I am glad. Keep making fun content😄
Content apart , the new animations that you've added are really nice. Extra efforts are appreciated! Keep em coming !
Man, you know how to teach with fun! I must say presentation and video editing is very well done in your videos.
Wow, excellent video. Love your unique style of breaking down the topics in chunks and explaining them neatly. Keep them coming.
holy shit! Just fall for these videos; Thank you from bottom of my heart! Please make more video on this topic!
What a beast! Loved your reference to Amazon S3 out of nowhere! 😎✔️
The material you have in this playlist is gold! I did not expect to learn so much from it. Thanks a lot!
@gkcs
2 жыл бұрын
You're welcome!
Blown by the simplicity you explained such a complex topic!
Bro - Thanks for the great video! Learned a lot. Just subscribed. Keep the great content coming!
Very helpful and simple for anyone who is preparing for software design interview. Your way of explaining is interesting and funny 😊
You have the most unique content on KZread, man. No bs, pure knowledge. Keep going.
@gkcs
4 жыл бұрын
Thanks!
It's amazing to see how Guarav improves the quality of his content, and I can tell you, guys, as a newbie tech KZreadr, it's a big deal. Keep it up, bro! ;-)
@gkcs
4 жыл бұрын
Thanks Victor!
1:23 AWESOME GAURAV, perfect example of what you are explaining, Huge respect bro
Whoa!!! Thank you so much for making such unique and great content videos ;)
As i have some experience in this area, i would like to say that you delivered crisp content on the same. Good job!
@gkcs
4 жыл бұрын
Thank you!
Another informative video from you. Nice explanation. Thanks. Just wanted to add something on the caching part. Many websites take help of CDN providers like Akamai to do the caching on behalf of them. The CDN providers have the required infrastructure across the globe wherein they have placed their caching servers in most of the countries.
@gkcs
4 жыл бұрын
That's interesting, thanks!
Awesome man! Simple and to the point. The best kind of online content.
Love the energy. Your videos are great.
Hi Gaurav, thanks for the video. Impeccable. Keep doing the great work!!
Have been working in this arena for a while now, you got everything correct man, other stuff you post usually goes over my head because I haven't dabbled in a lot of those things but for once, felt nice to already know what you were gonna say. Haha.
@gkcs
4 жыл бұрын
Thanks! Your replies on the comments are excellent btw :D
Showing Gupt in local popular movie would make Booby Deol really Proud. Savage AF Gaurav! :D
To be honest learned something useful today, and willing to learn from you more like this kinda of topic!
Gaurav, I can't tell how interesting this video was ! Thanks a lot ! I totally enjoyed learning this ! Keep making more such videos ! :)
@gkcs
4 жыл бұрын
Thanks!
I was waiting for this video, awesome
Wow this is sweet! Super smart solution to the issues of video streaming, will be interesting to see where it is improved upon in the future too!
Thank You for sharing. Very nice details and very good description
I'm very impressed! Great explanation! I just subscribed! Keep going!
Thanks for this wonderful insight on the engineering side of Netflix, looking forward to more system design videos.
@gkcs
4 жыл бұрын
Thanks!
You’re awesome. Thanks 😁😍 this was so helpful
That was a nice one ! Thanks for the video.
Short, simple and full of interesting concepts. Thanks!
@gkcs
Жыл бұрын
Thank you!
great efforts thanks a lot ! i remember my childhood days after seeing the GUPT poster! :-)
Moral : Steal that Open Connect Box
That bit at the beginning was hilarious 😂
Thanks for your descriptions
@gkcs
5 күн бұрын
Cheers!
Your videos are pure delight to watch !
@gkcs
4 жыл бұрын
Thank you!
Champion presentation. Just wanted to see a graphical representation of the adaptive bit rate ladder through Netflix's eyes.
I just started watching your videos. Your content is Amazing.
Nice and depth knowledge bro!
Awesome bro ..!! Simple and very informative.. loved it .. keep it up 👍
Don't be surprised if you hit 500K subscribers by the end of 2019. Your content quality is skyrocketing.
@gkcs
4 жыл бұрын
Thank you!
@DiaryOfMuhib
3 жыл бұрын
@@talesara74 It's system design and it's not for everyone to understand, what did you expect? People mainly use this channel for technical system design interviews.
@pratsway961
2 жыл бұрын
You said that from your perspective. But the sad reality is People really don't like good contents😅😂. They can watch a 3hrs movie, but couldn't watch this type videos for 10mins😑
@letsrock7354
Жыл бұрын
Gaurav I love your content and soon I guess will be joining your InterviewReady platform...Please offer me some special discount bhai 😍
@vinayak186f3
Жыл бұрын
It's almost 2023 and he still has 488k 🥲
Very detailed. Great work!
The content of this video was very dense. The breather at 5:29 was very much wanted :-). You have very good audience engaging skills. Good Luck !!
As always one another brilliant content
Hey Gaurav, I was just wondering how the Open Connect or KZread Red boxes are any different from the long existing concept of CDN (Content Delivery Network). The only change is, being big companies they have the money and resources to create their own personal CDNs and brand them which is kind of building your own house instead of renting one. Is there something I am missing here ?
@gkcs
3 жыл бұрын
They are very similar, yes.
@samiles171094
2 жыл бұрын
I was having the same question as how these Oppen connect boxes are different from CDNs.
that's pretty interesting concept... very well explained 👏👏.....
Can you explain some of the advanced concepts of video streaming too? For eg., Bandersnatch? I wonder how Netflix is providing conditional streaming of video chunks based on user selection.
I just love your system design videos.
@gkcs
4 жыл бұрын
Thanks 😁
I was the first one to watch this video after him! The content had no views but a like!!
@Lokesh_Karki
4 жыл бұрын
That's a improper system design for KZread, probably slow.
Keywords = CDN , predictive content delivery , content pre-fetch and so on...
@vinayshrivastava266
4 жыл бұрын
Yes Content Delivery Network plays a really important role if the geographic location of physical server is far and also I think what these OTTs would be doing is what we are getting in recommendation might be based on geo location rather than your watch history , this way if one guy from ur location has seen some movie it gets cached to a CDN and then when u request for the same content , since it comes in ur recommendation its extremely fast
Very nice explained and made it interesting. Will be waiting for more good system design question.
Thanks Gaurav. I am not an engineer but always wanted to get some resource to learn the concepts. I would love to see more of your videos and on advance concepts.
@gkcs
4 жыл бұрын
Thanks Ankit!
Thank you Gaurav for the content . Hope to see many more useful content .
Paaji ek dum kamaal ka video hai ...even understoodable for non cs
This was a very informative video! Awesome job.
You are God of System Design!!
Thank you for your reading on this topic...this is enlightening (at least for me)
must say u made it really Interesting with editing :P and great content!
@gkcs
4 жыл бұрын
Thank you!
Great, Nicely explained !!!
ISP cache is really interesting, wondering how they have handled security on top of it across multiple geography??
Thank you for the explanation
Great content always!
That's just awesome Keep it up
OMG!!!! the opening😂😂
real effort here, good job mate
Nice explaination, Thank you so much.
Excellent video, well explained!
this isp cache thing is just simply amazing,
Brief and precise.. Nice.
Good job explaining!!
Happy Teachers Day Sir! U taught me a lot! I really appreciate that!
@gkcs
4 жыл бұрын
Thank you!
Great video ! Thanks for sharing
This is great work Gaurav. Thank you for this!
@gkcs
3 жыл бұрын
😁
Great breakdown!
Outstanding 😨😱
Great presentation skills and content🔥🔥🔥
Superb explanation bro
Fantastic explanation
Very well explained, thank you, Gaurav!
@gkcs
2 жыл бұрын
Thanks Veronica 😁
Really helpful man!
Bro u are great am a 14+ software engineer really appreciate ur content
Please do some lectures on docker internals like Cgroups, namespaces, union capable file systems
So well explained 🤟
Awesome!!
Awesome Video. Very good explanation. Video content caching is not an easy job. The video is really good.
@sumitlahiri209
4 жыл бұрын
Software Architecture is the most important thing today. No matter how much you you improve on ML algos or buy hardware etc... It is the archtecture that defines quality and reliability. How you connect components together using a good deisgn tactic and make each component work flawlessly by the deisgn pattern matters the most. It is archtecture everywhere. The fact that you make videos on it is really a good thing. Developers today must understand the imporatnce of it. It is not just writing code but many other things too.You are doing a great job by making such videos.
Very informative!
You're a gem you should have a teaching startup like byjus
Great content.. keep up the good work..
Not a programmer or anything. But I love your videos. It's interesting.
Thats a really great explanation. Thanks
@gkcs
4 жыл бұрын
Thank you!
Nice work.
Gupt yrr, haha Very wonderfully explained Thanks for that
Not gonna lie, you got me in first second, sorting in O(n) lol
Awesome Video 😍
Hi Gaurav Thanks for awesome video. What are the technologies Netflix uses to display the videos?.
:) Thanks this is amazing
Bhai saab.. Creative content...
Thank you!
What about user authentication and authorisation when dealing with open connect??
Thanks for the great info I wonder why I like Netflix more than any other OTT