Why was Facebook down for five hours?

Facebook was down for five hours last week. What happened and what do DNS and BGP have to do with it?
Facebook's explanation: engineering. 2021/10/05...
Support these videos on Patreon: / beneater or eater.net/support for other ways to support.
0:00 DNS
7:13 Caching DNS
10:34 Hop-by-hop routing
14:07 Default-free routing
18:28 Peering
19:50 BGP
26:08 The outage
------------------
Social media:
Website: www.eater.net
Twitter: / ben_eater
Patreon: / beneater
Reddit: / beneater
Special thanks to these supporters for making this video possible:
Aleksey Smolenchuk, Anders Carlsson, Andrew C. Young, Anson VanDoren, Anthanasius, anula, Armin Brauns, Ben, Ben Cochran, Ben Kamens, Ben Williams, Benny Olsson, Bill Cooksey, Binh Tran, Bouke Groenescheij, Bradley Pirtle, Bradley Stach, Bryan Brickman, Burt Humburg, Christopher Blackmon, Dale Andrew Darling, Daniel Jeppsson, Daniel Pink, Daniel Tang, Dave Burley, Dave Walter, David Clark, David Cox, David Dawkins, David House, David Sastre Medina, David Turner, David Worsham, Dean Bevan, Dean Winger, Dilip Gowda, Dissy, dko, Dmitry Guyvoronsky, Dušan Dželebdžić, Dzevad Trumic, Emilio Mendoza, Eric Dynowski, Erik Broeders, Eugene Bulkin, Evan Thayer, Eveli László, fisksvett, fxshlein, George Miroshnykov, Harry McDow, hotwire33, Ingo Eble, Ivan Sorokin, James Capuder, james schaefer, Jared Dziedzic, Jason DeStefano, Jason Dew, JavaXP, Jaxon Ketterman, Jay Binks, Jayne Gabriele, Jeremy, Jeremy Cole, Jesse Miller, Jim Kelly, Jim Knowler, Jim Van Meggelen, Joe Beda, Joe Pregracke, Joel Miller, John Fenwick, John Hamberger jn., John Meade, Jon Dugan, Jonn Miller, Joseph Portaro, Joshua King, Jurģis Brigmanis, Kai Wells, Kefen, Kenneth Christensen, Kitick, Koreo, Lambda GPU Workstations, Larry, László Bácsi, Lucky Resistor, Lukasz Pacholik, Marcos Fujisawa, Marcus Classon, Mark Day, Marko Clemente, Martin Noble, Martin Roth, Mats Fredriksson, Matt Krueger, Matthäus Pawelczyk, Matthew Duphily, Max Gawletta, Maxim Hansen, melvin2001, Michael Tedder, Michael Timbrook, Michael Weitman, Miguel Ríos, mikebad, Mikel Lindsaar, Miles Macchiaroli, Muqeet Mujahid, My Yiddishe Mama, Nicholas Counts, Nicholas Moresco, Nick Chapman, Not Yet Wise, Örn Arnarson, Paul Pluzhnikov, Paul Randal, Pete Dietl, Phil Dennis, Philip Hofstetter, PixelSergey, Porus, ProgrammerDor, Randal Masutani, Randy True, raoulvp, Renaldas Zioma, Ric King, Richard Butler-Malone, Richard Ertel, Rick Hennigan, Robert Comyn, Robert Diaz, Robey Pointer, §çīŤøşHï ŇåĶąýŐbŕÔ, Scott Holmes, Sean Patrick O’Brien, Sergey Kruk, solderspot, SonOfSofaman, Stefan Nesinger, Stefanus Du Toit, Stephen Kovalcik, Stephen Riley, Stephen Smithstone, Steve Jones, Tayler Porter, TheWebMachine, Thomas Bruggink, Thomas Eriksen, Tim Walkowski, Tom, Tom Yedwab, Tommaso Palmieri, Tyler Latham, Vincent Bernat, Walter Montalvo, Warren Miller, William, Wim Coekaerts, Wraithan McCarroll, xisente, Yee Lam Wan

Пікірлер: 1 700

  • @sykskysyk
    @sykskysyk2 жыл бұрын

    Facebook going down was a small price to pay for getting this incredibly informative video from you, thanks Ben!!

  • @froller

    @froller

    2 жыл бұрын

    Let's bring down some other large network in some weird way to get next video! ...just kidding.

  • @lilstubthumb

    @lilstubthumb

    2 жыл бұрын

    I would pay for Facebook to be down.

  • @froller

    @froller

    2 жыл бұрын

    You forgot to add "forever". :)

  • @lwizzit

    @lwizzit

    2 жыл бұрын

    It’s a win/win

  • @francesco45

    @francesco45

    2 жыл бұрын

    Not for Mark ahaha

  • @penguin1714
    @penguin17142 жыл бұрын

    I'm pretty sure Ben Eater is actually a physical manifestation of knowledge.

  • @12crenshaw

    @12crenshaw

    2 жыл бұрын

    He not only knows everything, he also have time to live, breathe and share his knowledge

  • @charstringetje

    @charstringetje

    2 жыл бұрын

    All you need is a default route and the ability to read, and you can acquire knowledge too. Exciting times.

  • @youkofoxy

    @youkofoxy

    2 жыл бұрын

    That sounds like he is a named version of eaters from Digimon cuber sleuth. Eaters are only interested in collecting and sharing information across it's hive mind.

  • @martinkuliza

    @martinkuliza

    2 жыл бұрын

    Ben Eater is an A.I. this physical form that we see is a computer manifestation of his real self, Ben Eater is like the M.C.P. on Tron Part 1 LOL

  • @cyber_8778

    @cyber_8778

    2 жыл бұрын

    @Penguin Hey I remember you! You commented on my video once :)

  • @xXRedTheDragonXx
    @xXRedTheDragonXx Жыл бұрын

    "I'm connected to a router on AT&Ts network that I happen to have access to" As a former NOC tech for an ISP this was absolutely hilarious to hear you say this so casually. They don't just give anyone console access to random routers!

  • @yves.dantas

    @yves.dantas

    9 ай бұрын

    I think he still works as a nw engineer. If im not wrong he even worked for juniper

  • @tecno2053
    @tecno2053 Жыл бұрын

    Im a network engineer for a medium size ISP (primarily enteprise fiber). This has become my goto video to explain to people what I do. I work with BGP and the like daily.

  • @dddd6606
    @dddd66062 жыл бұрын

    This is better explanation of DNS than the one I had at university, thank you.

  • @dm3on

    @dm3on

    2 жыл бұрын

    That's because he is self thought, not sure if Ben has high school diploma.

  • @Ripsaw460

    @Ripsaw460

    2 жыл бұрын

    @@dm3on "self thought"

  • @Ripsaw460

    @Ripsaw460

    2 жыл бұрын

    @The Poison Donut not only that but for even a person that isn't very knowledgeable in this field, he makes it very interesting to watch and all his videos are very easy to follow and i almost never find myself lost trying to figure out what is going on, ben is the best.

  • @Nick-lx4fo

    @Nick-lx4fo

    2 жыл бұрын

    @@Ripsaw460 oH nO, a slight spelling mistake that totally obfuscates the meaning of his message, what will I do?

  • @Ripsaw460

    @Ripsaw460

    2 жыл бұрын

    @@Nick-lx4fo you dont have to do anything, just thought it was funny.

  • @ryanallen2001
    @ryanallen20012 жыл бұрын

    "A router on AT&T's network that I happen to have access to" WHAT A FLEX :D

  • @esepecesito

    @esepecesito

    2 жыл бұрын

    He works there...

  • @jirensan828

    @jirensan828

    2 жыл бұрын

    Not really a flex. That server is public and available for anyone to use to validate their BGP routes

  • @TomStorey96

    @TomStorey96

    2 жыл бұрын

    There's a few of them spread around, usually referred to as a "looking glass".

  • @tiporari

    @tiporari

    2 жыл бұрын

    Lol I like the "my address has changed by now" disclaimer. Don't poke me please. You are just poking my mouth breathing neighbor.

  • @ratbag359

    @ratbag359

    2 жыл бұрын

    @@tiporari It's bad enough with bots scanning ports and trying to dump payloads into the open ports they find yet alone the comment section of youtube seeing what holes or services are avalible.

  • @laykefindley6604
    @laykefindley66042 жыл бұрын

    Watching one person setup global router configurations is seriously invoking wizardry levels of mastery. I am truly humbled to watch a work of fine art like this and gives me goosebumps thinking how much power one can wield with the right knowledge and access to modern society.

  • @autohmae

    @autohmae

    2 жыл бұрын

    Now multiply this times 70 000 or so, because that is the number of AS-numbers in use in the world, thus close to the number of independent Internet networks who connect with everyone else.

  • @critical_always
    @critical_always Жыл бұрын

    Nobody can explain like Ben does. Everything is always at exactly the right pace and detail. I kinda feel smarter learning from this man.

  • @jonasdatlas4668
    @jonasdatlas46682 жыл бұрын

    To quote a random Tweet: "it's hard to even explain to a layperson how badly you have to fuck up to get BGP and DNS both as part of the same problem. Like, you need multiple diagrams."

  • @nickwallette6201

    @nickwallette6201

    2 жыл бұрын

    LOL -- that was my exact reaction when I heard about this. Like, what happened to break DNS and BGP, that took 6 hours to recover from? That is one COLOSSAL screw-up. With such huge stakes, too. It did make me suspicious what _really_ happened.

  • @johnm2012

    @johnm2012

    2 жыл бұрын

    It took them 5 hours and 50 minutes to find someone with a physical key to let them into the building.

  • @NavidIsANoob

    @NavidIsANoob

    2 жыл бұрын

    @@johnm2012 That's false. It DID take them that long to get someone with the necessary protocol expertise to fly in and fix the mess. It's silly to assume that no one on site would have a key to the building. It's reasonable to assume that a corporation like Facebook understaffs their infrastructure and doesn't retain experts to maintain that infrastructure.

  • @zacker150

    @zacker150

    2 жыл бұрын

    On site, they use badge readers to access the building. Those badge readers ran on Facebook's network.

  • @NavidIsANoob

    @NavidIsANoob

    2 жыл бұрын

    @@zacker150 Again, this is false. Why would a badge lock system run on Facebook's global network, as opposed to an internal, segmented network? It makes no sense.

  • @infinitenex8165
    @infinitenex81652 жыл бұрын

    Man, networking is so convoluted to me. Its a miracle that the global internet works as well as it does on a daily basis.

  • @GummieI

    @GummieI

    2 жыл бұрын

    The number 1 reason it works so well really is the redundancy. Notice how multiple time through this video he keeps talking about dozens if not hundreds of this and that. And that is what makes things work so well, everything is connected in hundreds of ways, so if a route or 2 goes down for whatever reason, who cares, there are still hundreds of routes left to take instead.

  • @LMB222

    @LMB222

    2 жыл бұрын

    There are further tools that make automatization much easier. You set up a few things in the beginning, and then sit back and relax.

  • @jdpruente

    @jdpruente

    2 жыл бұрын

    I tell people that the entire internet only works with gentleman's agreements and handshakes. And that's when I'm being nice about phrasing it.

  • @edenrose2374

    @edenrose2374

    2 жыл бұрын

    @@GummieI except its rarely the case with preferential routing. ISP's sign contracts with third parties to reduce overhead and operating costs by having them operate long-distance runs. - This gives these single-points of failure a Scapegoat if it fails. US and UK are the worst offenders of this, literally blocking or banning competitors networks at hops that would route through their network rather than around it. This means that regardless of speed or outage state, a singular ISP can ensure the bulk of their traffic goes through and is on their and partner networks. This gets worse when we start talking undersea cables. Exit Nodes (backbone) connections to externally connected networks are entirely priority based. When it comes to routing overseas, these contractors get priority. Why? rather than operating their backbone at a flat-cost for a nation, and telling the Goverment to cover the cost of operation so all may benefit... They instead choose to charge rate-based, with tiered flat based contracts available to ISP's. - This means that, your provider has to both pick a rate limit(and quality[jitter] limit) for its clients and limits routing to the connecting country to ONLY this backbone. | Sometimes they have failover's to competitors backbone lines, but since its rate-based there is caps client and SEVERE data-rate penalties. Amazon is the worst offender of this, offering basement priced backbone lines that are PAINFULLY slow(except to /some/ of their services) and this results in much of the worlds internet being measurably slower as a result... Australia has the worlds slowest internet exclusively due to no one hosting services inside australia (draconian encryption laws), and relying exclusively on Amazon and Telstra's dedicated Undersea cables to route outside of Australia(to the UK, the longest possible run). Since nearly everything is hosted in the US or germany. This means reaching these places is usually several hundred hops. - Turning the Fibre network that makes up Australia's rather robust internet infrastructure into a cesspool of garbage slow performance.

  • @vintyprod

    @vintyprod

    2 жыл бұрын

    @@edenrose2374 this is actually super interesting. thanks for sharing.

  • @askthisscientician6040
    @askthisscientician60402 жыл бұрын

    That had to be the best explanation of this I've seen. Not dumbed down, no hand-waving explanations, but carefully and effectively explained with a live demonstration to boot. Thank you.

  • @casualyoutuber6239
    @casualyoutuber62392 жыл бұрын

    I just enrolled in a CCNA/CCNP course being offered at my University and yes I am as vanilla as they come as long as Networking is concerned but this video was so crystal clear that I may have just revised effectively about 16 credit hours in a single 30minute video!

  • @lightdark00
    @lightdark002 жыл бұрын

    At the end should have been a cutaway to a team of breadboard computers all networked together, that made all this possible. 😊

  • @TNTsundar

    @TNTsundar

    2 жыл бұрын

    Cutaway scene like from the movie “The Prestige” with an eerie music that shows a lot of those breadboards, ICs, wires and then Ben using the monitor and keyboard connected to it.

  • @BertGrink

    @BertGrink

    2 жыл бұрын

    Maybe that´ll be the topic of an upcoming video?

  • @K-o-R

    @K-o-R

    2 жыл бұрын

    @@TNTsundar You mean the breadboards that generate the AI known as Ben Eater? 🙃

  • @Aighthandle

    @Aighthandle

    2 жыл бұрын

    The actual servers aren’t that different, just a lot of modularized sheet metal chassises in big housing units with so, so many hard drives

  • @mjsvitek

    @mjsvitek

    2 жыл бұрын

    Next video: "I built 30 breadboard computers so I could make my own internet... The Eaternet."

  • @crimsonfart
    @crimsonfart2 жыл бұрын

    I must've watched a dozen videos about the Facebook outage, but somehow only Ben is able to explain it not only in a way that I can understand, but it actually seems simple, even though it absolutely isn't. This is the best tech channel on KZread, no contest.

  • @thewhitefalcon8539

    @thewhitefalcon8539

    2 жыл бұрын

    Except he didn't explain it, he talked about what the servers were doing and how the rest of the internet would know they were down and then said "oh and the reason was they typed something wrong"

  • @stephaniecurry45

    @stephaniecurry45

    2 жыл бұрын

    I mean complexity is built on understand and learning the simple stuff. This still isn’t like rocket science or anything lol

  • @caleballen4721

    @caleballen4721

    Жыл бұрын

    You know a person is a good teacher if they can communicate clearly about DNS

  • @Backyardinstallers
    @Backyardinstallers2 жыл бұрын

    Ben, just found your channel and I have to say this is refreshing to see a DIY learning channel starting from the beginning... you keep learning strong... thank you for your time...

  • @jonathansteward1
    @jonathansteward12 жыл бұрын

    Ben this is an amazing video, as a network engineer in the industry I always like to listen along to these kind of videos to refresh my knowledge and remind myself of ways in which to explain various content and this was perfect! I was really thinking of making a video on the topic but this does the topic justice and I don't think I could add anything else to the topic! Keep up the great work!

  • @---nw9qu
    @---nw9qu2 жыл бұрын

    One of the best explanations of dns, routing, and BGP that I have seen.

  • @ahensley
    @ahensley2 жыл бұрын

    Wow! That seems like a lot of work to set up the demo you did, but then again I don't happen to have access to a router connected to AT&T's live network 😏

  • @hexagonist23

    @hexagonist23

    2 жыл бұрын

    How does he even have access to that?

  • @SirLightfire

    @SirLightfire

    2 жыл бұрын

    @@hexagonist23 my guess is that his full time job is a network engineer at AT&T?

  • @Jeff-ss6qt

    @Jeff-ss6qt

    2 жыл бұрын

    Maybe he rents a server there or installed his own? He has his own website that likely gets a lot of traffic.

  • @SirLightfire

    @SirLightfire

    2 жыл бұрын

    @@Jeff-ss6qt but that wouldn't give him access to a top level route table for an AT&T router To even be able to login to something like that is a huge deal

  • @scott3489

    @scott3489

    2 жыл бұрын

    AT&T and several other ISPs host public route servers and/or a BGP looking glass where you can view their BGP routes. Their around to assist with troubleshooting multi-carrier internet routing issues. Maybe that’s what he’s connected to? But yeah, the demo he built is incredible.

  • @bobbyb42
    @bobbyb422 жыл бұрын

    I'm studying for the Network+ right now, and this video is the best explanation on these topics I have found anywhere. I could go on and on about how great your videos are, but mainly I just wanted you to know that I appreciate you sharing your knowledge with us and putting in the effort to make it easy to understand and fun to watch.

  • @Will_of_Iron
    @Will_of_Iron Жыл бұрын

    Subscribing to your channel is one of the best decisions I've ever made in my life. I didn't even know anything about these and you explained it so smoothly. I guess I'm blessed to have a teacher like you. Keep up the good work!

  • @BobHolowenko
    @BobHolowenko2 жыл бұрын

    Hi Ben. I am a network engineer for an ISP and this was a VERY well done video that talked about the "house of cards" effect of the outage. Great work!

  • @riz94107
    @riz941072 жыл бұрын

    I've been an internet professional since 1994, and i knew all this already, but your explanations are SO GOOD that i watched it just for fun. Great job!

  • @L0j1k

    @L0j1k

    2 жыл бұрын

    Yea I spent years banging away on dn42 but this is BY FAR the best and most succint and insight-dense "class" on DNS and BGP I've ever seen.

  • @TheRukisama

    @TheRukisama

    2 жыл бұрын

    He does give excellent explanations of things, he lays it out better than I've ever been able to a non-IT person.

  • @mattboardman2148

    @mattboardman2148

    2 жыл бұрын

    The subnet mask description was so succinct that I'm using it internally when explaining networking to both our techs and clients.

  • @Mine8maniak

    @Mine8maniak

    2 жыл бұрын

    not much changed since 90s, i mean the protocols. right?

  • @EdGriebel

    @EdGriebel

    2 жыл бұрын

    Same. I took a grad networking systems course, Ben’s explanation of BGP is *so* much better than the course, I wish I had this then

  • @thewelder3538
    @thewelder35382 жыл бұрын

    You are one of the most truly educating people on KZread. Unlike so many others that are just so superficial with their details, mainly because they don't really understand the subject matter, you still down and give the real information. Now I understand DNS and how it works and it was just nice to see it properly explained. Would have been a bit nicer to have had an explanation about the DNS protocol, but this video is still an amazing explanation. Great work.

  • @tsraikage
    @tsraikage2 жыл бұрын

    you know the video will be as informative as it possibly can when the tutor is building a GPU on a breadboard on his free time

  • @meeDamian
    @meeDamian2 жыл бұрын

    You are undeniably one of the greatest content creators on this platform.

  • @ndupontnet
    @ndupontnet2 жыл бұрын

    That outage just happened seconds after the update of my local Pi-Hole DNS instances, that was quite a head-scratching experience. Thanks a lot for those explanations.

  • @nebula_wolf3132
    @nebula_wolf3132 Жыл бұрын

    I must say, you're one of the best teachers out there, not only is your explanation simple and easy to understand, but it also lends itself to allowing people to learn on their own, rather than shoving information into our faces (thanks school). Whatever you do keep teaching, you're amazing.

  • @nathantron
    @nathantron2 жыл бұрын

    Great job explaining this by the way. It's extremely challenging to explain this in a simple way for everyone to grasp, but you manage to do it perfectly. Great work with the lab stuff too.

  • @gnsoares_
    @gnsoares_2 жыл бұрын

    I just had a class about this today and I understood it so much better in this video. Thank you so much for your work Ben

  • @realdragon

    @realdragon

    2 жыл бұрын

    It says a lot about me when I still don't understand it

  • @SteveJones172pilot

    @SteveJones172pilot

    2 жыл бұрын

    That's kind of what I was going to say - He boiled down probably 2 or 3 days of a CCNP class into 20 minutes or so.. Of course the "real" class would have confused you with all the other options you could put on those bgp routing statements, but it does make me think about maybe this video should be a prerequisite for diving into those sections of a network course. It's really good to see the basics working first!

  • @To-mos

    @To-mos

    2 жыл бұрын

    It's a shame the education system is failing people, I didn't start truly learning until I dropped out of college. It's bizarre how much they charge for schools considering the word is Greek for "free time".

  • @einarabelc5

    @einarabelc5

    2 жыл бұрын

    That's called applying knowledge. Next time, explain it to someone else. Try the Feynman method.

  • @indeepjable

    @indeepjable

    2 жыл бұрын

    Isnt The Current Educational System Still For Factory Jobs?

  • @AndreaTerenziani
    @AndreaTerenziani2 жыл бұрын

    everybody gangsta till Ben pulls out the heckin AT&T router

  • @MxSherwood

    @MxSherwood

    2 жыл бұрын

    @Maxime St-Louis , Ben has ssh access to a router on the AT&T network, and he blurred out its IP address in the video. That is not public.

  • @henrychoo4361

    @henrychoo4361

    2 жыл бұрын

    @Maxime St-Louis do u have access to an AT&T router? Not every Tom, Dick and Harry has it

  • @HaraldSangvik

    @HaraldSangvik

    2 жыл бұрын

    @@MxSherwood Wasn't that just his home router?

  • @krmr

    @krmr

    2 жыл бұрын

    @@HaraldSangvik No, he specifically said he's connected to an AT&T router happens to have access to and blurred out the hostname.

  • @thewhitefalcon8539

    @thewhitefalcon8539

    2 жыл бұрын

    @@HaraldSangvik No, it's part of the actual for real AT&T network, the one that may suddenly stop working and cost millions of dollars if you type a wrong command

  • @andrewohanian5132
    @andrewohanian51322 жыл бұрын

    I am in awe at how many views a video, that is in large part a Juniper router configuration demo, has. This has got to be the most viewed video that details Juniper router BGP configuration. Amazingly well done.

  • @zacharybearden9950
    @zacharybearden99502 жыл бұрын

    Awesome visuals. As a fellow route/switch guy - I can't pass up a video with BGP in the title. About 7 years ago when I was starting my journey in networking, I watched nearly all of your network videos. The fundamental concepts that series covered set me on the right path.

  • @_Stin_
    @_Stin_2 жыл бұрын

    Now, THIS is how you explain things!! Foreword, introduction, background, examples and real-world demonstration... The best explanation for how the DNS architecture works (and how the rest of the Internet works lol)

  • @GeoffreyThornton-TheWinGuru
    @GeoffreyThornton-TheWinGuru2 жыл бұрын

    Ben, I just wanted to say how much I enjoy watching your videos. I've learned breadboarding techniques, logic design, microcode and processor design, and hardware bit error detection and correction (among others). Then, when you create a video that is my wheelhouse, you do so in such a concise and informative fashion that I'd be hard pressed to do the same. You cover the fundamentals of Internet name resolution and route exchange in 30 minutes in what would normally be an intense week to month long seminar. I'm impressed with the level of detail you provide, along with the time it took to create a lab environment to demonstrate the operation of BGP routing. I had not noticed the Facebook outage myself, but at the start of your video, I suspected it might have been a result of BGP route dampening due to network interface flapping, but that would have been for the entire AS and not just specific subnets. Your insight and use of BGP looking glass servers to demonstrate how specific routes were missing is remarkable. I don't have any particular insight as to how this occurred either, but perhaps Facebook is using an internal route reflector or confederation to handle routing Anycast traffic for DNS and ICMP. I don't have any experience with Facebook's network infrastructure, but this outage should hopefully help Facebook and other networks design and maintain more resilient networks. Thanks again for all your time and effort in creating these videos. You have always been a favorite content creator of mine, and this latest video just adds to my adoration. I could tell you were running JunOS, but I'm curious if you were using GNS3 or some other simulator. In any case, kudos for your time and devotion to create such an informative video!

  • @hariranormal5584

    @hariranormal5584

    2 жыл бұрын

    isnt the commit command a thing only on junos?

  • @GeoffreyThornton-TheWinGuru

    @GeoffreyThornton-TheWinGuru

    2 жыл бұрын

    @@hariranormal5584 It's a thing for JunOS and Palo Alto firewalls (PanOS). Juniper devices also let you do a confirmed commit where if you lock yourself out by doing something dumb, it will automatically rollback to the previous configuration. Cisco devices typically commit changes immediately and so can cause issues if commands are entered incorrectly or in the wrong order. The only recourse then is to reboot the device and let the "startup-config" replace the running config. That's why you typically see commands like "copy running-config startup-config" or old school "write mem" to finalize a configuration.

  • @Dejan357
    @Dejan3572 жыл бұрын

    Beautiful video. That's a whole lot of effort that went into demonstrating routing and BGP. Very clearly explained and easy to understand. Thank you for you effort.

  • @HopeRunsDeep22
    @HopeRunsDeep222 жыл бұрын

    I have been studying for my Network+ Exam and am taking it on Monday. This video was so awesome to be able to relate some of what I am learning and actually see it in a real life situation to really get a better understanding of how it works rather than just reading about it. Awesome video! Thank you!

  • @F4LDT-Alain
    @F4LDT-Alain2 жыл бұрын

    Fantastic. I came for some more details on the outage than given in Facebook's statement and I got the best BGP tutorial I've ever seen. I'm more of a system guy myself, so local routing and DNS don't have much mystery left for me. But internet routing and especially BGP has always been kind of black magic to me. Thanks to you, I've got a grab of all this and the whole picture of that outage makes sense now.😁 Great video. Your simulated networks (and routers, I guess, apart from the AT&T one?) are impressive.

  • @TheUnofficialMaker

    @TheUnofficialMaker

    2 жыл бұрын

    I'm just gonna keep it as Black Magic!

  • @N0Xa880iUL
    @N0Xa880iUL2 жыл бұрын

    You're the only person I think who understands all the systems from the ground up.

  • @niklas6576
    @niklas65762 жыл бұрын

    Ben you absolute legend, I had a good portion of this already figured out, but the way you explain it and glue it all together is just next level!

  • @memoryvice398
    @memoryvice3982 жыл бұрын

    The sound of Ben's keyboard is totally ASMR for a guy like me. Brings back lovely memories of my first PC back in the 90's.

  • @NickiRusin

    @NickiRusin

    2 жыл бұрын

    yeah the clackity clack clack just sounds super comfy

  • @javabeanz8549

    @javabeanz8549

    2 жыл бұрын

    Goes back further for me, at the keyboards for an IMSAI and a Polymorphics in the late 1970's, and the college SOROC terminals attached to a PR1ME mini in the mid 1980's.

  • @izzurzuhri

    @izzurzuhri

    2 жыл бұрын

    what keyboard/switch he use?

  • @lordkell1986

    @lordkell1986

    2 жыл бұрын

    @@izzurzuhri probably built it himself from breadboards and electrical switches. Each key is the size of a foot and he jumps on them

  • @kaitlyn__L

    @kaitlyn__L

    Жыл бұрын

    @@izzurzuhri sounds like clicky Alps to me, might be wrong though, a few other switches “ring” the same way

  • @SubspaceEmber
    @SubspaceEmber2 жыл бұрын

    I actually had to do DNS config for my website a few days ago, but wasn’t really sure what I was looking at, but watching this made that make a lot of sense. You can actually almost see how all this grew from just connecting a few computers to each other!

  • @rymaples
    @rymaples2 жыл бұрын

    I feel more ignorant every time I watch one of his videos.

  • @HandyFox333

    @HandyFox333

    2 жыл бұрын

    Same here.

  • @rainbowbunchie8237

    @rainbowbunchie8237

    2 жыл бұрын

    Dig your feet in and don't give up.

  • @opendstudio7141

    @opendstudio7141

    2 жыл бұрын

    What is Facebook? The only one I remember is one from the movie ARMY of DARKNESS or Ash v.s. The Evil Dead. 👻🙃

  • @ZeroUm_

    @ZeroUm_

    2 жыл бұрын

    Haha, and this one is one of the most beginner-level videos he has around.

  • @NoahK216

    @NoahK216

    2 жыл бұрын

    @@ZeroUm_ I'm big into his breadboard PC series but not into networking. They're very different, this may be beginner but still wack for someone not familiar

  • @alimibrahem8120
    @alimibrahem81202 ай бұрын

    Thank you so much Eng. Ben for this pro explination, I think you have to do another one for what happen yesterday.

  • @Mattboy300
    @Mattboy3002 жыл бұрын

    I have to say that the way you explain things in all of your videos by showing all the steps and distilling things into simple to understand chunks is really fantastic. If you were my college professor, you'd be my favorite.

  • @dantheman4700
    @dantheman47002 жыл бұрын

    Can you please do a video incorporating NAT. You honestly have explained subnets, dns, and ip better than anyone Ive ever seen.

  • @Hugh_I

    @Hugh_I

    2 жыл бұрын

    that would fit into the explanation of this video quite well. I'll try, probably not as clear as Ben could do it: His router has two addresses, one public IP address that is routable in the internet (i.e. his ISP advertises routes to that address via BGP) and one on his local network, that is not. Only his router knows what to do with addresses on that local network. Now when he sends a packet from a host on his local network to the internet, it gets send to the router (the default gateway). What a NAT router does is now in addition to just passing the packet on to his ISP's router, is to slightly modify it. It replaces the source address (i.e. the 'return' address that the host on the receiving end should send its answer back to) with his own public IP address. It also stores the info of where it actually came from in a table in memory. That way it looks to the host on the other end like the request was sent from the router's public IP address, not the local IP address of the host in Ben's network. Any reply will be sent back to that router's public address, that everyone knows how to route to. Once the router receives a reply, it'll look up in his table where the request actually came from and rewrites the destination address for that packet back to the original local IP. Now it can route that reply packet via the local network, and sends it to that local address. This allows one to use a big number of local addresses but still access the internet, even though the ISP gave you only one single publicly routable IP address. As Ben mentioned, IPv4 address space is limited (to about 4 Billion addresses), so NAT is an essential crutch to keep IPv4 working today, since that address space would otherwise be to small to address every device on the planet.

  • @javabeanz8549

    @javabeanz8549

    2 жыл бұрын

    @@Hugh_I not bad, a little bit simple, since the NAT typically ends up rewriting the port number as well as the IP address, but that's a minor point compared to the rest of your explanation.

  • @kaitlyn__L

    @kaitlyn__L

    Жыл бұрын

    @@Hugh_I my favourite is when there’s two or three layers of NAT, so port-forwarding breaks because it’s only aware of its own internal NAT table, and not the layers beyond itself.

  • @aelolul
    @aelolul2 жыл бұрын

    Thank you for this deep dive! I really appreciate it. I'm a computer professional, but my knowledge of networking is limited to the ip command, modifying /etc/network/interfaces, and cursing at systemd. I hadn't even heard of BGP before this kerfuffle. I watched a few other videos on the topic since the outage, but this one is far and away the best I've see. I feel like I actually understand what's going on. Thank you again.

  • @12-343

    @12-343

    2 жыл бұрын

    Same. Best I can do is IWD and copy paste commands

  • @xtdycxtfuv9353

    @xtdycxtfuv9353

    2 жыл бұрын

    @@12-343 same LOL. Networking is really hard :(

  • @fedemtz6
    @fedemtz62 жыл бұрын

    This is a really good video. I just finished watching your Networking playlist and would love more videos like this one focusing on setting up routers.

  • @FrankGraffagnino
    @FrankGraffagnino2 жыл бұрын

    thanks so much Ben. Thanks for putting in all the work to make those simulated setups and walk through those commands, plus the work to show access to an actual router setup. We appreciate it!

  • @Kaepsele337
    @Kaepsele3372 жыл бұрын

    I really like your style of explaining stuff completely. That really helps me with understanding, most other people would gloss over "details" like the root DNS servers.

  • @sneedsfeed757
    @sneedsfeed7572 жыл бұрын

    Nice to see someone practically explaining dns lookup.Even I have never tried to do it

  • @doodmonkey
    @doodmonkey2 жыл бұрын

    I've been waiting for the technical analysis, ty

  • @junkmail4613
    @junkmail46132 жыл бұрын

    Ben, I'm no scholar at this, but again (as I say 'bout most everything you do) you'vedone a superb job of detailing intricate(unfamiliar) connections in a clear and lucid manner. Thanks!

  • @qpSubZeroqp

    @qpSubZeroqp

    2 жыл бұрын

    Agreed, thank you!

  • @mustafaark4762
    @mustafaark47622 жыл бұрын

    Again, a great video. Never disappoints man!

  • @EdgarStgo
    @EdgarStgo2 жыл бұрын

    Hats off @Ben Eater, you are a gifted man, the way you flow explaining all this is so smooth. I used to work as a Network Engineer but reached up to OSPF from the client-side, not ISP. I just learned a ton of useful aspects about BGP and DNS from the ISP Internet perspective. Cheers!

  • @docouto
    @docouto2 жыл бұрын

    Brilliant explanation! Tks for taking the time!

  • @orchidejczyk
    @orchidejczyk2 жыл бұрын

    plot twist: it was not simulated network, Ben just was recording while repairing facebook xddd he just needed few days to do montage : D also without recording he would repair it much faster! thanks for vid Ben!

  • @anti7gn

    @anti7gn

    2 жыл бұрын

    Ben took down Facebook just to create an explanation video.

  • @Nate-hf8hm
    @Nate-hf8hm2 жыл бұрын

    That was a seriously intersting video, would love to see more stuff like this!

  • @waldolemmer
    @waldolemmer2 жыл бұрын

    I like the nice illustrations And your way of teaching, I always learn so much from your videos :)

  • @ironmati
    @ironmati2 жыл бұрын

    Your work is great and you have an amazing way of showing and explaining things that is almost hypnotizing

  • @jasonwilliams8730
    @jasonwilliams87302 жыл бұрын

    Another amazing video! Thanks for sharing.

  • @sophiemilton5939
    @sophiemilton59392 жыл бұрын

    "Why was Facebook down for five hours?" A better question is why was ot ever up in the first place.

  • @TheLamefeed

    @TheLamefeed

    2 жыл бұрын

    Agreed, the time Facebook was down probably got families to actually talk with each other, and hopefully get a better relationship. I hope all these unsocial networks would cease to exist.

  • @henrydorsett6076

    @henrydorsett6076

    2 жыл бұрын

    Technical question != interweb-social question But agreed in general :)

  • @UncleKennysPlace

    @UncleKennysPlace

    2 жыл бұрын

    @@TheLamefeed Actually, it made people crazy.

  • @ELYESSS

    @ELYESSS

    2 жыл бұрын

    @@TheLamefeed back in my days we used sms instead of facetime

  • @friendly0

    @friendly0

    2 жыл бұрын

    @@TheLamefeed pornhub usage increased massively 😂

  • @jarred7721
    @jarred77212 жыл бұрын

    Damn this was really cool, I'm glad you took the time to explain this in such detail.

  • @vikramsandhu9779
    @vikramsandhu97792 жыл бұрын

    Hey Ben, Your videos are always a treasure trove of knowledge. You've inspired me a a lot. Keep it coming. Best wishes to you man.

  • @YandiBanyu
    @YandiBanyu2 жыл бұрын

    My god, all this knowledge for free! Thank you Ben!

  • @deathsheir2035
    @deathsheir20352 жыл бұрын

    I learn a lot from your videos. May have to watch videos multiple times to get a firmer grasp on what you're saying, but my god have I learned a lot. Also, I don't think Facebook is telling the truth about why it was down for 5 hours. Then again, I never fully trusted facebook from the beginning.

  • @Coloneljesus
    @Coloneljesus2 жыл бұрын

    Another very clear and well done video. Looks like I'll be working with BGP at work very soon, so this came at an opportune time!

  • @therealchayd
    @therealchayd2 жыл бұрын

    I've never managed to get my head around BGP until now (despite working with network engineers whose explanations just become white noise). Massive thanks!

  • @Levy1111
    @Levy11112 жыл бұрын

    Wonderful explanation, thank you. Could you in some future video show how did you prepare mock networks?

  • @nicholaswest9146
    @nicholaswest91462 жыл бұрын

    Your way of explaining things is incredible! I've worked in web hosting for about 6 years and now I really understand DNS :)

  • @crit7_

    @crit7_

    2 жыл бұрын

    You're working for 6 in years in web hosting and you didnt know what dns was?

  • @ilyadaemon

    @ilyadaemon

    2 жыл бұрын

    @@crit7_ for web hosting dns is a table like /etc/hosts :)

  • @kaitlyn__L

    @kaitlyn__L

    Жыл бұрын

    @@ilyadaemon plus you don’t need to know anything after the point where your web server network connects to your ISP!

  • @minikretz1
    @minikretz12 жыл бұрын

    Amazing graphics! Made it real easy to follow as a fellow network engineer. Great explainations as well, I learned a lot about bgp

  • @yongluo7121
    @yongluo71212 жыл бұрын

    great video, very clear explanation and illustration. good work👍

  • @SONGOKU-tl3ht
    @SONGOKU-tl3ht2 жыл бұрын

    Amazing explanation.

  • @youtubeisfascism
    @youtubeisfascism2 жыл бұрын

    One funny thing, I've always been saying IPv6 will not become mainstream, because no network engineer can make sense of the numbers. Its funny it was actually Ben's only reason not to use it.

  • @DFX2KX

    @DFX2KX

    2 жыл бұрын

    I mean, for trying to explain something, it's impractical. But Copy-paste can be your friend. And it works in a fairly similar way to v4, far as I can tell.

  • @ioanbustean7442
    @ioanbustean74422 жыл бұрын

    @Ben Eater, I came here to see why it happened but I definitely leave this video with a good baggage of information that would probably took me waaay longer to get. Thanks! You've done a great job explaining not only what happened but also the theory behind it.

  • @Steven9675
    @Steven96752 жыл бұрын

    Well now. Mr. Eater, thank you, thank you, thank you! Also, thank you! Clear, concise, full overview I will be reviewing and attempting to understand for years! I have been setting up networks for BAS for years, not often, but as a need would pop up. Struggling through it. First time I have had a complete view of the picture . I just had a look at your other videos…. More to learn. Thanks for that too. I do wish i could type even a quarter the speed you do.

  • @bassman87
    @bassman872 жыл бұрын

    when you said AS number 7018 I immediately said AT&T. Ive peered with AT&T so many times now that their AS number is permanently engrained in my head. haha.

  • @hariranormal5584

    @hariranormal5584

    2 жыл бұрын

    I do know 174 is cogent and 6939 is HE, and 1299 is telia, too much interest on the hosting community so that's just stuck on my head xD

  • @BobHolowenko

    @BobHolowenko

    2 жыл бұрын

    ​@@hariranormal5584 Years of working in carrier networks and looking at BGP tables has me reading these like maps...It's funny how numbers stick inside the human brain.

  • @JonathanRockway

    @JonathanRockway

    2 жыл бұрын

    I decided to look up who AS 1 is, and it's Level 3 these days. Ironically, they are not AS 3; that's MIT. They should broker a trade ;) (I used to work at 111 8th Ave, and Level 3 proudly had space on the 3rd floor. Always enjoyed that.)

  • @minirop

    @minirop

    2 жыл бұрын

    @@JonathanRockway then they move to England and everything breaks x)

  • @meme-hj5rs

    @meme-hj5rs

    2 жыл бұрын

    @@minirop is it because of break-xit ?

  • @samiurkhan
    @samiurkhan2 жыл бұрын

    If I were to guess, they have something roughly inspired by OpenFlow SDN internally to propogate config changes. Configs store things like DOS protection thresholds, routes to key services (likely algorithmically generated and pushed based on an understanding of machines and distances between them). I suspect a config change they are talking about damaged the route generation.

  • @myname377

    @myname377

    2 жыл бұрын

    nohh. they just disconneced Mark Zuckerberg 10 years old laptop :)

  • @samiurkhan

    @samiurkhan

    2 жыл бұрын

    @@myname377 fb hire this man right now

  • @nickwallette6201

    @nickwallette6201

    2 жыл бұрын

    LOL -- yup. The SDN controller was running on Phil's workstation, because that's the IP address it was configured to use when they were first configuring the dev POC. And unbeknownst to anyone, including Phil, it never got changed. So, when Phil shut down his workstation to do an OS update, just two minutes before a scheduled schema replication, the entire controller cluster went offline and started dropping routes. You know... or something like that. :-) If I've seen it once, I've seen it a thousand times.

  • @kaitlyn__L

    @kaitlyn__L

    Жыл бұрын

    @@nickwallette6201 damn you’ve genuinely seen that before? That’s depressing. I’ve had stuff like that happen with my NAS and various desktops, but to imagine professionals in a (public) company doing it… actually no, that makes perfect sense. The feet tend toward the well-trodden path and all that.

  • @grzegorzkowalski1507
    @grzegorzkowalski15072 жыл бұрын

    I love how you just casually answered my questions about the internet that have been absolutely boggling me my whole life and that I didn't know how to ask for

  • @skinnyflea2628
    @skinnyflea26282 жыл бұрын

    Awesome! Thanks Ben. Your content is always making stuff easy to learn.

  • @DaedalusYoung
    @DaedalusYoung2 жыл бұрын

    The fact that I missed the entire event says a lot about how much I value Facebook.

  • @paulstelian97

    @paulstelian97

    2 жыл бұрын

    And WhatsApp and Instagram as well.

  • @whynotanyting

    @whynotanyting

    2 жыл бұрын

    Only reason I knew about it is because my mom was flustered she couldn't reach me on Whatsapp... ugh

  • @ornessarhithfaeron3576

    @ornessarhithfaeron3576

    2 жыл бұрын

    Only reason I know it is because I don't have the apps on my phone but I was checking if my professor answered my question in the private Facebook group we have for a lesson

  • @MmeHyraelle

    @MmeHyraelle

    2 жыл бұрын

    My mom was crazy about accessing her cadle and resetted the internet modem because even if she understands its the first diagnostic step, she does not understand why and when to do it. You still had internet, mom. Just not those websites.

  • @dieselgeezer18

    @dieselgeezer18

    2 жыл бұрын

    i didn't even notice it. My friends were talking about it and thats how i learnt about it

  • @iKrizNL
    @iKrizNL2 жыл бұрын

    Hey Ben what software do you use to simulate the routers and peering? Great explanation btw, pitty we don't get more insights from Facebook

  • @sschueller

    @sschueller

    2 жыл бұрын

    I would like to know as well.

  • @Themoonisachees

    @Themoonisachees

    2 жыл бұрын

    I assume since he has access to an at&t router that he works in the industry, which would mean he has access to disk images of the router operating system. Then it's just a matter of putting it in several VMs and connect them together, using the VM provider of your choice.

  • @TomStorey96

    @TomStorey96

    2 жыл бұрын

    My guess would be either GNS3 or some vSRXs running in VMware or similar. Logical systems are a possibility, but the interface names suggest one of the first two.

  • @geordish

    @geordish

    2 жыл бұрын

    He is using a Juniper vMX virtual router.

  • @basedbulgarian511

    @basedbulgarian511

    2 жыл бұрын

    Cisco Packet Tracer and GNS3 are the most commonly used ones

  • @watvannou
    @watvannou2 жыл бұрын

    The production value of this is just insane! Amazing job.

  • @dapengliu4753
    @dapengliu47532 жыл бұрын

    This is one of the best videos that I had watched ! Subscribed. As to the reason, when software engineers stopped the access to data, backup procedure will be fast.

  • @funkykong9001
    @funkykong90012 жыл бұрын

    Thanks!

  • @inspirated
    @inspirated2 жыл бұрын

    Thousands of outlets and channels covering the Facebook outage: zero knowledge and a clusterfuck of buzzwords. Ben Eater covers the outage and ends up creating a video that (informally) explains DNS & BGP better than almost everyone else out there. This is in fact a perfect video now to show to someone who asks "how does the internet work".

  • @artursmihelsons415
    @artursmihelsons4152 жыл бұрын

    Great deep explanation, as always. I did know it before, but now I know a little bit more.. 👍 Thanks Ben!

  • @LasithaDenipitiya
    @LasithaDenipitiya2 жыл бұрын

    Thank you Ben Eater for this valuable explanation. I also struggled in understanding BGP and this video explained it clearly as a highlevel explanation.

  • @danieladelodun9547
    @danieladelodun95472 жыл бұрын

    14:20 but do they know you have access? 🤔 imagine getting rce on an at&t router just make a youtube explainer

  • @rebmcr

    @rebmcr

    2 жыл бұрын

    He could have a read-only account for auditing security, nothing he did on the live box needed rce.

  • @__8120
    @__8120 Жыл бұрын

    "So I'm in the console on one of AT&T's routers that I just happen to have access to" I love that he just casually glosses over that and none of us even bat an eye. "Yeah ofc he has access to it, why wouldn't he, nothing to see here"

  • @diydad7704
    @diydad77042 жыл бұрын

    That was awesome! Please don't stop making these super informative videos.

  • @b00lean
    @b00lean2 жыл бұрын

    Great job explaining! Thanks from Switzerland!

  • @ochanlee4414
    @ochanlee44142 жыл бұрын

    Bridging Gap Protocol :) Basically the reason for the outage was a human error that was propagated to >1000s of Facebook's backbone BGP routers through the massive use of network automation and management from a single pane of glass! It goes to show that automation doesn't get rid of human error but lets it propagate at a large scale...Don't forget the basics and think thrice before you hit that push button :) Excellent BGP and DNS overview Ben!

  • @codingstudent3163
    @codingstudent31632 жыл бұрын

    You are Inspiring, You are my Knowledge Angle, Bless me. Praneeth from India.

  • @khealer

    @khealer

    2 жыл бұрын

    Angle?

  • @wol2231

    @wol2231

    2 жыл бұрын

    @@khealer I think they meant "angel".

  • @jamesinonati7608

    @jamesinonati7608

    2 жыл бұрын

    @@khealer Don't be surprised if tomorrow there is a temple dedicated to Ben in India.

  • @clonkex

    @clonkex

    2 жыл бұрын

    "knowledge angle" ahahaha

  • @cygnusx7
    @cygnusx72 жыл бұрын

    Very well explained and edited!

  • @herzogsbuick
    @herzogsbuick2 жыл бұрын

    Loved how you went through all of this so fluidly. Well done. When I try to explain these things -- nope, now sending people to this video heh

  • @jaredteaches894
    @jaredteaches8942 жыл бұрын

    For context, Ben Eater's expertise is actual computer networking. He speaks about his amazing journey on his podcast. He is absolutely brilliant and his story is inspiring.

  • @mattym8

    @mattym8

    Жыл бұрын

    Yeah, this is the subject he started out learning and his pro experience. The rest is just for fun.

  • @nicolascuervo9910
    @nicolascuervo99106 ай бұрын

    Hey Ben, Here in Australia today there was a nation wide outage for a major ISP and the outage lasted 9 hours. It seems that bgp was the culprit. It would be great if you could explain to us how, because in this case no dns was involved

  • @mattiviljanen8109
    @mattiviljanen81092 жыл бұрын

    I had to come back to comment. It's unbelievable how well you explained BGP. We touched the subject at school and I couldn't get it. Now I did. Not that I'll ever need it, for reasons, but I did get it. Thanks!

  • @waldolemmer
    @waldolemmer2 жыл бұрын

    I've finally watched all of your videos from beginning to end.

  • @foxonrails4140
    @foxonrails4140 Жыл бұрын

    This guy went from looking at a routing table in his LAN to a BGP table on Tier 1 ISP's network within 10 seconds. Damn.

  • @ShainAndrews

    @ShainAndrews

    Жыл бұрын

    Nope...