How Roblox Went Down For 73 Hours

Ғылым және технология

A look into what happened behind the scenes during the longest outage in Roblox history.
Sources:
blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/
www.hashicorp.com/resources/how-we-used-the-hashistack-to-transform-the-world-of-roblox
roblox.fandom.com/wiki/2021_Roblox_outage/
roblox.fandom.com/wiki/Timeline_of_Roblox_history/2016#August_2016
news.ycombinator.com/item?id=30013919
raft.github.io/
www.lmdb.tech/media/20130329-devox-MDB.pdf
www.lmdb.tech/doc/
db.cs.cmu.edu/mmap-cidr2022/
kzread.info/dash/bejne/eniDy9KddrPIcpc.html
Chapters:
0:00 Intro
0:33 HashiStack Explanation
4:47 Outage Investigation
8:20 Root Causes Found
11:30 Return to Service
12:19 Slow Leaders
15:56 Resolution
Corrections:
- At 9:44, the default unbuffered channel in Go does not hold any items and has a buffer size of 0. Sends to such a channel are blocked until another goroutine is ready to receive the value. The illustration in the video shows a **buffered channel of size 1** however the overall point still stands.
Music Credits:
- Firecracker by LEMMiNO (kzread.info/dash/bejne/p6Ca0bdrfd3Pf8Y.html)
- Impact Prelude by Kevin MacLeod
- We're Finally Landing by Home

Пікірлер: 531

  • @sirgamsay3596
    @sirgamsay359623 күн бұрын

    Imagine just doing a Hobby project to understand a piece of Software and suddenly the complete Roblox Infrastrucure is build on it.

  • @AjaxGb

    @AjaxGb

    23 күн бұрын

    Open source developers: "Hey guys check out this thing I built in my spare time! It's not perfect but I'm making it freely available so other people can learn from--" Large corporations: "FREE?? 👀👀🥵🥵👀👀"

  • @JusticeNDOU

    @JusticeNDOU

    23 күн бұрын

    i thought this was a joke,

  • @xelspeth

    @xelspeth

    23 күн бұрын

    xkcd 2347

  • @shantilkhadatkar1195

    @shantilkhadatkar1195

    23 күн бұрын

    @@xelspeth what is xkcd

  • @michael_betts

    @michael_betts

    23 күн бұрын

    @@shantilkhadatkar1195 webcomic. type that text into goold and youll get the comic.

  • @agnor9978
    @agnor997823 күн бұрын

    every time I hear that someones hobby project caused a major outage somewhere I get the feeling that maybe big corporations should maybe check what software they are built on and support it's development/maintenance

  • @fisch37

    @fisch37

    23 күн бұрын

    XKCD 2347

  • @humza890

    @humza890

    23 күн бұрын

    Except that software relies on another software, which then relies on another software, which then relies on another software.... It can turn into an endless loop

  • @agnor9978

    @agnor9978

    23 күн бұрын

    @@humza890 it can't circular dependencies are usually rare and you can stop looking for dependencies once you've seen it. I also didn't mean, that every company has to look through all of their dependencies and maintain them all, but maybe picking a few or doing an audit of some of them every now and then would be beneficial to not only them, but the world as a whole

  • @GoogleDoesEvil

    @GoogleDoesEvil

    23 күн бұрын

    The Unix philosophy of "do one thing" and link against a ton of dependencies was a mistake.

  • @bookle5829

    @bookle5829

    23 күн бұрын

    It's what FUTO stands for.

  • @c1ph9r
    @c1ph9r23 күн бұрын

    the negative 900 million dollars hits hard 😭

  • @_GhostMiner

    @_GhostMiner

    22 күн бұрын

    Why? Trash game has trash income

  • @Luna5829

    @Luna5829

    22 күн бұрын

    @@_GhostMiner its not a game tho its a game engine and hoster

  • @_GhostMiner

    @_GhostMiner

    22 күн бұрын

    @@Luna5829 **H O S T*

  • @_GhostMiner

    @_GhostMiner

    22 күн бұрын

    @@Luna5829 ackhually

  • @ducksongfans

    @ducksongfans

    22 күн бұрын

    @@_GhostMiner plenty of trash games hosted on roblox, plenty of great ones

  • @ratm0
    @ratm023 күн бұрын

    "A massive company with ... -$924 million net income" 💀

  • @klafbang

    @klafbang

    23 күн бұрын

    "Each minute of downtime costs us negative $1750, this must be fixed ASAP!"

  • @kosmonautofficial296

    @kosmonautofficial296

    23 күн бұрын

    @@klafbanglolol

  • @Mempler

    @Mempler

    23 күн бұрын

    That is absurd lmao

  • @zaper2904

    @zaper2904

    23 күн бұрын

    @@klafbang So does that mean they were earning money when they were down? 🤔

  • @LibertyMonk

    @LibertyMonk

    22 күн бұрын

    ​@@zaper2904no, because they still had expenses (developers trying to fix the servers) but reduced income (no micro transactions available).

  • @k4l1hm4n
    @k4l1hm4n23 күн бұрын

    Turns out, this video could be a great introduction to modern backend architecture and development.

  • @SrKinko

    @SrKinko

    23 күн бұрын

    I think all of his videos are a good resource for understanding different architectures and subsequently how fragile they can be lol

  • @wangjiefan8939

    @wangjiefan8939

    23 күн бұрын

    I worked at a global e-commerce company a year ago and their platform infrastructure is pretty similar, down to their use of etcd and go channel spaghetti 💀

  • @百合仙子

    @百合仙子

    21 күн бұрын

    and a great counter-example for troubleshooting....

  • @bounceysteve

    @bounceysteve

    13 күн бұрын

    the leaks are too

  • @erie7452
    @erie745213 күн бұрын

    Crowdstrike video when?

  • @yeetyeet7070
    @yeetyeet707023 күн бұрын

    github repo: "it was a toy project never meant for production" multibillion dollar company: "YAYEET"

  • @Mihacappy

    @Mihacappy

    17 күн бұрын

    "multibillion dollar company" _-$924 million net income_

  • @bruhmoment5974

    @bruhmoment5974

    12 күн бұрын

    L pfp

  • @ccccy-o7x
    @ccccy-o7x23 күн бұрын

    Hi Kevin, amazing content as always! One minor correction @9:54 tho: Go unbuffered channel's length is 0, instead of 1, and it means the sender will get blocked until a receiver receives the value. What the video @9:54 showcase is actually a buffered channel with length 1 (e.g. result of make(chan string, 1)).

  • @mrdabup
    @mrdabup23 күн бұрын

    I still remember the day that it went down, people were blaming Chipotle (american fast casual chain) because they had an event that same day where you could claim a free burrito. People suspected that it was due to a mass influx of people, I knew (and a bunch of people too) that this wasn't an issue with influx of people. At the end of the day, it was a fun journey (more or less with the conspiracies, guessing correctly that it went down for 3 days months before this outage, and youtubers just milking on the outage). Thank you for making a video about this.

  • @frezzingaces

    @frezzingaces

    23 күн бұрын

    Wait. How in tf could Chipolte's traffic affect Roblox's servers. Whats the theoretical connection?

  • @baribari1000

    @baribari1000

    23 күн бұрын

    @@frezzingaces it was a sort of partnership between chipotle and roblox, so if you installed roblox and did a bunch of stuff you'd get a free burrito. I think that's what it was, roblox has done tons of these

  • @fitmotheyap

    @fitmotheyap

    23 күн бұрын

    Oh this happened during that time? Man the memes about the roblox crashes during its downtime were so enjoyable

  • @bruhlake

    @bruhlake

    18 күн бұрын

    @@baribari1000 Yeah, it was super easy too, you could do it in like 2 minutes on a new account, it gave you a free entrée instead of a free burrito, so you could actually choose most meals you wanted. The few times they did the event with chipotle, I probably earned like 35 or so free entrees, which is pretty decent!

  • @argynews2825

    @argynews2825

    17 күн бұрын

    wasnt there also a massive adopt me update at the time which also probably caused a large increase of active accounts

  • @theprantadutta
    @theprantadutta23 күн бұрын

    This is one of the biggest challenges of modern programming, depending on various 3rd party packages, not knowing what that package is, what it does, or whether it's even reliable, and moreover knowing what are the dependency of that 3rd party package and whether they are safe or not.

  • @Paulo27

    @Paulo27

    23 күн бұрын

    Also never update anything

  • @juniorwmg

    @juniorwmg

    23 күн бұрын

    *If its not a security fix ​@@Paulo27

  • @tbuk8350

    @tbuk8350

    23 күн бұрын

    Or HashiCorp, being a multi-billion dollar company, could just maintain the fucking project themselves instead of blindly using a 4-year-old abandoned pet project from some random person's GitHub page and trusting it to work in a large production environment.

  • @ironcanon4920

    @ironcanon4920

    22 күн бұрын

    And that's before the issues of relying on additional 3rd party companies to supply the correct 3rd party packages. Supply chain issues the whole way down.

  • @mahnibba2674
    @mahnibba267413 күн бұрын

    Came here to look for crowdstrike, seems like im way too late🤣

  • @michaellin7936
    @michaellin793613 күн бұрын

    Crowdstrike video incoming in 2 years

  • @andreyabrz
    @andreyabrz13 күн бұрын

    Well.. now we know the next video

  • @useruser-ti1og
    @useruser-ti1og23 күн бұрын

    This is like the XKCD of all of the world depending on a toy project someone abandoned 10 years ago

  • @imgladnotu9527

    @imgladnotu9527

    22 күн бұрын

    probably 2347... as someone mentioned in some comment above.....

  • @shalodey
    @shalodey13 күн бұрын

    IT global outage vid gonna go crazy

  • @Sam_Hue
    @Sam_Hue13 күн бұрын

    The Crowdstrike video is going to hit pretty hard

  • @levimatheri7682
    @levimatheri768212 күн бұрын

    Waiting for the Crowdstrike outage video!

  • @sergelorenzvillasica2361
    @sergelorenzvillasica236113 күн бұрын

    Can't wait for the CrowdStrike episode 😀

  • @arnavn2554
    @arnavn255413 күн бұрын

    You gotta make a video about the CrowdStrike outage

  • @pdlbackup
    @pdlbackup23 күн бұрын

    Roblox players figuring out about the DNS steering and sharing ips for early access is kinda crazy 💀

  • @TheeSirRandom
    @TheeSirRandom21 күн бұрын

    Imagine how it must feel, starting a free project just as a hobby, and planning to abandon it eventually, then pretty much half the internet starts using it as an important building block to support the web. Now you're just sitting there, and have a choice to make. Stop maintaining the software, and pretty much break half the internet or keep going, getting zero thanks, and zero dollars for your work.

  • @mortred4144
    @mortred414413 күн бұрын

    yo when is the CrowdStrike video coming

  • @ski3r3n
    @ski3r3n23 күн бұрын

    the kids enter angry the kids leave confused

  • @darthmaul5413
    @darthmaul541313 күн бұрын

    can you do a video about the current CrowdStrike Outage?

  • @KieranMahoney
    @KieranMahoney12 күн бұрын

    WHENS THE CLOUDSTRIKE EPISODE COMING OUT???

  • @nebufabu
    @nebufabu23 күн бұрын

    Whatever it took to make a video about a Roblox server crash and not use the "oof" SFX even once... I salute it.

  • @MartijnvanBerkel

    @MartijnvanBerkel

    23 күн бұрын

    It's on 6:45

  • @yeetyeet7070

    @yeetyeet7070

    23 күн бұрын

    @@MartijnvanBerkel gottem

  • @aze4308

    @aze4308

    23 күн бұрын

    6:45

  • @nebufabu

    @nebufabu

    23 күн бұрын

    @@MartijnvanBerkel I stand corrected. Frankly, using it only once is even more impressive.

  • @vincentschumann937

    @vincentschumann937

    21 күн бұрын

    i read this 2 seconds before the oof sound played, well done sir

  • @TheRealStevenPolley
    @TheRealStevenPolley13 күн бұрын

    Kevin Fang, big fan here. Please cover the clownstrike incident

  • @ChineseKiwi
    @ChineseKiwi13 күн бұрын

    Kevin, get busy and make the Crowdstrike video 😂😢

  • @bummbumm6
    @bummbumm623 күн бұрын

    This happened in the middle of my friends sleepover, when we were COMPLETELY into Roblox. He pretty much just came to play it. We checked like every 5 minutes if it got better. We eventually just slept. THROUGH THE WHOLE THING Edit: Are some of you really watching videos on Roblox and just hate people in the comment section who used to like the game? Find something better to do jeez

  • @ProblematicParag0n

    @ProblematicParag0n

    23 күн бұрын

    Seems like you guys need to find better games

  • @N30ZUK1

    @N30ZUK1

    23 күн бұрын

    ​@@ProblematicParag0n Isn't your avatar from a ripoff of Minetest?

  • @Hellscaped

    @Hellscaped

    23 күн бұрын

    @@N30ZUK1 minetest is a clone of minecraft...

  • @tbuk8350

    @tbuk8350

    23 күн бұрын

    @@N30ZUK1 Calling Minecraft a ripoff of Minetest is the most sweaty nerd Redditor thing you could do

  • @dexahtheman

    @dexahtheman

    22 күн бұрын

    @@tbuk8350 Tbh nothing is correct here. Minetest is not trying to be minecraft it's trying to be a general purpose voxel game engine (check out it's other gamemodes there's some pretty unique cool stuff in there)

  • @yaakovwaxman4807
    @yaakovwaxman480723 күн бұрын

    This is by far my favorite documentary channel on yt

  • @BananasAintCheap
    @BananasAintCheap23 күн бұрын

    It’s crazy how much of the internet as a whole is in the hands of solo developers who made a thing in their spare time for fun

  • @MaximumADHD
    @MaximumADHD23 күн бұрын

    Oh shit I was gonna suggest this as an idea, awesome to see that you did it!

  • @0x7f2c

    @0x7f2c

    22 күн бұрын

    Lol nice you're here

  • @glefyr

    @glefyr

    22 күн бұрын

    is that

  • @use2l

    @use2l

    21 күн бұрын

    ​@@glefyrhello call of duty black ops guy

  • @superbobsaget9000
    @superbobsaget900023 күн бұрын

    Thank you for all the work you put into making this!!

  • @HarishDoredla
    @HarishDoredla13 күн бұрын

    Next video on Crowd Strike update causing global outage!!

  • @ChineseKiwi

    @ChineseKiwi

    13 күн бұрын

    It was Crowdstrike, not Microsoft

  • @_tylerkinney
    @_tylerkinney23 күн бұрын

    Thank you for this, been waiting for this one for awhile now!

  • @tekratek4077
    @tekratek407723 күн бұрын

    Nice technical aspect of the outage!

  • @Aunarky
    @Aunarky23 күн бұрын

    I'm glad you made a video on that. I had no idea how it went down behind the scenes! :D

  • @Komas19Gaming

    @Komas19Gaming

    16 күн бұрын

    there was an blog post made after the outage

  • @iqmal
    @iqmal13 күн бұрын

    Great. Hopefully you'll make a video about Windows bsod due to CrowdStrike

  • @rusprice
    @rusprice20 күн бұрын

    Thanks! I submitted this in as a suggestion a while ago, never thought it’d be published.

  • @pompomaddons
    @pompomaddons23 күн бұрын

    KEVIN FANG JUST DROPPED A VIDEO ABOUT THE HALLOWS OUTAGE OH MY GOD

  • @jakfjfrgnei

    @jakfjfrgnei

    21 күн бұрын

    The burrito incident

  • @pompomaddons

    @pompomaddons

    21 күн бұрын

    @@jakfjfrgnei the slippery cord incident

  • @Hopgop1
    @Hopgop123 күн бұрын

    Man I love your videos, this was a particularly technical one, but still really well presented and interesting.

  • @i-am-linja
    @i-am-linja23 күн бұрын

    I'd imagine programmer Hell is just a bug like this which takes all of Eternity to fix, also it takes down the company's internal issue tracker and communication system.

  • @davidslevs
    @davidslevs23 күн бұрын

    Roblox is actually a bigger company than most think. Thanks for doing a video on it.

  • @zenobikraweznick
    @zenobikraweznick23 күн бұрын

    Amazing CGI as always, thanks !!!

  • @asmith7966
    @asmith796621 күн бұрын

    Haven't finished the video yet, but this already makes me feel better about the half-day internet outage I fixed at work

  • @poketopa1234
    @poketopa123419 күн бұрын

    Great great video, I seriously love the format and I learn so much

  • @xFrednet
    @xFrednet23 күн бұрын

    Awesome summary, as always. Thank you! :D

  • @Cmanorange
    @Cmanorange22 күн бұрын

    daily appreciation of kevin's visual style, i love how you're able to break down the language i might take for granted and make it easily followable

  • @patahgaming
    @patahgaming23 күн бұрын

    Saddest day ever for 7 Y.O i hope they can recovery from this 😢

  • @dagdnoob

    @dagdnoob

    22 күн бұрын

    😂😂😂😂😂😂😂😂😂😂😂

  • @gn2b445

    @gn2b445

    15 күн бұрын

    developers probably missed out on millions of dollars too!

  • @pitust
    @pitust23 күн бұрын

    9:45 "A default channel can only hold one piece of data at a time" It's actually even worse than this: an unbuffered channel also requires that this piece of data is received before a send can complete (!)

  • @MohamedAruham
    @MohamedAruham23 күн бұрын

    Damn I was waiting for this one

  • @Evercreeper
    @Evercreeper23 күн бұрын

    YAY GLAD YOU DROPPED THIS

  • @wormonastring6888
    @wormonastring688823 күн бұрын

    Another super interesting well researched + explained video! As a back end game dev, thanks for the nightmares!

  • @TheGrimravager
    @TheGrimravager23 күн бұрын

    > And probably some machine learning and block chain for good measure lmao nice

  • @mwalton9526
    @mwalton952612 күн бұрын

    How fast can you pop out a video? I think there might be something video worthy.

  • @hdgrove5567
    @hdgrove556722 күн бұрын

    Love these videos please keep them coming!

  • @passenger175
    @passenger17523 күн бұрын

    Good work, these are both interesting from the tech perspective and just plain fun hah

  • @hasanpatel9029
    @hasanpatel902923 күн бұрын

    The oof sound was a chef kiss to this master piece of video. Great work as always.

  • @Dudex11a
    @Dudex11a22 күн бұрын

    This video is very well executed!

  • @warw
    @warw23 күн бұрын

    Great video!

  • @_xord
    @_xord23 күн бұрын

    new kevin fang video today is a good day

  • @ElioAllen-sb6by
    @ElioAllen-sb6by23 күн бұрын

    I like your stuff keep it up make more security related stuff!

  • @ishan6771
    @ishan677121 күн бұрын

    Well done as always

  • @flokibyarian6832
    @flokibyarian683221 күн бұрын

    Thank you for the great information and entertainment video like always😊

  • @JustDeeevin
    @JustDeeevin19 күн бұрын

    Tons of love for captioning your videos❤❤

  • @3rdalbum
    @3rdalbum17 күн бұрын

    Another great video, I really enjoyed it. There's probably heaps of outages you can do next, but perhaps you could do a video on the "OpenOffice can't print on Tuesdays" bug?

  • @mat-hu5ys
    @mat-hu5ys23 күн бұрын

    love your vids! please make more

  • @frosty4769
    @frosty476922 күн бұрын

    the goat's back with another banger

  • @gareth2021
    @gareth202123 күн бұрын

    great video, thanks dude

  • @matze489
    @matze48923 күн бұрын

    its a good day when there ia a new kevin fang video

  • @vash47
    @vash4722 күн бұрын

    your videos are quality over quantity

  • @Spiffycaius
    @Spiffycaius23 күн бұрын

    Man I remember when this happened this was crazy.

  • @ibis8566
    @ibis856622 күн бұрын

    these videos make me feel like im watching a some type of CSI crime documentary

  • @Viniter
    @Viniter22 күн бұрын

    I love this series. It's like true crime or airplane disaster videos, but it can be fun, because nobody really gets hurt. Except for big corporations and Roblox players, and well... screw them.

  • @hagangray8006

    @hagangray8006

    22 күн бұрын

    That’s a bit harsh on Roblox players… I mean most of them are like 9 years old

  • @absoultethings4213

    @absoultethings4213

    22 күн бұрын

    @@hagangray8006if they aren’t 9 there’s a 50% chance they’re a predator or another kind of scum

  • @apersoniguess_

    @apersoniguess_

    21 күн бұрын

    @@absoultethings4213 or… just normal people. Big shocker I know

  • @adityaramadhan1708

    @adityaramadhan1708

    20 күн бұрын

    ​@@apersoniguess_impossible😱😱😱😱😱

  • @enthuscimandiri1640

    @enthuscimandiri1640

    19 күн бұрын

    unti money some rando get involved, yeahhh its really fun

  • @NoobieNoodle89
    @NoobieNoodle8922 күн бұрын

    I love the way that you explain these complex incidents. You deserve a 冰淇淋🍦

  • @ImTotallyTechy
    @ImTotallyTechy23 күн бұрын

    In life... you have roblox (another BANGER kevin fang video, cant wait for the next)

  • @thegammingbros6231
    @thegammingbros623112 күн бұрын

    love this video makes everything understandable!

  • @H-E-S-C
    @H-E-S-C19 күн бұрын

    finally, a good video on the infamous outage

  • @heyjakeay
    @heyjakeay23 күн бұрын

    yo honey wake up, new Kevin Fang video to watch while at work

  • @greatcanadianmoose3965
    @greatcanadianmoose396522 күн бұрын

    Always love kevin fang videos... but would you mind using I Home's we're finally landing closer to the end of the video please? Thx

  • @GardenOfUna
    @GardenOfUna23 күн бұрын

    I don't understand a single thing but I'm so incredibly curious that I want to know more I genuinely really love this for some reason

  • @Mihacappy
    @Mihacappy17 күн бұрын

    Ah yes, that day in 2021 that i was working in Studio and the toolbox stopped working, and my ass almost had a heart attack because i though i got banned.

  • @No-day-off
    @No-day-off11 күн бұрын

    Let’s go bro. CrowdStrike is giving free material to your next video.

  • @fang-penlin4482
    @fang-penlin448223 күн бұрын

    Oh man, I love your channel so much. I can't wait to see the XZ backdoor video made by you, it's gonna be fun 😂

  • @5TC
    @5TC23 күн бұрын

    Wasn't expecting him to talk about this but man I remember when this happened

  • @randomazzy11
    @randomazzy1122 күн бұрын

    5:04 I also heard avatars broke before the whole game went out, and some players were able to play roblox but most of the scripts were missing so it was pretty unplayable. Is it because the game couldnt fetch those? Wow

  • @CiY3
    @CiY323 күн бұрын

    Finally, a Kevin Fang video about an outage I was witness to.

  • @brawldude2656
    @brawldude265621 күн бұрын

    These server incident always feels like a surgery where you have to save a person in its current form ASAP

  • @siz1700
    @siz170022 күн бұрын

    Nice! I wish Roblox never recover from that!

  • @Jmcgee1125
    @Jmcgee112523 күн бұрын

    I paused the video when that perf screenshot came up. 5 seconds later I'm like "why the hell did nobody check this before?" We love lock contention.

  • @chawrx3
    @chawrx323 күн бұрын

    STARTING THE DAY OFF NICE !

  • @leosh9026
    @leosh902617 күн бұрын

    Thanks for the explanation dude"

  • @Core533
    @Core5335 күн бұрын

    Waiting for the crowdstrike video

  • @imchillingbro
    @imchillingbro17 күн бұрын

    i remember this outage, me and my friends kept playing muck while waiting

  • @06NinjaKid06
    @06NinjaKid0623 күн бұрын

    best roblox video

  • @mementomori8856
    @mementomori885623 күн бұрын

    GO mentioned! So you're telling me that I should continue to be paranoid about how every single line of code of my personal projects is not efficient or secure enough? Deal! Love this thank you!

  • @caduhidalgo4996
    @caduhidalgo499623 күн бұрын

    Baby, wake up! New Kevin Fang lore just dropped 🎉

  • @soulsmwc
    @soulsmwc23 күн бұрын

    Great!

  • @dogeextras
    @dogeextras10 күн бұрын

    My friend was stuck in a game during the roblox outage and saw lots of things happen

  • @BeaStScoPesHD
    @BeaStScoPesHD22 күн бұрын

    3 days to figure out that they turned on a bad feature a day ago is actually insane