CHECK OUT VERCEL AND UPSTASH BECAUSE THEY KEPT US ALIVE THROUGHOUT THIS vercel.com/?ref=theo upstash.com/?

@dasrite

Жыл бұрын

Am i correct in presuming the only reason you're not gonna be billed for this is because you're sponsored? So if it was me i'd be getting billed for a DDOS attack? that's enough of a yikes for me tbh lol i'd rather it crash than it staying up and them charging me tenfold lol

@t3dotgg

Жыл бұрын

@@dasrite no, they would help any customers going through this

Either they bought botnet time and it costed them money directly; or they used their own botnet (instead of renting it to other people during this time), and that's money they could have earned but didn't. So the result is the same : it cost them money. Probably a decent amount. Which I find hilarious considering the ridiculous impact they had.

@sardines7436

Жыл бұрын

not to mention theyre giving him monetizable content lol. from his pockets to theo’s

@Hexalyse

Жыл бұрын

@@sardines7436 and good publicity for Vervel too, seeing how easy it was to handle the problem with them. Conspiracy theory plot twist : it's actually Vercel themselves who conducted the attack so that Theo make this video to further PR their services to his followers, for free.

@Manas-co8wl

Жыл бұрын

@@Hexalyse i'm glad i'm not the only paranoid one who thought of this. i even went a step further..

@jordixboy

Жыл бұрын

he gain internet fame thats better than money

@sajanah1253

Жыл бұрын

Just curious, how much it could have cost for this attack?

We had a DDoS attack about a year ago where it was about ~10TB/minute and we are hosted behind cloudflare, so just couple clicks inside cloudflare panel (there is a button "we are under attack") and this attack is gone, next minute I checked where is it comes from and every single IP of attack came from outside of my country (No one wants to ddos from same country or your business since police could investigate it and attacker could end up in jail), so I did just enabled captcha for any request form outside of my country (since our business doesnt have international customers) and disabled "we are under attack" and never had an issues since then while they still trying (one year later). So may be something like cloudflare could help you.

You should make a dedicated video about DDoS-protection on the T3 stack, as clearly there is a possibility of creating unreasonable cost for the service provider and not everyone will have their bill refunded. You mention as a side note that you could put rate limiting on each route for a personal CloudFlare, maybe you expand on that and/or provide a package as framework for that.

Who has that much time to waste on a DDOS attack that gains them absolutely no benefits? It's clear that your tech stack handled the attack quite well, it didn't even cost you much at all. If you get enough views you'll probably even get money out of this video that they helped create. Someone really hated you, lol.

@NihongoWakannai

Жыл бұрын

For some reason there are random tech nerds who REALLY love to die over tiny molehills for no reason

@augustoeduardo209

Жыл бұрын

cant understand why someone waste his time to do that....

@josemfcheo

Жыл бұрын

Maybe written in Rust...

@brahimbenfares1464

Жыл бұрын

Apparently aws didn't like that video lol

@milanpatel3159

Жыл бұрын

@@josemfcheo beware of usage of that word bro 😂 (the R word)

This video got hundreds of hits in the first few minutes. Maybe your KZread is getting DDOS’d 😮

@emanuelfarauanu1760

Жыл бұрын

KZread alg really loved this video, it showed it to me in the recommendations the minute it was published. That was with me not watching many of the previous videos.

The key is that you need to rate limit these attacks whether you use server or serverless. So this demonstrates serverless has tools to handle it. Ofc if you had a server and rate limited it could handle it too. Re one of your last statements, it doesn’t mean servers would’ve been worse. It means not rate limiting would’ve been bad, server or serverless

This happened to me too! It might not have been targeted towards you because it happened to a test deployment of mine which didn't even have any real traffic. It was also on Vercel, and I get blocked pretty quickly (although support helped me get unblocked) Edit: My attack caused 462 GB-Hrs within like 20 minutes

Considering that one 10gbit server can (in theory) handle 1.5tb raw traffic (in 20min) I don't believe that this was a big DDOS attack. Also I believe that having multiple cheap vps with high bandwith automatically deployed when needed, would be probably way cheaper than vercel. Obv. the developer experience will be worse, especially when setting all the servers up or other cluster related issues occure.

@perc-ai

Жыл бұрын

Yes this was just 15 yr old some kid with a very small botnet. A pro would have used 10k IPs and Vercel would have to shutdown their dns for a period of time

This was not even that big of an attack. The traffic is literally less than 1 GB per second. If anything this was a skid attack which is further supported by them literally just loading one JS file over and over. This wasn't a DDOS attack, this was some kid trying out their $5 booter.

@OfoeNelson

Жыл бұрын

So this is not on the same level as the ddos attack that took down google

It is actually really cheap to buy residential proxies (pools with millions of IP addresses) and then use them to bombard requests to services. These residential proxies exist to enable scraping of SERP content as well as regular sites with hardened DDOS protections. Some residential proxy services also bypass recaptchas (using AI and sometimes even humans) for premium. Residential proxies have legitimate use cases but can be misused to create botnets too. That is what I am suspecting is happening here. They haven't actually paid for those 600 IPs. Rather, they are tapping into a pool of million IPs provided by residential proxy services.

@NiSiRewinD

Жыл бұрын

It's free, even on that scale. There are a ton of private projects for geting around state cencorship, x100.000 of IPs, since the Ukraine war shined a light on how those countries block access to Tor itself. No idea how big those requests were, but generating a couple Gbps traffic is still negible, as far as paid botnets would go. Those networks typically limit access to specific websites, but I guess it wouldn't be crazy to get around that, as user, depending on how they filter. I suspect this didn't run via Tor, but I guess we would with a list of the IPs

You got lucky because you are publicly sponsored ... but this brings up a BIGGER point. When you pay for metered services, the providers NEED to indemnify you against DDOS attacks or other potentially ruinous events. This could be a huge selling point, because not all services have built in caps. As a matter of experience, I witnessed a peer who used AWS, his application while still in beta had a memory leak and AWS sent him a $13k bill. Insane! Risk avoidance is important to any business especially if there is no ceiling or price cap. As someone who uses these services this keeps me up at night.

I am glad that there is an official report, from the DDOS Foundation, on this incident and that Theo is now a part of it! 👍😉

It was probably @theprimeagen...

If you weren't on the pro license and sponsored by vercel this might have been a different story. I can imagine a normal person would have to suck up the big fees or take down their application

@mormantu8561

Жыл бұрын

It depends. Cloud providers more commonly than you might think cover ridiculous fees in case of an error or attack like this. Because they don't want to lose your business.

@Knightfall23

Жыл бұрын

They’ll most likely cover it the first time but if it happens again your on the hook

@mormantu8561

Жыл бұрын

@@Knightfall23 Agreed.

@dasrite

Жыл бұрын

@@Knightfall23 I'd rather get a straight response from Vercel directly than try to imagine what might happen

@samdcbu

Жыл бұрын

If you are a normal user on the free tier of vercel you would just put your application behind Cloudflare for free and let them handle the DDoS traffic.

I mean, having a punchable face and arrogant personality is bound to provoke someone when exposing yourself to thousands of strangers. Even so, it takes some extra thick emotional issues to waste any amount of time and resources to get revenge on a parasocial relationship.

@unnaturalatrophy5377

Жыл бұрын

Damn dude 😂

I think both CloudFlare and Linode (Akamai CDN) has ddos protection included.

Love that the stack you are recommending is the one that you use for your stuff. I can imagine that the people behind this were just absurdly annoyed that you are recommending tools that don't fit their certificates or what they consider is the "right move".

It would be great to know how to prevent a DDOS attack against AWS and GCP (Cloud Run and Cloud Functions).

@QckSGaming

Жыл бұрын

GCP: Toggle the DDOS shield on. Cloud Armor it was called I think.

Did Vercel give more details? Such as if the IP addresses were all from the same IP block or dispersed across many, whether or not they were residential IPs, their own IPs, IPs from other cloud providers, etc, geolocation lookups of the IPs? All of this seems like it would be super useful to know about to prevent future attacks both for you and them

@bluesteelbass

Жыл бұрын

Willing to bet those IP addresses got put on one of the many naughty lists that are distributed to rulesets for firewalls.

Love it, keep them coming

Side note - The vid I'm most looking forward to is the one you mentioned about syncing clerk with your own db 🙂

huh, so vercel has no rate limitting by default? I would have expected a managed service to handle this, not have me set up my own edge middle ware (upstash?) It doesn't look good on their part

Is it me, but why are they are targeting static assets? I mean if you want to increase Theo's bill, DDoS the api route which has the upstash rate limiter as well? It will cost him 0.20 cents - 0.40 cent per 100k request for upstash and probably far more for serverless/edge functions on vercel. Thus sending 100 milion request will at least cost 1000 * 0.20 + 500 GB hours ( 5*40) = 400 dollar + rest of vercel

@perc-ai

Жыл бұрын

Tbh this attack probably came from a 15 yr old… this is not a serious attack tbh it’s very easy to rotate 10k residential proxies and force vercel to temporarily shutdown all ping services I could probably do it

Would be good to get an in depth video on the specifics of how you (or the tech) delt with it 🙏

at what point does vercel consider the requests as a ddos attack do they use any tools? what happens if a tiny dev's app gets ddossed, would vercel refund 100% of the money by all requests that day? how long do they take to answer from the point where you're under attack to when the situation gets resolved?

@nickwoodward819

3 ай бұрын

after 1 minute according to their webpage. not great really given the number of requests you could be on the hook for at that point. and no, if they don't say they'll refund your money, assume they won't.

>>> I think it’s quite impossible to take us down with this stack Bro, you’ve just broke the main rule of opsec 😅😅

if you're putting the upstash ratelimiter infront of everything, how are you not hitting ratelimits on upstash?

That would be an awesome tutorial setting up ddos protection using upstash

Where is redis in your data fetching flow?

Can you do a more in depth video on how to stop DDOS and other random attacks?

It's laughable how much the attacker likely spent vs what you incurred. Perfect example of mitigation, you can't stop it from happening, you have to make it too expensive for bad actors to continue.

@name_less227

Жыл бұрын

Don’t these type of attacks usually use hacked computers to help them attack?

@Hexalyse

Жыл бұрын

@@name_less227 They do. It might not "cost" them anything in the literal sense. They didn't spend money most probably, if they own the botnet. BUT... usually when you own such a botnet, you can sell it, or rather rent it to people who want to conduct such attacks. So all in all, either their bought botnet time and is cost them money directly; or they used their own botnet instead of renting it, and that's money they could have earned but didn't. So the result is the same : attacking cost them money. Which I find hilarious considering the ridiculous impact they had.

@jason_v12345

Жыл бұрын

But you can stop it from happening. Rate limiting middleware can be written in a just a few lines of code.

@perc-ai

Жыл бұрын

@@jason_v12345 Theo doesn’t even know what he is doing lol

Can you not put a Vercel site behind a Cloudflare proxy?

Wouldn't surprise me if some from /g/ were part of this, your videos have started making thier rounds over on the board.

@t3dotgg

Жыл бұрын

Oh really? Fuck yeah finally they're gonna HATE my ass

@hyper_channel

Жыл бұрын

/g/ cares about nothing but LLMs right now, highly unlikely

@varma8669

Жыл бұрын

What is /g/?

@hyper_channel

Жыл бұрын

@@varma8669 4chan's technology board

@g-rexsaurus794

Жыл бұрын

LLM?

DDOS is my main concern with Serverless. With an nginx proxy you can get sub 1ms 503 responses in a DDOS and cap the number of requests per IP so that it doesn't touch your actual app code when it happens. So for a free or cheap server vs a Serverless platform, at least getting started out, I know someone can't rack up costs for me.

6:09 I dont understand this part. Why would you fare worse if you had actual servers? They have rate limiting and IP blacklisting as well

@jacoblockwood4034

Жыл бұрын

I think he’s saying that if they were hitting the actual servers, all the endpoints would be destroyed pretty quickly

ddos a static file :genius:

Can someone describe the AWS scenario? What would have happened and how to do rate limiting?

@lapulapucityrider3227

Жыл бұрын

Use ec2 not serverless much better

Where is your video on the rate limiting with UpStash?

dude, im sold on this platform you are using.. what a way to advertise..

Yep, serverless rocks. Won't ever go back to dedicated hardware

Good stuff.

your hair looks majestic dude !!!

Honestly Theo, now I am very relaxed about the decision of using T3 Stack and the services you recommend us. If even Chirp handled this insanity! Then we’re in safe hands as Solo-preneurs 😊

upstash rate limit is good for backend but what about fronted to save from ddos attack

I have a question. Are those people generating these attacks going to be held accountable or there are ways to generate botnets attack and get away with it

Jeff Bezos hired DDOS assassins to protect his stack

This won me over!

I can't imagine that someone decided to waste any significant ammount of money doing this. I'm wondering how they had access to 600 static ip addresses.

How much would it have cost? Isn’t the point they were making is that on demand computing can make your costs sky rocket?

we got ddosed, and vercel did nothing. 6.8TB

How do you know the attackers have used static IP addresses?

@filda2005

Жыл бұрын

if it did not chang within 2weeks, it is not so much dynamic IP

How much did it actually cost you?

But, what exact cost for you of this DDOS attack?

Wow! Holy crap!

it migth be done from the rust foundation

Just wondering why it’s costing the attacker more? The hundreds of IPs used by the botnet are probably someone else’s.

What do you think about IP based rate limiters? Would they prevent such attacks? That's the only protection I have against DDOs.

@ttrss

Жыл бұрын

ddos atks are (Distributed)DOS

@sortof3337

Жыл бұрын

@@ttrss so its like wearing a hat to be bulletproof. Nice.

@ttrss

Жыл бұрын

@Sort of 😄i guess. And then cloudflare protection is like letting a government protect you, but they're like super authoritarian.

Does anyone have any examples of using upstash's rate-limiter with tRPC? Been using it more, and I haven't really gotten around the concept of how rate-limiting could be added to it. It most likely would be done via a middleware, but just putting up the flag for any existing repos that have it.

@liam.brewer

Жыл бұрын

he covers it in his newest t3 stack course where he builds a twitter clone called chirp

@SeanCassiere

Жыл бұрын

@@liam.brewer thanks! My bad for not fully checking the repo.

1400 GB costs USD 180 on Vercel. (First 1000 is USD 20) What're you talking about?!

Nice promotional video :-)

Would a simple solution be - count how many requests for each IP address, and if it goes above 100 per second you block them?

Theo makes a video about Twitch dying then gets DDOS’d hmmm… 😂 all jokes aside your services handled well!

Because you were Ddosed you have got a subscriber.

Oh nice!! Long live rate limiting.

Who has the resources to pull this off? Amazon does, that's who...

It was probably the angular team trying to make t3 look bad.

Absolutely hilarious that some people will spend thousands out of pure spite for absolutely nothing. Still, I can't help but be skeptical of these new cloud providers you're showcasing. When the 'growth' period ends and the 'taking profits' period begins, is it still going to be more attractive than AWS? We'll see!

Theo what are these thumbnails 😂

i see what you did there

Instead of bringing down your services, they just gave you a topic to talk about? hilarious!!

How would you be dead if you had actual servers running this? Wouldnt it be better because then you wont get charged a lot of money? Thanks for helping a newb like me understand

this is a pretty cool vercel ad to be honest.

What contents do you have about Vercel? I guess we all have some app to protect.

People can be nasty, that’s why it’s better sometimes to build in private. Oh and good for you man the infrastructure really held up

Vercel needs bun

@RedStone576

Жыл бұрын

bun on the edge?

@cowabunga2597

Жыл бұрын

Pork bun ?

Gotcha

"impossible to take down our services" my guy, you are challenging the wrong community here 😂, and for those that say it cost them money, ego doesn't care about money. We do things some time just to prove that we can, no need to hate you to do something like this (I'm not saying I'm even capable of this), but if i could, i wouldn't do it because hate, but ego or passion, and seems you already have enough of both

Did Vercel try to charge you?

I thought hacker is always smart, they absolutely stupid in this case.

introducing captchas might also help.

@t3dotgg

Жыл бұрын

For fetching a JS file?

👍

Must be rust foundation

Haters are just mad you can beat them in games of skate and they are mongo

Well you sold me on Vercel thats for sure.

lmao they literally gave you content

oof

Primeagen testing out his Rust pen test code?

If the haters attack again, please have a bowtie for the next vid.

Considering how much this cost the attackers and how little it affected you, it had to be someone with disposable income. Seems like Elon wants to get his revenge after you told him how ads work.

Seems someone's back-end needs Rust Framework 😊 Edit: idk why my reply multiple times got deleted. so i am sorry, i cant explain due to no freedom of speech

@IvanRandomDude

Жыл бұрын

explain

@NorthernChimp

Жыл бұрын

@@IvanRandomDude hint: kzread.info/dash/bejne/ZKOcma-oddmwaJs.html

@alessandrosomigli

Жыл бұрын

This really just shows that TS on the backend can handle this kind of load as well most of the time...

@t3dotgg

Жыл бұрын

Comments like this are why it's hard for me to give a crap about Rust tbh - like it's obvious it wouldn't have helped here at all

@christophersherman8036

Жыл бұрын

Hope this comment is ironic lol

no they didnt burned 500 slots. you burned yourself 500 slots and from all that who knows how many of them was players:))