For those that did not read though the reports, It was FIRST discovered in SSH and those details are in the Openwall OSS post I was reading from that is also linked in the description. But yes, openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. And it discovered by the researching because the timing delays in SSH.

@nagoranerides3150

2 ай бұрын

Don't use systemd. It's not designed for your convenience, it's designed for commercial entities' convenience. And it was designed by an idiot.

@lilblackduc7312

2 ай бұрын

Wendel Wilson on Level1Techs said SSH wasn't safe probably 2yrs ago and no one in this community seems to have heard it besides me, because some are acting like it's a big revelation. 🤔😳

@lilblackduc7312

2 ай бұрын

I haven't checked in with you for months, now. Your appearance has become absolutely shabby! I don't understand how a successful person can expect to conduct a professional business, and not maintain a tidy appearance. A psychiatrist might say your self-image has been degraded. I hope that's not true. Customers judge you based on your personal hygiene & kept look, weather they say so, or not.

@nagoranerides3150

2 ай бұрын

@@lilblackduc7312 Yes, but this isn't a vulnerability in ssh. It's a vulnerability in systemd. Systemd is the open door for hacking your system because it reaches into so many places where it has no business being, opening routes from any number of places which should be isolated.

Andres Freund, who found and reported this, should be called "The XZorcist".

@_sneer_

2 ай бұрын

Underrated comment

@PeterHonig.

2 ай бұрын

Superb comment!!

@timmitchell9021

2 ай бұрын

Dude

@Simte

2 ай бұрын

Lmao.

@NickDoddTV

2 ай бұрын

This needs to go into any paper about this CVE

Not comforting to know this was find by accident by someone bench-marking Postgres and it made it into the wild.

@user-in2cs1vp6o

2 ай бұрын

Don't look up how LogoFail was discovered. Greedy motherboard manufacturers will backdoor their own hardware to allow branding on boot the screen but won't bother hardening access to the processes.

@PowerUsr1

2 ай бұрын

Im with you but at least the code was there to be looked at instead of being proprietary. Super fortunate we all are right now.

@Alan.livingston

2 ай бұрын

I’m sure it was another case of absolutely critical dependencies in the Linux stack being barely maintained and liable to breaks in the chain of trust.

This is exactly the reason why i love OSS. OSS doesn't stop people from doing bad things but it makes it much more likely that when bad people do things someone can investigate. With some group effort involved a supply chain attack was spotted and remediated (in progress). My trust in OSS is the trust that I have in the community not necessarily the product. Whether its pfsense or XCPNG or your flavor of Linux, the stewards of the project are the people and I'm damn proud of the folks who are fixing it. Lets keep on supporting and staying viligent.

@villandoom

2 ай бұрын

This wouldn't have been noticed without a bug in the backdoor

@Dje4321

2 ай бұрын

@@villandoom Probably eventually but for how long? This was only found because one person happened to need precise system timings and had experience with past software versions. If the code was just always in there, or non impactful, this would not have been discovered nearly as soon as it did.

@sqlexp

2 ай бұрын

Why has there been no mention of the person who provided the code containing the backdoor? Any trace of the origin?

@U20E0

2 ай бұрын

@@sqlexp There isn't any information beside the basics, afaik. He's the maintainer of the project, who took over as the old maintainer gave up (with the attacker's alts' "help" starting from 2021). The malicious code was sprinkled in over a period of about a year ( iirc )

@ZombieLincoln666

2 ай бұрын

This is way off and it was only caught bc some Microsoft engineer miraculously caught a slight performance drop in his ssh calls

thanks for this video tom. nice to see a quick and informative response to it.

Good work Tom. No hype, no BS. And I think we’re pretty lucky it was found so quickly. This could have lain under the radar for ages.

Thanks, Tom! As usual, I get more information about threats from your channel than most other places. I'm doubly glad now that all my SSH access is behind a firewall. I'll be able to use this as an example to customers and maybe get the hardheads to secure things better.

Thanks for the update Tom!

Thanks. I was going to have to review this. Knowing where the exposure starts lets me ramp down the priority.

I hope this leads to undermining the security stance that newer is always better. I'm always fighting an uphill battle when I don't want to install a patch in prod 5 seconds after it came out.

Thanks Tom for posting this during your Easter weekend. Don't forget, Arch Linux is also bleeding edge too 😵‍💫

@insu_na

2 ай бұрын

Arch hasn't been bleeding edge for years now... Arch is barely even leading edge.... Debian sid is usually more recent than arch

I think it's time to go through all the packages and check if any other packages are "fiddling" with any blobs during build like this one did. We may be able to find other packages that have been tampered with and potentially put in place rules to prevent this in the future. "Massaging" blobs seems a bit suspicious by itself, especially the cut a bit and join something and then decompress it-like activity... 🤔

damn it just took me whole day to install and customize arch linux and this video pops up.

@Gunzy83

2 ай бұрын

At least you can say I use Arch BTW

@Agnemons

2 ай бұрын

Obviously you needed the practice 😛

Thanks Tom

The xz package is on Arch Linux as well. I had the corrupted version and just updated the package to 5.6.1-2, the safe version of the package.

Great overview. Goes to show what is possible when that itch needs to be scratched. Those who watched Mr. Robot know what I'm talking about.

Nice explanation, thanks sir.

Thnaks Tm for the detailed information. I understabd better now the origin of the compromission. I tested with the little script provided (detecte.sh), on some ubuntu machines, and they don't appear to be affected...

I literally searched for "CVE-2024-3094" and i saw this video literally released 30 seconds ago lmao

@mavfan1

2 ай бұрын

Using "literally" twice in one sentence when neither use was necessary, impressive!!!!

@Powerful4u

2 ай бұрын

@@mavfan1literally agreed. I literally wanted to say the same thing but saw your reply. Literally took it out of my mouth. Literally.

@Mempler

2 ай бұрын

@@Powerful4u literally all of your opinions

@uexodus1

2 ай бұрын

@@mavfan1 I literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally literally cannot believe how often the word 'literally' can be used literally in a sentence, literally overwhelming any sense of literal meaning.

0:30 slight correction, it's not a bug, it's a malicious payload since it was placed

@LAWRENCESYSTEMS

2 ай бұрын

Correct

Makes me wonder how many back doors closed source bios firmwares have?

Said it before and repeat again: No system or code is perfect or impossible to infiltrate! If a human made it, another human can break it!

I wonder what other places we might find similar attacks

This has been discovered by chance. If not, it would have been a nightmare and would have triggered massive attacks. Open source means lot of people are working on it now, but how can this situation be prevented ? More controls are necessary to prevent injecting attack payloads.

Love how the Fedora people went back to xz 5.4.6 because they don't understand the 5.6.0 garbage code ;)

@iuse9646

2 ай бұрын

Context?

@Batwam0

2 ай бұрын

5.6.0 is also compromised so they had to. The updates to 5.6.1 were made to make the backdoor harder to detect.

@peterpain6625

2 ай бұрын

@@iuse9646 Maybe watch the video before commenting? ;)

@peterpain6625

2 ай бұрын

@@Batwam0 Makes sense. I bet if they released 5.6.1 as 5.6.0 it would still be undetected.

@Batwam0

2 ай бұрын

@@peterpain6625 i believe so too. Seems like he did a relatively sloppy job all in all as the implementation resulted in slowdowns and crashes. Had he been good at this, we probably wouldn’t know it’s there. Scary stuff.

The best part is that whoever made this is going to be FUMING that they were rumbled by shitty performance of what they wrote.

A good followup would be how you'd lockdown a vps that has to be public facing such as an email server or webserver. I can understand the VPN recommendations if you how the network your server is in, but if your server is in the cloud it's a little bit harder to lock down

@sleepyostrichthing3599

2 ай бұрын

Actually it makes no difference. Like he said, firewall rules restricting SSH to known IPs would work. Or cut SSH off from WAN entirely and make a VPN to your VPS' "LAN" and SSH through that.

@__Brandon__

2 ай бұрын

@@sleepyostrichthing3599 VPN to your server doesn't sound like it would protect you from a similar supply chain attack. I don't think that locking it down to 1 up address and then making that IP address be your vpn server so you do client-vpn-ssh-server and then blocking everything except that IP on the firewall and server could do it. Just seems like every method is vulnerable to atleast a few ways of breaking in given a 0day

@LAWRENCESYSTEMS

2 ай бұрын

Most all hosting companies have some sort of firewall option that will allow you to filter ports by IP.

Yup, ssh behind the vpn, thank you very much.

Law enforcement ain't gonna get squat on this guy. Micronesian IP, but almost certainly a proxy. It WAS the 2nd (newer) maintainer, and there was a 3rd (possibly sock) person who kinda goaded the original maintainer into handing things over to the 2nd. This was YEARS in the making and if you think there's even a CHANCE this was not government sponsored, well, you're new at this I guess? …the question is which government? The answer is we probably will not find out if people purged their systems (and assuming this backdoor didn't do anything more than we KNOW it does…) We _might_ have dodged a bullet here. Maybe. Let's hope.

The 'almost' hidden dependencies used by the attack seemed very well thought. The social plot planed in years. The possibility of state sponsoring is very SUS in my opinion. Its "target" would have to use some latest distro. My question is why would a high-steak entity uses non-stable distro?

that 'you're on mute' tshirt is sooo true!!! lol!!!! i am realy on mute since you canot hear me as its a video recorder alredy!! lolol!

need to read "Reflections on Trusting Trust" again. Who/what company made the change ?

@handspiker1994

2 ай бұрын

The change was committed by a maintainer that took over the project last year after the creator of the project burned-out. It was slowly added over months. The committer has also been adding test binaries for multi-years so basically everything needs to be audited now. Investigation so far has found the 0.1 version that was actually used to hide an update to the backdoor to prevent detection by Google (who does their own testing and packaging) because it started flagging warnings in memory debuggers. The backdoor appears to have been the result of years of planning, so it's likely a country's intelligence bureau that created the backdoor. Even some of the "people" supportive of the new maintainer as far back as 2022 have been found to possibly be sock-puppets that went silent soon as maintainer access was granted.

@muhdiversity7409

2 ай бұрын

@@handspiker1994 Thanks. I figured it was something like that. Playing the long game. I'm guessing there's other packages that have been similarly compromised.

@Agnemons

2 ай бұрын

As always. Trust but verify.

I looked at my system but Ubuntu 23.10 has not that version

This was super lucky. How many such backdoors did not have shitty implementation that can be spotted with valgrind?

Rest assured it's one of many.

so if my xz version is 5.4.5, am i still affected?

thanks how clouds data center deal with this backdor ?

@LAWRENCESYSTEMS

2 ай бұрын

The good ones do it with lots of monitoring and watching of connections.

The real problem with opensource is not properly supporting/compensating the contributors.

What uses SSH, that we know we want to keep up-to-date, SFTP transfer software. Is this vulnerable if just recently updated? Would it use that library?

@LAWRENCESYSTEMS

2 ай бұрын

Are you running a bleeding edge distribution or a stable production one? This was really only pushed to bleeding edge.

@JonathanSwiftUK

2 ай бұрын

@LAWRENCESYSTEMS like most enterprises, and especially those which are heavily audited, such as insurance and finance, we have to take released patches quickly after release, but it should only be stable releases, the problem would be if some dev just decided to take non stable release code, perhaps to fix some problem they think they have, because they can't wait, working to a release schedule, and incorporated it into other stable code. When you get the next patch from supplier XYZ you cannot be absolutely sure what is in it. I'm being over cautious, probably :D

Wasn't in the code it was in the make file that's what made it hard to find.

xz --version -> 5.6 contains the problematic code (xz-utils) So if you are 5.4 or below you are fine. As it wasn't released to stable, server OS's are likely not going to be an issue. (lucky) But biggest thing to check is any OS you use that is raw/main - otherwise you are most likely fine :)

@legi0n99

2 ай бұрын

There si concern about malicious code in the previous versions as well.

@mitchellmnr

2 ай бұрын

@@legi0n99 yes, always. But the specific topic apparently was added in the latest so it in and of itself, isn't out on stable - at least.

@__Brandon__

2 ай бұрын

Arch is currently running 5.6.1, are they impacted

@mitchellmnr

2 ай бұрын

@@__Brandon__ 5.4.6 stable is the known stable and safe version. Uknown if impacted, but I would downgrade from 5.6 to 5.4 for safety ...

@-morrow

2 ай бұрын

@@__Brandon__ arch isnt impacted since arch doesn't use systemd which is required for the backdoor to be triggered. I'd still downgrade for safety.

Does this affect any unifi product?

@LAWRENCESYSTEMS

24 күн бұрын

None that I am aware of.

Attack you mean?

@LAWRENCESYSTEMS

2 ай бұрын

Haha, yup!

@elitedeciel

2 ай бұрын

@@LAWRENCESYSTEMS maybe it could be a new type of attack. Attaching malware while in supply delivery.

You know what would be cool? If it turned out the back door was an elaborate red herring and april fools joke to raise awareness for how undermaintained pivotal OSS projects can be subverted. I doubt it will turn out to be that way, but hope springs eternal.

Someone is cornered and has an IT guy that has to be dealt with.

I actually do not see the connection between 'xz' and 'ssh' ... none of my ssh/sshd uses xz/libxz at all. The title of this video talks about an attack in ssh ... the content is all about xz. Could someone clarify this. (I use some machines with opensuse tumbleweed (the rolling release version of opensuse), and I found my xz package version to be 5.6.1 .... this would be affected).

@LAWRENCESYSTEMS

2 ай бұрын

It was first discovered in SSH and those details are in the Openwall OSS post I was reading from and linked in the description. But yes, openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.

How the hell did someone manage to sneak a backdoor into open source without it getting noticed, and can anyone use the backdoor or is it one person or entity who can utilize it?

@zekicay

2 ай бұрын

The co-maintainer of the project did it - it was a long con. In the research so far it seems that only one person/entity can utilize it - only ones who know the secret root key the code is hardcoded to validate against.

@ihad2reload

2 ай бұрын

It got noticed

i was never a fan of the way kali loads soooo many packages, and all of them bleeding edge experimental

hmm.... maybe need to start filtering these source contributor.... for the past few year, we starting to see a lot of these (unethical) contributor, attempt to put these backdoor into the pool.....

How do I patch it? Anyone have the command?

@LAWRENCESYSTEMS

2 ай бұрын

The command depends on what distor of Linux you are running and only bleeding edge systems pulled the bug in.

State actor methinks

Do people really open SSH to the public still? If other peoples firewall logs look like mine on my home pfsense box, they are insane. Or dumb. The two ports banged on the most, by a huge margin are SSH and TELNET. Thousands of connection requests over a week.

Why was Microsoft mentioned?

@LAWRENCESYSTEMS

2 ай бұрын

Because it was found by a Microsoft employee and Microsoft owns GitHub

@Agnemons

2 ай бұрын

Because it's bad so Microsoft is included atuomatically?

I have windows 95, am I affected?

@zacanger

2 ай бұрын

Great strategy, avoid new vulnerabilities by just using an OS that's too old to do anything!

@_sneer_

2 ай бұрын

Yes you are, but not by this.

@bertblankenstein3738

2 ай бұрын

Just keep win 95 behind your firewall and don't connect to the interwebz.

@ArmageddonAfterparty

2 ай бұрын

Lol, i should have known people would take this seriously. I am so sorry.

@bertblankenstein3738

2 ай бұрын

@@ArmageddonAfterparty you mean you really are not using w95? 😯

The line about "law enforcement" being on task to investigate is worrying, because that's the last group i trust with security related investigation

@zacanger

2 ай бұрын

It's CISA, not, like, the NSA

Another obvious reason not to touch systemd distros. Many already know that. And much more don’t.

Imagine if the guy who found this is actually the malicious party and getting everyone to upgrade is all part of their plan

Can be said whoever inserted the backdoor has poor programming skills? With such way to get worldwide systems backdoored you make it recognizable for excessive resource use? wthell!

Any chance you could at least get the title right? The supply chain attack was found in xz not ssh.

@LAWRENCESYSTEMS

2 ай бұрын

The title is correct as it was found in SSH and SSH appears to be a target for this. And before you say that openssh does not directly use liblzma, Debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. Am I wrong?

@RobbyPedrica

2 ай бұрын

@@LAWRENCESYSTEMS Yes you're wrong. 1. the exposure of SSH does not require any fix in SSH, only in xz. There's a big diff between "supply chain attack in SSH" to "appears to be a target for this". A supply chain attack relates to the component that has been attacked, in this case xz (according to every definition of supply chain attack). 2. note, only xz packages have been updated, not SSHD. Again ref'ing the attacked package. 3. not a single news outlet indicated a supply chain attack in SSH ... it's understandable that you got this wrong, seeing as you're not a security researcher, specialist or consultant. And you can admit making a mistake - we all make these from time to time.

@LAWRENCESYSTEMS

2 ай бұрын

Yes, there is a difference in "Supply Chain Attack in SSH" and my title of "Supply Chain Attack Discovered in SSH" because that is both where it was first discovered and the target of the supply chain attack.

first

Seriously selling open source as a benefit here?

@LAWRENCESYSTEMS

2 ай бұрын

Even more so. This happens in closed source and is MUCH harder to audit and unwind.

Am I the only one that laughed uncontrollably to hear, "bleeding edge" and "Debian" as factors to be vulnerable? I know where my money is on this one. Safe bet.

@mrkesu

2 ай бұрын

What?

@LackofFaithify

2 ай бұрын

@@mrkesu Debian usually isn't associated with the phrase bleeding edge and the ones that added the code, well, you can google.

@fomxgorl

2 ай бұрын

this was bleeding edge cause it was caught before it made it to stable. we got very lucky for this one, but we may not be so lucky in other cases that we now need to investigate

@nobloat5702

2 ай бұрын

This affects Debian Sid and it is considered bleeding edge. May desktop users choose Sid for latest updates Debian Stable is not affected as far as I know. I am running it and it doesn't use the new package.

@relwalretep

2 ай бұрын

Probably

You're on mute

Very bad click bait title. You know better Lawrence.

@LAWRENCESYSTEMS

2 ай бұрын

What's click bait? It was a dependency attached to SSH that led to the discovery.

NSA must he really sad to have lost access to this gem

Wouldn't it only effect someone if they have port 22 open