Wow. Your video made me go back to my box (and review all my folders) and I had one folder that did not have the data integrity protection. Thank you for the information.

Great heads up, thanks for sharing.

Good information Frank. Thank you for sharing!

@WunderTechTutorials

Ай бұрын

Thanks for watching, Tony!

Bit rot does happen, I've had it happen and it's ugly if aren't protected from it when it does. I'd advocate for monthly scrubbing of volumes. Even just from a hardware testing point of view for the drives and not so much the data to ensure drives aren't accumulating errors also captured by SMART status reporting during the scrubbing process.

Excellent video Frank. you have touched on a super important topic that even linus tech tips was once a victim of. great video!!!

@WunderTechTutorials

Ай бұрын

Thanks, Avi! Appreciate you watching!

To enable checksums on a shared folder that does not have them enabled you can enable them when moving the shared folder to another volume. Of course you need 2 volumes.

@steveyg777

Ай бұрын

You absolute genius! Good call dude

Fantastic video, thank you very much. I've just gone in and saw that none of my folders have this setting on so I might just start creating new Shared Folders with this feature on and copy everything across. My question is that when this setting is off for all of the shared folders, do you know what does actually data scrubbing do when it automatically launches due to a power outage? (obviously when it's not connected to a UPS)

@WunderTechTutorials

Ай бұрын

Thanks! That's a good question - it must be checking the integrity of the data but I'd imagine it's not exactly what Data Scrubbing does, though I'm not certain of that.

It doesn't happen often but bitrot is real. It mostly affects larger files but can happen to smaller files too. This is most obvious on pictures and videos that appears corrupted when you open them. I've had seen this happen multiple times on some of my media that are over 10 years old that was just stored on regular hard drives without much protection like using a btrfs filesystem. More modern hardware should be more resistant to this. Great choice of topic to discuss! Back then I didn't know about bitrot and wondered why some of my files got corrupted. That's when I learned how to create par files to use to repair corrupted files. I have no NAS back then let alone running a raid set up. Fun times. 😅 I suggested this feature to UGreen's Kickstarter page. I hope they implement it on their NAS'es. 😔

Love your videos Frank. So I have a two bay nas…I have checksum enabled and have a data scrub set for twice a year. Since I have 2 drives in SHR , will I not get the fixing benefit of this feature?

@WunderTechTutorials

Ай бұрын

Thank you! I am hoping that Synology would inform you if it found a mismatch and in that case, it makes sense to run it so at least you know. The problem is, I have no idea if they notify you. I couldn't find any email notifications for it, but I'm hoping it just does it by default.

@Foton410

Ай бұрын

​@@WunderTechTutorials So RAID is not a backup and new one - checksum on 1 and 2-bay NAS do nothing because we need 3 disks xD slick topic so this explains the lack of information what actually do scrubbing in 1 and 2 bay NAS

I wonder what data scrubbing is doing, when scrubbing my single disks with a single volume and one shared folder on that volume? It checks something, since it runs for many hours on each drive :) Some of the drives/volumes/shared folders I have in the scrub list don't even have checksums enabled!

Great videos, thanks. After watching your video I found I have integrity checking on most shared folders and was going to fix it for the others but then thought since I have 3-2-1 backups in place for my data, do I need this on all folders? I can always get a copy of a file from a backup. Any thoughts?

@WunderTechTutorials

Ай бұрын

Short answer is the backup may or may not have the correct data, so it's possible, but if you're using SHR (3 drives), RAID 5, 6, or F1, it might make sense to just set it up.

Thanks!

Thank you! Uups… didn‘t know i need three drives 😢 does terramaster have something like integrity check for shared folders / bit rod protection?

@WunderTechTutorials

Ай бұрын

Yes they do, assuming you use Btrfs.

Hi Frank, you mention SHR (with at least 3 drives) to be “okay” (as long data integrity option is enabled). Is SHR2 with 4 drives also “okay”, or does “all this” only apply to SHR (and not SHR2)?

@WunderTechTutorials

Ай бұрын

SHR-2 will work too!

@markhuf

Ай бұрын

@@WunderTechTutorialsGreat, thanks Frank 😊

Great video once more. Thank you. 2bay nas here. JBOD configured. I understand that data scrubbing for me is useless and should be disabled, correct?

@WunderTechTutorials

Ай бұрын

Thanks! Someone else asked that and truthfully, I don't know. I imagine running it can't hurt, assuming it notifies you if it finds something wrong, but I'm not sure if it does. I've never received one, but I'm not positive I ever experienced bit rot either.

@user-lw4vq3yq6k

Ай бұрын

@@WunderTechTutorials thanks fror your reply!. I'll leave it on. Not big deal to run twice a year. Doesn't hurt, as you say.Thanks again for your effort and great videos

Had huge bitrot once... the nas files were on had a failing power supply that only found out about when it started generating a smokey smell...

Thank you. I've been wondering about this. I've been confused why data scrubbing and the enable advanced checksum option when creating shared folders were separate options. If I understand correctly, if you do NOT have the enable advanced checksum option checked, the only way to recover a corrupted file is via the fact you have a multi disk RAID 5, 6 or the equivalent SHR configured. What still confuses me, is why then is data scrubbing offered as an option on my 2 bay system when the enable advanced checksum option has NOT been selected on my shared folders? Also, is there a way of determining if the advanced checksum option has been set on a shared folder?

@WunderTechTutorials

Ай бұрын

I think it's mainly there to store the checksum incase you expand it later. I am not sure if the NAS will notify you if it finds a file where the checksum doesn't match (I'm hoping it does), and if it does, it could help you restore the file from a backup (assuming the backup has a better version).

I just checked my shared folders and almost all of my important folders are checksum enabled. I did a quick test to see if, when creating a new shared folder, if the checksum option box was enabled by default. It was not checked by default. Oddly, my ActiveBackupForBusiness did not have the checksum enabled. I don't know if that was my fault or Synology created that folder when I started using ABB. Fortunately, I have 2 other NAS where all of my data gets backed up to. Thanks for the eye opener Frank.

@mvrck-pb5pk

Ай бұрын

For all Btrfs volumes, new shared folders should be checked by default which isn't . I went through that exact same 'mistake'. But, ABB does not create the checksum, so I recreated it manually.

@WunderTechTutorials

Ай бұрын

It's funny because on my older NAS, it's not checked off for the ABB folder, but for the newer one that I use for testing, it is. Synology must have changed it at some point.

@mvrck-pb5pk

Ай бұрын

@@WunderTechTutorials Synology should address this issue for the less tech savvy users: check the checksum box by default, when creating a shared folder on a Btrfs volume. Simple and more user friendly. Having a strict 3-2-1 backup strategy is mandatory for all NAS users.

Other question, on a 2 bay nas (224+ in my case) with shr, should I unschedule data scrubbing ? Since it's enabled as well as data integrity

@raylopez99

Ай бұрын

Yeah I think the author misspoke. I don't see why a non-four-bay NAS, that is, a two-bay NAS, cannot do data scrubbing.

@nixxblikka

Ай бұрын

No you can still have data scrubbing active, just the self healing part won't work, but at least you know which files are corrupt

@WunderTechTutorials

Ай бұрын

Thanks for watching, and I totally agree on the UGreen NAS!

@Kevin-oj2uo

Ай бұрын

​@@nixxblikka I have a two bay NAS and the selfhealing option is there and can be enabled. So I don't understand why Synology would put it there if it doesn't work?/

@nixxblikka

Ай бұрын

​@@Kevin-oj2uo good question, probably because of raid 1, I would have thought due to parity information it works there too, but apparently it works on raid 5, I'm on holidays so can't test right now...

@wundertech you mention about not using an expansion unit (like the dx517? - which i own), why is that? (or did i get that wrong and you were talking about external disks attached via usb?)

@WunderTechTutorials

Ай бұрын

You can use one, but the expansion unit should be its own storage pool/volume rather than extending the first from a 2-bay NAS. The cable is a point of failure and if it goes bad or gets pulled out at the wrong time, etc, the storage pool will most likely be shot.

Most useful video I've watched this year! I went and looked at my shares and about half of them had data integrity enabled and half didn't. So, I've spent the last three weeks re-creating the shares, copying the data, verifying the data, renaming then deleting the old share, re-configuring Hyper Backup and Snapshots, etc. I'm working on the last share that I need to do this to: homes. I've copied the data, turned off the Home service, disabled the Synology Drive and "My Photos" services, but I still can't rename the existing "homes" share (I prefer to rename the old and new shares, re-enabling services, etc before deleting the old share). If I delete the existing, non-data-integrity-protected "homes" share, will I be able to rename my new, protected share to "homes"?

@WunderTechTutorials

12 күн бұрын

Thanks! No, I wouldn't do that. Honestly, I don't know in what context it gets created properly but at some point in DSM 7, it was enabled by default (and never was in the past for the homes folder). I don't exactly know what will happen if you delete it so I'm afraid to tell you to do that. Do you have a virtual DSM instance set up? It might make sense to test it there.

@tkeva

12 күн бұрын

@@WunderTechTutorials Thanks for the response! So I ended up firing up an old Synology box that had the latest version of DSM 7 on it. I was able to delete the homes share (after I turned off the Home service) on this box. I could NOT create a new homes share outright - the "homes" share name was reserved. But, when I turned the Home service back on, a new homes share was created. On this particular box, the volumes are not BTFRS, so Data Integrity Protection wasn't turned on, but I bet that on my current Synology box, it would be. So I may be brave and delete the homes share on my current Synology, re-enable Home service, and then copy the data back over to it. All of this is a huge pain in the butt, but I'm still WAY glad to have learned about this issue! Thanks again!

This is a great video! My follow up question is: What protects the RAID checksum? These checksum values are stored on the same drives. I know it’s far fetched, but if a bit in the RAID checksum gets corrupted for the file that also has a bit corrupted, that file cannot be recovered by Data Scrubbing. Do you have inside on how Synology protected the RAID checksums?

@xellaz

Ай бұрын

That's a very intriguing thought! I would assume maybe Synology will create an odd number of parity files and use something like a majority vote wins system to choose which is the correct parity file to use to fix the corrupted data. Still, there could be a chance these get corrupted too but that chance should be infinitely small and could further be mitigated by using ECC memory. Hopefully. 😅

@WunderTechTutorials

Ай бұрын

Great question! I honestly have no idea, but I'll see if I can find out!

Thanks for this great presentation ! I have 4 x bay NAS with two disks configured in RAID1. I have one volume configured as Btrfs and multiple shared folders with "Enable data checksum for advanced data integrity" checked. Data scrubbing is also scheduled. What I don't understand is why data scrubbing should not work if you don't have at least RAID5 configured (3 x disks). In my case with only two mirrored disks, if I understand correctly, data scrubbing is useless. I wonder why DSM present "Enable data checksum for advanced data integrity" as a viable option (not gray out) if my system does not have at least three drives. I understand the logic behind RAID5 parity so the only thing I can think of is that the checksum is calculated and stored if you have the option checked and data scrubbing will compared with the real time checksum value, if they don't match, you will receive a warning message, having a RAID5 will also repair the corrupted file. Anyway if you have corrupted files, I don't understand how a RAID5 will help you as the corrupted data is distributed over the three disks and the controller does not know if the data is corrupted or not, same for RAID1. RAID1 or 5 are not backup methods for corrupted files, it will help if a disc failed then data is reconstructed from the other disks using parity or mirror weather data is corrupted or not. Thank you !

@WunderTechTutorials

Ай бұрын

I would hope that Synology would notify you, but I don't know to be honest because I couldn't find any email alerts for it. I think they allow you to do it in case you have plans of expanding the pool later in time - it'll store the checksum at that time.

I think you missread the Synology page where it states "Data scrubbing is only supported on Btrfs volumes or storage pools of the following RAID types: SHR (consisting of three or more drives), RAID 5, RAID 6, or RAID F1." The key factor is the word OR, so you need Btrfs or RAID types: SHR (consisting of three or more drives), RAID 5, RAID 6, or RAID F1 You dont need both otherwise the word or will be and so its one or the other. Thats the way i read it.... I could be wrong. I run RAID 1 Btrfs with data intgrity and the data scrubbing runs no problem, I would think Synology would flag it in the DSM if it wasnt able to run?... I learn alot from you and the Seagull lover channel thank you for doing these videos.

@WunderTechTutorials

Ай бұрын

I could have definitely done a better job explaining the differences between data scrubbing and RAID scrubbing, but I was trying to keep it as simple as I could so I consolidated them both. When the data scrubbing process runs, it first performs the data scrubbing process using the Btrfs metadata (validating the checksum). This will work if you're using a Btrfs volume no matter how many drives you have. However, if it finds a problem, it must run a RAID scrubbing process to fix it - that's where the SHR + 3 drives, RAID 5, 6, or F1 comes in. You can run data scrubbing if you're using RAID 1, but you won't actually be able to repair any of the files, assuming the checksum doesn't match. Hope this clarifies it in case anyone else sees it. Thank you for watching!

as my current syset is build at the early dsm 6.0 timei don't thine we even have this option at the time eveni build it on btrfs it will be a huge pain to move 60tb+ of live data into a new share folder one thing I don't understand is when I check the note it comment that To ensure service quality, we recommend not to enabling data checksum if you are Hosting database / VM / storing Surveillance station recording etc does this mean it will create a lot of IO overhead and slow down the read and write of the system? if true how bad will this be?

Does bitrot happen on Solid State storage? I suspect that bitrot is due to deterioration of the strength of magnetic fields on a classic magnetic spinning media over time making the polarization of a data bit ambiguous.

@WunderTechTutorials

Ай бұрын

I think it can technically happen, but it's more likely to happen from a bad write (potentially due to memory) than from just sitting there. That's why Synology's switch to ECC memory (while losing things like hardware transcoding) is worth it for the majority of people.

would be good if the scrubbing have some kind of report output showing how much it fixed

Hi Frank , I have a ds923+ with 2x12 TB ironwolf. I’m around 20% max of storage usage … so basically I need to add an another 12TB so that the data scrubbing is working properly … it make sense since raid 1 is a mirror with no checksum I guess

@WunderTechTutorials

Ай бұрын

Correct! Thanks for watching!

@sylvainalain6637

22 күн бұрын

@@WunderTechTutorialsjust finished to migrate from raid 1 to raid 5. It took around 2 days from 2x12tb to 3x12 tb

Know this is late asking. Found all my folders this is not turned on. My NAS near 80%, so can I make a new folder with turned on and do a Move To? Or is it just only on Copy To? With different apps using different folders and such, I would like to make new folder move everything, then delete old remake it with turned on, and move everything back. Is that possible or do I need to figure a way to copy everything somewhere off NAS and then back.

@WunderTechTutorials

25 күн бұрын

I like to copy the data over, but if you have to, moving is fine. Just make sure you have a backup.

I thought data scrubbing also “refreshed” the 1’s and 0’s in the file by re-writing the file (exactly) over top of itself. This prevents bit rot because what looks like a ‘1’ can slowly degrade to a zero (or a bit flip). It refreshes the bit to a ‘1’ before it degrades too much. If that’s correct, then data scrubbing is still valuable even without checksums. I believe this type of bit rot happened to me. I had dozens of old photos get corrupted on an old hard drive - photos that I hadn’t looked at in years.

good info but have an issue w/the title...it's misleading ..click baiting >.> still good info for those that didn't get it

You say that 3 or more drives are needed for data checksum integrity operations? I have the 2 bay DS224+ NAS with BTRFS and I just checked my shared folders and this option is turned ON and greyed out so I cannot turn it off even if I wanted to.

@WunderTechTutorials

Ай бұрын

It can only be turned on when you create the shared folder. If it wasn't on when you did, you can't turn it on now.

@rafflesnh

Ай бұрын

@@WunderTechTutorials Thanks for the reply. Yes, I realise that, but my point was to do with the minimum number of required drives in the NAS before data scrubbing can be enabled AT ALL during the set-up process. e.g. @2:25 you state that SHR arrays (as well as other types of RAID setups) require a minimum of 3 drives. I have my 2 drive NAS setup with SHR, and yet I am able to run data scrubbing. What have I missed?

@WunderTechTutorials

Ай бұрын

You need 3 drives to repair files where the checksum doesn't match. So you can run data scrubbing with a 2 bay NAS, but it'll only identify problems, not repair them.

@rafflesnh

Ай бұрын

​@@WunderTechTutorials Ah ok, thanks for making that clear.

The need for at least 3 disks is new to me. I'm scrubbing at the moment with my 2 bay NAS. I've checked all the other boxes but apparently this is still useless. I'm wondering why Synology allows to enable data scrubbing if you disk configuration isn't sufficient.

@WunderTechTutorials

24 күн бұрын

It will still scrub through the data using the Btrfs portion (checking the checksum), it just can't repair it unfortunately. My hope is that it would notify you if it found something wrong.

I dunno why my comment does appear (maybe because it contains a link ?) . I found an interesting thread on reddit that explains that a scrubbing on a 2 disk shr volume does execute btrfs scrubbing (data checksum vérification) but not raid scrubbing (no fixing)

@WunderTechTutorials

Ай бұрын

That is correct - sorry, that's what I meant to imply but probably could have said it better. You can do data scrubbing, but you can't repair anything.

Is it better to run the data scrubbing every month ?

@WunderTechTutorials

22 күн бұрын

You can, but I wouldn't say it's necessary. I think that quarterly is a happy medium for all of it, but I know some people run it monthly so it's up to you!

Think of the surface of a drive platter. It’s nothing but ones and zeroes. What happens if the drive’s surface wears out to the point that it can no longer store data accurately? A one could be read as a zero, or vice versa. The checksum detects when this happens, and the file can be rebuilt from the parity information stored across the drives in the RAID. And this is why you need RAID 5 or RAID 6, rather than RAID 0 or RAID 1. RAID 0 and RAID 1 contain no parity data. If a file is corrupt on one drive of a RAID 1, it’s corrupt on the other drive as well.

So, set my backups with a very long term version retention period. Got it.

@jasonrowe6302

Ай бұрын

Your backups can also suffer bitrot.

@believeinheroes

Ай бұрын

@@jasonrowe6302 which is why I have multiple backups. I can also upgrade to punch cards.

@believeinheroes

24 күн бұрын

@@jasonrowe6302 cool I’ll try that.

Why minimum of 3 drives? Why wouldn't it work on 2 drives using SHR?

@WunderTechTutorials

14 күн бұрын

2 drives are a mirror and you need parity data for it to work, which is why 3 drives at minimum are required.

What about Raid 10 EXT4 ?

@WunderTechTutorials

Ай бұрын

No, that won't work.

I really don't get why RAID 1 wouldn't allow data scrubbing to do its job?!! it has a full copy of the file on the other disk of a two disk array, so why not?

@WunderTechTutorials

Ай бұрын

Since it's a mirror, if it's bad on one drive, it'll be bad on the other.

What about RAID 10?

@WunderTechTutorials

Ай бұрын

Sadly, since RAID 10 is basically RAID 1 + 0, no, it won't repair.

@EnriqueReyes

Ай бұрын

@@WunderTechTutorials But why does DSM allow one to Enable data checksum... for a shared folder on a RAID 10 storage pool? If data checksum does not work on RAID 10, DSM would not allow one to Enable data checksum for a shared folder on a RAID 10 storage pool, don't you think? ️ I asked the Large Language Model Llama3-8b-8192 the question "On my Synology NAS, does Data checksum for advanced data integrity work on a shared folder that is stored on a storage pool configured as RAID 10?" Answer below: "Regarding your specific question: if you have a storage pool configured as RAID 10 (also known as RAID 1+0), the answer is a bit more complex. RAID 10 is a combination of RAID 1 (mirroring) and RAID 0 (striping). This means that your data is written to multiple disks, and each disk has a duplicate copy of the data. This provides excellent redundancy and fault tolerance. In this scenario, the Data Checksum feature will still work as intended. The checksum will be calculated for each block of data written to the storage pool, and the system will detect any corruption. However, it's important to note that the RAID 10 configuration already provides redundancy and fault tolerance. If one disk fails, the system can still recover the data from the duplicate copy on another disk. In this case, the Data Checksum feature is more of a "double-check" to ensure data integrity, rather than a primary means of detecting corruption. In summary, Data Checksum will work on a shared folder stored on a RAID 10 storage pool, but the RAID 10 configuration already provides redundancy and fault tolerance."

@WunderTechTutorials

Ай бұрын

@@EnriqueReyes There are two steps in the data scrubbing process: the first validates the checksum, which is a function of the Btrfs volume. This can be done on RAID 10 because all it's doing is validating the checksum. If the checksum does NOT match, RAID scrubbing occurs which will take data from the parity drive(s) and fix the mismatched checksum. This is a function that requires SHR (3 drives), RAID 5, 6, or F1. Since RAID 10 is RAID 1 + 0, what exists on one drive, exists on the other, so if the checksum doesn't match on one drive, it won't match on the other either. There's no parity data to fix it using RAID scrubbing. So to answer your question, yes, your storage pool can validate the checksum does or does not match. No, it cannot repair the data if the checksum doesn't match.

@EnriqueReyes

Ай бұрын

@@WunderTechTutorials Got it 👍 So if one is running volumes on RAID 10, there is no need to enable checksum validation 👍 Thanks

@WunderTechTutorials

Ай бұрын

@@EnriqueReyes I didn't say that, lol. There's no information out there that I can find on this, but I am hoping that IF Synology finds a mismatch, they'll inform you. If they do and you know about it, you can check other device(s) to see if a copy exists without bit rot. Personally, I think you should run it for that reason, but if there's info out there saying that Synology DOESN'T inform you, then yes, waste of time.

So I purchased a two bay synology and I guess that was a mistake…

@WunderTechTutorials

Ай бұрын

Not a mistake necessarily, but something to be aware of at least.

Hello friend, please try to make tutorials which can be understandable for beginners and intermediate users, I first time heard about Data scrubbing. What is Data scrubbing?

@WunderTechTutorials

Ай бұрын

1:43

Damn. So owning a 2 bay nas with all things enabled for the last years was a lie. Synology is slowly becoming a papercut company.

@frankboyer1490

Ай бұрын

Learn how to properly configure your tech

So if I have a 5 bay NAS with just two drives set up in SHR, what happens when I add a third drive? I purposely bought two drives at first to space out costs and to hopefully save from having them all fail at once. I have had the NAS up about two months and I have data scrubbing set up along with checksum enabled on folders. Should I go ahead and add the third drive? Would the files already added benefit at all or just the new ones?

@WunderTechTutorials

Ай бұрын

I wouldn't rush to add it, but when you do add the third and the RAID array rebuilds, it'll function like RAID 5 (parity) rather than RAID 1 (mirror). As soon as the third is there, technically, data scrubbing will be able to fix data assuming it finds a problem. As long as the checksum is stored, it'll be used when data scrubbing runs whether there are two drives or three, it's just that 3 will be able to repair the file if there's an issue.

@ballsdeep9648

Ай бұрын

Thanks! Also, what happens if I have my data backed up from Synology to something like Backblaze and my Synology file gets bitrot? Can’t I just restore from Backblaze? Does Backblaze do anything for bitrot?

@WunderTechTutorials

Ай бұрын

@@ballsdeep9648 Hyper Backup should check the integrity of the backup regularly (normally once a week), so if everything is configured properly, yes, it should be able to fix Backblaze.