TELL ME YOUR STORAGE HORROR STORY! And use this link to get $25 off your PPA membership www.ppa.com/join/tony-chelsea-northrup/mar2022

@AlEbnereza

2 жыл бұрын

True story: Adobe Lightroom mobile early days, circa 2015ish they lost my entire online catalog (which was relatively small at that time, only around 800 raw files + edits). I was a paying member, was in between main desktop catalog machines and ADOBE had data loss with NO enterprise recovery or backup abilities at the time! All I got was a slew of uninformed tech support reps on the phone and a few very apologetic emails.

@Designsecrets

2 жыл бұрын

I've run a synology nas for years......might be time to double check it's doing what it should be doing :)

@cameraprepper7938

2 жыл бұрын

ALWAYS at least 3 different back-ups on 3 different types of harddrives/SSDs, NEVER use the same type of drives ! (a fault on a drive will be on ALL drives if it the same type !), so buy 3 different drives at 3 different times to be shure. ALWAYS check each back-up and NEVER delete/format the memory cards before you are sure you have 3 different back-ups that work well ! All drives must be offline when not in use ! When you do back-up, turn the internet OFF ! Do not use "NAS", if you do use 3 different types.

@SmallSpoonBrigade

2 жыл бұрын

A mirrored zfs array will automatically heal those corrupted files every time the file doesn't match the checksum.

@ericklauser2423

2 жыл бұрын

Just came across a picture today that had bitrot. Luckily the copy on my desktop was still in good shape.

I have been an IT person for over 20 years. I shot my first wedding for a coworker 7 years ago. I used a Canon 5D with 2 memory cards. After the wedding I backed up all my memory cards to my computer and then to an external hard drive. The next morning I left on vacation to go camping. I received a call from my neighbor at 2:00 am telling me that my house was on fire and the fire department was cutting a hole in my roof to let the smoke out. They busted out all my windows and threw all my computers and electronics out of the windows. When i got home my Mac was sitting on a pile of debris with water dripping out of it. My external hard drive and memory cards were in another soaking wet pile. I was devastated. I lost everything I owned except for the clothes I took on vacation. I found all the memory cards and my 2 hard drives while raking through the debris. I put them in a bag of rice for a week. I had to call my coworker and tell her that all of her wedding pictures were in the house during the fire. After a week i connected the hard drives to a USB device and amazingly I was able to get 100% of my data. Even the memory cards still worked. I learned a huge lesson that I should have been following, being an IT person for over 20 years! Get those files offsite ASAP. The 3-2-1 backup strategy simply states that you should have 3 copies of your data (your production data and 2 backup copies) on two different media with one copy off-site for disaster recovery.

@saganandroid4175

2 жыл бұрын

OMG. How did the fire start?

@TheKc2pxp

2 жыл бұрын

@@saganandroid4175 not sure. Undetermined.

Oh man have I been there done that! Replicated corrupted data is a nightmare. Offline solid state back ups plus data integrity checked Time Machine backups, plus cloud galleries that are unsynced and static. Long term data management is a deep rabbit hole. I so feel your pain TonyTone!

@kimmortensen9779

2 жыл бұрын

Think you mean offline as in not internet connected? Having the the backup air gaped (no internet access) is a good idea. But if you leave a SSD unplugged for long enough the taped electrons that makes a "1" in solid-state storage will leak over time and flipping that bit to a "0". How long that takes varies wildly. and i would guess older SSDs are likely to suffer from this than newer ones. HDD, DVD / blu-Ray have a not so good "shelf life". The reason why tape-drives are still a thing for on the shelf data storage.

Once a year , I go through all my kids pictures on my drives and make an old fashioned hard copy photo book and have backed up pictures on different drives - Thanks for this video lost memories sucks - I am in my 50’s and getting to the age I think more about memories and not the things we have acquired - When we are on our death bed I want to remember those moments while having headphones listening to music - life is all about memories with your family friends and moments -

This is such a good summing up of the situation. I have made 5 family albums but discovered some corrupted images in the process, which is incredibly sad. I laid out the albums in Affinity Publisher, exported the high res PDFs as well as email quality, and distributed these to family who can decide to print from the high res versions if they want. But after watching this, I realise even more the need to back up twice, in different locations. Thank you Tony!

In my early days of selling computer stuff, I heard and remembered a quote "There's data you've backed up, and data you haven't lost yet". I try to backup my photos on my desktop, laptop, wife's laptop, Google Photos, and photobooks I've created and had printed. Only saving grace in my case is my total storage is under 1TB total. Oh, and when I worked at an actual office vs home, I kept a backup drive there too. Anyway, great video. Thanks very much. As for a (lame sauce) horror story, I lost the photos I'd taken for my brother-in-law's wedding a few years after the event, and it was maybe another decade later my brother-in-law found the CD backup I'd made for him at the time and I was able to get the photos back.

Use a NAS that uses ZFS as a file system, since zfs has built-in bit rot and data integrity features. We build large enterprise storage SANs with sensitive data, all built on ZFS. In 18+ years haven’t had a single corruption issue with high io storage devices. I don’t remember the last time I had to do data recovery.

@SmallSpoonBrigade

2 жыл бұрын

Yes, but make sure to use a mirror or similar as well as an online backup. A mirror allows for healing the rot.

@eivinha

2 жыл бұрын

Modern filesystems that protect against bit rot and data integrity are definitely the way to go. I'm building a new setup with two NAS-es now (haven't had any data loss in 20 years, but trying to beat my bad luck here). Primary NAS will be TrueNAS Core running ZFS. Secondary NAS will be Synology running BTRFS. I also have backup in the cloud for everything except video (too expensive and slow to backup and restore). I'll also use snapshots at least for TrueNAS/ZFS, and I'm hoping to do this with BTRFS on Synology too. Snapshots every hour for a day, every day for two weeks, etc. Even so, I'm still constantly worried about bit rot or crypto virus. I'd like to incorporate some form of checking integrity automatically, perhaps using hashing, but I'm not really sure about the best way to go about this...

@eivinha

2 жыл бұрын

Maybe if there were good ways to declare parts of backups immutable, in order to prevent them from being overwritten...?

@PeterBrockie

2 жыл бұрын

Specifically make sure ZFS scrubbing is setup to repair the rot. Just having RAIDZ, etc. is not enough as it won't actively check for errors until something is accessed. Linus Tech Tips just had this exact issue when they weren't actively checking their old data.

@niduroki

2 жыл бұрын

Tony said, that he's using Btrfs for his backup system, though?

Had a programming teacher who said, “Blessed are those who make backups of backup”. I would always have a running backup and one stored in the bank vault. Macs use a concurrent backup. But to be safe, have a separate folder the Mac never uses. So, basically you have a concurrent backup on the backup drive and a separate folder that contains the photos. (X2). Much harder if you take thousands of photos weekly but it that’s your income, then invest. Memory is cheap cheap cheap compared to 20 years ago.

it is quite remarkable how often your photos get corrupted, especially on external drives. I had corrupted files very soon after copying them on the drive. Once I even discovered that whole directories just disappeared

I back up only new files or files where the date or size has changed. That means an error inside the file will not be duplicated to the backups.

This is why I got into Freenas (now truenas) which uses ZFS which uses scrubbing to protect against bit rot.

Great video, Tony - thank you very much! So sorry for your loss (we felt your pain). In a seminar years ago, an IT expert advised about backing up digital media - get in the habit (at least annually) of migrating your data to the next-best most secure storage available and have an off-site location. ChronoSync software really helps as you syncronize your drives. Your recommended use of battery backup power conditioners is spot on. I use 5 (from APC). Electrical spikes and sags can wreak havoc on sensitive micro electronics. The videos you and Chelsea produce are great!

One of the main reasons I'm using ZFS for storage at the moment, with scrubs every other month to check everything for corruption and to correct it if needed. With spinning disks I've found that basically since around the time they introduced shingled magnetic recording reliability has gone downhill (even on non SMR disks). I have old 1-2 TB drives that have been running for years and return less corrupted data than brand new drives of similar size. Using a filesystem that can recover corruption combined with regular scrubbing of the disks, and to basically not sync files older than a few weeks from your PC (that probably uses a filesystem that won't catch corruption) to the storage server anymore really helps prevent issues for me. Also I don't know the current state, but iirc btrfs did have issues with raid5 or 6 type usage when I last looked into it allowing data to get corrupted. Also, don't have the backup server "mounted" on your system as network drive or network source by default, set the server up to initiate the connection to your PC to get the data. That really reduces the chances of ransomware getting to the data

@esenel92

2 жыл бұрын

@@TDCIYB77 It kinda makes me wonder why Synology would even want you to use BTRFS for archiving. (I don't know their products from experience, but I assume they will have proper documentation for filesystem choices). Ive been eyeing BTRFS for ages as replacement for my own home server/NAS, specially because it allows you to spread data properly over existing striped disks when you add another one. something ZFS doesn't allow. (last time I checked anyway). But iirc it was even on their own wiki that RAID-5/6 like setups were called "experimental" (or something along those lines). So NAS brands allowing usage of it without clear warnings about hat is kinda shocking to me.

You might want to try reading your images with different software packages. I'm a software engineer who used to write low-level JPEG code long ago. Some JPEG software will just abort if a single bit is wrong, and either the whole image or all the image past the corruption will be lost. For many corruptions, JPEG software CAN be written in a way to recover and decode the rest of the image. You will end up with a little defect in the image, like a 8x8 block (or streak to the end of the the line) of garbage pixels, which can often be fixed up in Photoshop. JPEG also often stores the chroma separately from the luminance data, so serious corruption of the chroma data may still allow recovery of a black and white image. Corruption of JPEG files is not a binary thing, there are many gradations of how damaged is it.

@CaseyConnor

2 жыл бұрын

I had the same thought... there are software packages to attempt repairs of corrupted image files. Not sure if any of them get into raw formats, but it's definitely worth a look.

@matrixate

2 жыл бұрын

That was my point to ThioJoe and Tony. He needs software that modifies the arm mechanics to read the bits of the file backward and forward to get "around" the bad spot and do what you propose. So, yes...this can work because I've been successful in doing it.

@cookwal

2 жыл бұрын

Does a single bit of bit-rot flip the Archive bit on ? Then he should have noticed that several very old images suddenly needed to be backup and have stopped him from doing so. The software I use give me a preview of what the scripts I have setup will do , color coded if overwriting/copying /Creating new etc. This has stopped me from making mistakes many times.

@8khdrsoothingrelaxation276

2 жыл бұрын

Thanks for your comment. Is there any way to save many .MOV files in a way to recover and decode the rest of the video in case of bitrot in it? Or, for videos bitrots are not so dangerous as for pictures? Any input from anybody will be greatly appreciated!

Thanks for the video Tony! But I think you did not tell us how someone can prevent corrupted stuff from being backed up and also corrupting the backups.

@jmontgomery7394

2 жыл бұрын

Agree ... with all of the backup processes and equipment you have in place ... how did this happen? We could all really use a more detailed assessment. Just got thru doing archive scans of 3000+ slides and tossed the originals. Do agree about printing ... printing ... printing

Great info, as always, Tony. I think all of us - pro or enthusiast, must find the best way (that suits the individual) to address backup, recovery, and loss. I don't think there's any one way that's the correct way. Each photographer should find what works best -- from a perspective of value, cost, time, and importance. I'm not a fan of cloud storage -- but I'm only an enthusiast. For me, it's simple -- I don't store any photos on my desktop (or laptop) PC. After every shoot, I copy the contents of the memory cards to an external drive (connected to my desktop). Note - If I'm in the field for an extended shoot, I bring an external drive and copy the contents of the days shoot to that external drive, connected to my laptop, each evening. The external drive is backed up to my local Synology DS 920+, and a make a backup of the external to a 2nd external (then store the 2nd external offsite). At this point I have 3 sources of the photos (not counting the memory cards). I don't format the memory cards until I've confirmed the files on the Synology and the 2 external drives. I do all Lr/Ps post processing from the desktop-attached external drive -- not the fastest method, but then again, I'm not a fast editor. And, I replace the external drives every 3 years. Is it perfect? Nope - not even close. Do I sleep well? As well as one can, I suppose. I end up with my complete NAS backup - original and edited files; another copy of the edited and original files on the desktop-connected external drive; and a copy of just the original raw files on the non-connected external drive. For me - this sysem works. Like most of us, I don't go back and view the hundreds of thousands of old images on my drives, with any regularity. I just don't have the time. But I hope that keeping my external drives fresh, on a 3 year rotation minimizes the chances of storing corrupted files. Thanks again for the insight, detail, and passion. Truly appreciated.

Use your pictures! That’s the best way to keep our valuable memories! Printed copy is one of the best way to keep some valuable memories! Sometimes the old methods helps!

I'd like to start doing annual photo books this year to counteract this. Even with triple back-up updated regularly, I have no trust in digital storage anymore, and physical media is much more gratifying to go through. I used to do individual prints of my best photos, but I have so many envelopes of prints that photo books seem like a better option in the future. No one will go through your hard drives when you die. They'll be happy to leaf through a photo book, though.

@GarretGrayCamera

2 жыл бұрын

That is true. You can even print a few and give them to friends and family in the event you lose yours.

@georgedavall9449

2 жыл бұрын

Excellent comment Christian! And like my reply to Harry, will there be available, devices to be able to read today’s digital media, by tomorrow’s generation?

@TheChadOlson

2 жыл бұрын

This is a great idea, and one I definitely subscribe to. Love having 'coffee table' books people will look at, that I can also put other info in, like family tree or history/stories. And I can revise and get it reprinted, and give away the old copy.

@KyleWilson

2 жыл бұрын

" *No one will go through your hard drives when you die. They'll be happy to leaf through a photo book, though* " OH MY GOSH, so true!

Sorry to hear, like you its just happened to me. 20 years of work and memories. I will now place in multi locations. thanks for a great video

In some ways, that is where old fashioned film photography is superior to digital. I still have a very thick album full of negatives from 30 years ago still intact. Nothing beats making prints, and keep them somewhere safe. Favorite past time might just be having to look for that one photo, only to be flooded by the many memories from THAT album you look through every now and then.

Funny that you made this video after I had an issue with memory cards. Fortunately I didn't lose any pictures. My issue was that I started out shooting on a Canon APS-C camera and of course those only have 1 memory card slot. Last year I bought an R6. Of course those have 2 card slots. I use 2 Sony Tough cards in it. Last Saturday I was shooting and my camera stopped taking pictures. I thought one of the card became corrupt. I was smart enough to have listened to you Tony about keeping more cards handy. So I swapped the cards out and kept shooting. I got home and checked the cards. Come to find out my second Tough card was full! I had never deleted the old photos from it because I didn't realize I had to switch between cards to delete everything. Luckily I have everything backed up on an external HDD and 2 cloud serviced. So no more lost pictures. Thank you Tony and Chelsea for drilling the 'carrying the extra cards' into me all these years!!

I don't know enough about jpeg compression to really understand, but since it looks like the corruption on the images "starts" some ways down the images, I'd be really curious if there was some way to go into the image and manually start flipping bits in the data associated with the specific pixel it "starts" at (assuming that can be identified), and see if it can be fixed. I'd also be curious if raw files, presumably being uncompressed, would be affected the same way, or if a flipped bit would instead just appear as a messed up single pixel. (And if so, maybe compressed raw would be affected though)

@matrixate

2 жыл бұрын

No, that's not what's happening here. Jpegs use transformation matrices and coordinate transformations. So, if you lose a bit of that coordinate matrix, you're screwed...sort of. What's happening here is that the magnetic strength of the stored data is not readable using the conventional mechanics of the arm in the HDD. He needs to use the software that I used to recover my files. That software will cause the drive to read files backward and sort of simulating what you were hinting at but not quite. It should work on most of the files. There may be a couple that are really that bad where nothing will work but the software won't freeze and continue trying to recover every file.

@geraldmcmullon2465

2 жыл бұрын

Some JPEG recovery programs can get most of the file back. I have only seen that damage on JPEG files for which there was no RAW copy but I had analogue originals.

@PaganiTypeR

2 жыл бұрын

Not a single file is safe from what tony is saying. Pretty much everything gets corrupted over time. Same with RAW files. At the end of the day it's a matter of luck. If in jpeg a byte is changed in specific area it could potentially ruin whole image or just a single pixel, really depends on where that is going to happen. But where it's going to happen is unpredictable. The HDD's looses magnetism and SSD's looses charge so not a single storage is safe over time. Best thing to do is probably cloud storage. Also Tony has mentioned making videos with slides of images which is also a pretty good idea since to corrupt a video file it would require much more effort than with JPEG's or RAW files. In general video files are much more robust to glitches. Even if glitch would happen during the video it would still continue to play and recover at some part where the image is still good.

@HR-wd6cw

2 жыл бұрын

His is examples use JPEGs but this can also happen to RAWs and most other files. JPEGs were probably used for this example as most apps can display them and show the impacts (bitrot, or corruption) he was trying to show. But all files of any type are succeptible to this.

@chemicalburn

2 жыл бұрын

@@PaganiTypeR what about using .par files when making backups? I remember those from the old Usenet days. Random parts of the file would get corrupted, so redundant information was sent so that the file could be repaired.

Wise words from a wise man. Thank you for all the videos you've made over the years, I just started about a year ago with the DSLR and can't imagine how difficult it would have been for me to piece together all the information that you put in your videos. Thank you.

Also IT background. I've been running a ZFS storage array with two drive redundancy (RAIDZ2) for years now and it has been rock solid. I run pool scrubbing monthly and it has from time to time detected and corrected bit rot. One of the Amazon services I've been taking advantage of is free photo storage for Amazon Prime customers. This includes RAW photos too. My workflow works something along the lines of: Import from SD to laptop. The Amazon Photos app runs in background monitoring my pictures folder. I also sync to my ZFS NAS. On another system, I have the Amazon Photos app download to another system. I think run a hash calculation on both the NAS and the downloaded photos. When cleaning an SD card, I hash those files and if that hash exists on BOTH Amazon and NAS I'll delete just that file.

You've convinced me to move a few hundred GB of photos onto my one drive 365 subscription considering I pay for it every year, So far I've only had a handful of individually corrupt images over the years

Computer Information Systems degree, but professional cinematographer and wedding cinematographer here. This is so key, personal and business. My personal/business setup includes a QNAP 72TB ZFS NAS system with a UPS as well. As a business owner capturing footage that cannot be re-shot, a data backup is probably the most important aspect of your business. There is way too much on the line. The QNAP has Hybridbackupsync that allows you to back up footage to multiple online providers like Google Drive, Backblaze, and Onedrive. Once the footage has been shot, I upload it immediately to a Google Drive Workspace account. $12 per month per user and unlimited data. That way, if anything happens to the server locally, that data is up on Google's servers and can be re-downloaded at any time. From a personal perspective, it is true that we should be printing our photos and making albums.

Under Windows I use the system tool RoboCopy to copy my images from one hard drive to a different one. That does not copy unchanged files, so it does not "copy corruption". Corrupted files usually have same size and time stamp so they are not detected as new.

Film photographers also would send half the film from one of the two cameras to be processed, wait for the results before sending the remainder of film. Also one of my instructors would process one sheet of 4x5 film at the beginning of his shoot before continuing to shoot.

@CramerGallimore

2 жыл бұрын

Did this always for commercial shoots.

@roccop33437

2 жыл бұрын

I used to send my film from 2 different cameras to 2 different film processing houses.

Do it people ! I lost 15 yrs of work due to the Australian floods just recently . I had multiple backups on various HDDs stored in separate locations . Record flooding killed that plan . Even the drives stored at a previously flood safe location ended up under 2 metres of filthy , muddy water for a few days , along with my PC and laptop and half of my camera kit . Luckily I had recently started to use a cloud storage service , but sadly I had only backed up a fraction of my files when disaster struck . Do not hesitate , protect your work .

I was the quality manager for a large computer corporation and also a photographer and I can't stress your advice more. One of my jobs was projecting failure rates for new products so replacement parts were available in the field as well as helping our marketing team. 2 terms misunderstood are both identified as MTBF (Mean Time Before Failure and Mean Time Between Failure). ALL devices will fail eventually. MTBF was the point where 60% of the field population failed (before first failure) and the point where the mean time expected on a particular device (between failures). I keep my digitized photos from the 60's (film) and my newer digital camera files on USB drives that do not copy from each other as well as 2 cloud repositories. I also have the original hard copies should I need them in the event of a disaster. I have also been keeping files on Amazon photo, which is free and unlimited for prime members (at least so far). They even support the RAW file format and limited amount of movies.

Some warnings and tips: with NAS drives the issue is mechanical failure of the NAS device itself especially the power supply, and this may cause two problems: if your NAS is discontinued parts may not be available and with some NAS manufacturers there’s no backwards compatibility so you can’t just put your drives into a new NAS even with the same manufacturer. (I’ve had this happen with Western Digital). Tip1: to avoid the above creating your own PC-NAS is the way to go for the technically minded. Tip2: tag your most valuable photos which will cull the number to a manageable size and back them up to dual copies of M-Discs to be kept in separate locations. These discs are rated to last for 100s of years. The only issue is ensuring the disc technology remains current if ever a restore becomes necessary.

@Epic501

2 жыл бұрын

Backing important things up to a wholly separate kind of format is an important point left out of this video. Archival tape or disk as you mentioned. Also it's becoming more affordable these days to fully stock NAS' with flash storage which is a lot more feasible long term

@saganandroid4175

2 жыл бұрын

The only HDs I ever had fail on me were WD.

I'm old so I can speak from personal experience to back in the film days when we used to have two photographers to cover any wedding giving us multiple views but also two rolls of film from two different cameras cause not only were there film process failures but camera failures on site happened as well. We had a rule of backups of equipment in extra flashes and camera bodies for each photographer and backups for film with multiple camera backs, backups were always needed and are needed now more than ever. Oh and also one other point. If we lost a roll of film due to processing we only lost 15 frames (120 format film) out of the hundreds that were shot on a full day, not at all like the unimaginable risk of a single 256mb flash memory card with the complete day on it.

I'm sorry for what happened Tony. This is why i always PRINT the photo that matter the most to me

@ronaldmcyr156

2 жыл бұрын

I agree with you that prints will outlast any digital storage device (and the equipment to read it). I can still read 300-yr old books in my library, yet no longer have floppy-disc readers from the 1980s. During the early 1970s, I took a lot of Kodachrome prints; despite being stored in boxes, the colors had faded badly 30 years later. Looking at the options available today, it seems that pigment inks on cotton rag is the most permanent (and expensive) way to produce long-lasting prints.

Oh damn I was thinking about my backups and boom your video pops up!

@saganandroid4175

2 жыл бұрын

Same here!

For real Tony?? OMG, that's heartbreaking.. backups of backups of corrupted photos.. what a NIGHTMARE!

So sorry for this event! Thank you for sharing with everyone!

Hi Tony: I had an external HD become corrupt this past summer and because I back-up each night to a second locate drive it also became corrupt. As a result I couldn't access my LrC Catalogue or images. I never let my HD capacity go over 80% full so this wasn't the issue. We had some power outages over the summer and I think that these events contributed to my issues. Fortunately, I also store all of my images to Back Blaze with the 1-year version option selected so I had them send me a new HD with my 235,000 images on it. I dodged a bullet but I am still very careful when I look at them back-up strategy. Great PSA video. Everyone should be mindful of this topic. It is not a matter of if my HD fails, it is a matter of when. Cheers, Keith

Just watching the Man of Steel. After his mother’s house was destroyed, she said to Superman, it’s only stuff. It can be replaced. The great news is you still have your family. They can’t be replaced. I could tell you were totally pissed but it’s not the worst thing. Time enough for many more years of great memories!

THIS JUST HAPPENED TO ME YESTERDAY. I bought Dropbox 2 days ago to start double backing up files. I then tried to save all my work from the past year and my flash drive wasn’t loading. I took it to Best Buy and was told there was no way to access it. I immediately went to the bathroom and cried.

Thanks for the reminder and sorry abut the lost files. I set my Lumix G9 to record to both SD slots and save the files to two external HDDs attached to a desktop PC. Cloud storage would be a good tertiary option, but I could also periodically save to a third external HDD that I stored in another location (I'm not dealing with quite the volume you are). Years ago my travelling companion and I had all of our camera gear and over a dozen rolls of exposed but undeveloped film from a trip to Utah and Colorado stolen (apparently by a parking valet at a high-end hotel in Denver, but the issue was never resolved). It still bothers me so I can imagine how frustrated you feel, but at least we have the memories (but of course we still want the images)!

Easily the most useful video I’ve ever seen… didn’t know Bit Rot was even a thing.

Tony's last advice is my biggest takeaway. I'm not a professional but an enthusiast/hobbyist. I have external HDD's/SSD's where I store my pictures but only for a year. Every year I print a photo album with all of my favorite photos and hold on to the pictures on my storage devices for 6 months. After that, the old pictures in storage are expendable. I do this because I can't afford cloud/server or NAS based storage rigs like Tony's.

@SmallSpoonBrigade

2 жыл бұрын

You must be shooting a ton. A 4tb mirrored Pi ZFS raid can be had for maybe $400 and most of that is for the drives themselves. That gets you a ton of storage space, certainly enough to hold several years worth of images, and if you pare back the subpar ones, probably many years worth. In other words, this doesn't really make much sense. Even if you just delete the raw images and keep the JPGs, you'd have to be shooting hundreds of thousands of photos to have an issue with storage space.

I haven't had a hard drive failure in quite a while (knocking furiously on my wooden desktop). I have had memory cards fail, but not with anything on them (again - knock wood). They just suddenly couldn't be recognized by the equipment or computer. I do make redundant backups of my memory cards as part of my work flow - a back up to external USB drives, a backup in the cloud, and two copies on my working drive in separate folders and working only from one of those folders. Memory cards are too expensive for me to keep around, so the handful that I have get reformatted between projects once all the back ups are done.

Interesting how the digital age has made things like this more complicated and costly than in the analog age. Great benefits with digital, to be sure; I would not be using a camera now but for today's digital options. This is a worthwhile presentation, Tony. Thanks so much for creating and sharing it.

@veganpotterthevegan

2 жыл бұрын

Its less costly and less complicated. Stored, processed film can go bad too. And making film backups of film is far more expensive than making digital backups..

@AliasJimWirth

2 жыл бұрын

@@veganpotterthevegan It is mostly a moot point for me. The question is, do we live in the past, or do we live in the present...the present being the only time we actually have to live in? There is enough happening now to occupy our full attention everyday. I don't really agree with the cost and complication thing though. I think it best to print your best work onto archival media, display it, and enjoy it today and each day...while you can.

@veganpotterthevegan

2 жыл бұрын

@@AliasJimWirth We live how we want, it's person to person... odd to draw that comment from your own original comment though🙃 As for printing and displaying, a lot of us take far too many photos to do that. Or can't afford enough frames to do that. *Some people have piss poor memory. Some were too young to remember really old occurences, and sometimes other people that weren't there would like to see those images. I'd love to see photos from my mother's upbringing. But she was far too poor(in the Philippines) and there are less than 10 photos of her before she got out of her teens. My dad has one photo of his parents, they all burned in a fire while he was in college and his parents were already dead by then.

@AliasJimWirth

2 жыл бұрын

@@veganpotterthevegan In this life and in this world there are and will be many tragedies and sad moments and experiences. I am sorry for your losses and the general losses of your family. Not to make slight of that, I would point out that the time we have here on this earth, at best, is short of one hundred years. A more significant concern is the question of where each of us will spend eternity, which is not merely one hundred years, but forever. With that in mind, our photos and mementos, have little real and enduring value. Will we remember this life in eternity...I think not.

@veganpotterthevegan

2 жыл бұрын

@@AliasJimWirth There's definitely no life after death which is why it's important to do what you want(so long as you're not hurting anyone). Many people want to look into the past. Nothing is wrong with that. That said, looking at printed photos you hung on a wall is living in the past too.

So sorry to hear, heartbreaking. I use M-Discs as my ultimate archives. Future plans are for a zfs based NAS.

@-freespirit-3314

2 жыл бұрын

M-Discs are the best stable data backup solution and it’s cheap.

Excellent video that takes the world of IT and crosses it with the wonderful world of photography. I currently use the Adobe cloud to backup pictures and Acronis cloud to backup machines along with my own Synology NAS which has pictures. I don't have anything close to what you have in terms of TBs, but even 1 photo is worth a lot.

Years ago I set up a fancy computer for a client (a physician). I created an external backup with automatic backups, but he insisted it will only slow down the computer, and elected for Manual backup. A few years passed and he called to tell me that the computer was exposed to some power failure, and the hard drive is dead. "No problem," I said. I'll be over and restore all your files from the external backup to the new computer you just bought. I showed up and the last backup was the day I ran it for him, a few years back. The drive that failed contained all the pictures of their teen twin sons who died in a car crash a few weeks earlier. There were thousands of images his wife took, believing he is backing the data up as I showed them both years ago. We sent the damaged drive to California, to a special lab in an attempt to save the data from the drive. It didn't work. The wife walked away and divorced him. Edit: I use BackBlaze on all my computers, aside from DropBox sync, and another local backup. 35 years, never had an SD Card failure, but I do agree with what you said: Keep the files on the card for two more weeks, to make sure the files were backed up to the online service. Always check the online files and randomly restore a file to check quality. Thank you for another great and practical video.

What a nightmare. I'm so sorry this happened to you guys. I use a similar backup system to yours: multiple Synology DiskStations, BTRFS, SHR-2, multiple locations. I don't think most people appreciate the importance of using enterprise-grade drives like the WD (HGST) UltraStar models. The error rate on them is exponentially lower than consumer drives. Instead of the expansion units, I use three DS1821+ units which I regularly sync to each other using GoodSync with checksums to ensure data integrity. I physically transport one of the DS1821+ units back and forth to a second location to sync with a third DS1821+ to keep the units air-gapped. Whenever I replace the drives in a DiskStation, I take the old drives and copy data onto them with the intention that they will never be touched again except in case of emergency. If I had unlimited funds, I'd set up snapshot replication...but I don't. If I dealt with as much data as you, I'd also look at LTO tape. The tapes are surprisingly archival. The drives are expensive but the tapes aren't bad. They're ransomware proof too.

I came out OK, but despite living in the burbs, out of flood, earthquake, and forest fire zones an urban wildfire blew up here (Louisville, CO) at the end of 2021 forcing me to evacuate with about 30 minutes notice. I grabbed two backup drives (and a bunch of gear), but missed another drive and all my printed photo albums. I was looking into cloud storage when Tony posted this, good timing. Upshot: Backup, but have a plan and checklist for what to grab if you have to leave at a moment's notice.

Lost all my son's grad pictures from Simon Fraser University in BC. A suggestion for videos like this which is basically aimed at pros. You mention the one thing I now advise my non-photography friends to do 1) use the picture books offered by Facebook 2) use other picture book services. Make a picture book at the very least every five years for those photos that are important to you. I would suggest your suggestions for the non-pro's appear earlier in the video before you nerd out on equipment. I almost went to sleep when you were showing all your cool gear, I just hung in there to hear the "cheap" suggestions. I bet most non-pro photographers won't make it to the end of this video. But suggestions for non-pro photographers earlier in your video and totally nerd out at the end for your pro subscribers.

Having a backup of our photos is absolutely a necessity. If you love your photos, you should have a backup of backups. Losing data can be destructive to our passion so just do your best to not let that happen ;(

I once lost a chunk of photos, and I worry about this. Currently, I keep all my RAW photos on an external drive, and have a copy on a 2nd external drive. Also, I have JPEGs of all my photos on Smugmug. It is hard to have a perfect solution, and most people aren't IT professionals who know how to set up a complicated RAID drive with NAS. I think online (cloud) storage is the way to go. In a sense you are then letting an IT company worry about maintaining drives and backups in different locations, and they have the resources to do that. But my solution for myself is based on my being a hobbyist; professionals have different needs.

@SmallSpoonBrigade

2 жыл бұрын

I wouldn't assume that online services are going to be careful with the files. They don't own them and probably only have to refund your money if they're lost. But, the online service copy can be one of the 3 copies of the files that you've got. If you add a proper online backup company and a backup to an external drive, you're mostly fine. But, you do still have to go through regulalry and verify their integrity.

Thanks Tony for the wake-up call. I had never heard of bit rot.

A great video. I have 3 independent back ups. I have back up drives connected to my computer all the time. Soon as I get back from a photo shoot I put everything on the external back up drives. I also every few weeks back up those drives to some more drives which are only connected to the computer during back up. I have a bank safe so 1set of drives goes into the safe, another set I have at home for easy access in case of a natural disaster or fire so can grab them easily, if I am here at the time it happens that is. I never store any photos on my computer's they are all on external drives,, not connected to the computer except the one used to initially put the photos on from the cameras memory card. So even if my computer was hacked, broken or even destroyed I would not lose my photos. I also every year or two buy new drives to replace the oldest ones for more peace of mind. I also don't use cloud services as anything connected to the internet has potential to be stolen, no matter how secure they are supposed to be. If you photos are that important do not ever use cloud services.

I'm surprised nobody mentioned optical storage. I have 20 years of backups on CD's and DVD's. At least your relatives could find the pictures if something happened to you. I recently went through them and made sure that I have all the pictures copied to the cloud and on external hard drives. I think of it as the rule of three. Keep in at least three places.

@genius1a

2 жыл бұрын

I almost forgot about that, because regular Backup solutions seem to be so secure these days. All of my quality ancient CDs still work to this day! And, yes I also found out that writing a backup over a backup via a strange cryptic "backup Program" is nothing compared to a simple mirror hard drive of all my data done from time to time. For the short term. Not for long term, because all of my ancient hard drives have stopped working. They obviously loose the magnetic information if not used at least once every few years. So CDs and maybe magnetic tape seem to be the ultimate backup solutions. A data storage center can go bankrupt. I absolutely agree to Tony in that point! We should rally think more about saving our precious pictures at least once a year.

@WesternAustraliaNowAndThen

2 жыл бұрын

CDs and DVDs will also degrade over time. We have had it happen to discs that were carefully stored in individual sleeves in temperature controlled purpose built cabinets. Some disks are rated higher than others for long term storage but eventually they are all susceptible to data rot.

@Yankeededandy62

2 жыл бұрын

@@WesternAustraliaNowAndThen CDs, DVDs and BluRay Discs can last decades. The quality of the disc itself is of course a factor, but I can tell you from being in that business since the mid 90s, that the handling and way you store discs is the key to how long they will live. Don't put them in a sleeve. Anything in permanent contact with the surface of a disc will provoke a chemical reaction sooner or later. A classic Jewel Case is still the best way. Avoid fingerprints at all cost. Don't scratch them. And the most important tipp of all: Always burn 2 discs at the same time. Chances are high, at least one of those copies will survive. I have CD-R here that are 23 years old and work perfectly. If you want to be really safe, copy those archived discs every 5 years or so and you'll be fine.

@genius1a

2 жыл бұрын

@@Yankeededandy62 Wow, two CD copies (and maybe of a different brand) are a really good advice! That's something everybody can do at home easily. External burning drives and writable medis are still available and seem to be a really good way of preserving a limited amount of data over decades. For all mass data producers I would recommend magnetic tape drives, they are now in the Terabyte range (per cartridge!), they are affordable and last forever. Think of old music casettes. Ever had one not playing anymore because of magentic loss?

@geraldmcmullon2465

2 жыл бұрын

My personal store of videos, photos and documents is around 2Tb. 800Mb CD or 4Gb DVD is going to be some stack. 50Gb Blu-ray is only a little better. CD, DVD in homes is now uncommon. I don't know anyone with a Blu-ray drive or player. They dumped CD and DVD to streaming and also "lost" all their old files.

This is why I really like the Flac format for music. It has built in bit rot detection, tools can automatically scan flac files for corruption.

I use LTO 4 drives as a backup only 700Gb per tape but magnetic tape is very robust and cheap. All of my most important data is on at least 2 tapes. I think LTO 7 is up to 6Tb per tape. I come from a professional broadcast background where tape backup is very commonplace. While tape backup is very slow I do like being able to put tapes in a box and put them in another location.

@TomHofmann

2 жыл бұрын

Not 100% fail save. We had a full set of corrupt tapes a few years ago with a large database index on them. We sent them off to a data recover service but no luck. In the end we had to rebuild the index which took a few weeks. Of course these days I'd run this in AWS or Azure and use one of their multitudes of online storage solutions for cheap backup.

Video is a bit of a problem to back up because of the space it takes up. For most amateur photographers though, photos can be backed up to a number of different sources for free. You might not be able to store shots at the highest quality, but platforms like Facebook, Pinterest and Instagram have virtually no limit to the number of pictures you can store. As a backup of last resort, these platforms are very valuable. Evan as amateurs we still run several backup drives. We almost got caught a while ago when our work drive failed and when I went to make a new copy from the backup drive, it failed too. Thankfully it was under warranty from Seagate and they flew our drive to their recovery facility in the Netherlands and recovered ALL of the irreplaceable footage on the backup free of charge. It was ALL of the footage we have taken of Western Australia over decades and would have been soul destroying to lose. A couple of things I would add are, 1. NEVER try to recover damaged drives or files yourself. You will almost certainly do more damage and make otherwise recoverable data completely unrecoverable. 2. Backup irreplaceable files at least TWICE. One backup is never enough. Keep one backup off site. I carry one drive where ever I go, so if there was a catastrophe at home, I still have a copy of all the important files with me. KZread is another free option for storing video footage. You can upload to your account and keep the files private or if you just want to share them with friends you can make them unlisted and just give the links to people you want to share them with. Paying to back stuff up is essential if you are a professional but for most others, there are many free services available that you can use to keep copies of your files. Oh and one more REALLY important thing to remember, Don't just back your files up and then put them in storage for months, or years at a time. Drives maintain themselves best when they are powered up and just left to run for a few hours. The data gets refreshed by the OS and there is less chance of corruption. If you just unplug a drive and store it for a very long time, chances are that the data on it will be corrupt the next time you power it up.

I finally found the fix This is the excat same way my photos got corrupted. The reason - 1 if you use an sd in in both an apsc and a full frame camera it will corrupt soon. Some SD cards can't handle both smaller and bigger files at the same time. If u use multiple SD cards it will corrupt soon. So if you use 1 perticular card in one perticular slot of one perticular camera - keep doing that and don't change. 2 don't use 1 sd in a different camera system and back. That will also lead to corruption 3 format in the camera only. If you format in the computer and put it in the camera the chances of corruption are high . 4 most importantly do not insert an slow sd card in a fast reader and a fast port / also Don't insert an usb 2.0 pendrive in a usb 3.0 or 3.2*2 or a thunderbolt port. The faster usb 3.1 port will try to read the slow pendrive and it will try to rip all the info to the pc as fast as possible. And it will end up reading only half the data will trying to open the old pics asap. To avoid this copy data from a slow pc with usb 2.0 to the cloud and then download it to the faster pc then copy it to an ssd or HDD. 5 older gen cellphone also suffer from this problem. So don't plug in your old hard drive with your precious old memories in a moden pc. ( I did it and opened my old photos and saw them get corrupted before my very eyes) If you have an old hardisk, don't open the folder immediate after plugging it in. Give it a lot of time. Let it completely go for a complete spin. Then do a full virus scan before opening it (in a old slow pc ) 6 the worst part is switching from windows to Mac .Mac is fast and hates old files especially it has a lot of issues with the formatting part. It will ask you to format the entire Hardisk to use it. If you format you will loose all your stuff. So don't do cross platform stuff. Finally have some respect for the time I took to type this comment out for your benefit and up vote this so that other could see this too.

i must have missed the part where you avoid backing up corruption. does you Synology system do some sort of special verification? if your files are corrupted does it alert you?

I had a hard drive crash in 2013, and I didn't have a serious backup strategy. Now I have a 100T off line back up facility where I back up photos and videos at least twice a month, and often once a week. I lost All the photos that I took that year. Fortunately I had prior photos backed up already through dumb luck on another computer. Now i have a system, and that system seems to work pretty well. I'm thinking of doubly backing up photos, but haven't done that yet.

Every backup scheme works until it doesn't. My first rule: make multiple permanent archives. Bit rot can't go backwards in time. So I have every photo I've ever shot on at least two BD or BDXL HTL discs, most of them M-Disc media. Unlike CD and DVD, BD uses a non-organic medium, if you get the original formulation, called HTL these days (high to low -- the laser makes the shiny bits less shiny). Not a great solution for tens of terabytes of video a year, but it's okay for my 1/2-3/4TB of photos every year. I also copy my current files to a RAID locally, and run SyncTrayzor to sync between computers at two houses. If Godzilla stomps one house, all my photos will survive. If he stomps both houses, well, I have some 14TB of stuff backed up on BackBlaze, but I'm thinking if that happens, I've got larger problems. I don't worry about messing with custom file systems (though a specialized RAID can do better). HDDs and SSDs use more powerful ECC data to check and, if things are going bad, correct small bit flips that lead to bitrot. This has been the case ever since IDE drives. The problem, of course, is that transparently correcting a bit flip fixes the immediate problem, but it doesn't tell you that the problem actually occurred. Most modern discs run SMART, which actually does track errors, but for most people, the only time they're checking this is after the errors start creeping in. My ideal backup would also create a hash of each file, and for incremental backups, compare the old hash with the new and, unless it's an obviously edited file, treat the changed file as a different file. That way, even if bitrot sets in, there's no overwriting of the old stuff. I would assume Enterprise-class backups do something like this.

I'm sincerely horrified to hear/see this Tony; another way to create backups (though you're using up some time creating them) would be to burn important images into M-Discs, which now come in capacities up to 100 GB for BDXL disc... Since they do NOT work off of organic dye systems like CD's/DVD/s but utilize 'rock-like' materials for it's writing layer they suffer no data rot and literally are guaranteed for 100's of years, as the bits are permanently 'engraved'. They are still readily available online in places like B&H and Amazon and can be burned by most BD burners. I think these are a great cost-efficient option for people who can't afford an elaborate backup system.

@Vertifuge

2 жыл бұрын

Truth. M-DISCs are a very important PART of my data backup strategy. At rest, they should be good for a century or more and they are readable in any blu-ray data drive. Even if drives are no longer mainstream in consumer PCs, I expect commodity/surplus blu-ray drives to be available and accessible (without excessive difficulty) for decades. The other two prongs of my strategy are cloud and local hard drive RAID array. Notably, one branch of a sane backup strategy should be OFF-SITE, so that any single event (such as a fire) does not destroy everything at once. I archive my optical media in a safe secondary location, and my cloud provider's primary datacenter is over a thousand miles away.

@georgedavall9449

2 жыл бұрын

Excellent excellent comment! This is what I’m looking into as well. A good Blue Ray player that will burn M discs (Archival quality) is a solid alternative, but it should be used in addition to other back up measures. It’s almost ‘conspiratory’ how all of us have to invest so much money and time in trying to hold onto our cherished images. Tony touched on a very real thing in regards to ‘ Bit Rot.’ In the end, It shouldn’t cost so much or be so much effort involved to work with and hold onto images. Digital Photography promised so much and was so liberating from using film, but ends up carrying its own set of ‘baggage?’ Happy Shooting. 👍✌🏻📷

@DaveHaynie

2 жыл бұрын

I have used BDXL and M-Discs myself for years. It's actually the case that standard Blu-ray discs -- what are now called HTL discs -- are also non-organic. They use a layer of copper against a layer of silicon. Like M-Disc, they start out "high" (reflective) and the laser burns the "low" parts (in the HTL Blu-ray, it's fusing the copper and silicon). They don't last as long as M-Disc, of course, but if there's a difference between the two in my lifetime, that's a high-class problem. However, the dye-based companies added the "LTH" Blu-ray disc, which is back to organic dyes, making those even more perishable than DVD or CD. You're always safe with M-Disc.

@DaveHaynie

2 жыл бұрын

@@Vertifuge Now you're touching on format rot, which has also been a problem. The optical discs, being a consumer format rather than a computer thing, are certainly the best candidates for longevity.

OK but how do I protect myself against the bit-rot? I use a Synology NAS (only four disks with some type of RAID) and I keep copies on multiple computers. But as you say I am not going to know until I open them years later.

Very good advice there Tony, also coming from an IT rich background I’ve seen plenty of the old writable CD and DVD’s turn to custard after a few years , very dependant on brand. Ironically the slides my folks took in the 1950’s of us kids still look good so long as all those other catastrophes you mention don’t befall them. Fully agree that as you get older and people pass away it is great to be able to look back. Cheers Chris from NZ

@SmallSpoonBrigade

2 жыл бұрын

Optical media just sucks as a backup medium. Apart from preventing deletions and modifications, it just doesn't do anything compelling and is a massive pain in the backside to check. Best thing is a hard disk based backup system with online backup and automated file verification. Preferably using a filesystem that can detect and repair bitrot, but as long as you detect it while you still have a good copy, that's OK.

@Chris-NZ

2 жыл бұрын

@@SmallSpoonBrigade agree entirely, I haven’t backed up using CD’s since usb drives were affordable. These CD’s were backups we made in the office in the 90’ s and claimed to be “archival” 😀😀 our first Pc with a ten mb hard drive for my engineering team plus a dot matrix printer, green mono screen, plus a pen plotter in 1985 cost $47,500 in todays money ($33,000 US) I think my toaster has more computing power and a single image from my mirrorless camera would fill the hard drive five times over and yet back then we though we would never fill it. In the mid 90’s ram for my 486 was $100 a megabyte !!!

@trevorsowers2202

2 жыл бұрын

That is why I still use slide film for important photos. I’ve lost too many digital files and almost no slides.

@Chris-NZ

2 жыл бұрын

@@trevorsowers2202 Interesting approach and of course I’ve still got my parents Kodachrome ll slides from the 1950’s still in great condition. I did have a complete failure with Agfa slide film I used in Australia in 1980 much of it now has an orange tinge. Of course easy enough to scan all that old stuff. Printing is my long term archiving option plus photo-books of course. Not sure I could go back to film after spending 35 years in that medium but you are correct in thinking long term digital storage no matter how carefully you are hasi it’s pitfalls and I wonder what will become of all the images taken by todays youth taken on phones when they want to show the grand kids in years to come. ☹️ I do have a mistrust of online services and there subscription models that’s for sure.

@trevorsowers2202

2 жыл бұрын

@@Chris-NZ some films are more stable than others for sure. Kodachrome was very stable but early E6 slides were not great but the later E6 is great. I have lots of Ektachrome that is 30+ years old and looks as good as new. I’m currently using Ektachrome E100 and I’m very satisfied with that film. I use a EOS 1V and a 5D MKIII so I can carry one set of lenses and speedlites and shoot digital or film as the situation warrants but the more important photos are usually done with Ektachrome as I have more faith in that medium

Customer files get backed up on two external drives. I try not to reuse the SD cards aside until the project is over. Deliverables get stored in OneDrive. BackBlaze also runs continually to keep everything backed up another layer, with the 1-year history add-on in case you need to roll-back to a previous version. Haven't had an SD card go bad since I switched to the Sandisk Extreme Pro cards. They're (much) more expensive, but haven't had any failures in the last 10 years. During my university days in the early 2000s I had a HDD go bad on me where I lost a few weeks worth of files, but otherwise fortunate to have all my files since the early 1990s.

Are there checksum or even parity-bit solutions to combat bit rot?

This is pretty basic for someone who does know something about data protection and storage, but in reality, it's not common knowledge and if you want to do it properly it can be quite annoying and expensive if you have lots of terabytes of data. The key concept anyone should know is the 3-2-1 rule: 3: you must have at least 3 copies of your data 2: in at least 2 different physical drives 1: and 1 of them must be in another physical location What does that mean in layman's terms? the easiest method would be: A.- get a couple of external disk drives and backup (duplicate) your data on both disks. Then pay for a decent online storage service and also make a backup there. B.- get a couple of pre-built NAS hard drives and backup your data with redundancy on both units. Then grab one of those units and leave it on another place like your office or something like that, away from your home. Then keep them synced online via internet. This method is far superior but it has a higher initial cost and it requires some configuration. Hope this helps someone. Sorry for my English, it's not my first language.

@SmallSpoonBrigade

2 жыл бұрын

This is true and it's especially annoying given the high quality tools that currently exist to prevent this. From database programs like imatch that have built in file verification functions, to file systems like ZFS that can be set up to detect and fix bit rot to various online backup utilities that can mirror the data to multiple data centers in different regions. But, it's still not on by default and requires a bit of work to get right.

Thanks! I won't make the mistake of taking photos!

My horror story isn't about total loss, but partial corruption, and I want people to know: USE AN INTEGRITY-CHECKING NETWORK TRANSFER PROTOCOL TOO ! What happened to me was the switch inside my router had started silently corrupting parts of some network frames from time to time. It did not impact transfer speed so I didn't realise something was wrong, and as the default setting on client Windows versions is not to enable signing for SMB - which provide integrity, my PCs were transferring data that was being silently corrupted. I only noticed when I copied executable files over the network and Windows then said they were corrupted once I tried running them. That is the moment terror struck me as I realized my whole music collection that I had recently moved from an old machine to a new one was now corrupted and all my FLAC files now had glitches in them... As I kept that synced up, the synced copy also now had the glitches. I cried a bit.

Wow, this is horrible news. So sorry it happened. The problem came about because hard drives were the only foundation for the back-ups. For still photos and family videos, I feel everyone needs to upload to a website separate from their hard drive solution. I have uploaded my finished JPGs from every commercial and personal shoot since 2008 to Zenfolio. I am so glad I have those pristine files online there. However, I realize Zenfolio could have issues at some point. So, for the past couple of years I also have been adding my exported JPGs to the Adobe and Apple clouds. In a nutshell, I export my JPGs at full size with a quality level of 90 and have them on my Synology at home and online with Zenfolio, Adobe and Apple. Hope this helps someone.

man so sorry to hear this Tony. You've really got me thinking about this ....

I feel for you Tony, but I'm confused. Don't cloud storage services use hard drives of some sort? Aren't they too subject to bitrot? I didn't find this video helpful in combating bitrot losses. i always keep a backup copy of my raw files, which I make prior to importing to lightroom. But now it seems like these backup files could rot over time. And I backup my lightroom catalogues, including the raw files, but they could suffer bitrot too. How did you lose your files with all these backup NAS drives you use? If you lost your photos, what could the rest of us more casual backuppers expect?

Tony, you have an amazing IT background so you probably know how to set up a RAID storage solution... LOL! I am watching this in real time so I see that you DID suggest a RAID solution... curious to see how you set it up...

Thank God I used to back up to archive quality DVDs I had a PC crash about 10 years ago and lost everything or so I thought till I went through a box of data DVDs . The disc on top was the latest back up of my photos. It contained all but some of my oldest images which were on another disk deeper in the box. I now use a plug in hard drive and 2 cloud services. Every once in a while I’ll get nostalgic and burn a disc of the latest files

Last year I lost a 1TB drive with lots of media on it. I don't yet go to the extreme of off-site storage or fire/waterproof safes, but I decided to make multiple copies of files past a certain age and spread them across various drives (Laptop, NAS, Desktop, Google Photos and older drives that I leave unplugged in a closet). Since HDDs always get larger, every time I upgrade, I make a new copy from a verified-good backup. It's pretty manual, but it hopefully prevents the automatic backup of corrupted files.

As a geek, I nowadays only use filesystems with checksumming - btrfs on the laptop and less critical single disk systems, ZFS on the desktop and the rest. The desktop, which also uses ECC memory, is where all photos are copied to and live. Its data partition a RAID6 array allowing for 2 disk failure, but since upgrading it to be NVMe-only, that amount of redundancy was not feasible both in terms of cost and number of PCIe lanes. 😉 Not a big issue, as the system is backed up nightly to my local backup server that has a ZFS mirror as well as two different online storage providers - all these are encrypted and fully deduplicated, growing only by actually new data. QR-code printouts of the cryptographic key material stored off-site in a vault. Photos are recorded to 2 memory cards, I prefer more smaller ones, so e.g. on vacation, after each day of shooting, they can be taken out and stored in different locations. Back at home, the secondary cards stay in the car, while the primary ones are copied onto the desktop ASAP. Long since, the (semi-automated) commands for the copy process involve a second pass of verifying the files on disk are bit by bit identical to the ones on the card. Tanks to the absurd failure of the camera industry to put a GPS chip into every modern camera 😡, the files have to touched one last time to fix eventual timestamp offsets and geotag them. I go quickly over them once to eventually sort them to individual folders (if multiple locations were visited on one day), enjoy a first quick review of what I ended up with and make sure all photos do display correctly (ie. are not obviously corrupted, e.g. by the camera writing garbage or the primary card having an issue). Before using checksumming filesystems, I often did create checksum files (actually MD5 hashes, most of the time) of the photos to allow detection of bitrot after that point. The cards only ever get formatted and put back into the camera bag for reuse once the next backup of the desktop has completed successfully. Before having a camera with 2 card slots, I managed to corrupt and loose a few photos in one incident: After copying the previous load of photos off the SD card, I forgot to properly unmount it before yanking it out of the card reader. Put into he camera, I formatted it in there and went out taking photos. So far things were all fine, the problem arose when finally reinserting the SD card in the card reader and the system did not realize that it had been modified since being yanked out, writing some outdated data on it, ruining the directory structure. Whether the camera is to blame for formatting every card with the same serial number and maybe not even applying a different time stamp, or the at that time early implementation of exFAT not bothering the check those before assuming a reappearing block device being the same as before and happily writing stuff onto it, I never bothered to research. What I did was making an image of the whole SD card and then recovered all but very few photos from that using photorec.

I‘ve lost a film roll from the wedding of my very best friends, who relied on me taking good photos from that event-I absolutely know what you are talking about. I‘ve also seen several hard drives suddenly crash. It’s not the question if such things happen, it’s only about when-and you’ll never know that in advance. By the way, having one redundant backup system is not enough. I have seen two redundant raid-5 systems electronically fail in two days, so before the first of them was repaired … 1 TB of photos needed te be recovered from tape.

The best advice you have given, fortunately, although I had a laptop that suddenly failed, I was able to remove the hard disk and transfer. Not had any failure of cards, though now I am tempting fate. I always transfer SD cards to iMac or SSD Sandisk drive. Still have thousands of negatives though that are in their protective slide sheets. However, will invest in further back up, i Cloud looks promising. I did know, after being a geek like you, that digital storage can get corrupted over time. All the best, thanks for the reminder! Share you sadness, that is a photographers nightmare, it is like robbed and having all your equipment stolen when insurance could not be afforded. Yep that happens.

Sorry for your loss. A hard lesson to learn.

Spot on Tony, and timely too. Only two weeks ago I had a massive storm where a very close lightning strike meant we lost power for just over two days, and flooding caused all sorts of other problems. I have at least one UPS on almost all my pc and tv equipment, but I was only using it for power, not for the network cables, so the lightning strike ended up frying my WAN port on the router (connected from the modem by a long network cable, which picked up the lightning RF), so once I got power back it took a while to work out the issue. One very scary part was that without the router, I couldn't know if I lost my 12TB NAS box, as it was hard to get it to boot, and I didn't know if I lost it all for ages. I'm currently organising my online backup, which will be terribly expensive, but I can't afford to lose it all. Your bitrot reminder is very scary though!

been looking at joining PPA for a little while - just joined thanks to y'all :-)

So... since you have all these devices - synology, back ups etc. How did you end up being a victim of a bit rot? Your advice is what to do for us - and yes - you have done everything right and still something went wrong. What did actually went wrong specifically in your case? What did you miss that you are going to change?

@ThioJoe

2 жыл бұрын

Apparently Synology doesn't have scrubbing enabled by default (at least it wasn't for me), which is what repairs bit rot if it's run periodically (anywhere from monthly to yearly depending on who you ask)

@vancouverbluesea

2 жыл бұрын

@@ThioJoe looks like a critical miss. Is this option not promoted as a must have? It would be nice if @Tony does spell it out because the video points to recommendations (all valid) but the viewer is left guessing "what did really happen"

@matrixate

2 жыл бұрын

@@StefanHundhammer This is absolutely NOT true. Bit rot is not an attribute of the date, it's an attribute of the magnetic strength. If you copy files that are barely readable from one drive to another, the copy is refreshed with a strong magnetic strength. You then format or rewrite over the old hard drive.

@definingslawek4731

2 жыл бұрын

@@matrixate does that mean you could simply refresh the data on a hard drive by the writing all of it over itself ? Or is it that once this hard drive is weak magnetically it won’t be stronger again after writing again ?

@adrianvanleeuwen

2 жыл бұрын

The lesson is that nearly all hard drives will eventually fail (I discussed this with an IT person). Some take 7 years, some even less years, some 20 years, so having multiple backups and the methods Tony used is the best solution. Tony may have lost some photos, but learned along the way to remove this problem with better systems he is sharing here. (I have had both HDD and SDD drives fail in past, but the backup may not have failed.) For personal photos, always a good idea to print some of the best images from your life at a lab for more permanent memories.

Fantastic content! Professional Photographers of Washington just published an article I wrote about "Digital Decay." With the usage of cellphones and reliance on social media for photos, I strongly encourange people to print their pix and backup. I dropped a 4Tb hard drive--bye bye to all client pictures prior 2017! I now backup to 2 external harddrives. A lesson was learned!!

@DaveHaynie

2 жыл бұрын

Ha! One of my cats recently knocked an 18TB hard drive onto the floor. Not all that far, either, it was on top of the PC, and she decided she ought to be on top of the PC instead. It's warm there, but it's the drive that's toast :-( Aside from cat-proofing one's office, this is a good example of not having a single copy of anything.

I'm sorry to hear about that! Data loss is a real nightmare. I feel the urge to check on my files asap... After my NAS and the cloud, my third way to backup photos is actually Blu-Ray discs! Optical media rated for a 100 years. If my files are intact at the point I burn them to the disc, bitrot is rather unlikely. I keep the discs in my basement which is basically a bunker, so they are relatively safe from fire or other disasters. Of course this is only for long term storage and not for files I use regularly.

I'm keeping the pictures backed up automatically on a QNAP NAS and separate manual backup one in a while on 2 separate drives attached to NAS (externally) just as another back-up that doesn't overwrite automatically. I lost pictures in the past, professional and personal, an entire year of photos.

The following solution is not cost effective for those with really large photo libraries, but it is a good option for 1 or 2 TB libraries. At the end of each year, I buy a new backup drive for about $100, back up all of my photos on it, and then put that drive in a security deposit box with all of the others from previous years. If in 2030 I find that my photos from 2015 have bit rot. I just go through every year's backup drive until I find the backup without bit rot.

For most people, I would just check your onsite backup before syncing to the cloud, and then for files that don't change, don keep resyncing them, as this can lead to problems if your onsite backup (that is used as source to sync with your cloud backup) becomes corrupt, then you're syncing corrupt data. What I do is verify my files both onsite and then after they are in the cloud (using verification tools) and then once that's done, I disconnect the backup (no need to continue backing up files that don't change, which is usually my RAW files). Then I do backup my catalogs and edit data (and any files I create with PS or LR) and sync those. At least my RAWs are untouched as I don't sync them, but rather just do a one-way backup once, and then I'm done (so more like archiving your RAW files in the cloud, not syncing them, which is two way on most systems). In fact as I'm writing this, I'm doing a second cloud storage backup of my files on another cloud storage platform just to be safe.

If you use a cloud backup service that lets you restore from a particular point in time, it stores the changes (deltas), you can at least recover your original uncorrupted version.

Thanks for reminding me - I have bought enough space on G-Drive to at least upload a couple of thousand of my "best" images as RAW - but have to be honest forgot to do it. I will now actually dedicate a couple of hours in the weekend to finally do it! 😊👍

@SmallSpoonBrigade

2 жыл бұрын

Use mirrored zfs along with online backups.

Currently I have been storing my photos and Videos on External Harddrives which are disconnected from my computer. I call these Collection Drives. Each Collection drive has a backup copy which is stored at my storage unit away from my house in case of fire or flood. The pictures or videos that I use are stored on a separate drive I call the Active Drive. This way I has less of a chance of creating a User Error. (Saving Over, Corrupting, or Deleting Files) I follow a protocol of only using my Collection drives as a read only method. Just copy the files over to my Active drive. Do not modify or delete anything on my Collection drive unless I am specifically checking it or doing an intentional update on it. The less I tamper with my files, the safer they are. As for Bit Rot. I don't have a solution to this as of now. In the future I may do a full transfer over to SSD's as a new medium and refresh all my data. to a New medium. Some very important files I might copy to a third drive for an extra layer of protection. (Manually checking backed up without syncing. I do wonder if blockchain technology will solve the issue of data rot through its verification process. It would really suck for someone to have the wrong amount of coin in their digital wallet due to bit rot.

Sorry to hear what happened. Now about the video itself : I understand the advice of making some use out of the photos instead of letting them sit on a drive (which not only creates some redundancy, but also brings back the point of taking photos in the first place) ; but shouldn't the video also cover measures that would prevent the problem from happening in the first place ? I would think that generating a hash for each picture and periodically checking for it would help detecting corruption on a given drive as soon as it emerges. Of course that's probably a naive assumption on my end but sure enough something along those lines should be possible to at least prevent the blind backing up of already corrupted files.

This is why I sometimes still use film. My negatives won't disappear anywhere.

Thank you for this video. You did a great job demonstrating that even if you practice the 3-2-1 method of data backup, you can still have bad data replicating through your backup system. Did you ever figure out where the source of the corruption occurred? Did it happen long ago when you put photos on external HDD's? Then when you upgraded to a more sophisticated NAS you were just copying bad data? Is there a way to review all your pictures in bulk to make sure that they retain their original data integrity? I am naive to this, but I would think that each file would have a CRC check or something equivalent that would alert you that the data is corrupted.

I burn my data onto recordable blu-ray disks. I burn two copies and store each set in seperate locations.

Hi Tony, Sorry for your loss. As much as I know quick format deletes only the content table of the card but not the pictures so you can try to recover them with recovery software for deleted files as long as you do it asap after the format.

Back to film photography, Robert Capa lost 90% of his D-Day's photographies on Omaha Beach, "thanks" to the mistake of a lab technician in London: out of 106 pictures, only 11 survived... When I started (film) photography, I was always anxious about giving my rolls to a lab... I'm only six years older than Tony by the way 😉 Cheers Tony!!!

Well that truly sucks - so sorry this happened to you - did you send any of the lost files to other relatives back in the day? Maybe they still have some of the files that they could send back to you?

I once wanted to insert a sd card in the reader... And it broke in the middle... Now I have a camera with 2 sd card slots. I tend to make a backup every month and store once a year on a separate drive that will probably never be touched again. Lastly I have a small (100GB) storage server rented for the best.

Ten years I had a few problems with pictures suddenly getting massive color stripes. That was enough for me to make multiple backups as a habit. I also made checksums for files so errors could be caught before a file was reviewed by a human. Rule #1: never backup a bad file. Now I use WINRAR on windows. It handles its RAR and other formats such as ZIP. It includes checksums for the files and optionally encryption and error correction encoding.