there is another option using write-behind caching on EXT4 so it caches all the writes to ram and then using the COMMIT=xx seconds mount option it will write the changes from ram to disk every so often, the best of both worlds

@AndreasSpiess

2 жыл бұрын

Good point! And it seems to be easy. Similar to log2RAM, I think.

@_gawen

2 жыл бұрын

It doesn't ensure that writes will be cached to ram though. Just that it will cache up to "commit" seconds of writes before automatically syncing the filesystem so if your system suddenly crashes, you know that you have lost at worst this amount of writes. Applications can request their writes to be immediately recorded to the disk/SD Card. Applications who use databases (such as SQLite) or unbuffered logs will do that frequently.

@magoostus

2 жыл бұрын

@@_gawen i forgot to mention, that its a setting to enable journal_data_writeback using "tune2fs" on ext4 filesystems, as well as "nobarrier". this will run it all to ram first at the cost of data security and reliability, obviously

@paaao

2 ай бұрын

I have a pi3 running on the same sd card I started with, probably 6 years of running 24-7 with not a single issue. Pi4, has already burned 3 sd cards up in less than 2 years. Can't for the life of me remember what commands I did to limit the card writes, or whatever I read at the time, but it's worked. Wondering if this is it...

This came just as I needed it! Releasing 24 dashboards driven by raspberry pi's next week and was worried about configs being changed and other (python) code config that needs to write to the memory! Much appreciated Andreas!

@AndreasSpiess

2 жыл бұрын

Indeed this should prevent bad things from happening.

great features Thanks for sharing your experience with all of us 👍😀

@AndreasSpiess

2 жыл бұрын

Thanks for watching!

Thank you! Extremely useful, extremely inexpensive, reversable and can be done in seconds. Love it! The overlay wins for me.

@AndreasSpiess

6 ай бұрын

Glad it was helpful!

I was able to reduce my sd card consumption drastically by logging to my remote homeserver. (almost) No local logs anymore. There is another advantage in remote logs: you can find suspects, even after the device in question died. Since there is a homeserver up and running 24/7, it was just a matter of some simple configuration.

@AndreasSpiess

2 жыл бұрын

Remote logging seems to be an interesting alternative (or extension) to this concept. Maybe I have to try it too...

@JB-fh1bb

2 жыл бұрын

@@mixxx2005 For anyone considering this: make sure to put security in your network config. Unless you want a random device in the wild to give an attacker access to the files on your home servers/computers.

@SteveJones172pilot

2 жыл бұрын

Have any details on how you did this? I'd love to do the same.. have an NFS share or similar on a pre-existing NAS that would support logging from all of my various PIs, which could then run completely read-only.

@peter.stimpel

2 жыл бұрын

@@SteveJones172pilot I enabled one of my Linux hosts to accept remote syslog messages, and enabled all clients to log to this remote logging facility. This is quite straight forward. How you have to do i depends on your software, of course.

Thanks Andreas, I was worrying about my sd card in. my headless Raspberry Pi 4 recoding NOAA weather images 24/7 so I made images of the card if it fails, I haven't had any card fail yet but you saved the day and me worrying, thanks for this tip will implement it : )

@AndreasSpiess

2 жыл бұрын

As said: A backup is always good! But your application probably is a good use case to reduce the chance for failure.

Great hint, Andreas! Thank you!

@AndreasSpiess

2 жыл бұрын

You are welcome!

I've been using raspberry pi for quite some time It didn't realize that was there. I really enjoyed this video. I really enjoyed listening to you on the ham radio workbench podcast as well. Thanks

@AndreasSpiess

2 жыл бұрын

There were a lot of discussions in the past about this problem. It was also a pleasure to be guest in the workbench podcast.

Thank you very much! It was useful and interesting 🙂

@AndreasSpiess

2 жыл бұрын

Glad to hear that!

With the newer Raspberry Pi 4 with the higher RAM this overlay makes a lot more sense now. I guess you could use read/write Boot drive and periodically copy the overlay to the SD or to an external network drive for data that may need to be kept longer term. Thanks for the great information.

@AndreasSpiess

2 жыл бұрын

You are right, many combinations are possible if you include network disks in your setup.

great video. happy christmas

@AndreasSpiess

2 жыл бұрын

Thank you! Happy Holidays, too.

Thanks Andreas. I watch your videos and like them every time but this is the first time I comment. I did not know about this feature and it is very useful for me because my children use Pi 400 for school. They love to mess with the OS and I have to reimage their systems often. Well, not any more. Thanks again

@AndreasSpiess

2 жыл бұрын

Perfect use case, I think...

thanks! this helps me a lot!

@AndreasSpiess

2 жыл бұрын

You are welcome!

Very useful and simple, many thanks.

@AndreasSpiess

2 жыл бұрын

Glad it was helpful!

YES! Finally I can cut the power to my rasppi whenever I want without the fear of destroying the SD-Card!

@AndreasSpiess

2 жыл бұрын

True. I like this, too!

@accdiabolus

2 жыл бұрын

I tried it with hdd. I've had to re-set it up twice. 😒

Useful and interesting as usual.

@AndreasSpiess

2 жыл бұрын

Glad you think so!

Great video, as always. Also, a great tip I will try both ways. Overlay and with a small SSD drive.

@AndreasSpiess

2 жыл бұрын

I use both ways, too. For different use cases.

Thank you this was a simple but effective trick!

@AndreasSpiess

2 жыл бұрын

You are welcome!

excellent... i wondered what that setting was for but never looked into it!...thanks Andreas

@AndreasSpiess

2 жыл бұрын

You are welcome!

Great. I just needed this.

@AndreasSpiess

2 жыл бұрын

Very good!

IIRC, OpenWRT uses OverlayFS, but their implementation is pretty static without lots of hackery. Really neat the Pi Foundation was able to simplify it down to a toggle switch! 👍

@AndreasSpiess

2 жыл бұрын

As a Linux noob I was not aware of it until it was simplified ;-)

Useful AND interesting! I was unaware of the read only setting.

@AndreasSpiess

2 жыл бұрын

Glad you like it!

@jmr

2 жыл бұрын

@@AndreasSpiess I realized I was unknowingly using the read only feature on my pistar hotspot. 🤣🤣 I thought it was part of the project and didn't realize that was a feature of the OS.

@AndreasSpiess

2 жыл бұрын

AFAIK Pistar does not use Raspbian and therefore probably implemented a different concept. But it seems to work ;-)

Nice info! Thanks!

@AndreasSpiess

2 жыл бұрын

Glad it was helpful!

Thank you for these useful tips. Much easier that I had imagined Great comments on this video too. :)

@AndreasSpiess

2 жыл бұрын

I agree, I have a lot of knowledgeable viewers and commenters!

thanks for the knowledge

@AndreasSpiess

2 жыл бұрын

You are welcome!

Wow..I didn't know about this feature..?...thanks!

@AndreasSpiess

2 жыл бұрын

You are welcome!

Very useful video, Andreas👍- this setup is indeed ideal for 24/7 applications that do "something" and send the results to a remote server or such. Or another idea: Run Minikube on the Pi4 and push containers to it from remote. That way the Pi would become application-agnostic?

@AndreasSpiess

2 жыл бұрын

So far I never used Kubernetes. Maybe a project for winter ;-)

Thanks for the info

@AndreasSpiess

2 жыл бұрын

You are welcome!

Great tips, nice video! To be a bit pedantic, overlayFS itself doesn't mean writes are put in RAM. Rather, overlayFS is one piece of the puzzle - it allows redirecting writes to somewhere else, and when combined with tmpfs (RAM-based filesystem) it achieves the reboot-to-discard-temporary-changes feature.

@AndreasSpiess

2 жыл бұрын

You are right, of course!

I also tend to install ZRAM on every pi I own. It removes the need for swap space on the sdcard and it allows you to fit about 50% more stuff into the physical available RAM.

@AndreasSpiess

2 жыл бұрын

I never used it because I only have 2GB and bigger Pi4. So memory is no more a big problem for me. Probably good for a Pi zero if it does not consume a lot of CPU.

@misteragony

2 жыл бұрын

@@AndreasSpiess I even use it on my rpi 8gb model. Sometimes I notice raspian swapping stuff out even with minimal applications running and plenty of main mem free.

@jg374

2 жыл бұрын

​@@misteragony For some reason I think the default "swappiness" value is a bit too high in raspbian (how likely the os is to use swap). I adjusted it to 10 from its default 60 and have only noticed minimal swap usage since then, but it is still there for emergencies when you are running close to the limit (I have also done this on my laptop with similar results).

Indeed interesting. Guess I should make work of adapting my RPi asap.

@AndreasSpiess

2 жыл бұрын

If you have one which does not have to keep data it is a good thing.

SD cards are not by definition unreliable. They come in different flavors for different use cases. If you pick the right one, an SD card can be more reliable than a big cheap SSD. The most important criteria are the cell technology used and the Bill of materials. Cell technology: * Most consumer cards use TLC or QLC (3D) cell technology. A single cell can typically be rewritten approx 200 to 300 times. * Some consumer cards advertise with 'long endurance' or 'industrial' terms. Often such cards use MLC cell technology (but this is not guaranteed!). In this case a single cell van typically be rewritten approx 2000 to 3000 times. This type is also often used for EMMC memory and modules. * For more demanding environments (automotive, medical, extreme conditions,..), often (p)SLC cards are used. For these type of cards, a single cell can be typically rewritten up to 30000 times. That's >100 times more than a typical consumer card!. You will not find these cards in a regular (online) shop. Cards utilizing this technology often have more advanced flash controllers onboard providing additional security and features compared to consumer grade cards (e.g. power failure protection, better wear leveling...). Bill Of Materials: * Most consumer cards do not have a fixed Bill Of Materials (BOM). This means the internal components can change without notification and the manufacturer will choose whatever is available and the cheapest that complies more or less with the specifications. So for one batch of cards, the cells can be TLC, and a few months later, QLC becomes cheaper and the manufacturer switches to QLC components, or uses another flash controller... This is also the reason why almost no manufacturer will give the complete specifications of the used components for these cards. * Most manufacturers also offer fixed BOM cards. These are cards where the used components are fixed and will not change without notice. This guaranties that if you buy the same card next year, it will behave exactly the same as the ones you already have. An example of a 'cheap' (p)SLC card with fixed BOM is the SDSDQED-016G-XI model by Western Digial/Sandisk. It is a 16GB SLC card with an extended temperature range (-40+85C) and an endurance of 480TBW. It costs about 30 euro's at Mouser. This is typically cheaper than a usb->sata/M2 adapter + decent SSD. For comparison: a Kingston A2000 SSD of 500GB has an endurance of 360TBW. And if you only look at smaller very low cost ssd's, then the endurance is often 40 TBW) So for projects that do not depend on very high speed disk access or require a lot of storage, such SD card is unrivaled. Both for price and reliability. (The explanation above is a bit simplified... there are other factors playing also a role, but this comment is already long enough for now :-)

@AndreasSpiess

2 жыл бұрын

Thank you for bringing light into the dark of the SD cards! So far I have only used SD cards from online suppliers. Good to know other variants exist. Concerning SSDs: Even my home automation server only writes MB and not GB because I store sensor data and not videos. So with my Raspis, I can live with very low TBW values (other than with my KZread production machine ;-) ) I also only use the smallest SSDs I can get because the volume is small (

@Adrian_M_Boyko

Ай бұрын

Best comment ever! Thanks.

@Adrian_M_Boyko

Ай бұрын

I see that there is also a SDSDQEC with up to 3200 TBW. Search for "brochure-removable-flash-storage.pdf" for a good summary of current offerings.

Very very good video and here I was wondering why my card would die after a few months of usage. Thanks a lot for this video. Now to the terminal window!

@AndreasSpiess

2 жыл бұрын

I hope this will solve your problem. Other viewers commented also that they were successful using "endurance SD cards"

nice and useful trick, thanks for the info

@AndreasSpiess

2 жыл бұрын

Always welcome

It is also very useful to mount the filesystems with noatime,nodiratime options in order to avoid updating access time in the file and directory attributes each time the file or directory is accessed, no matter it was just read.

@AndreasSpiess

2 жыл бұрын

Another viewer also mentioned noatime. I am not aware of this feature. So I have some reading ahead ;-)

@NikolayH

2 жыл бұрын

@@AndreasSpiess This is how I use these mount options, on each and every extX filesystem which resides on SD, SSD or CF on my machines. root@raspberrypi:~# cat /etc/fstab proc /proc proc defaults 0 0 PARTUUID=60650d97-01 /boot vfat defaults 0 2 PARTUUID=60650d97-02 / ext4 noatime,nodiratime 0 1

@NikolayH

2 жыл бұрын

@@AndreasSpiess I just noticed on the mount(8) man page that nodiatime is implied if noatime option is used. There is always something new to learn. From the mount(8) man page: noatime Do not update inode access times on this filesystem (e.g. for faster access on the news spool to speed up news servers). This works for all inode types (directories too), so it implies nodiratime.

Thank you Adreas for the easy to understand ex[;amation of the Overlay File System. It's just what I needed for my weather station which uses an ePaper display controled by a Pi Zero W.

@AndreasSpiess

2 жыл бұрын

That seems to be a perfect fit!

@Gigamegawatts

2 жыл бұрын

​Is it a perfect fit, though? You mentioned in the video that 2G of RAM is recommended, and the Pi Zero W only has 512M. I don't know how the Overlay File System decides how much RAM to use: is it dynamically allocating memory based on the amount of data written, possibly running out of memory over time? Unfortunately, Googling doesn't return any tips on reducing its memory usage, and I couldn't find confirmation that anyone is using it with a Pi Zero W successfully. I guess the only way to be sure is to try it. Thanks for pointing out the option, though: I didn't know it existed.

Excellent information, I already put UPS boards on all my Pi’s but this will is a great trick.

@AndreasSpiess

2 жыл бұрын

UPS is also a good think, But not cheap ;-)

@NexGen-3D

2 жыл бұрын

@@AndreasSpiess No they are not, but I wont run a PI without one, I use the PiSuger2 on all of mine, they come with a remote web interface for configuration and status, totally worth it.

@AndreasSpiess

2 жыл бұрын

Looks like a good solution!

thank you for the job

@AndreasSpiess

2 жыл бұрын

You're welcome!

Great find! I'm going to have to check to see if it works on my OctoPi server 😃

@AndreasSpiess

2 жыл бұрын

I never tried so far. But it should not need to keep a lot of data after reboot.

Nice Video, learned something new. Now it would be great if specific log operations could be made to go to the SD card and the majority just to ram...

@AndreasSpiess

2 жыл бұрын

You an du that by using RAMFS or log2RAM to only store logging in RAM. But it needs a little more Linux knowhow. Or you mount a network drive for data as other viewers suggested.

I saw this option before in dietPI, I think it's a cool feature, glad the ported it on raspbian too

@AndreasSpiess

2 жыл бұрын

Another viewer asked if it is available on dietPi too. So here we have the answer...

@anocco

2 жыл бұрын

@@AndreasSpiess Not sure if it is exactly the same. I know that in DietPI logs are not saved directly to SD and when this functionality was introduced, it was not yet in Raspbian. It is a really interesting concept to preserve SD life expectancy

thanks to the guy with the Swiss accent, another great tip and information

@AndreasSpiess

2 жыл бұрын

Happy to help!

Thank You

@AndreasSpiess

2 жыл бұрын

You are welcome!

This was a great video.. I had been wondering why there wasn't an easy way to do this.. Next step would be to map a network drive and use it for logging and other data you dont want to lose, and it would be the best of both worlds! Thanks!

@AndreasSpiess

2 жыл бұрын

Agreed. Other viewers also suggested that method. So far I never used it.

I use a different approach, and I find it much simpler and scalable: I use a NAS, so shared directories to save logs and data. So I get automatically backups (I need to set up only once for the base directory in the NAS), fast disk (but it depends also on network), cheap disks. And I'm able to split or merge services and rpi much easier.

@AndreasSpiess

2 жыл бұрын

Storing the files on a NAS is for sure a good thing for many applications (if you own one).

@PvGeens

2 жыл бұрын

When a Raspberry is booted without connection to the NAS (e.g. NAS is down, or no network connection), the Raspberry runs without NAS connection forever! No retries to connect later. Solution: autofs mounts the NAS directories when needed, and releases the connection after 5 minutes when not in use, so this also costs less memory. Thus a retry is provided after connection-loss. When speed is not needed, you can make a NAS from a Raspberry and 2 USB disks (even USB-thumbs) as RAID-1 using mdadm.

@nigratruo

2 жыл бұрын

@@AndreasSpiess It is super easy to make your own NAS, just install Linux on a machine (could be another raspberry pi) and configure a RAID 5 (preferably not on USB, which is too unreliable for it, as it can disconnect randomly from time to time, which breaks the RAID and makes it go into degraded mode) and then share certain folders via NFS, very easy. The advantage is that you don't have a proprietary NAS like Synology that is both inflexible, hard to fix when it breaks and expensive and often uses a non standard proprietarily modified Linux.

@denastewart-gore800

2 жыл бұрын

I have a NAS as well and I prefer using it as well. I just need to get 1 that's a few teraflops my problem is the $.

@denastewart-gore800

2 жыл бұрын

@@PvGeens Sounds like I will need to be careful with doing that.

After 2 SD cards busted, I put an SSD for my Home Assistant RPi4 and it is running since then very reliable, hope I don't jinx it. But very good tip to make the SD card last longer.

@AndreasSpiess

2 жыл бұрын

I also use an SSD on my server. Also for speed...

Nice!, this will also improve the performance. I am thinking about to create a minimum raspbian version to improbe the boot speed and memory usage on a Raspberry Pi Zero.This will help me for sure.

@AndreasSpiess

2 жыл бұрын

And it is needed for the Pi zero. I do not like it because it is very slow. Maybe with your changes it is ok...

Thank you very much!!!!!!!!!!!!!!

@AndreasSpiess

2 жыл бұрын

You are welcome!

Add commit=300 as a mount option in /etc/fstab. EXT4 default is to commit all writes every 5 seconds. If you mount with commit=300 it will cache writes in memory then write them to disk once every 5 minutes. You can reduce your SD wear by 60x by adding that line.

@AndreasSpiess

2 жыл бұрын

Good point. Thank you. I will try it

thats a neat trick, plus is more secure, becasue if you loose your rapsberry pi whilst transporting it, you wont have to worry about still being logged into google etc and saved passwords. just enable it on a new install after you have setup wifi and customized the dektop

@AndreasSpiess

2 жыл бұрын

Good point!

Instead of a SSD, you can also use a network filesystem and redirect things there through symbolic links. Handy if you have a NAS on the same network. Beware that SAMBA/CIFS does not work well for that and some services will randomly fail if their data are put on a SAMBA mount. NFS on the other hand works fine with everything. You can also use the iotop command for instance 'iotop -o -b -a' to log the cumulative writes of the different processes so you can see who is writing and how much.

@AndreasSpiess

2 жыл бұрын

Thank you for your input. Remote logging indeed would be a possibility. Maybe even with these switches (if you want to keep your logs)

Just a quick question about your description, why the anti-docker statement? Is it against docker corp (understandable) or the concept of containerization itself? Thanks for your wonderful work towards better IoT and wireless communication!

@AndreasSpiess

2 жыл бұрын

No. It is only for KZread. I use Docker in my videos…

Clever ways to save on writes :) Phones uses F2FS file system to save on writes. F2FS supports ZSTD compression. Image with bear minimum of applications would write less compared to stock Rasbian. There are MLC SD cards that last ~10x at same capacity compared to most micro SD cards in the wild. Jet MLC cards cost ~4x (on SSDs the price difference is more reasonable ~2x). Higher capacity cards will last longer (grows lineary with the size). As for disconnecting the drive unsafely - I had luck with usb sata bridge and microSD readers, they usually go before the storage.

@AndreasSpiess

2 жыл бұрын

Thank you for the additional info!

Years ago, my daughter had been using my core 2 duo laptop running ubuntu 12LTS booting(with pre-loaded apps) from the dvd drive loading it on ram for several years till the internal cooling fan conked out. You could still save on the hard drive but everything else resets after shutting down.

@AndreasSpiess

2 жыл бұрын

Others also suggested that the overlay FS was used for USB stick booting which was probably similar to your concept. So nothing new, but still useful and now integrated into Raspbian (good for noobs like me)

brilliant!

@AndreasSpiess

2 жыл бұрын

Thank you!

Very useful for making changes and checking they work correctly. One question, once you make some changes can you then commit the changes to SD so the changes take effect on reboot?

@AndreasSpiess

2 жыл бұрын

Not with this method, AFAIK

I use RAMFS and mount it over the top of the log directories. Then, when a program writes to the log files in the log directory they are intercepted by RAMFS instead. When the max size is reached they are discarded. They are never written to the SD card. My SD card is in its 4th year of operation (as a LoRa gateway) with no problems so far.

@AndreasSpiess

2 жыл бұрын

I never used RAMFS, only log2ram which seems to have a similar strategy. This is another valid way to solve this problem.

@jannepeltonen7493

2 жыл бұрын

I have a bunch of raspi 3's running on the field and they have been working nice for the past 4 years with this tmpfs trick. A couple of sd-cards have became corrupt during this time, but I suspect its due to power outages, since after dd'ing they seem to work ok again.

@SteveJones172pilot

2 жыл бұрын

I would love to see any scripts or configs you have to accomplish this.. Sounds cool.. Also sorry you're "not" mark Knopfler.. it would be cool to be Mark Knopfler! :-)

@inconnu299

2 жыл бұрын

Can you write the command for setting this up? I have found this: sudo nano /etc/fstab And added tmpfs /var/log tmpfs defaults,noatime,nosuid,mode=0755,size=20m 0 0 Is this correct or are there other directories? Thx.

Thanks Andreas for the tip. What I do is to use a high endurance SD card, specially designed for high number of writing cycles ( they are used in security cameras). Is a very lazy solution though.

@AndreasSpiess

2 жыл бұрын

I hope this works (because I ordered a few of them after comments of other viewers...)

@electricwhirl5175

2 жыл бұрын

@@AndreasSpiess I personally have been lucky using Samsung pro endurance in Rpi for a while after testing them 1+ year on dashcam. Only 1 out of 10 have made one file corruption (as far I have noticed). Before that the Sd cards was not saving all the files or saving them empty. Just a note, that bigger sizes have bigger large ratios before failure. Other option is to add some USB drive, which comes in micro size, so fairly extends out of devices itself

@jdsan6009

2 жыл бұрын

@@AndreasSpiess Nothing is failure-free. I used to work with MDVR (mobile video recorder) and moved from a Sandisk Ultra type SD to a SanDisk high endurance. From a batch of 20 units, we used to have one SD dead every 2 weeks; after the change the first failure happened 6 months after. Using Transcend High Endurance SDs in three Raspi working as LoRa Gateways for one year, no problem so far.

@Darkk6969

2 жыл бұрын

@@jdsan6009 Thanks for the info about Transcend SD card. I will try that one next time. Currently using SanDisk 128GB High Endurance in my 4K dashcam. Working fine so far. My old 64 gig SanDisk SD card worked fine for 2 years before I replaced it. I always get the biggest SD card that the device can support to extend the life out of it.

@jdsan6009

2 жыл бұрын

@Andreas Spiess. You will like to know Tesla uses industrial type SD Cards to save logs. Here is the reference kzread.info/dash/bejne/gXiYq7SrZbXgeZs.html

I've usually mitigate writes to the sdcard by making sure the noatime and nodiratime option is on the sdcard in the fstab and mounting a small flash drive to the /var directory, and using a larger flashdrive for the home directory (you could put both partitions on one flashdrive). This should eliminate all writes to the sdcard except when installing and updating software. Not on PIs, but on PCs, I frequently use a virtual disk and keep my work on flashdrives, and have scripts with clickable icons to setup, load, and when my session is done, save and destroy the virtual disk.

@AndreasSpiess

2 жыл бұрын

Thank you for sharing how you do it. Virtual environments are also a good thing if you want to protect the original.

@nmjerry

2 жыл бұрын

Sometimes, at good points, I hit the save work icon, and it saves only the files that have changed, using rsync, and I continue on.

True, I've lost a few SD cards due to unexpected power off

@AndreasSpiess

2 жыл бұрын

Thank you for sharing your experience!

Very educational content. Just wanted to point out that I'm personally using High Endurance SD by SandDisk which meant for CCTV application (run 24/7 + consistent read & write application). So far works well (more over 8 month with multiple time of trips by thunder storm - in turn will be a sudden power loss) for my simple LAMP project without overlay file which is running 24/7 when ever possible. Maybe can give those a shoot for extensive test?

@AndreasSpiess

2 жыл бұрын

Other viewers also commented about those SD cards with different results. I do not plan such a test because I do no more use SD cards for heavy usage. I changed to SSDs, also because they are faster.

Thanks Andreas, I believe a short lifecycle is typical for low-cost SD. I use several SD cards in raspberries for 3 years already, incl my Majordomo home automation (decades of mqtt sensors, etc.) and had issues while using sandisk-like SDs). After changing to less cheap I've forgotten about it.

@AndreasSpiess

2 жыл бұрын

Thank you for sharing your experience. It seems you were lucky. At least I found mentioning Sandisk and other branded SD cards in the forums complaining about failures. Maybe the operating systems also got more robust against corruption of SD cards.

@paulcohen1555

2 жыл бұрын

Please let us know which are the BAD and the GOOD SD cards you used.

@alexanderskorokhodov9453

2 жыл бұрын

@@paulcohen1555 Sandisk were the worst, now Samsung, no issues yet. I can't remember particular models, sorry.

@keepsformat

2 жыл бұрын

@@paulcohen1555 I don't think it would be a healthy data because of the chip shortage. Even good SSD brands like samsung started to use less expensive less durable components. It's safe to assume that same tactics are deployed in the SDcard production.

@alexanderskorokhodov9453

2 жыл бұрын

Day of installation: # ls -ld /lost+found drwx------ 2 root root 16384 Nov 29 2017 /lost+found

Hi. Can I do this with twister OS? This OS is so beautiful. I ve seen a review but I want to follow your advice. If Twister OS has this set up for overlay, would be great.

@AndreasSpiess

2 жыл бұрын

I only use Raspbian:-(

Does the SSD have the same corruption issue as the SD cards resulting from power outages? I have corrupted Octoprint a few times from someone (me) cutting the power before shutting down. I am definitely going to try the Raspian Function. Thanks!

@AndreasSpiess

2 жыл бұрын

Corruption often comes from the file system (the computer cannot finish a transaction) and can also happen to the SSD.

I would like this, but is there some way to make certain write process still work, like apt-get upgrade and for my moonraker to keep Klipper and mainsail up to date? Perhaps this is the case, but I am afraid to turn it on and have issues.

@AndreasSpiess

2 жыл бұрын

As shown in the video, all changes are lost during reboot. But you can switch it it off, do your updates, and switch it on again. I would try it with a copy of your productive SD card.

Good video, no mention of emmc SD card adapters?

@AndreasSpiess

2 жыл бұрын

I would classify them as SSDs (price, form factor)

@gedgicat2063

2 жыл бұрын

@@AndreasSpiess eMMC SD keys or adapters are small and directly replace the micro SD card, much smaller than an SSD, although I expect the SSD will beat on performance

@AndreasSpiess

2 жыл бұрын

I did not know they are so small! I have to look at them

Add autofs against an NFS box with your homedir, we did diskless workstations that way for years back in the 90’s.

@AndreasSpiess

2 жыл бұрын

You are right. Remote file systems can be a good solution instead of an SSD if you need to keep the changing data.

I have a ton of raspberry pi computers (mostly 3b & zero W models) and I've done the 'barefoot' method before. I haven't tried the built-in raspbian feature yet but I wonder, does the feature disable write to other storage? If it doesn't, then you could redirect or save 'persistent' files and such to another device like a simple thumbdrive.

@AndreasSpiess

2 жыл бұрын

I assume it only blocks the SD card from writing, but I did not try. Other viewers also suggested to use network drives for valuable data.

I have few application that move files from 1 folder to another in my Pi. I assume that the overlay file system will revert everything back on restart? I getting a SSD for my 3B+ soon, which is a better solution for my situation (i think)

@AndreasSpiess

2 жыл бұрын

1. Yes, it will revert to exactly the initial state 2. An SSD is a good thing. However it will not be very fast on the USB2 of a Pi3B+

What would be the difference when leaving the boot partition on read/write while having overlay enabled?

@AndreasSpiess

2 жыл бұрын

I never thought about that :-(

I have the same SD-card running since my Raspi 3B+ was new. 2018? 19? Never got corrupted (headlesss lite raspian doing DNS and Cups). I still did that, thanks.

@AndreasSpiess

2 жыл бұрын

So you were lucky and got a good SD card, I think.

Does this also help in the event of a sudden power loss? I imagine as long as there's no writes occuring at the time of loss it could be okay?

@AndreasSpiess

11 ай бұрын

Yes. Because nothing is written directly to the disk (or SD card). So, your Pi always starts with a "fresh" image

In addition to these tips I would strongly recommend getting an SD card that's more suitable for this kind of usage. I've personally had good results with the WD purple SD cards, I've got 3 pi's that have been running 24/7 for about 3 years now on an unmodified raspbian installation with full logging enabled (and one of them is constantly writing due to a pihole installation). So far I haven't had any issues. But... I would recommend partitioning only a small part of the SD card (i.e. 25% or so) so the wear leveling algorithms can do their thing and make sure your SD card won't run out of working blocks.

@AndreasSpiess

2 жыл бұрын

Thank you for sharing your experience. Because other viewers also recommended these WD purple cards a few are in the mail by now. I hope they implemented wear levelling. The descriptions I found were not clear. Just that they monitor "bad cells".

@rickvanhattem1713

2 жыл бұрын

@@AndreasSpiess I've read on reddit that they are supposed to have wear leveling according to WD suport. Anecdotally however, I've had many SD cards fail before I switched. Even high quality ones

@Darkk6969

2 жыл бұрын

I do not believe having a small partition would make any difference to the wear leveling since it uses all the cells to write data regardless what file system you use. The idea is get the biggest card the device can support to extend the life out of it.

@WoLpH

2 жыл бұрын

@@Darkk6969 You're right, assuming you have TRIM enabled it won't make any difference in that aspect. But if you only partition a small part of the disk you leave the SD card with enough space to keep wear-leveling without any worries and you can't accidently fill up the disk with logs or other files. Given how cheap memory cards are and how much space you actually need, this has been a good tradeoff for me.

@Andrew-dp5kf

2 жыл бұрын

Sandisk do a range of rugged IoT targeted SD cards using more robust flash. Makes sense.

So, why isn't there an option to write the ram files back to sd on close? Thus dramatically reducing the writes

@CodeMonkeX

2 жыл бұрын

My guess is it’s too hard to guarantee that. If people expect their data to be saved when the pi reboots then they will be angry if 100’s of megs get erased when it looses power or crashes. If the system expects files to be written on shutdown then the file system would be corrupted on ever single unexpected reboot. It’s much safer to just assume all data is temporary in the RAM disk.

@wktodd

2 жыл бұрын

@@CodeMonkeX Yes I see if the rewrites were automatic, there could be a problem. My thoughts were for a Manual/programmatic option - a simple 'do you want to save' or cmdline save

@AndreasSpiess

2 жыл бұрын

I am sure a lot of things can be done. But, as a Linux noob I am happy to have a simple way. All my use cases can be appointed to one or the other scenario: No longterm storage needed --> SD card or longterm storage needed --> SSD

@jmr

2 жыл бұрын

Interesting question. That would be a nice option. I'm all about those SSD drives because of the speed. I only use SD cards for the portable builds like my hotspots.

@vlya1533

2 жыл бұрын

In some way a similar functionality can be achieved with "apt install log2ram". If I'm not mistaken "log2ram" is installed by default in armbian-os, on raspberry-os we can install it ourselves.

Perfect; I'll set the SD to RO, and write data to a NAS.

@AndreasSpiess

2 жыл бұрын

That should work and protect your installation

Hi Andreas, Last week I wanted to update a 2017 installed Raspbian Jessie on its 8 GB µSD card. Of course, I dumped the card beforehand ... or at least I planned to. Because dd reported many defective blocks. The dump ran for 48 hours because of the many retrys. Out of interest, I checked the file systems in the dump: were both okay. Obviously the mechanism that hides defective cells in the µSD card works very well. But I'm also sure that the card was on the verge of total failure ;-) vy 73 de Micha, DD0UL

@AndreasSpiess

2 жыл бұрын

Good that you were able to get the content. This was not always the case here... I stopped to use my 8GB cards when I saw how slow they are compared with the bigger cards, not because of unreliability. So I spent a few bucks.

Brilliant way to make a Live OS

@AndreasSpiess

2 жыл бұрын

True!

Just to be clear when I want to update the OS, I need to turn the overlay feature and RO off and then reboot, and update the system and then turn it back on and reboot again correct?

@AndreasSpiess

2 жыл бұрын

Yes. At least if you want to keep the changes.

I'm testing the overlay file system on 11 Raspberry Pi, running camera applications. Users were instructed to just unplug the power without powering off the system, so I could see if this was a viable solution. So far no issues with sd card corruption. 2 of the Raspberry Pi were running without interruption all day (9-18) for 3 month and they still work perfectly fine. I don't know if i'm lucky or if it works well, so take it with a grain of salt.

@AndreasSpiess

2 жыл бұрын

Thank you for sharing your test data. For me it looks like it works as designed!

It was interesting. I do have an original Pi 1 that has the operating system on an attached powered Hard Drive. Just uses SD card to boot.

@AndreasSpiess

2 жыл бұрын

I only started with the Pi3.... I tried once to attach a HDD but had to solve the power issue...

@ajward137

2 жыл бұрын

@@AndreasSpiess yes, definitely works. Actually, I use nfs mode to develop o/s image updates, then use squashfs to compress it and send it over the network and into ram from a server at boot time. Runs as normal, no sd card required. Needs 2gb ram minimum. I have 5 pi4s on my network doing the home automation (node red and home assistant) - I can't have part of my house dropping out because of a worn out sd card!

Not Pi related but Linux related. In the early days of SATA SSDs I had one at 16GB from Imation. I had the most 'write intensive' volumes mounted on an LSI RAID volume (e.g. /tmp /var/ and SWAP). It lasted over 6 years in 24/7 use before it was replaced by a new computer. I did pull the drive before sending the computer to freegeek. Those are the 'high write cycle' mounts for a system drive. Data drives are not addressed.

@AndreasSpiess

2 жыл бұрын

Thank you for sharing your experience. So far I also had no problems with SSDs, also not on the PC. Even if I do some video editing. For the Pi I do not expect any issues because the data volumes are very small (mostly sensor data and logging).

Hi Andreas, what are your thoughts about eMMC storage like the RasPiKey as a halfway house between SD cards and SSD storage?

@AndreasSpiess

2 жыл бұрын

I never used it. I would use it if it would be mounted on the board (instead of an SD card). Otherwise it makes no sense for me.

@bartjsmit

2 жыл бұрын

@@AndreasSpiess I think it is sort of mounted on the board, although admittedly not within its outline. I run a pair of Pi 4's with PoE+ hats that boot from SD. I tried to get them to netboot/pxe boot with partial success. If I can't fix that before the SD cards die, I'll try the RasPiKey.

@AndreasSpiess

2 жыл бұрын

Strange that network boot does not work. Other viewers suggested that as an alternative to my proposal. I never used it...

@bartjsmit

2 жыл бұрын

@@AndreasSpiess It does but it then runs the rancher labs k3s kubernetes systemd service and that won't start. I suspect that the NFS mount needs a user space component for that part. Eventually I'm hoping to move to k3os instead of RasPi OS. I appreciate that this is a bit off topic for storage reliability though.

The medium article instructions, while time consuming are better when it comes to usability With overlay fs, you need to reboot every time you need to make any changes and then reboot again to return

@AndreasSpiess

2 жыл бұрын

Yes, you need to reboot twice for changes.

Have you tried using high endurance SDCards sold for devices like dashcams that repeatedly overwrite the storage?

@AndreasSpiess

2 жыл бұрын

Not yet. But a few viewers suggested to use WD purple. So they are in the mail now ;-)

@68HC060

2 жыл бұрын

@@AndreasSpiess They should be a lot better, but they will also wear out at some point. This also depends a bit on the file system you're using. Unfortunately I don't think LittleFS can be used seriously with Linux yet. I know I stated that SSD would wear out and that harddisks would last longer. This also depends on the use case. If for instance your harddisk is placed in a -20° shed, then the SSD will probably last longer. Harddisks may die if turned off and put on the shelf for 15 years (SSD likewise). -So it all depends on things we know and things we don't know. :/

Nice hint I was not aware of! It's there a known command to write down all memory changes to disk aka SD? That would be great if you are experimenting and at the end think, ah I should have saved this. As I understood it, writing or not ALWAYS requires a reboot first, correct?

@AndreasSpiess

2 жыл бұрын

Log2RAM does something like that. It stores the data in memory and saves them every hour or so as well as before a proper shutdown.

@AndreasLink

2 жыл бұрын

@@AndreasSpiess awesome, that is what I'm interested in, I will look into it. Thank you!

a pxe boot with a tftp server hosting the image of the operating system ? or even , clients on a vnc environment that runs elsewhere and maps the hardware of the pi to that user profile .

@AndreasSpiess

2 жыл бұрын

Using network boot can also be a good possibility if you trust the network connection. I never used it.

Are any of these options available on the old Raspberry Pi model B boards (not Pi 3B but the original one)? Would it for instance be possible to boot from SD card and then re-mount to an SSD on USB? I'm not a unix expert, so commenters forgive me if my question is very stupid.

@AndreasSpiess

10 ай бұрын

The feature presented in this video is a software feature and should be available also for older models. But I do not own such boards anymore. So I do not know if there are restrictions.

@nkronert

10 ай бұрын

@@AndreasSpiess thank you kindly for your reply, Andreas.

after the loss of 2 SDcards, ive switched to using a USB Stick in my Pi, worked sofar without issues.

@AndreasSpiess

2 жыл бұрын

Thank you for sharing your experience.

How does Overlay File System in Read-Write mode works? As in RO mode but writes on SD Card before shutdown or restart? Or is just a temporary solution when updating and just passes the data through RAM to SD Card? Thank you

@AndreasSpiess

2 жыл бұрын

As shown in the video all changed data is lost when you reboot the Pi.

@anducampu

2 жыл бұрын

Thank you. Also for all the videos on your channel. I learned a lot of useful stuff.

The overlay system looks like a cool feature but I almost never use raspbian. Tbh, I've never had a problem with tweaking fstab but this is handy to know to recommend to others as it's simpler to explain.

@AndreasSpiess

2 жыл бұрын

If you use different distros I assume you are an experienced Linux user and do not need such a simple solution. For a Linux noob like me it is different ;-)

@23wjam

2 жыл бұрын

@@AndreasSpiess not sure I’d call myself experienced but I am comfortable with *nix. Like I said, didn’t know about this feature and I will recommend it to those that don’t really care about what’s under the bonnet. Thanks for sharing.

Do you have any data or experience on the reliability advantage of an ssd over an sd card? After all, both have a limited number of write cycles.

@AndreasSpiess

2 жыл бұрын

I do not have evidence. But SSDs are made for computer usage and implement wear levelling (According to the manufacturers). Most SD cards are not made for computer usage (if you trust the datasheets of the manufacturers) and did not implement wear levelling because it was not needed/beneficial. Now you get some high-endurance SD cards which seem to be useable in computer scenarios. One viewer reported that Sandisk refused to replace a defective SD card with this explanation.

@paulj4178

2 жыл бұрын

@@AndreasSpiess our need is 24x7 controlling a complex system with 40 relays and about 40 inputs. We also have about 200 operating state variables that have to be retained through a restart. We have a twin 15 farad super cap, mcu controlled ups on board, gives us 30 seconds safety. To simplify safe shutdown writes, we use a virtual drive space (memory) for sql lite db, all variables are stored in this db. So when we have a power issue (many of our customers do have frequent power issues), it is detected by interrupt, and the virtual db is saved to the sd card, and it code puts the pi into a tight do nothing loop, which is only broken by power coming back on, or the caps drain. This way sd card writes are minimized, and the pi will automatically restart after a power outage of more than 30 seconds.

is it possible to use the ssd on the usb 2.0 ports aswell? my 2 usb 3 ports are busy with the hdds i'm using for my nas already

@richards7909

2 жыл бұрын

As far as I'm aware, yes, but it would be slower. You *might* find putting the SSD on USB3 and HDDs on USB2 to be better but that would be dependent on what you're using the storage for, and how fast each storage device actually is. Be an interesting tinkering experiment if you're up to it :-)

@AndreasSpiess

2 жыл бұрын

It works on USB2. But extremely slow. Nothing to try ;-)

I have installed "remote-iot" to my pi3B/3B+ and am monitoring total writes -the only tool I found that would read microSD stats. I added "/dev/root / ext4 rw,noatime,commit=60 0 0" to mtab" to minimize writes. I have computed writes over many days using the "total writes" count, and it is about 10/sec. I am trying to figure how "many" this actually is...... Doesn't seem like much. Hard to translate this. Thanks for your video! Wish stretch/jessie had overlay FS.

@AndreasSpiess

2 жыл бұрын

I did not know of remote-iot. Its SD monitoring function is very interesting! The only problem: Nobody seem to know how many write cycles a typical SD card supports before it dies...

Awesome, so I guess no UPS will be needed if uptime is not desired? am I correct?

@AndreasSpiess

2 жыл бұрын

Yes, I think so.

@sshah876

2 жыл бұрын

@@AndreasSpiess excellent discovery , thanks and hats off to you sir!

I usually use a μSD card only to boot from and I have the storage on network with Ceph.

@AndreasSpiess

2 жыл бұрын

Also a good possibility if you know what you do...

I feel you should look at using a network drive so you do not need to use so much RAM. One simple way to make a network drive is to use an old desktop as a Linux box and have it share the hard drive to the raspberry pi.

@AndreasSpiess

2 жыл бұрын

You are right. Other viewers also mentioned remote logging as an opportunity. I assume it would need at least some Linux knowledge.

Hey: what's a LoraWAN gateway and what do you use it for?

@AndreasSpiess

2 жыл бұрын

Maybe you watch my LoRa introduction video?

@ErikUden

2 жыл бұрын

@@AndreasSpiess good idea!

WD line Purple of microsd cards for video security has a write cycle of up to 16tbw for a capacity of 32gb.

@AndreasSpiess

2 жыл бұрын

Thanks for the tip. I ordered a few of them ;-)

This is a discussion about two topics: 1. How flash works. 2. BC/DR. Lets solve 'two' first. If your data/system is critical make sure you take/keep a remote copy and keep it off site. End of story. 'Local resiliency' doesn't help you if your data center - or in this case raspberry pi - goes up in smoke. For 'one' let's talk about flash's 'peculiarities.' Flash devices - and by the way both memory cards and SSDs work the same way - have the following characteristics: 1. Fixed block size for writes. Unlike spinning disks flash can has a fixed write size. i.e. If a device writes at 32kb and your write is 4kb. The device writes 4kb of data and 28kb of zeros. If your write is 128kb the same device writes 4 × 32kb writes. 2. SLC, MLC, TLC. Denotes the number of bits in a cell. 1, 2 or 3 respectively. Additionally, it indicates 'burn rate' how quickly cells those cells die. The greater the number of bits in a cell the faster they wear out. Most consumer grade SSDs and flash cards are MLC. 3. Overprovisioning. Every consumer flash device SSDs and flash cards are overprovisioned to take wear and tear into consideration. SSDs have wayyyyyy more than Flash cards as they have to deal with the writes an OS throws at them. So don't buy an SSD buy a bigger memory card. It will last a lot longer, be easier to deal with and work out cheaper. 4. Fade. As magnetic media is hit by 'bit rot' flash gets hit by 'fade.' Over time the bits flip or the cells fail and the files become unreadable. Magnetic interference and heat accelerate this process so keep your pi cool. 4. Device firmware. Not all flash devices are created 'equal' and - no disrespect to the engineers who do this - the firmware on the a given generation of devices often gets written in a hurry. So the lifespans of two different cards with the same size advertised size and identical workloads may differ drastically in terms of lifespan. 5. Reads are free, writes are poison. High levels of random writes are absolute cancer for flash. Everything slows down if your device is bangoed with high write traffic. So if you are logging tons of data? My advice send it to a NAS backed by spinning disk DON'T store it locally on your memory card it will kill your flash device quickly. So in summary: 1. Keep a back up. 2. Buy a bigger memory card than you think you need. 3. Keep your PI cool. 4. For all random write traffic send it to spinning disk. 5. Check the product reviews for angry consumers complaining about 'failures within weeks of purchase.' How do I know this? I was employee 186 for the best enterprise flash array vendor on the market. As an aside - but related to this - 'middle out' from the 'Silicon Valley' series is based on a real algorithm that optimised wear and tear on MLC using meta data references to avoid writes if possible.

@AndreasSpiess

2 жыл бұрын

Thank you for your additional information. Everything is true. Do you have information about the wear-levelling of SD cards? I read a lot that they do not have such an algorithm and manufacturers explicitly forbid to use them in file system scenarios (and refuse warranty as one viewer wrote). Just a few additional thoughts about the use cases of the video: I talk about Raspberry Pi computers which are not often used in data centers nor do they have to deal with lots of data, because its performance is weak compared to servers. On the other hand its data often is not very valuable for the long run or does not change a lot. Many of our Pis just do some data transfers (sensors, gateways). or are used to drive displays, etc. and are sometimes operated in harsh situations. Fortunately, we will hardly hit any of the specified transfer volumes of SSDs. However, we all experienced crashes of SD cards. Sometimes also because the power was switched off during a bad time which corrupted the FS.

Does this only work with Raspberry Pi 4? I don't see the Performance options on my 3B+ which is running the latest OS.

@AndreasSpiess

2 жыл бұрын

I now tried it with a Pi3 (desktop version) and it was available as with the Pi4.

@BillYovino

2 жыл бұрын

@@AndreasSpiess It seems to be missing from the lite version. I’ll try again with the full desktop version. That opens up a lot of possibilities for automotive use.

@BillYovino

2 жыл бұрын

I had to update the firmware and then was able to get the the latest OS installed. I have it working the Pi I used for Fruitbox (jukebox software) and it works great! This should greatly prolong the life of the card. Thank you!