Page 1 of 2

Tmpfs disk savers, from ~/.profile

Posted: 2024-09-05 17:41
by CwF
In ~/.profile I have expanded the past thumbnail tmpfs to the entire ~/.cache, still on watch.

Code: Select all

#create tmpfs disk savers
systemd-mount -t tmpfs -o size=4G dram ~/Downloads
systemd-mount -t tmpfs -o size=2G ramdesk ~/Desktop 
systemd-mount -t tmpfs -o size=2G ramcache ~/.cache

Code: Select all

$  df
Filesystem     1K-blocks    Used Available Use% Mounted on
udev             1920964       0   1920964   0% /dev
tmpfs             388016     628    387388   1% /run
/dev/vda1        7894752 2653004   4815764  36% /
tmpfs            1940068       0   1940068   0% /dev/shm
tmpfs               5120       8      5112   1% /run/lock
tmpfs             376132      36    376096   1% /run/user/1001
ramdesk          1048576       0   1048576   0% /home/user/Desktop
ramcache         1048576  226852    821724  22% /home/user/.cache
The directory declared for use should be empty. So add the profile info, delete ~/.cache, reboot. To be sure you could do so with root in a tty while the user isn't logged in, systemd-mount will create the mount point if it doesn't exist.

Note, sizes are negotiable. They can add up to more than you have and no they don't use memory if not used. Dumping the cache on reboot can effect 'continuity'. Programs do benefit from the last session cache, it's really a incoming data consideration - do you want to wait, or pay to reload previously cached data. It's your call. Some may not like looking at generic favicons until updated. Some things that cache maps or /grits or audio/video temps and the like may be able to redirect their cache. For the most part, if something in cache is critical to the next boots operation we should consider it a bug and file on the offending package.

If we notice the example has a vda1 and not a sda1, we get into my main purpose of avoiding qcow2 size creep. Since copy-on-write can grow when overwriting the same info again and again, and deleted or unused data gives a new dimension to 'cruft'. This also is a 'backing' file where a second layer is used in numerous instances. Without the continuous cache cruft growing the 2nd layer qcow2, the file stays tidy - MB's instead of GB's. It's small enough for a temp vm to fit into a tmpfs on the host, say ~/Public/ram where the backing layer, and persistent 2nd layers are on a disk mounted in ~/Public/.images, both linked into the default /var/lib/libvirt/images for simplicity, and yet another discussion.

In other post I have mentioned the same ~/.profile method for additional devices, perhaps at ~/Music or ~/Videos. I have mentioned my TV DVR uses ~/Videos/vram as an absurdly large tmpfs. Also, I have an update not written out for the usb mounting...to be seen here eventually
https://forums.debian.net/viewtopic.php?t=155840

User permission needs set up for systemd-mount,
for my preferred polkit methods on a personal machine, see here
https://forums.debian.net/viewtopic.php ... it#p795066

I should track these older post better!
I use ~/.profile systemd-mount for most mounts of any kind. Once executed in profile they are active until shutdown. Any logoff/logon cycle ignores absent devices and existing declarations. All prior contents persist. Any 'ssh -X user@' will also safely run the sequence in that user profile.

Enjoy

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-06 05:49
by DebianFox
Great post. Thanks @CwF. A few doubts though. Can you please clarify these

1) You have put the creation of these tmpfs file systems in the file .profile. This will work when the user logs in for the first time. What happens when the user logs out or the system is shutdown and rebooted? Where would you put the unmount command?
2) You said the following in the above post. What is meant by that?
Dumping the cache on reboot can effect 'continuity'.
3) Many applications use .cache as a persistence directory. For example Firefox stores its profile under .cache. So does Thunderbird. Many File Managers also use .cache to store the configuration. Also many Desktop Environments use the .cache directory for storing configuration. So putting .cache in tmpfs will result in a loss of all the settings and cache. For example in case of Firefox the entire set of extensions, add-ons, settings along with the cache will be lost. How do you deal with this?

It would have been so better if all the applications were to use .profile or .config or .local directory to store the settings, configuration,add ons,etc. And only use .cache for its intended purpose like keeping browser cache or application cache which can be deleted. Maybe like the scenario that /tmp is cleaned/emptied at every boot/restart/shutdown, the .cache directory is similarily emptied at every boot/restart/shutdown.

Just for my reference, what Desktop environment are you using on your Debian? And are you on Debian 12.7 with Kernel release 6.1.0-25?

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-06 09:20
by sunrat
DebianFox wrote: 2024-09-06 05:49...
3) Many applications use .cache as a persistence directory. For example Firefox stores its profile under .cache. So does Thunderbird. Many File Managers also use .cache to store the configuration. Also many Desktop Environments use the .cache directory for storing configuration. So putting .cache in tmpfs will result in a loss of all the settings and cache. For example in case of Firefox the entire set of extensions, add-ons, settings along with the cache will be lost. How do you deal with this?
Firefox stores its profile in ~/.mozilla and Thunderbird in ~/.thunderbird. cache stores site data but browser settings and add-ons remain in the profile.

I wrote a how-to which will save a lots of disk writes:
[HowTo] Move browser profile and cache to RAM

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-06 18:08
by CwF
DebianFox wrote: 2024-09-06 05:49 Where would you put the unmount command?
You don't. It is handled by systemd. Mentioned below, drop that 'n'! it's umount!
You can manually 'systemd-umount' or 'systemctl stop'.
systemd gracefully shuts down all types of mounts and I have thousands of accumulated hours with no issues. I do use --automount=yes for physical device mounts to avoid unclean dismounts. There is some testing still ongoing for that with no issues yet. Technically there is no such thing as an improper unmount of a tmpfs, it doesn't care. I will mention though that it is possible to orphan memory when an open file handle in tmpfs is deleted while a program has the file open. We can remove and recreate the tmpfs in that case, or reboot. Again, the DVR has been up for a few years now, it seems safe enough...
3) Many applications use .cache as a persistence directory.
No, they don't.
if something in cache is critical to the next boots operation we should consider it a bug and file on the offending package.
I hold fast, that's a bug. @sunrat clarified it. Passwords are not in .cache, all configuration is in ~/.mozilla. Personally, it is a bug that it is not ~/.config/mozilla. I don't use thunderbird - so there's some homework for you. By 'continuity' I mean appearance, not function. Think of something like tumbler that needs to read and create all those little icons again - an annoying pause maybe but not a show stopper.
Example - notice the date
https://bugs.debian.org/cgi-bin/bugrepo ... bug=747850

Again, I have years of practice of transfering (migrating) only a chosen slice of user home when a new backing is introduced and have never included ~/.cache. I suppose that is the origin of this realization to wipe it on every boot. It is doing exactly what I wanted btw, minimizing qcow2 file size creep. On bare metal it will save some wear.

Most 'dot' files are integrated into the backing layer and not backed up. This practice is a form of 'immutable' without being immutable. the layer2 qcow2 can be mangled, the backing cannot. The equivelant on a bare metal system is similar, only a small slice of home is backed up and injected when a new image is written to the disk. I 'forwardup' instead of backup the OS. All systems have an up to date vm ready to take over a running system, after the short time it takes to image to disk. All xdg directories (Documents, Music, Videos, etc) are separately maintained and often a profile mount. Mostly a future thing for bare metal once virtfs block drivers are matured - which is now and I'm late...

An example teaser - disk_ext4_music_0000_run.qcow2 to be served up and mounted in ~/Music by ~/.profile.
Backed up offline as disk_ext4_music_2403_archive.qcow2, no change since week 3 of 2024. I have yet to find a flaw in storing large qcow2's as a backup method. And yes there is more than one copy.

Someday. I mention this tangent info since it is ~/profile systemd-mount that ties it all together. Despite the complexity of the back end, the user foreground config is very simplified.
Just for my reference, what Desktop environment are you using on your Debian? And are you on Debian 12.7 with Kernel release 6.1.0-25?
Yes, I should mention. I'm always talking about Xfce, and only Xfce. I do ask for and appreciate when others test my wild ideas under alternative desktops, but I do not do so myself. Also I'm biased towards desktops and up with ample memory. I'll also invite anyone to test smb, nfs, or other uses for ~/.profile. I think I've hinted at the request a few times, nobody has taken the challenge. I no longer use them preferring temporary jit access (just in time).
History;
Actually this goes back to Stretch, all kernel and Debian versions since. Elements of implementation have matured continuously and I've backtracked to earlier versions mostly for curiosity, but also to demonstrate (to me) how slow we change habits - and how long this functionality has been baked in. On these forums I think I have mentioned tidbits about polkit use for example, which is important here, that were not included until Buster or Bullseye and still absent for some DE's. Some examples are the policy files that declare an action need to be included with the appropriate package. Or missing agents when password entry is desired. For example, /usr/share/polkit-1/actions/org.xfce.xfce4-terminal.policy is still not included. That is from 2018 and in all my images since Stretch. One reason is fear, beginners do stupid things. A better reason is 'conflation factors' that we don't find immediately, or near term, or sometimes for years. So I do stupid things in vms where it's safe, mostly. I try my best to not mention certain methods until there is a history of known good without complications. The mentioned unmount is an example that was not present until later, requiring 'systemctl stop' instead of systemd-umount. Hard to believe the missing 'n' was preserved... Details run deep!

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-10 05:00
by DebianFox
sunrat wrote: 2024-09-06 09:20
DebianFox wrote: 2024-09-06 05:49...
3) Many applications use .cache as a persistence directory. For example Firefox stores its profile under .cache. So does Thunderbird. Many File Managers also use .cache to store the configuration. Also many Desktop Environments use the .cache directory for storing configuration. So putting .cache in tmpfs will result in a loss of all the settings and cache. For example in case of Firefox the entire set of extensions, add-ons, settings along with the cache will be lost. How do you deal with this?
Firefox stores its profile in ~/.mozilla and Thunderbird in ~/.thunderbird. cache stores site data but browser settings and add-ons remain in the profile.

I wrote a how-to which will save a lots of disk writes:
[HowTo] Move browser profile and cache to RAM
Thanks @sunrat for the post. It helps. I have a further question on your post. I have put those over there. If possible do have a look at it.

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-10 05:26
by DebianFox
@CwF how about creating a link in .profile file which points the $HOME/.cache directory to say /run/user/$USERID/.cache? Will that work too?

Here $USERID has to be replaced with the output of the command

Code: Select all

$ id -u
$
So the command to be entered in the $HOME/.profile file becomes

Code: Select all

ln -f --symbolic $HOME/.cache /run/user/`id -u`/.cache
Symbolic has been chosen because /run in on a separate file system as compared to the file system where $HOME typically resides.

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-10 15:45
by yvs222
CwF wrote: 2024-09-05 17:41 In ~/.profile I have expanded the past thumbnail tmpfs to the entire ~/.cache, still on watch.
...
User permission needs set up for systemd-mount,
Good approach,
beside of user permissions: is ~/.profile guaranteed to be executed for most widely used login shells, and in both xsession and wayland-session?

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-10 16:19
by CwF
yvs222 wrote: 2024-09-10 15:45 is ~/.profile guaranteed to be executed for most widely used login shells, and in both xsession and wayland-session?
Good question! This -->
CwF wrote: 2024-09-06 18:08 Yes, I should mention. I'm always talking about Xfce, and only Xfce. I do ask for and appreciate when others test my wild ideas under alternative desktops, but I do not do so myself.
I'll also note I don't use a login manager but what I call 'autologin by-tty'. This also utilizes ~/.profile on the last line to startx. I'll need to find a link...
@yvs222
https://forums.debian.net/viewtopic.php ... y2#p796656

Otherwise I assume most methods read and use ~/.profile at start. I did mention that 'ssh -X' will read the profile also. Wayland is nowhere on my radar.
DebianFox wrote: 2024-09-10 05:26 @CwF how about creating a link in .profile file which points the $HOME/.cache directory to say /run/user/$USERID/.cache? Will that work too?
Perhaps, you should try it and see how it works!
I don't think that is as graceful of a solution.

Using run/user is nice for programs, I think there was a thread asking about that, yours?
I commented that I use it in my tickle programs. I've actually considered revising that to use ~/.cache. But I don't always use tmpfs for ~/.cache on a system using those programs, so a stalemate. I consider things like this on a Debian time scale, slightly quicker than geologic. When it doesn't matter much, I wait for ‘compelling’. Browsers causing qcow2 creep was compelling.

On a hypervisor there are virt-manager, libvirt, and qemu, and some others in ~/.cache that are both tidy and forensically useful so I do not use a tmpfs for all, only /thumbnails. Those cases use my herder.tk which uses /run/user/$(id -u)/herder for slop which could go in ~/.cache. Without a browser of any kind on those system to clog things up, I think I'll leave it the way it is.

Remember to weigh options for tangible gain, not simply trickery.

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-11 09:48
by DebianFox
CwF wrote: 2024-09-10 16:19
DebianFox wrote: 2024-09-10 05:26 @CwF how about creating a link in .profile file which points the $HOME/.cache directory to say /run/user/$USERID/.cache? Will that work too?
Perhaps, you should try it and see how it works!
I don't think that is as graceful of a solution.
I will try it out. @CwF why do you think that it is not a graceful solution? Any hints would be welcome.
CwF wrote: 2024-09-10 16:19 Using run/user is nice for programs, I think there was a thread asking about that, yours?
I am not sure which thread you are referring to over here. If you can paste the link over here I will confirm. Is it the Have TMPDIR/TMP environment variables been deprecated thread?

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-11 11:53
by yvs222
Besides systemd-mount and symlinking via .profile, there's also another option with log2ram deb package and its scripts.
I think that option can be used to put ~/.cache content into RAM too (and periodically, relatively very rare, to sync it to disk).
With log and user cache in its config (PATH_DISK="/var/log;/home/yvs/.cache") it looks for example like

Code: Select all

% df -h | grep 2ram
log2ram         500M  187M  314M  38% /var/log
log2ram         500M  287M  214M  58% /home/yvs/.cache
I haven't used this package for user directories before, so that cannot say pros and cons with this option for cache (or side effects if they exist).

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-11 11:56
by DebianFox
log2ram is a new filesystem that this package uses?

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-11 12:16
by yvs222
DebianFox wrote: 2024-09-11 11:56 log2ram is a new filesystem that this package uses?
if I've not missed something, it's tmpfs
https://github.com/azlux/log2ram/blob/m ... g2ram#L140

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-11 18:21
by CwF
DebianFox wrote: 2024-09-11 09:48 why do you think that it is not a graceful solution?
I refer to how it fits into the bigger subject of systemd mounting as simply one of the options. A link method would be specific to the single purpose. With this systemd technique all the various declarations of how/what/where are delivered in a single common format. So the tmpfs in one section, mounts and automounts in another grouping, all with a common syntax.
DebianFox wrote: 2024-09-11 09:48 Is it the Have TMPDIR/TMP environment variables been deprecated thread?
Ya, that's what I was thinking of. I'd use /run/user only for things that are not already in /home and never need manually browsed with a gui FM. Minor points really.
yvs222 wrote: 2024-09-11 11:53 there's also another option with log2ram
Thanks for the alternative mention. I think journal already solves logging with configuration options that make log2ram redundant on a systemd system.

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-11 21:46
by yvs222
CwF wrote: 2024-09-11 18:21
yvs222 wrote: 2024-09-11 11:53 there's also another option with log2ram
I think journal already solves logging with configuration options that make log2ram redundant on a systemd system.
Why not, there's still a lot of systemd-journald's IO write operations indicated by 'iotop -Pao' unless volatile storage settings used,
with storage=volatile logs aren't synchronized to disk (if I got it correctly from its manual) like with log2ram.

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-12 00:15
by CwF
yvs222 wrote: 2024-09-11 21:46 misreference a bit (not mine)
Weird, sorry about that, didn't mean to, odd. I'll fix it.
yvs222 wrote: 2024-09-11 21:46 with storage=volatile logs aren't synchronized to disk
Good point. I think there is a flush command to write to disk when the journal is not persistent. I haven't looked at that lately.

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-13 07:25
by DebianFox
yvs222 wrote: 2024-09-11 12:16
DebianFox wrote: 2024-09-11 11:56 log2ram is a new filesystem that this package uses?
if I've not missed something, it's tmpfs
https://github.com/azlux/log2ram/blob/m ... g2ram#L140
@yvs222 this looks promising. Can we schedule log2ram such that at shutdown/restart the logs which are stores in RAM are actually flushed to the disk?
Also this looks dicey scenario for the cases where there is a catastrophic shutdown/reboot, for example power failure or some other scenario.

Have you used it? Can you please share your feedback?

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-13 07:38
by DebianFox
DebianFox wrote: 2024-09-11 09:48
CwF wrote: 2024-09-10 16:19
DebianFox wrote: 2024-09-10 05:26 @CwF how about creating a link in .profile file which points the $HOME/.cache directory to say /run/user/$USERID/.cache? Will that work too?
Perhaps, you should try it and see how it works!
I don't think that is as graceful of a solution.
I will try it out. @CwF why do you think that it is not a graceful solution? Any hints would be welcome.
@CwF this was done. Now $HOME/.cache is a syslink to /run/user in RAM. So all the applications are using RAM as a cache. Till now have not faced any issues. Hope this helps in preventing unnecessary writes on the disks.

Currently /run/user is limited to 2-4 GB would that be an issue? Also what would happen if a big compressed/zip file were to be uncompressed? Would it use .cache directory?

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-13 17:39
by yvs222
DebianFox wrote: 2024-09-13 07:25
yvs222 wrote: 2024-09-11 12:16 log2ram
@yvs222 this looks promising. Can we schedule log2ram such that at shutdown/restart the logs which are stores in RAM are actually flushed to the disk?
it's scheduled with systemd's oncalendar/cron, by default once a day (maybe for cache it could be more frequently)
Also this looks dicey scenario for the cases where there is a catastrophic shutdown/reboot, for example power failure or some other scenario.
yes, so it is, as that said on their page "The script will also make this copy of RAM to disk in case of machine shutdowns (but, of course, it still won't do it in case of power failures)."
Have you used it? Can you please share your feedback?
I have, for several years but not for critical resources (it's used for /var/log data which logs aren't critical for me on desktops and on rpi devices).
~/.cache was added just a couple days ago on home PC, there's one unknown x to me: estimated maxsize for browsers' cache there (in case of systemd-journald data it's evaluated with help of its SystemMaxUse setting)

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-13 19:35
by CwF
DebianFox wrote: 2024-09-13 07:38 Currently /run/user is limited to 2-4 GB would that be an issue? Also what would happen if a big compressed/zip file were to be uncompressed? Would it use .cache directory?
That's why I prefer the graceful individually declared profile method, you set the quota. And No, unzip location is under your control.

Re: Tmpfs disk savers, from ~/.profile

Posted: 2024-09-16 03:53
by DebianFox
Is there a way to increase the size of /run/user or /run to say 2 GB or 3GB?