Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Upcoming how-to...RAMboot to a ramdisk for blazing speed!

Here you can discuss every aspect of Debian. Note: not for support requests!
Message
Author
User avatar
Aubrey_Snoddie
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#16 Post by Aubrey_Snoddie »

Perhaps Isaac has not read the questions in my last post...

User avatar
Pick2
Posts: 790
Joined: 2007-07-07 13:31
Location: Decatur Il

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#17 Post by Pick2 »

This thread had been dead since 10 Jun 2008 ,that's 4 3/4 years.
I would not get to impatient or get your hope set to high :D

User avatar
Aubrey_Snoddie
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#18 Post by Aubrey_Snoddie »

Strange, Isaac replied yesterday.

I woul have thought that with the normal man in the street being able to afford 16GB RAM nowadays, a RAMboot (persistent) would have aroused more interest.

Silly me.

User avatar
IsaacKuo
Posts: 316
Joined: 2008-04-24 20:06
Contact:

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#19 Post by IsaacKuo »

Sorry, I was a bit busy. I'm not sure exactly what you mean by "checkpointing". Periodic use of rsync could be good enough for what you want...but it sounds like you want to do periodic differential backups while running. I haven't experimented with it, but perhaps you could find what you're looking for with rsnapshot (which uses rsynce on the back end).

How about this:

1) Main OS image is normal partition on SSD.
2) Main OS is copied to tmpfs ramdisk on boot.
3) One or more "snapshot" images use aufs, attached to other partitions on SSD. (These can be smaller.) Alternatively, if you have plenty of space on the SSD, these can also be normal partitions--equal in size to the main OS image. I would recommend the latter, at least to get things going.
4) Periodically use rsync to backup tmpfs ramdisk to "snapshot" image.
5) On shutdown, rsync to copy tmpfs ramdisk back to main OS image.

I would recommend another backup image that you manually copy over periodically in case the main OS image gets messed up.
Isaac Kuo

User avatar
IsaacKuo
Posts: 316
Joined: 2008-04-24 20:06
Contact:

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#20 Post by IsaacKuo »

Thinking about it, I'd recommend skipping aufs and just using two or more normal partitions. The big disadvantage of aufs, besides the added complexity and inefficiency, is that if your main OS image gets messed up all aufs images based on it get messed up also. So you need another full size image anyway as a backup.

So...I'd just go with two or more normal partitions sized for your OS. The main one is what the boot copies from and rsyncs to on startup/shutdown. The secondary ones are the snapshots rsynced periodically while running. The danger of corruption of a secondary one is not such a big deal. It's really mostly the same effect as accidental loss of power, if something goes wrong in the middle of an rsync.
Isaac Kuo

User avatar
Aubrey_Snoddie
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#21 Post by Aubrey_Snoddie »

Isaac,

again, thank you for your time and the brain cells (if any) expended on thinking about my wacky idea.

I have spent 35 years working with IBM mainframes and am not a complete *nix novice but "copying my main OS partition into a tmpfs RAMdisk at boot" and everything else that that entails is beyond me.

Is there a cook book / howto on how to do same somewhere?

Would it be advisable to have a sandbox / test environment / chroot available to do all of this?

Thanks again.


Aubrey

User avatar
llivv
Posts: 5340
Joined: 2007-02-14 18:10
Location: cold storage

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#22 Post by llivv »

Aubrey_Snoddie wrote:Isaac,
I have spent 35 years working with IBM mainframes and am not a complete *nix novice but "copying my main OS partition into a tmpfs RAMdisk at boot" and everything else that that entails is beyond me.

Is there a cook book / howto on how to do same somewhere?
ibm should have a few,
Also, a short description of what you have done so far while implementing your " wacky idea"
would help the rest of us a lot in figgering out where to point you.
Aubrey_Snoddie wrote:Would it be advisable to have a sandbox / test environment / chroot available to do all of this?
in my mind that is kinda a redundant question because the ramboot build processs
kinda parallels all three to some degree.

original hint:
have you looked at the debian debirf package yet?

I don't know about anyone else, but

I'm eagerly awaiting the upcoming howto

and hoping for more stripping hints.
In memory of Ian Ashley Murdock (1973 - 2015) founder of the Debian project.


User avatar
Aubrey_Snoddie
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#24 Post by Aubrey_Snoddie »

In one sentence have memory speed when doing I/O or anything type of fetching, just like with the good ole Mainframe.

In layman's terms -

Boot to Grub, copy OS to memory, checkpoint (write changes to SSD) every 5 mins for example - run as normal / daily stuff etc. etc. At shutdown time write OS-in-memory back to SSD and power off - simple or not?

Have a gander at these figures -

root@Carcassonne ~: df -ah
Filesystem Size Used Avail Use% Mounted on
rootfs 12G 5.8G 5.1G 54% /
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
udev 10M 10M 0 100% /dev
devpts 0 0 0 - /dev/pts
tmpfs 1.6G 712K 1.6G 1% /run
/dev/disk/by-uuid/fd78685e-ea1b-4ad3-8583-0aff167de1fe 12G 5.8G 5.1G 54% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.1G 0 3.1G 0% /run/shm
tmpfs 9.0G 20K 9.0G 1% /tmp
tmpfs 7.8G 0 7.8G 0% /var/tmp
tmpfs 7.8G 340K 7.8G 1% /var/log
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
root@Carcassonne ~: dd if=/dev/zero of=/tmp/tt bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 1.97469 s, 4.4 GB/s
root@Carcassonne ~: df -ah
Filesystem Size Used Avail Use% Mounted on
rootfs 12G 5.8G 5.1G 54% /
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
udev 10M 10M 0 100% /dev
devpts 0 0 0 - /dev/pts
tmpfs 1.6G 712K 1.6G 1% /run
/dev/disk/by-uuid/fd78685e-ea1b-4ad3-8583-0aff167de1fe 12G 5.8G 5.1G 54% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.1G 0 3.1G 0% /run/shm
tmpfs 9.0G 8.1G 1.0G 89% /tmp
tmpfs 7.8G 0 7.8G 0% /var/tmp
tmpfs 7.8G 344K 7.8G 1% /var/log
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
root@Carcassonne ~: free -g
total used free shared buffers cached
Mem: 15 8 6 0 0 8
-/+ buffers/cache: 0 14
Swap: 0 0 0
root@Carcassonne ~:

As you can see I have a supposed throughput of 4.4 GB/sec

I must state here that with having a fast SSD (Samsung 840 Pro) I will probably never feel the "difference" between the relatively slow SSD transfers speeds compared to RAM-to-RAM.

I am running on a miniscule Thinkpad X230 with 16GB RAM, am not a gamer, overclocker or media weirdo. I have a few Iceweasel browser windows open, VPN perhaps also, otherwise no heavy load.

It would be nevertheless nice to see the memory (RAM) being used more efficiently though...

Pray tell what is "figgering"??


Aubrey

User avatar
Aubrey_Snoddie
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#25 Post by Aubrey_Snoddie »

I have had a look at debirf and was not impressed - too convoluted and not intuitive,

User avatar
Aubrey_Snoddie
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#26 Post by Aubrey_Snoddie »

Quote from Isaac "Thinking about it, I'd recommend skipping aufs and just using two or more normal partitions...."

I notice that my Debian 3.8.x needs AUFS(3) patches.

I received this message -

"mounting aufs on /root/ failed: no such device.." using this information I found on my google travels - http://mce.commsbyte.com/index.php/arti ... isk-how-to

In the great (Live) scheme of things, how does one substitute a "normal" (tmpfs?) partition instead of AUFS.

Perhaps Daniel Baumann is listening?

Would be most appreciative of any enlightening pointers / ideas.




Regards,


Aubrey

User avatar
IsaacKuo
Posts: 316
Joined: 2008-04-24 20:06
Contact:

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#27 Post by IsaacKuo »

AnInkedSoul wrote:is this the new and improved version?
http://forums.debian.net/viewtopic.php?f=16&t=29774
Ah, yes--that was indeed the new and improved version. Of course, I wrote that ages ago--but still, I never made any significant improvements over that.

If it's not clear how to use it, maybe it would be best to ask questions about it on the other thread.
Isaac Kuo

User avatar
IsaacKuo
Posts: 316
Joined: 2008-04-24 20:06
Contact:

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#28 Post by IsaacKuo »

Aubrey_Snoddie wrote:Quote from Isaac "Thinking about it, I'd recommend skipping aufs and just using two or more normal partitions...."

In the great (Live) scheme of things, how does one substitute a "normal" (tmpfs?) partition instead of AUFS.
I'm sorry for not being clear--the "normal partitions" I refer to would be normal hard drive partitions using whatever normal file system you prefer--like ext3. So, you have two or more ext3 partitions on your SSD. In RAM, you just have one tmpfs file system for the OS.
Isaac Kuo

User avatar
Aubrey_Snoddie
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#29 Post by Aubrey_Snoddie »

Thank you Isaac.

Seeing as the original article was written in 2008 before SSD and cheap memory became mainstream, is there any chance of an update to your masterpiece to reflect today's state of things?

I would like to achieve all of this without installing multiple OSes on the machine as I have enough space to copy / rsync things around...

Again, apologies for the bother.


Aubrey

User avatar
IsaacKuo
Posts: 316
Joined: 2008-04-24 20:06
Contact:

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#30 Post by IsaacKuo »

Aubrey_Snoddie wrote:Thank you Isaac.

Seeing as the original article was written in 2008 before SSD and cheap memory became mainstream, is there any chance of an update to your masterpiece to reflect today's state of things?
I'm actually using the same hardware I was using back then. My main computers are still the same Socket 754 Sempron builds. The motherboards are limited to 2gigs of RAM, so I really haven't had the chance to do much more.

In the years since, basic web browser requirements have ballooned, so 2gigs just isn't enough. I could turn off flash in the old days, and still have something usable for the vast majority of websites. But that's not possible with javascript. So, I eventually abandoned RAMboot in favor of normal installs, until I got more capable hardware. So far, I haven't.
I would like to achieve all of this without installing multiple OSes on the machine as I have enough space to copy / rsync things around...
I'm not sure what your goals are. If you want to have multiple "snapshots", they have to go somewhere. I don't see any point to putting them in RAM.

If you just want to periodically save changes back to the SSD, periodically running rsync to incrementally backup from the tmpfs OS partition to the SSD partition will work.
Again, apologies for the bother.
No problem! It's my pleasure to assist!
Isaac Kuo

User avatar
Aubrey_Snoddie
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#31 Post by Aubrey_Snoddie »

Isaac,

thanks for the speedy reply.

In your original documentation you mention copying to a directory called /snapstrip/ and creating a snapstrip.tar etc., in the /usr/share/initramfs-tools/scripts/local script, you do a "mkdir /ijkijk". How do the /snapstrip/ and /ijkijk directories get concatenated / merged together e.g. /ijkijk/snapshot/. ??

Here is what I have on disk at the moment -

root@Carcassonne ~: ls /bootie/ -als
total 3442288
4 drwxr-xr-x 3 root root 4096 Mar 17 16:40 .
4 drwxr-xr-x 27 root root 4096 Mar 17 16:30 ..
16 drwx------ 2 root root 16384 Mar 16 16:34 lost+found
3442264 -rw-r--r-- 1 root root 3524874240 Mar 17 16:36 snapstrip.tar


Regards,


Aubrey

User avatar
Aubrey_Snoddie
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#32 Post by Aubrey_Snoddie »

Done.

root@Carcassonne ~: free -g
total used free shared buffers cached
Mem: 15 5 9 0 0 4
-/+ buffers/cache: 0 14
Swap: 0 0 0
root@Carcassonne ~: df -ah
Filesystem Size Used Avail Use% Mounted on
rootfs 12G 4.8G 6.9G 41% /
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
udev 10M 0 10M 0% /dev
devpts 0 0 0 - /dev/pts
tmpfs 1.6G 708K 1.6G 1% /run
none 12G 4.8G 6.9G 41% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.1G 0 3.1G 0% /run/shm
tmpfs 14G 56K 14G 1% /tmp
tmpfs 7.8G 0 7.8G 0% /var/tmp
tmpfs 7.8G 352K 7.8G 1% /var/log
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
root@Carcassonne ~: mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=2012534,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1617580k,mode=755)
none on / type tmpfs (rw,relatime,size=12131840k)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3235140k)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,size=13749420k,mode=777)
tmpfs on /var/tmp type tmpfs (rw,noatime)
tmpfs on /var/log type tmpfs (rw,noatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
root@Carcassonne ~:

I tried to post a screenshot, but alas, the forum here had a problrm with that...

I will keep you posted on my progress.

Thank you all for your help.


Regards,


Aubrey

User avatar
IsaacKuo
Posts: 316
Joined: 2008-04-24 20:06
Contact:

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#33 Post by IsaacKuo »

Oh, I forgot to reply to your last update!

In your situation you should eliminate everything in the snapstrip script involving removing files (anything with the rm command).

The snapstrip script is described in Step 4 of my how-to: http://forums.debian.net/viewtopic.php?f=16&t=29774

The reason the files are put into a big tarball is because the initial busybox environment has a limited version of cp. It just can't handle a massive recursive cp call of the size and scale of copying over an entire OS--even when stripped down. In contrast, the tar command is available in busybox and it can handle humongous archives.
Isaac Kuo

GlenWalker
Posts: 2
Joined: 2013-04-18 21:15
Location: UK

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#34 Post by GlenWalker »

Hello everyone!

I've been using Debian for years but only just registered on here...to join in this discussion!

For a few months now I've been mulling over creating a virtual machine (ever since I found out about VGA passthrough using Xen) so I can install Windows inside Debian and dust off some of my old games that refuse to work in WINE. Since RAM is relatively inexpensive at the moment (and I need to build a new computer capable of doing IOMMU for passthrough to work) I am hoping to get 16GB (or possibly even 32GB if I'm feeling greedy) and I thought it would be really cool if I could load the host operating system (dom0 in Xen) all into RAM so it would run at the fastest possible speed.

On top of the host I would have a Windows virtual machine and possibly a BSD/other Linux virtual machine to play around with. Since the virtual machines would have their own file system (on their own drives) it would get around the issue of persistence.

Curious to see what you all think about that? Would having the Xen dom0 running in RAM actually make a difference to the performance of the virtual machines? Obviously they'll still be reading from and writing to an SSD but the virtual side of things such as the network interface and USB/PCI cards all goes through the host OS doesn't it? Or am I confused...

(oh and Isaac I feel for you and your old hardware...I have a bunch of stuff hanging around (especially if I build a new PC!) so please PM me if you are in the UK and I'd be happy to send you some of it)
This is for work: http://www.technicalwriting.org.uk
And this is for fun: http://www.bluepenguin.me :--)

DebbyIan
Posts: 158
Joined: 2013-05-09 12:12

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

#35 Post by DebbyIan »

Way to go.
What once seemed an academic exercise now seems quite a plausible avenue for the industry to pursue in the foreseeable future. I can readily see a serious use and benfits of a RAM disk for the entire OS by using it with a Solid State Disk/Storage.

Post Reply