Upcoming how-to...RAMboot to a ramdisk for blazing speed!

[Aubrey_Snoddie]

Isaac,

again, thank you for your time and the brain cells (if any) expended on thinking about my wacky idea.

I have spent 35 years working with IBM mainframes and am not a complete *nix novice but "copying my main OS partition into a tmpfs RAMdisk at boot" and everything else that that entails is beyond me.

Is there a cook book / howto on how to do same somewhere?

Would it be advisable to have a sandbox / test environment / chroot available to do all of this?

Thanks again.


Aubrey

[llivv]

Isaac,
I have spent 35 years working with IBM mainframes and am not a complete *nix novice but "copying my main OS partition into a tmpfs RAMdisk at boot" and everything else that that entails is beyond me.

Is there a cook book / howto on how to do same somewhere?

ibm should have a few,
Also, a short description of what you have done so far while implementing your " wacky idea"
would help the rest of us a lot in figgering out where to point you.

Would it be advisable to have a sandbox / test environment / chroot available to do all of this?

in my mind that is kinda a redundant question because the ramboot build processs
kinda parallels all three to some degree.

original hint:
have you looked at the debian debirf package yet?

I don't know about anyone else, but

I'm eagerly awaiting the upcoming howto

and hoping for more stripping hints.

[AnInkedSoul]

is this the new and improved version?
http://forums.debian.net/viewtopic.php?f=16&t=29774

[Aubrey_Snoddie]

In one sentence have memory speed when doing I/O or anything type of fetching, just like with the good ole Mainframe.

In layman's terms -

Boot to Grub, copy OS to memory, checkpoint (write changes to SSD) every 5 mins for example - run as normal / daily stuff etc. etc. At shutdown time write OS-in-memory back to SSD and power off - simple or not?

Have a gander at these figures -

root@Carcassonne ~: df -ah
Filesystem Size Used Avail Use% Mounted on
rootfs 12G 5.8G 5.1G 54% /
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
udev 10M 10M 0 100% /dev
devpts 0 0 0 - /dev/pts
tmpfs 1.6G 712K 1.6G 1% /run
/dev/disk/by-uuid/fd78685e-ea1b-4ad3-8583-0aff167de1fe 12G 5.8G 5.1G 54% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.1G 0 3.1G 0% /run/shm
tmpfs 9.0G 20K 9.0G 1% /tmp
tmpfs 7.8G 0 7.8G 0% /var/tmp
tmpfs 7.8G 340K 7.8G 1% /var/log
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
root@Carcassonne ~: dd if=/dev/zero of=/tmp/tt bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 1.97469 s, 4.4 GB/s
root@Carcassonne ~: df -ah
Filesystem Size Used Avail Use% Mounted on
rootfs 12G 5.8G 5.1G 54% /
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
udev 10M 10M 0 100% /dev
devpts 0 0 0 - /dev/pts
tmpfs 1.6G 712K 1.6G 1% /run
/dev/disk/by-uuid/fd78685e-ea1b-4ad3-8583-0aff167de1fe 12G 5.8G 5.1G 54% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.1G 0 3.1G 0% /run/shm
tmpfs 9.0G 8.1G 1.0G 89% /tmp
tmpfs 7.8G 0 7.8G 0% /var/tmp
tmpfs 7.8G 344K 7.8G 1% /var/log
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
root@Carcassonne ~: free -g
total used free shared buffers cached
Mem: 15 8 6 0 0 8
-/+ buffers/cache: 0 14
Swap: 0 0 0
root@Carcassonne ~:

As you can see I have a supposed throughput of 4.4 GB/sec

I must state here that with having a fast SSD (Samsung 840 Pro) I will probably never feel the "difference" between the relatively slow SSD transfers speeds compared to RAM-to-RAM.

I am running on a miniscule Thinkpad X230 with 16GB RAM, am not a gamer, overclocker or media weirdo. I have a few Iceweasel browser windows open, VPN perhaps also, otherwise no heavy load.

It would be nevertheless nice to see the memory (RAM) being used more efficiently though...

Pray tell what is "figgering"??


Aubrey

[Aubrey_Snoddie]

I have had a look at debirf and was not impressed - too convoluted and not intuitive,

[Aubrey_Snoddie]

Quote from Isaac "Thinking about it, I'd recommend skipping aufs and just using two or more normal partitions...."

I notice that my Debian 3.8.x needs AUFS(3) patches.

I received this message -

"mounting aufs on /root/ failed: no such device.." using this information I found on my google travels - http://mce.commsbyte.com/index.php/arti ... isk-how-to

In the great (Live) scheme of things, how does one substitute a "normal" (tmpfs?) partition instead of AUFS.

Perhaps Daniel Baumann is listening?

Would be most appreciative of any enlightening pointers / ideas.




Regards,


Aubrey

[IsaacKuo]

is this the new and improved version?
http://forums.debian.net/viewtopic.php?f=16&t=29774

Ah, yes--that was indeed the new and improved version. Of course, I wrote that ages ago--but still, I never made any significant improvements over that.

If it's not clear how to use it, maybe it would be best to ask questions about it on the other thread.

[IsaacKuo]

Quote from Isaac "Thinking about it, I'd recommend skipping aufs and just using two or more normal partitions...."

In the great (Live) scheme of things, how does one substitute a "normal" (tmpfs?) partition instead of AUFS.

I'm sorry for not being clear--the "normal partitions" I refer to would be normal hard drive partitions using whatever normal file system you prefer--like ext3. So, you have two or more ext3 partitions on your SSD. In RAM, you just have one tmpfs file system for the OS.

[Aubrey_Snoddie]

Thank you Isaac.

Seeing as the original article was written in 2008 before SSD and cheap memory became mainstream, is there any chance of an update to your masterpiece to reflect today's state of things?

I would like to achieve all of this without installing multiple OSes on the machine as I have enough space to copy / rsync things around...

Again, apologies for the bother.


Aubrey

[IsaacKuo]

Thank you Isaac.

Seeing as the original article was written in 2008 before SSD and cheap memory became mainstream, is there any chance of an update to your masterpiece to reflect today's state of things?

I'm actually using the same hardware I was using back then. My main computers are still the same Socket 754 Sempron builds. The motherboards are limited to 2gigs of RAM, so I really haven't had the chance to do much more.

In the years since, basic web browser requirements have ballooned, so 2gigs just isn't enough. I could turn off flash in the old days, and still have something usable for the vast majority of websites. But that's not possible with javascript. So, I eventually abandoned RAMboot in favor of normal installs, until I got more capable hardware. So far, I haven't.

I would like to achieve all of this without installing multiple OSes on the machine as I have enough space to copy / rsync things around...

I'm not sure what your goals are. If you want to have multiple "snapshots", they have to go somewhere. I don't see any point to putting them in RAM.

If you just want to periodically save changes back to the SSD, periodically running rsync to incrementally backup from the tmpfs OS partition to the SSD partition will work.

Again, apologies for the bother.

No problem! It's my pleasure to assist!

[Aubrey_Snoddie]

Isaac,

thanks for the speedy reply.

In your original documentation you mention copying to a directory called /snapstrip/ and creating a snapstrip.tar etc., in the /usr/share/initramfs-tools/scripts/local script, you do a "mkdir /ijkijk". How do the /snapstrip/ and /ijkijk directories get concatenated / merged together e.g. /ijkijk/snapshot/. ??

Here is what I have on disk at the moment -

root@Carcassonne ~: ls /bootie/ -als
total 3442288
4 drwxr-xr-x 3 root root 4096 Mar 17 16:40 .
4 drwxr-xr-x 27 root root 4096 Mar 17 16:30 ..
16 drwx------ 2 root root 16384 Mar 16 16:34 lost+found
3442264 -rw-r--r-- 1 root root 3524874240 Mar 17 16:36 snapstrip.tar


Regards,


Aubrey

[Aubrey_Snoddie]

Done.

root@Carcassonne ~: free -g
total used free shared buffers cached
Mem: 15 5 9 0 0 4
-/+ buffers/cache: 0 14
Swap: 0 0 0
root@Carcassonne ~: df -ah
Filesystem Size Used Avail Use% Mounted on
rootfs 12G 4.8G 6.9G 41% /
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
udev 10M 0 10M 0% /dev
devpts 0 0 0 - /dev/pts
tmpfs 1.6G 708K 1.6G 1% /run
none 12G 4.8G 6.9G 41% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.1G 0 3.1G 0% /run/shm
tmpfs 14G 56K 14G 1% /tmp
tmpfs 7.8G 0 7.8G 0% /var/tmp
tmpfs 7.8G 352K 7.8G 1% /var/log
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
root@Carcassonne ~: mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=2012534,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1617580k,mode=755)
none on / type tmpfs (rw,relatime,size=12131840k)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3235140k)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,size=13749420k,mode=777)
tmpfs on /var/tmp type tmpfs (rw,noatime)
tmpfs on /var/log type tmpfs (rw,noatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
root@Carcassonne ~:

I tried to post a screenshot, but alas, the forum here had a problrm with that...

I will keep you posted on my progress.

Thank you all for your help.


Regards,


Aubrey

[IsaacKuo]

Oh, I forgot to reply to your last update!

In your situation you should eliminate everything in the snapstrip script involving removing files (anything with the rm command).

The snapstrip script is described in Step 4 of my how-to: http://forums.debian.net/viewtopic.php?f=16&t=29774

The reason the files are put into a big tarball is because the initial busybox environment has a limited version of cp. It just can't handle a massive recursive cp call of the size and scale of copying over an entire OS--even when stripped down. In contrast, the tar command is available in busybox and it can handle humongous archives.

[GlenWalker]

Hello everyone!

I've been using Debian for years but only just registered on here...to join in this discussion!

For a few months now I've been mulling over creating a virtual machine (ever since I found out about VGA passthrough using Xen) so I can install Windows inside Debian and dust off some of my old games that refuse to work in WINE. Since RAM is relatively inexpensive at the moment (and I need to build a new computer capable of doing IOMMU for passthrough to work) I am hoping to get 16GB (or possibly even 32GB if I'm feeling greedy) and I thought it would be really cool if I could load the host operating system (dom0 in Xen) all into RAM so it would run at the fastest possible speed.

On top of the host I would have a Windows virtual machine and possibly a BSD/other Linux virtual machine to play around with. Since the virtual machines would have their own file system (on their own drives) it would get around the issue of persistence.

Curious to see what you all think about that? Would having the Xen dom0 running in RAM actually make a difference to the performance of the virtual machines? Obviously they'll still be reading from and writing to an SSD but the virtual side of things such as the network interface and USB/PCI cards all goes through the host OS doesn't it? Or am I confused...

(oh and Isaac I feel for you and your old hardware...I have a bunch of stuff hanging around (especially if I build a new PC!) so please PM me if you are in the UK and I'd be happy to send you some of it)

[DebbyIan]

Way to go.
What once seemed an academic exercise now seems quite a plausible avenue for the industry to pursue in the foreseeable future. I can readily see a serious use and benfits of a RAM disk for the entire OS by using it with a Solid State Disk/Storage.

[IsaacKuo]

Curious to see what you all think about that? Would having the Xen dom0 running in RAM actually make a difference to the performance of the virtual machines? Obviously they'll still be reading from and writing to an SSD but the virtual side of things such as the network interface and USB/PCI cards all goes through the host OS doesn't it? Or am I confused...

To be honest, it's new technology to me, since my computers are so weak. But with Debian 7's release including OpenStack and Xen Cloud Platform, I've got to try it out myself!

Well...at least the cloud computing aspects. I'm severely limited by the fact that my motherboards are limited to 2gigs of RAM. When I get a new job, I'm going to celebrate with a Motion Computing LE1700 SXGA slate, but even that will be limited to 4 gigs of RAM (two slots of DDR2 laptop memory; I actually have one 2gig stick already).

[GlenWalker]

Seems to be new to everyone I've asked so I think my solution is to try it and see!

I have settled on what hardware I am going to use and just need to scratch together some funds for it.

For now though I have managed to get Windows 7 working inside OpenSUSE (it was already on the hard drive). They both run really slowly though and the motherboard I have doen't support the right kind of virtualisation for graphics/PCI passthrough so functionality is limited.

Going to try again with Debian instead of OpenSUSE and see if I can boost performance then I'll try the ramdisk...though like you I am limited to 4GB for now.

Once I've done some experimentation (and possibly raided the piggy bank for funds and acquired new hardware) I'll post back my findings.

[IsaacKuo]

I'm bumping this because I've just put together my latest and greatest version of RAMboot - this time for Debian 8 Jessie! Honestly, the exact same techniques still work even after all these years, and regardless of the switch to systemd.

But I've cleaned things up, and made better notes on how to maintain the system and what to expect from update-grub.

http://www.linuxquestions.org/questions ... sie-37165/

Enjoy!