Upcoming how-to...RAMboot to a ramdisk for blazing speed!

Here you can discuss every aspect of Debian. Note: not for support requests!

Upcoming how-to...RAMboot to a ramdisk for blazing speed!

Postby IsaacKuo » 2008-06-06 07:38

I've figured out a way to boot Debian into a nice big uncompressed tmpfs ramdisk. The result is stupendously fast--programs pop up instantly--and silent. There are no hard drive seek noises, and the hard drive can be spun down entirely.

I do wonder, though...is there a vaguely "standard" way to do this? My method involves a hack.

The basic outline of my method is:

1) Do two normal hard drive installs. One will be the "main" normal OS, while the other is an auxiliary install used only for creating a new initramfs image and for processing the other OS partition into a stripped down tar archive.

2) Prepare an automated script to snapshot/strip down the "main" OS into /snapstrip.tar.

3) Do a hack on /usr/share/initramfs-tools/scripts/local to change how it "mounts" the root partition so it copies /snapstrip.tar into a tmpfs ramdisk instead of the normal behavior.

4) Make the custom initrd.img file with that "ramboot hack", and create a new grub entry for it.

I'm not sure how far the OS files can be stripped down. So far, I'd say you probably need at least 512megs of RAM to load up a comfortable fully functional system.

If you're interested in seeing just how FAST your computer can go, and you've got the RAM, RAMboot is definitely worth checking out. It's also great for silent computing on a budget.
Isaac Kuo
User avatar
IsaacKuo
 
Posts: 316
Joined: 2008-04-24 20:06

Postby naithen » 2008-06-06 08:44

No wonder you can get a blazing speed because of the RAM access time; which is far less that the HDD seek and read times.

I don't know about a standard method but this is again like booting from a live CD but store the data in your HDD. So I guess it is all about a custom initrd where it will load all the data to the RAM and continue and you won't need a seperate installation for that.

You can just use the custom initrd at boot and use the existing installation. But then again you'll have to have some sort of a shutdown script to save any changes back to your HDD (or have your /home folder in the HDD and rest in the RAM) and if you want to make any changes to the system (e.g: install something) you can use the normal boot and do the changes so when you boot back for the RAMboot you can access those changes. But if you have two systems you'll have to make some clever scripts to save the system changes or do it manually.

Think these might help:
http://www.squad17.org/sites/default/files/initrd.pdf
http://www.squad17.org/linux_hints

:roll:
User avatar
naithen
 
Posts: 213
Joined: 2008-03-03 15:33

Postby IsaacKuo » 2008-06-06 13:38

There are a number of advantages to having two installs rather than just one. The main one is convenience. The "auxiliary" OS is convenient for making backups and restorations of the "main" OS without worrying about stepping on your own toes. Also, you can clutter up the "auxiliary" OS with utility tools that you won't need on the "main" OS.

At first, I thought of just using "cp" to copy over the files without using a tar archive. Unfortunately, the "cp" included in busybox isn't powerful enough to deal with such a large job, and the "cp" in /usr/bin fails if you attempt to run it from the initrd because it has some lib dependency.

I was annoyed that a straightforward "cp" wouldn't work, and didn't think very hard about what alternative tools might do the job. Tar was the first thing that popped into my head, and it worked like a charm.

This is certainly related to the way liveCDs work, but it's also MUCH simpler in many respects. There's no need to fuss over which portions of the file system need to be writeable, or to deal with a compressed filesystem.

There's just one way in which it's more complicated than a liveCD. You need to copy over the OS files within initrd--which wouldn't be a problem except for the non-obvious limitations of "cp" within initrd.

As for saving changes back to the hard drive: I wouldn't bother. The version of the OS copied into RAM is extremely stripped down and crippled in many ways. In particular, apt is ham-strung. You can save a LOT of space removing apt and unused modules and such. Instead, if there are any changes you want to make, then boot up into the "main" OS on the hard drive.
Isaac Kuo
User avatar
IsaacKuo
 
Posts: 316
Joined: 2008-04-24 20:06

Postby IsaacKuo » 2008-06-06 13:41

Here's a first draft; it needs a lot more on stripping down the OS:
----------------------------------------------------------------------------

How to RAMboot

This details a method of loading your entire OS into an uncompressed ramdisk. The result is lightning fast performance, and elimination of hard drive noise and power consumption (if swap is not used and the hard drive is spun down).

The basic steps are:

1. Install Debian 4.0 twice on the hard drive

2. Create a modified /etc/fstab which has tmpfs for the root partition

3. Create a script which makes a stripped down OS image

4. Create a custom initrd.img which loads the OS image into a tmpfs ramdisk

5. Modify /boot/grub/menu.lst with an entry for the custom initrd.img

After completing these steps, you will have a triple boot system with the following boot options:

A) Boot into the "auxiliary" OS, where you run the OS snapshot/stripping script

B) Boot into the "main" OS, where you install new software or change settings

C) Boot into the "ramboot" OS, for high speed silent computing

-------------------------------------------------
Step 1. Installing Debian 4.0 twice

Create three partitions:

hda1 ext3 4 gigs
hda5 ext3 2 gigs (unless you have lots of RAM and plan on making a BIG image)
hda6 swap

First, install onto hda5 do NOT select either Desktop Workstation or Base software suites.

Upgrade the kernel, and install the following:

apt-get install hdparm localepurge debconf-english (will remove debconf-i18n)

Do some stripping down by removing the following:

apt-get remove --purge aptitude tasksel nano
apt-get clean

Second, install onto hda1. This is the auxiliary OS, and you do not need to do anything special to strip it down if you don't want to. Upgrade the kernel to the same version as the other install.

In the auxiliary OS, set it to mount /dev/hda5 to /mnt/hda5.
-------------------------------------------------
Step 2. Create a modified /etc/fstab

Go to create and edit a new fstab using these commands:

cd /mnt/hda5/etc/
cp -vax fstab fstab.ramboot
vi fstab.ramboot

Comment out the swap entry and the /dev/hda5 entry. Create a new / line like this:

none / tmpfs defaults 0 0

-------------------------------------------------
Step 3. Create a script which makes a stripped down OS image

Boot up into the auxiliary hda1 OS, and login as root. Then create a basic script like this:

Code: Select all
#!/bin/sh
#
# Takes an OS snapshot from /mnt/hda5, strips it down, and wraps it up into /snapstrip.tar

# Clean up anything previous
touch /snapstrip
touch /snapstrip.tar
rm -fvr /snapstrip
rm -fvr /snapstrip.tar

# Copy the files over
mkdir /snapstrip
cp -vax /mnt/hda5/* /snapstrip/

# Move over the modified fstab
cd /snapstrip/etc/
cp -vax fstab.ramboot fstab

# Strip down unnecessary stuff
cd /snapstrip/
rm -fvr /snapstrip/boot/*
rm -fvr /snapstrip/var/lib/apt/lists/*
rm -fvr /snapstrip/usr/share/doc-base/*
rm -fvr /snapstrip/usr/share/doc/*
rm -fvr /snapstrip/usr/share/man/*

rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/drivers/bluetooth
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/drivers/ieee1394
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/drivers/parport
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/drivers/pcmcia
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/drivers/telephony
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/drivers/isdn
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/drivers/md

rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/fs/ntfs
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/fs/reiserfs
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/fs/hfs
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/fs/hfsplus
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/fs/xfs

rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/net/appletalk
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/net/bluetooth
rm -fvr /snapstrip/lib/modules/2.6.18-6-486/kernel/net/irda

#############ADD IN MORE STUFF TO DELETE HERE

# Create the tar archive
cd /snapstrip/
tar cf /snapstrip.tar *

Run the script to create the tar archive. You'll run this script after making changes to the main OS to create a new snapshot file.
-------------------------------------------------
Step 4. Create a custom initrd.img which loads the OS image into a tmpfs ramdisk

This is step is a hack. It works with Debian 4.0, for now at least. There's probably a less "hackish" way of doing this.

On the auxiliary OS, go to /usr/share/initramfs-tools/scripts/

cd /usr/share/initramfs-tools/scripts/

Create a backup of "local" and edit a modified version with:

cp -vax local local.bak
vi local.ramboot
cp -vax local local.ramboot

You'll want to modify the portion where the actual "mount" command is done. Comment it out and insert something like this:

Code: Select all
[...]
########################################ramboot
        # FIXME This has no error checking
        # Mount root
###     mount ${roflag} -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} ${rootmnt}

################################ mount the filesystem
        mkdir /ijkijk
        mount ${roflag} -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} /ijkijk

################################ create ramdisk (note hardcoded size)
        mount -t tmpfs -o size=400M none ${rootmnt}

################################ copy the files over to the ramdisk
        cd ${rootmnt}
        tar xf /ijkijk/snapstrip.tar

################################ umount the filesystem and set to spin down
        umount /ijkijk
   hdparm -S 23 /dev/hda

########################################ijk
[...]

After making these modifications, create the initrd.img with this command:

mkinitramfs -o /boot/initrd.img-2.6.18-6-486.ramboot

After creating this ramdisk make sure to copy back the backup file with:

cp -vax local.bak local

This is important! If you forget to do this, then your system will be screwed up if your kernel is upgraded!

Note that at first, I tried using "cp" to copy over the filesystem, but that failed since the version of cp included in busybox is apparently not up to the job. Tar worked fine.
-------------------------------------------------
Step 5. Modify /boot/grub/menu.lst with an entry for the custom initrd.img

Modify /boot/grub/menu.lst with a new entry. Copy the auxiliary OS's entry where root=/dev/hda1. Then modify the initrd to use your new initrd.img. It will look something like this:

title RAMdisk Debian GNU/Linux, kernel 2.6.18-6-486
root (hd0,0)
kernel /boot/vmlinuz-2.6.18-6-486 root=/dev/hda1 ro
initrd /boot/initrd.img-2.6.18-6-486.ramboot
-------------------------------------------------

After following these steps, you'll have a very basic working system. Now you can boot into the "main" OS and install things like X (only install the xserver you need) and other programs like icewm and iceweasel. For example:

apt-get install xserver-xorg-video-vesa xserver-xorg-video-ati xfonts-base alsa-base alsa-utils icewm menu iceweasel xfe aterm

The default icewm theme is rather ugly, so you can copy over a nice theme like /usr/share/icewm/themes/IceCrack2 from another install. Obviously, you don't want to install all of the themes in icewm-themes since they'll be consuming RAM just sitting there.
Isaac Kuo
User avatar
IsaacKuo
 
Posts: 316
Joined: 2008-04-24 20:06

Postby naithen » 2008-06-06 15:30

You are missing my point man, just think you just wanted to make your existing installation to a RAM boot just to make it faster. You don't need any extra utilities. All you need to do is create a initrd which does the job and create a boot entry with that initrd. Simple as that, you achieve your high speed silent computing.

And for the concerns with the custom initrd all you have to change is the init script; which should load you data to the ram to create the root file system and chroot to it. And you don't even change a tiny bit of the original system. Of course you can have you /home (or any other) mounted from the HDD if ou want to save user data.

EDIT---
Then again, you can pack it all into the initrd (like in the installation CDs) so you get a seperate OS in your initrd.
User avatar
naithen
 
Posts: 213
Joined: 2008-03-03 15:33

Postby IsaacKuo » 2008-06-06 15:45

Code: Select all
You are missing my point man, just think you just wanted to make your existing installation to a RAM boot just to make it faster.

I also wanted the RAM boot to actually fit in RAM. In order to do this, stripping down the image to the bare bones is necessary.

So, for example, one utility is the script which does the stripping. This may cull away hundreds of megs, so it's worth the effort.

Another nice utility is rsync, which can be used to backup OS images efficiently, and/or rsnapshot for incremental images. Why clutter up the main OS with these utilities?

Also, it's nice to have a full install of icewm-themes to copy over from. It's convenient to install icewm-themes on the auxiliary OS and just copy over the desired theme(s) to the main OS.

Ultimately, it's not just a question of hacking something that works, but doesn't work well and is a pain to work with. I wanted to set up something easy to maintain. It's simply easier to do various things on the auxiliary OS without worrying if anything I'm doing wastes some disc space. I only want to boot into the "main" OS to install new software or reconfigure something.

BTW, I disagree on mounting /home on the hard drive. That hamstrings performance and eliminates silence since almost all applications use files in /home. I save personal files on a centralized nfs network file server.
Isaac Kuo
User avatar
IsaacKuo
 
Posts: 316
Joined: 2008-04-24 20:06

Postby IsaacKuo » 2008-06-06 15:52

naithen wrote:Then again, you can pack it all into the initrd (like in the installation CDs) so you get a seperate OS in your initrd.

That was the first method I considered. However, I discarded that option as being more effort to set up and more dissimilar to a standard Debian install.

The hack I set up makes the RAMboot very similar to a standard Debian install. There's a small modification to /etc/fstab, and a hack to the rootmount script. Everything else proceeds as in a normal boot.

I think that stuffing everything into initrd would involve a bit more code hacking. This increases effort and reduces the chances of the hack working with other/newer Linux distributions (as initramfs-tools are updated or even replaced).

I wanted to do a minimal effort minimal change job. One unavoidable complexification, though, was the process of heavily stripping down the OS files. The image is simply too big otherwise, unless you have oodles of RAM at your disposal (I don't).
Isaac Kuo
User avatar
IsaacKuo
 
Posts: 316
Joined: 2008-04-24 20:06

Postby naithen » 2008-06-06 16:05

Well IsaacKuo,

There is no correct way of doing things in Linux, it all comes down to what you need and how you want it to be done.

My point is why bother installing a seperate "auxiliary" OS and create a initrd when you can do the same with just creating the initrd. You can strip things down when you are creating the root file system in RAM (using the init script/process in initrd) and you don't have to have an "auxilary" OS for that.

Also, it's nice to have a full install of icewm-themes to copy over from. It's convenient to install icewm-themes on the auxiliary OS and just copy over the desired theme(s) to the main OS.

You can do it from the init script/process while you are creating the RAM root file system.

And as concerned with the maintanence what is more easy to maintain? Just one script or a whole OS?

It's simply easier to do various things on the auxiliary OS without worrying if anything I'm doing wastes some disc space.

Well, you have already wasted disk space by installing an "auxiliary" OS.

I only want to boot into the "main" OS to install new software or reconfigure something.

You can do the same by just using the custom initrd. All you have to do is select the your main OS entry at the boot menu.

Mounting the /home to the HDD is not worthy if you are using NAS. I agree with you for that.

So as I said, my point is why do you want to do two things when you can get the same result from doing just one thing. And I'm not saying that your steps are wrong in any way. :D
User avatar
naithen
 
Posts: 213
Joined: 2008-03-03 15:33

Postby IsaacKuo » 2008-06-06 16:27

naithen wrote:There is no correct way of doing things in Linux, it all comes down to what you need and how you want it to be done.

My point is why bother installing a seperate "auxiliary" OS and create a initrd when you can do the same with just creating the initrd. You can strip things down when you are creating the root file system in RAM (using the init script/process in initrd) and you don't have to have an "auxilary" OS for that.

How about you try it yourself and find out first hand what's wrong with it?

1) Good luck copying over the files directly! cp won't work. Go ahead and confirm it yourself. Have fun! I know I did.

2) It takes longer to copy all of the files first (if they'll fit in RAM!), and then to delete the excess, than it does to only copy the needed files from a stripped down tarball.

3) You're talking about putting a somewhat lengthy stripping down script into initrd. This means that every time you want to add or remove items to strip, you need to rebuild the initrd. As opposed to just building one initrd and it just plain works.
Also, it's nice to have a full install of icewm-themes to copy over from. It's convenient to install icewm-themes on the auxiliary OS and just copy over the desired theme(s) to the main OS.

You can do it from the init script/process while you are creating the RAM root file system.

See above. Or better yet, try it out yourself. What you're suggesting is a pain.

Sometimes it's easier to only copy over only what you need. Sometimes it's easier to strip away what you don't need. My method offers the flexibility to do either, whichever it easier.
And as concerned with the maintanence what is more easy to maintain? Just one script or a whole OS?

The auxiliary OS is EASY to maintain. One reason I use Debian Stable is because I'm lazy about maintaining an OS. For the most part, I simply don't maintain it. It's Debian Stable. It's fine.

Now if we're talking about maintaining the stripping script, it's easier to simply edit that script than it is to edit it and then rebuild the initrd.img each time you edit it.

It's simply easier to do various things on the auxiliary OS without worrying if anything I'm doing wastes some disc space.

Well, you have already wasted disk space by installing an "auxiliary" OS.

Sure, but I don't WORRY about it. I don't care if I waste a few hundred megs of space on an 8gig hard drive. I DO care about wasting a dozen megs of space on my RAMboot image, because my machine only has 512megs of RAM.
I only want to boot into the "main" OS to install new software or reconfigure something.

You can do the same by just using the custom initrd. All you have to do is select the your main OS entry at the boot menu.
[...]
So as I said, my point is why do you want to do two things when you can get the same result from doing just one thing. And I'm not saying that your steps are wrong in any way. :D

Because my way is easier and will...actually work.

Hey, I'm open to suggestions for different/better ways of doing things. But in this case, I think you're just going to have to try it first hand to comprehend the problems. (And maybe even solve them!)

In any case, trying it out first hand is worth it, just because you'll get to see first hand the sheer SPEED of RAMboot. It's breathtaking.
Isaac Kuo
User avatar
IsaacKuo
 
Posts: 316
Joined: 2008-04-24 20:06

Postby Penguin Skinner » 2008-06-06 18:52

IsaacKuo wrote: ... The auxiliary OS is EASY to maintain....

I know this is a bit of a side topic, but I agree completely. I maintain what I call "parallel" installs of Debian Testing on two separate drives on my personal system. There is very little extra work involved in maintaining the second installation, and it is invaluable as a "working backup" alone, in the event of a drive failure. About once a week I boot into the second install, backup the primary, and do upgrades. So no question in my mind, this is a great solution to a variety of concerns.

I hope you go ahead and post a final "how-to" on your method for setting up Debian to run from ram, Isaac. Then let others post their methods. I'm sure the information will be a valuable contribution.
Penguin Skinner
 
Posts: 709
Joined: 2005-09-15 20:37
Location: North by Northwest

Postby naithen » 2008-06-10 09:33

IsaacKuo,
Sure dude, I'll try it out. May be you are write about saying it is a pain. I'm actually creating a live CD system (to back-up your existing system to a live CD) these days n' trying to find time to finish it. So I guess I based my facts on that. So maybe your method could be the easiest way or maybe not.

I'll keep this in my list of things to do n' hopefully I'll find some time to work on it.
User avatar
naithen
 
Posts: 213
Joined: 2008-03-03 15:33

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

Postby Aubrey_Snoddie » 2013-03-14 12:09

Hello all,

anyone there?

I have a few questions..

My setip is as follows - Thinkpad X230 with 16GB RAM and one 6 Gb/sec SSd drive (120GB).

I am already running Debian 7 (Wheezy) in the conventional way i.e. booting from the abovementioned HDD.

I would like, via an initramfs hack/tweak (aufs, tmpfs?), to load my complete OS (appx. 4.0 GB) into memory on startup.
This aufs should be read-writeable.
A checkpoint time should be specified so that all changes in RAM are written back to the SSD prior to a shutdown and reboot.
Of course the checkpoint file(s) should be read at startup to keep all changes in sync.

I have read quite a few articles in various forums and it would seem to be technically possible to do - but where are the pitfalls etc?

I would appreciate some pointers and / or ideas from your learned selves.


Regards,


Aubrey

PS I recall that a chap called Isaac Kuo was working in this direction a few years ago - is he still active?

PPS I know Puppy and Fatdog Linux attempt something similiar but I find it too kludgy.
User avatar
Aubrey_Snoddie
 
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

Postby IsaacKuo » 2013-03-14 12:58

I haven't been playing around with RAMboot lately, and I never bothered with rewriting changes back to disk.

tmpfs is quite fast, but it makes no attempt to keep track of changes for any sort of "checkpoint" files like you want. Instead, you could use rsync to copy back just the changed files.

My experience with aufs is that it's really only good as a hack to let you run an OS from a read only device (like a liveCD or a compressed file). Trying to force it into other uses seems to be rather unhappy.

I don't know for sure, but I expect that you won't want to be using any compressed file system at all. Your SSD is likely very fast already, so the performance hit from decompressing files outweighs any benefit from reading from RAM.
Isaac Kuo
User avatar
IsaacKuo
 
Posts: 316
Joined: 2008-04-24 20:06

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

Postby Aubrey_Snoddie » 2013-03-14 13:07

Appreciate the quick reply Isaac.

What I am trying to achieve is to have the whole OS <<uncompressed>> in RAM after startup. I am not concerned if startup / shutdown take a while to do their rsync thing.

What is important however, is that all changes, when everything is up and running are "checkpointed".

Maybe I am being naiive in thinking that if I have 16GB RAM I might as well use it...

Would be appreciative of your views on this.


Regards,



Aubrey
User avatar
Aubrey_Snoddie
 
Posts: 11
Joined: 2013-03-14 11:22
Location: EU

Re: Upcoming how-to...RAMboot to a ramdisk for blazing speed

Postby llivv » 2013-03-14 14:34

WOW!


I've had this thread bookmarked for a long time.
And I was following it and got most of the way through
and still have a couple of "TINY" minimal installs
I was experimenting with to use as protypes for a ramboot
image,
when mzilzaki (still trying to remember the spelling)
the official forum ninja
mzilikazi posted this howto
http://forums.debian.net/viewtopic.php?f=16&t=34902

I really have to get back and finish up my ramboot image project
and try the debian debirf package comparing the results.

IsaacKuo
many - many thanks for this howto
In memory of Ian Ashley Murdock (1973 - 2015) founder of the Debian project.
User avatar
llivv
 
Posts: 5488
Joined: 2007-02-14 18:10
Location: cold storage

Next

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 8 guests

fashionable