Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

e2zerocat

Programming languages, Coding, Executables, Package Creation, and Scripting.
Post Reply
Message
Author
dryden
Posts: 80
Joined: 2015-02-04 08:54

e2zerocat

#1 Post by dryden »

This is not a help request but there are not many subforums for general discussion that are "topic" based.

Just trying to see what I can do to invent some kind of incremental block based backup solution out of the ground seeing that apparently it does not really exist for Linux.

I mean, it does not exist for me.

I just wrote a shell script that transforms dumpe2fs output into a list of used and unused block ranges:

Code: Select all

$ ./e2zerocat.sh /dev/msata/boot

<snip>

unused: 229825-229952
used: 229953-230208
unused: 230209-230400
used: 230401-230912
unused: 230913-232192
used: 232193-232320
unused: 232321-237568
used: 237569-237826
unused: 237827-238338
used: 238339-239362
unused: 239363-239616
used: 239617-239911
unused: 239912-245759

blocks: 245760
size: 1024
Of course many programs do this or something of the kind, such as partclone. But I didn't feel like learning someone's format right off the bat.

In effect it is easy to generate a (huge) stream of an entire volume by intermittent "dd" reads from the disk, and "dd" reads from /dev/zero, on stdout.

This would effectively be the same as catting the device itself, apart from the fact that all unused blocks have been zeroed!! That's why I call it "e2zerocat".

There are other solutions such as squashfs which is great for creating single backups but not for anything incremental wise.

In effect, you can create a sparse copy of a device by doing:

e2zerocat /dev/msata/boot | cp --sparse=always /dev/stdin outfile.sparse-dd

This will only read used blocks, and all others will be made sparse by cp.

Then you would have a sparse image file that you can mount with loopback!!.

It is just not compressed yet and I'm not sure how to accomplish that but for my purposes a sparse image of the filesystem (the entire device) is already very nice.

I mean how to mount a compressed image.

Besides, gzipping the file will un-sparsify it (it will just compress the zeroes).

So at that point I have a space efficient image.

day
Posts: 56
Joined: 2015-03-03 00:00

Re: e2zerocat

#2 Post by day »

Where is e2zerocat?

User avatar
GarryRicketson
Posts: 5644
Joined: 2015-01-20 22:16
Location: Durango, Mexico

Re: e2zerocat

#3 Post by GarryRicketson »

day wrote:Where is e2zerocat?
You are looking at it:

Code: Select all

     $ ./e2zerocat.sh /dev/msata/boot 
by dryden » Just trying to see what I can do to invent some kind of incremental block based backup solution out of the ground seeing that apparently it does not really exist for Linux.

I mean, it does not exist for me.
I just wrote a shell script that transforms dumpe2fs output into a list of used and unused block ranges:

Read the post more care fully, the OP wrote a script , to perform
the task they needed, because other scripts or "programs"
do not do the job, or work the way they want them to.
by dryden » Of course many programs do this or something of the kind, such as partclone. But I didn't feel like learning someone's format right off the bat.

I do not know why the OP choose to not share the actual script,
maybe it is very long, maybe , maybe , maybe-----???
Only the OP can answer that.

dryden
Posts: 80
Joined: 2015-02-04 08:54

Re: e2zerocat

#4 Post by dryden »

Yes I was about to publish some of it.

But I was at a crossroads of whether to use the "bmap-tools" product or not.

https://source.tizen.org/documentation/ ... ls-project

It does pretty much the same thing and would have saved me some time ;-).

But I thought it would be cooler to write some C myself, so I created a little program that could find the holes in files myself (using the FIEMAP system of kernel 2.6.26) :P.

Well not completed but I will post it inline:

Code: Select all

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <linux/fiemap.h>
#include <linux/fs.h>

#include <linux/fs.h>
#include <fcntl.h>
#include <unistd.h>

int fd;

struct fiemap *map;
const int SIZE = 4096;
int start;
int newblocksize; // !!

int nextblock(int start, int *next, int *length, int *value) {
	map->fm_start = newblocksize * start;
	int ret= ioctl(fd, FS_IOC_FIEMAP, map);
	if (ret >= 0) {
		*next = map->fm_extents[0].fe_logical / newblocksize;
		*length = map->fm_extents[0].fe_length / newblocksize;
		*value = map->fm_extents[0].fe_flags;
		printf("Starting at %d, next block is at %d with length %d\n", start, *next, *length);
	}
	return ret;
}

int main(int argc, char **args) {
	fd = open(args[1], O_RDONLY);
	
	map = malloc(sizeof (struct fiemap) + (2 * sizeof(struct fiemap_extent)));

	if (!map) {
		perror("Could not allocate fiemap buffers");
		exit(1);
	}

	start = atoi(args[2]);

	map->fm_length = ~0ULL;
	map->fm_extent_count = 2;

	if (ioctl(fd, FIGETBSZ, &newblocksize) < 0) {
		perror("Can't get block size");
		close(fd);
		exit(1);
	}
	printf ("New block size is %d\n", newblocksize);

	// to find the next hole.
	printf ("First hole is at:\n");

	int s = 0;
	int n, l, v, prevlength;
	int count = 0;
	while (nextblock(s, &n, &l, &v) >= 0 && count <= 10) {
		if (s == n) {
			s += l;
		} else {
			printf ("Hole from %d to %d with length %d\n", s, n-1, n-s);
			s = n + l;
			count++;
		}
	}
}
Just based on sample code by others (Oracle).

This was the slightly not correctly working version that I just uploaded to github:

https://raw.githubusercontent.com/dryde ... zerocat.sh

The problem with sparse files without a map is that a sparse file may contain zeroes you need to write back but you cannot know whether they are real zeroes or sparse zeroes.

This is because cp --sparse=always adds sparse regions however much it likes.

So you NEED a map to go with the image file or else it is worthless. Not entirely, but it is worthless for fast restore or anything of the kind. Not necessarily for mounting the sparse image, that works fine. But putting it back on a volume you need to zero the volume first, unless you know which regions are actually allocated in the sparse image.

So the version you have linked here has a small bug causing it to copy some of the wrong areas :p.

I can fix it but this was the version I wrote about earlier.

My new version has checksumming code at will output that on fd 3 (>&3) if you redirect to something (such as a file).

A terrible slow way of making backups obviously --- having to checksum the entire block space you use each time you make a backup but for now this is the idea while being lazy enough to not use actual filesystem code :p.

This is the sucky new version that has the bug fixed:

Well I was trying to post it but git destroyed my commits now.

dryden
Posts: 80
Joined: 2015-02-04 08:54

Re: e2zerocat

#5 Post by dryden »

Actually that hole finding program is pretty much nonsensical since cp adds new ones anyway, so that was pretty much as waste of time.

Post Reply