e2zerocat

Need help with C, C++, perl, python, etc?

e2zerocat

Postby dryden » 2016-11-08 11:55

This is not a help request but there are not many subforums for general discussion that are "topic" based.

Just trying to see what I can do to invent some kind of incremental block based backup solution out of the ground seeing that apparently it does not really exist for Linux.

I mean, it does not exist for me.

I just wrote a shell script that transforms dumpe2fs output into a list of used and unused block ranges:

Code: Select all
$ ./e2zerocat.sh /dev/msata/boot

<snip>

unused: 229825-229952
used: 229953-230208
unused: 230209-230400
used: 230401-230912
unused: 230913-232192
used: 232193-232320
unused: 232321-237568
used: 237569-237826
unused: 237827-238338
used: 238339-239362
unused: 239363-239616
used: 239617-239911
unused: 239912-245759

blocks: 245760
size: 1024


Of course many programs do this or something of the kind, such as partclone. But I didn't feel like learning someone's format right off the bat.

In effect it is easy to generate a (huge) stream of an entire volume by intermittent "dd" reads from the disk, and "dd" reads from /dev/zero, on stdout.

This would effectively be the same as catting the device itself, apart from the fact that all unused blocks have been zeroed!! That's why I call it "e2zerocat".

There are other solutions such as squashfs which is great for creating single backups but not for anything incremental wise.

In effect, you can create a sparse copy of a device by doing:

e2zerocat /dev/msata/boot | cp --sparse=always /dev/stdin outfile.sparse-dd

This will only read used blocks, and all others will be made sparse by cp.

Then you would have a sparse image file that you can mount with loopback!!.

It is just not compressed yet and I'm not sure how to accomplish that but for my purposes a sparse image of the filesystem (the entire device) is already very nice.

I mean how to mount a compressed image.

Besides, gzipping the file will un-sparsify it (it will just compress the zeroes).

So at that point I have a space efficient image.
dryden
 
Posts: 57
Joined: 2015-02-04 08:54

Re: e2zerocat

Postby day » 2016-11-08 23:50

Where is e2zerocat?
day
 
Posts: 52
Joined: 2015-03-03 00:00

Re: e2zerocat

Postby GarryRicketson » 2016-11-09 00:16

day wrote:Where is e2zerocat?


You are looking at it:
Code: Select all
     $ ./e2zerocat.sh /dev/msata/boot


by dryden » Just trying to see what I can do to invent some kind of incremental block based backup solution out of the ground seeing that apparently it does not really exist for Linux.

I mean, it does not exist for me.
I just wrote a shell script that transforms dumpe2fs output into a list of used and unused block ranges:




Read the post more care fully, the OP wrote a script , to perform
the task they needed, because other scripts or "programs"
do not do the job, or work the way they want them to.

by dryden » Of course many programs do this or something of the kind, such as partclone. But I didn't feel like learning someone's format right off the bat.



I do not know why the OP choose to not share the actual script,
maybe it is very long, maybe , maybe , maybe-----???
Only the OP can answer that.
User avatar
GarryRicketson
 
Posts: 4260
Joined: 2015-01-20 22:16
Location: Durango, Mexico

Re: e2zerocat

Postby dryden » 2016-11-09 01:02

Yes I was about to publish some of it.

But I was at a crossroads of whether to use the "bmap-tools" product or not.

https://source.tizen.org/documentation/reference/bmaptool/bmap-tools-project

It does pretty much the same thing and would have saved me some time ;-).

But I thought it would be cooler to write some C myself, so I created a little program that could find the holes in files myself (using the FIEMAP system of kernel 2.6.26) :P.

Well not completed but I will post it inline:

Code: Select all
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <linux/fiemap.h>
#include <linux/fs.h>

#include <linux/fs.h>
#include <fcntl.h>
#include <unistd.h>

int fd;

struct fiemap *map;
const int SIZE = 4096;
int start;
int newblocksize; // !!

int nextblock(int start, int *next, int *length, int *value) {
   map->fm_start = newblocksize * start;
   int ret= ioctl(fd, FS_IOC_FIEMAP, map);
   if (ret >= 0) {
      *next = map->fm_extents[0].fe_logical / newblocksize;
      *length = map->fm_extents[0].fe_length / newblocksize;
      *value = map->fm_extents[0].fe_flags;
      printf("Starting at %d, next block is at %d with length %d\n", start, *next, *length);
   }
   return ret;
}

int main(int argc, char **args) {
   fd = open(args[1], O_RDONLY);
   
   map = malloc(sizeof (struct fiemap) + (2 * sizeof(struct fiemap_extent)));

   if (!map) {
      perror("Could not allocate fiemap buffers");
      exit(1);
   }

   start = atoi(args[2]);

   map->fm_length = ~0ULL;
   map->fm_extent_count = 2;

   if (ioctl(fd, FIGETBSZ, &newblocksize) < 0) {
      perror("Can't get block size");
      close(fd);
      exit(1);
   }
   printf ("New block size is %d\n", newblocksize);

   // to find the next hole.
   printf ("First hole is at:\n");

   int s = 0;
   int n, l, v, prevlength;
   int count = 0;
   while (nextblock(s, &n, &l, &v) >= 0 && count <= 10) {
      if (s == n) {
         s += l;
      } else {
         printf ("Hole from %d to %d with length %d\n", s, n-1, n-s);
         s = n + l;
         count++;
      }
   }
}


Just based on sample code by others (Oracle).

This was the slightly not correctly working version that I just uploaded to github:

https://raw.githubusercontent.com/drydenp/e2zerocat/4360cf8ebc7bc478f84aea9d5bcd8cf8a796a6c5/e2zerocat.sh

The problem with sparse files without a map is that a sparse file may contain zeroes you need to write back but you cannot know whether they are real zeroes or sparse zeroes.

This is because cp --sparse=always adds sparse regions however much it likes.

So you NEED a map to go with the image file or else it is worthless. Not entirely, but it is worthless for fast restore or anything of the kind. Not necessarily for mounting the sparse image, that works fine. But putting it back on a volume you need to zero the volume first, unless you know which regions are actually allocated in the sparse image.

So the version you have linked here has a small bug causing it to copy some of the wrong areas :p.

I can fix it but this was the version I wrote about earlier.

My new version has checksumming code at will output that on fd 3 (>&3) if you redirect to something (such as a file).

A terrible slow way of making backups obviously --- having to checksum the entire block space you use each time you make a backup but for now this is the idea while being lazy enough to not use actual filesystem code :p.

This is the sucky new version that has the bug fixed:

Well I was trying to post it but git destroyed my commits now.
dryden
 
Posts: 57
Joined: 2015-02-04 08:54

Re: e2zerocat

Postby dryden » 2016-11-09 01:17

Actually that hole finding program is pretty much nonsensical since cp adds new ones anyway, so that was pretty much as waste of time.
dryden
 
Posts: 57
Joined: 2015-02-04 08:54


Return to Programming

Who is online

Users browsing this forum: No registered users and 1 guest

fashionable