Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[Discussion] virt-sparsify return values?

Off-Topic discussions about science, technology, and non Debian specific topics.
Post Reply
Message
Author
CwF
Global Moderator
Global Moderator
Posts: 2751
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 45 times
Been thanked: 206 times

[Discussion] virt-sparsify return values?

#1 Post by CwF »

virt-sparsify is a command used to shrink the size of virtual disk image files. It is essentially a trim operation performed on an inactive image.

Code: Select all

$  virt-sparsify --in-place amd64_xfce_bees_2410-5_archive.qcow2
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ --:--
[   7.2] Trimming /dev/sda1
[   7.7] Sparsify in-place operation completed with no errors]
It can also be used during an archival of a regular disk image. For that I don't have an example output, but it is essentially the same. First we'd qemu-img -convert from a real disc to a qcow2 file, then run virt-sparsify on the image, and lastly another qemu-img operation this time from qcow2 to qcow2 with the compression option enabled.

There is a difference in the feedback when doing images from different sources with differing amounts of runtime. The [ 7.2] and [ 7.7] values are what I'm wondering about. I have tried to look it up without much luck and lose interest after modern search and AI give me absolutely nothing of use.

7.x is rather high, I suppose. I have nothing to go on but suspicion. On a clean, unused golden image where the recent runtime is the time it takes to update minor things, and also run a package update and test - the number is usually in the 4's. I seem to remember that without something large like a kernel update the number may be in the 2's or lower. If this image is a disk from a machine with multiple updates, even a release upgrade, and more significant runtime the number may be 14+.

Also, multiple runs may possibly reduce from an initial higher to a subsequent run lower number. I'll get two postings of this result when processing a real disk into an archive with my herder.tk gui, and the sparsify result spit out through a notify-send bubble where the final output is a qemu-img info. Each time I see it change I wonder...

Anybody have a clue what those numbers are possibly reporting? My guess is some metric concerning freed space.

User avatar
fabien
Forum Helper
Forum Helper
Posts: 768
Joined: 2019-12-03 12:51
Location: Anarres (Toulouse, France actually)
Has thanked: 69 times
Been thanked: 176 times

Re: [Discussion] virt-sparsify return values?

#2 Post by fabien »

There are a couple options that could give some clues.
--verbose (Enable verbose messages for debugging)
--machine-readable (in conjunction with other options to make the regular program output more machine friendly)
Isn't it just a certain number of seconds?
ImageShare your Debian SCRIPTS
There will be neither barrier nor walls, neither official nor guard, there will be no more desert and the entire world will become a garden. — Anacharsis Cloots

User avatar
pbear
Posts: 410
Joined: 2023-08-27 15:05
Location: San Francisco
Has thanked: 2 times
Been thanked: 66 times

Re: [Discussion] virt-sparsify return values?

#3 Post by pbear »

Off-topic, I will mention that an rsync backup of a QEMU VM also copies only space-in-use. Moreover, the backup can be compressed for archiving by any of the usual utilities. Which isn't to suggest rsync backup and virt-sparsify are interchangeable. They do different things. Merely pointing out the latter isn't the only way to trim a VM archive of unused space. [/off-topic]

CwF
Global Moderator
Global Moderator
Posts: 2751
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 45 times
Been thanked: 206 times

Re: [Discussion] virt-sparsify return values?

#4 Post by CwF »

pbear wrote: 2024-05-15 14:02 They do different things.
They are not in the same category. Perhaps complimentary but not at all synonymous. rsysnc and compression utilities are operating at the file level, not on the disk, partition, or file system structure. Furthermore, the resulting image is directly usable - never needing to be restored, or uncompressed, and there is no source and destination with --in-place. The size of the image in the example is 997MB, in use it reports 3.2 used of 8.4GB.
In-place sparsification works using discard (a.k.a trim or unmap) support.
As I recall, and 'empty' disk with no file system is about 197k. Add a file system and the image size grows in proportion to disk size up to a few gigs with no files at all. The image compression operates on that structure. Add a large test file and delete it, virt-sparsify operates on that. This is the only way the reclaim used non-file space.

Sometimes extra operations are necessary outside the scope of file based utilities. Take a random disc, fill it half way - then delete the partition and make a new one half the original and fill it to half. Upon imaging will it be 25% the disk size, or 50%? It will be 50% without an extra zero-block step. Without that extra step the info in the deleted partition beyond the new partition is in fact recoverable, and still clogging up the works.
fabien wrote: 2024-05-15 10:59 Isn't it just a certain number of seconds?
seconds? Possibly.
Maybe related, maybe not, during qemu-img copies from disk to image there is a similar digit added to the progress. Very clean and new the ending progress can be 100.0%. Known disk with issues I'll see 100.x% IIRC on one ntfs drive I saw that x number climb where I expected it to (at 35-40GB lets say) and the final was something odd like 100.13% That .13 means something! I suspect some retry count or error reporting. Both of these examples are just curiosities.

Let's see

Code: Select all

$  virt-sparsify --machine-readable --in-place amd64_xfce_bees_2410-5_archive.qcow2
[   2.3] Trimming /dev/sda1
[   2.7] Sparsify in-place operation completed with no errors
It just drops the progress bar, so does notify-send! See, the numbers are different, and maybe took 2.3 seconds!?

Added, I updated my herder.tk;

Code: Select all

grid [ttk::button .f1.nb.cow.btrim -text "trim" -width 4 -command {exec  virt-sparsify --machine-readable --in-place $basefile > ${tfs}/response; moO] -row 0 -column 2 -sticky w
grid [ttk::button .f1.nb.cow.barchive -text "archive" -width 7 -command {.f1.nb.cow.btrim invoke ; exec notify-send "[exec cat ${tfs}response]"; exec qemu-img convert -c -O qcow2 $basefile "$cowpath/${arch}_${dE}_${domain}_${versionN}_${eType}.qcow2"; lset basefile $cowpath/${arch}_${dE}_${domain}_${versionN}_${eType}.qcow2; newFile; .f1.nb.cow.binfo invoke; moO}] -row 0 -column 2 -sticky e}

Post Reply