Are you using Parity for big data in Cloud?

[bester69]

I was wondering How you dude do to upload huge files to the Cloud keeping the integrity of them..

Im using par2 command line, it is an erasure code system that produces par files for checksum verification of data integrity, with the capability to perform data recovery operations that can repair or regenerate corrupted or missing data.
https://en.wikipedia.org/wiki/Parchive

I alwasy keep the same steps:
1. I split the file in 25Mb chunks with Peazip
2. I move all the chunks to a subdirectory and then i generate the parity files par2 need to repair possibles corruptions.
par2 c -R parity-files.par2 Allparts/
3. Upload all files (parity files and chunks)

This way i can garantee i will be able to merge all the chunks into the zipped file and uncompress them witout corruption. I'd use parity file if needed to repair possible losses.

[tomazzi]

Essentially the cloud storage is very safe - because data centers are using distributed storage systems and ECC-class memory, and therefore it's very unlikely that Your data will get damaged in any way...
Unless ...
Someone will damage Your data intentionally.

Rule number one: encrypt Your data if You want to use cloud storage...
Rule number two: verify Your data *after* uploading - there's a *relatively* high chance that Your PC is damaged and it can malform Your data - i.e. normally nobody is using ECC memory in a home PC - bugs are common today (HDD error rate of 10^-14bits is rather safe - a single malformed bit in the RAM can make Your data unreadable)

Regards.

[bester69]

Essentially the cloud storage is very safe - because data centers are using distributed storage systems and ECC-class memory, and therefore it's very unlikely that Your data will get damaged in any way...
Unless ...
Someone will damage Your data intentionally.

Rule number one: encrypt Your data if You want to use cloud storage...
Rule number two: verify Your data *after* uploading - there's a *relatively* high chance that Your PC is damaged and it can malform Your data - i.e. normally nobody is using ECC memory in a home PC - bugs are common today (HDD error rate of 10^-14bits is rather safe - a single malformed bit in the RAM can make Your data unreadable)

Regards.

Perhpas premium services has those features i think, or paid cloud services, i think dropbox, mega and other free cloud services are not so secure as for data corruption, but i really dont know. But i think, the big risk is in the data transfer (upload/dowload stream bits), microcuts can alter the signature of the file making it usless. That's the reason they give you a md5sum's signature whith every distro, so you can check onece downloaded if the big fle has been corrupted with the download.

Whenever i upload a big/huge file to the cloud, i use splitting and Par2 (parity recover), this way i can almost always garantee my file wont get corrupt.

Now Im uploading a Win10 DVD to MEGA around 5GB, and i used par2, if not, im pretty sure that DVD image will come back corrupt.

[bester69]

..
Rule number one: encrypt Your data if You want to use cloud storage...
...
Regards.

For my incremental backups
- Im using local encryption with deduplication software (attic), so i can update my backup and uploaded to the cloud,
- and Im using duplicty without encryptation to make the transfer/upload easyly

[millpond]

I was wondering How you dude do to upload huge files to the Cloud keeping the integrity of them..

Im using par2 command line, it is an erasure code system that produces par files for checksum verification of data integrity, with the capability to perform data recovery operations that can repair or regenerate corrupted or missing data.
https://en.wikipedia.org/wiki/Parchive

I alwasy keep the same steps:
1. I split the file in 25Mb chunks with Peazip
2. I move all the chunks to a subdirectory and then i generate the parity files par2 need to repair possibles corruptions.
par2 c -R parity-files.par2 Allparts/
3. Upload all files (parity files and chunks)

This way i can garantee i will be able to merge all the chunks into the zipped file and uncompress them witout corruption. I'd use parity file if needed to repair possible losses.

Par2 works fine, but you can add an extra layer of security (or even replace par2) by using Rar with the rr flag for adding a data recovery record.