volsize as a maximum limit

On 08.08.2012 07:21, Timandahaf wrote:
> New question #205270 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/205270
>
> Currently --volsize <N> means "create volume sizes of <N megabytes + a fudge factor>". The use case that I would think would be more useful is to ensure that the volume size never exceeds N. This is useful in cases where there is a file size limit on the target media, for example. Does this make sense to others?
>
> If so, should I submit a feature request to modify the behavior of volsize? If it is not very simple, I'd be happy to take a look and perhaps submit a patch.
>

as far as i understand this is technically impossible because

- data is streamed and counted into a tar writer file object
- after count x is achieved the writer uses the compressor or gpg on it for the final volume

the difference between the compressed/encrypted and raw input is the fudge factor. but writing it, i realize that the files should be smaller than the raw input due to compression. so i must have got something wrong here.

@Ken: can you shed light on this?

Timandahaf you are welcome to have a look at this and deliver patches. there is actually another pressing issue regarding volumesize. manifests and signatures currently do not honor the --volsize setting, which makes duplicity unusable for some backends with enforced max filesizes.

..ede/duply.net

Okay, I took a look. Actually the way it works is:

- Uncompressed data in blocks of 128kB are sent to GPGFile or GZipFile
- the size of the written (compressed) gpg or gzip file is checked against volsize
- this process continues until volsize is hit

Where it gets tricky is this: both Gpgfile and Gzipfile write a footer when close() called. Unfortunately, I haven't found a way to predict the size of this footer in advance. The footer size seems to vary quite a bit depending on the data. I don't even know if I can find a bound on this footer - if I could, this would solve the problem. As a kludge, I could assume that the footer will take up to 128kB, and reserve that much space for it. This will probably work in 99% of the cases, and I've tried it on volsizes of upto 250MB. What do you think?

As an aside, the GzipWriteFile uses 128 kB blocks as shown above, but inconsistently uses a value of 32 kB to check for the last block. Fixing this will essentially apply my idea above to the gzip case. For the gpg case, we'll simply reduce the specified size by 128 kB.

I noticed the problem about volsize not being respected by the manifests and signatures earlier too because it is an issue for me as well. I'll happy to work on this at some point when I have some time available.