Back in time much slower than corresponding rsync command

Asked by Sébastien Pierre

After my hard drive crashed, I started a restore from my Back In Time archives, and after having let Back In Time run for around 12h, the backup was far from being complete (I would say no more than 5%).

Running the corresponding rsync command myself proved to be MUCH faster

rsync -rtDH --progress --links --no-p --no-g --no-o -R -v --chmod=ugo=rwX "/media/sebastien/Backups/XXX.local/backintime/XXX/sebastien/1/20131217-120546-840/backup/home/./sebastien" /home/sebastien/Restored/
sending incremental file list

Here are a couple of things that I've noticed that might really slow down the restore:

- When starting BIT (I'm using the gnome version), BIT seems to automatically compare the backed-up directories to the latest snapshot, which keeps the hard drive busy
- When starting a restore, the GTK log window that is displayed is flickering, apparently being constantly refreshed. I imagine that when doing a full restore with millions of files, a significant amount of time is spent drawing and refreshing the window content. It would be probably better to display the rsync command and then the last 1000 entries from the log.
- When restoring permissions (I did not get there for the full restore, but saw that when restoring sub folders individually), a "chmod" is called for each restored file -- which I imagine is done through a popen/subprocess. The result is that restoring permissions seems to take much more time (I would say 3 to 4 times) than restoring the actual files. Using system calls directly would probably greatly reduce the time.

In essence, I find Back In Time is amazing, but it does not seem to be able to do a full restore with Gb of data within a reasonable amount of time.

Are there some options that might speed up the process?

Thanks!

Question information

Language:
English Edit question
Status:
Answered
For:
Back In Time Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Benjamin Schmid (benbuntu) said :
#1

I'm not an expert to BIT, but did you see & try the "full rsync mode" option under Preferences > Options?

This and the "SSH encrypted" mode appeared after upgrading to the latest 1.0.34 via ppa.

Revision history for this message
Germar (germar) said :
#2

> Are there some options that might speed up the process?

Jepp. Just install the latest version :p
Current version is 1.0.34. If you are on Ubuntu you can use our PPA bit-team/stable

This is bug #1079479 (take a look at question #213871). BIT did redraw the complete restore log everytime a new line was added. As more lines this was, the longer it took. This is fixed since 1.0.18.

BIT will only take new snapshots at times you scheduled in Settings or if start it manually. It shouldn't start a new snapshot if you just open the GUI. Otherwise it would be a bug I don't know yet. Please double check this.

Restore doesn't call chmod/chown with subprocess. It uses os.chmod / os.chown (python) which is lot faster than subprocess calls. The log line only shows you what have been done.

'Full rsync mode' as recommended by Benjamin has nothing to do with you problems.

Regards,
Germar

Revision history for this message
Meffy (dameffy) said :
#3

Hi there!
I'm facing now the same problem with low speed during creation of snapshots with version 1.1.4, i.e. rsnyc alone works quite fine but BIT seems to be frozen (well, it is doing something, but as said very slowly...). The destination folder lies on a NAS-server, mounted via NFS. BTW, the same setup works fine on my Debian machine connected via LAN-cable; the problem occurs on my laptop with Ubuntu 14.04 installed connected via WiFi.
I would be very happy if you could give me a clue. I will of course give you any supplementary information needed; I did not send any because by default these are always the unnecessary ones ;-).
Thanks in advance!
Greetings
Matthias

Revision history for this message
Germar (germar) said :
#4

This is not the same as the original Question (that was a bug in restore dialog).

Could you please switch over from mounting your NAS via NFS to BackInTime's SSH mode? Syncing to a local mounted NFS is very slow because for each and every filesystem action there has to be a tcp package send to the NAS and back. With SSH mode rsync will run on your NAS and will only transfer those parts of files which had changed.

If your NAS has a native Linux filesystem (e.g. EXT) you could also enable 'Full rsync mode' which will speed up the snapshot process, too.

Regards,
Germar

Revision history for this message
Meffy (dameffy) said :
#5

Hi Germar!
Thanks for the answer!! I'm now struggling a little bit with the ssh-connection which works fine in the console. But (there is always a but ;-)) when trying to set up BIT (root or non-root) I get the message "Host does not support rsync..." (tried to translate it from German to English...). Trying the given rsync-command in the console does not work (just to be sure...). The first step to solve it is to give a additional rsync option for the rsync-path on the remote machine because it's not located under /usr/bin, but when setting this path in the BIT GUI under KDE the rsync command in the error dialogue does not change, i.e. the additional option is not added...??? I created a symbolic link in /usr/bin to rsync, but rsync is not interested in this link...Any idea?

Revision history for this message
Germar (germar) said :
#6

The automatic tests running after pressing OK in Settingsdialog doesn't always get all new settings (including the additional options). This is a bug and I'll fix this for next version.

For now you can either add the option manually in ~/.config/backintime/config:
profile1.snapshots.rsync_options.enabled=true
profile1.snapshots.rsync_options.value=--rsync-path=/opt/bin/rsync

or add rsyncs path to the $PATH environ on your remote host:
ssh user@host echo "PATH=/opt/bin/:\$PATH" > ~/.ssh/environment

You would also need to activate "PermitUserEnvironment yes" in your /etc/ssh/sshd_config on remote.

Revision history for this message
Germar (germar) said :
#7

The bug about ignoring new settings in automatic tests is fixed in current testing release. If you would like to test this please run

sudo add-apt-repository ppa:bit-team/testing
sudo apt-get update
sudo apt-get upgrade

Revision history for this message
Meffy (dameffy) said :
#8

Hello Germar!
Thanks for your reply! So I'm now one step further but it still doesn't work on my Synology NAS...There is a instruction how to set up the server (http://wiki.qedinux.be/index.php?title=BackInTime_to_Synology_DSM_5), but this doesn't work (ssh, sshfs and sftp are using some different roots as far as I understand...), but this gave me a clue. Fortunately your code is great to read so with some debugging I should be able to find a workaround, either on server side or on BiT side; depending what's more fun ;-). Hope the kids will give me some two hours to play around with it ;-).
Greetings
Matthias

Can you help with this problem?

Provide an answer of your own, or ask Sébastien Pierre for more information if necessary.

To post a message you must log in.