Problem: Rsync makes a new copy of files with each backup. Not incremental

[userzero]

Hi All,

I'm using rsync to back up local files to Google Drive. The first backup is successful but the second backup adds duplicate files with each backup. Its just doing a whole full new backup- not incremental. Below is rsync as I'm using it:

rsync -hvaPHL /home/hudson/Desktop /mnt/GDrive/backup/hudson-2 1> /mnt/livelog-hudson.txt 2> /mnt/errors-hudson.txt

Edit: The source computer in question is the second computer that is mounting Google Drive. I'm using rclone to mount GDrive and set it up after the first computer. The rsync command is exactly the same on both computers and the first computer isn't having the above problem. The only difference is that rsync is backing up to different directories:

Computer #1:
rsync -hvaPHL /home/hudson/Desktop /mnt/GDrive/backup/hudson-1 1> /mnt/livelog-hudson.txt 2> /mnt/errors-hudson.txt

Computer #2:
rsync -hvaPHL /home/hudson/Desktop /mnt/GDrive/backup/hudson-2 1> /mnt/livelog-hudson.txt 2> /mnt/errors-hudson.txt

Note the different directories in bold.

Any help much appreciated!

 

[f33dm3bits]

Do you actually have any symlinks or hardlinks in your directory structure that you are syncing to that location? It will probably be enough if you do something like this:

rsync -av /home/hudson/Desktop/* /mnt/GDrive/backup/hudson-2

This would sync all the files on your Desktop to /mnt/GDrive/backup/hudson-2.
What files are you actually trying to backup, your whole homedir or just what is on your Desktop?
Are you getting any error messaging when you are doing your sync from the second computer to the hudson-2 backup directory?

 

[userzero]

Hello and thanks for the reply. I don't currently have any symlinks but could in the future. I'll try just
-av and see if that helps. No error messages at all- rsync just adds copies of all files with each backup. For testing purposes, I deleted the rclone job from the old GD account and mounted via rclone, a new GD account to test. The same thing happens. I'll try the -av and report back. Thanks again!

 

[f33dm3bits]

As far as I am understanding what you are saying is you use rclone to mount your google drive directory on /mnt/Gdrive. Then you rsync from computer 1 and computer 2 to the corresponding backup directories, hudson1 and hudson-2. Can you share the output of the command you run when you report back?

 

[userzero]

I've created a fresh GDrive account for testing and mounted it via rclone as before. I used the rsync -av only, as recommended above and performed two rsync backups. Again, two copies of each file on Google Drive. If I do a third backup as previously, it should create a third copy. Needless to say I only want one copy of each file. There were no error messages to stderr but I've included the information to stdout from both backups below. This time I used a smaller directory called cat: (thanks again for your interest and help)

Backup #1: (to stdout)

sending incremental file list
cat/
cat/HAL9000-blue.png
cat/HAL9000.png
cat/Lime-icon.jpg
cat/aqua-black.png
cat/ceiling-hanging4.png
cat/dark-amarok.png
cat/dragon-middlewall.png
cat/dressers3.png
cat/emblem-money.png
cat/emerald-theme-manager-icon.png
cat/firefox.png
cat/gftp.png
cat/gnome-apple.png
cat/gnome-cd.png
cat/gnome-favorites.png
cat/gnome-home.png
cat/gnome-system-monitor.png
cat/green-black.png
cat/green.jpg
cat/left-hanging3.png
cat/lights3.png
cat/lime.jpeg
cat/pauf-leftfloor.png
cat/polaris.gif
cat/right-corner4.png
cat/snowflake.jpg
cat/stove3.png
cat/swfdec.png
cat/tvtiki-tv.png
cat/wall5.png

sent 1,141,373 bytes received 590 bytes 11,833.81 bytes/sec
total size is 1,139,018 speedup is 1.00

Backup #2: (to stdout)
sending incremental file list
cat/
cat/HAL9000-blue.png
cat/HAL9000.png
cat/Lime-icon.jpg
cat/aqua-black.png
cat/ceiling-hanging4.png
cat/dark-amarok.png
cat/dragon-middlewall.png
cat/dressers3.png
cat/emblem-money.png
cat/emerald-theme-manager-icon.png
cat/firefox.png
cat/gftp.png
cat/gnome-apple.png
cat/gnome-cd.png
cat/gnome-favorites.png
cat/gnome-home.png
cat/gnome-system-monitor.png
cat/green-black.png
cat/green.jpg
cat/left-hanging3.png
cat/lights3.png
cat/lime.jpeg
cat/pauf-leftfloor.png
cat/polaris.gif
cat/right-corner4.png
cat/snowflake.jpg
cat/stove3.png
cat/swfdec.png
cat/tvtiki-tv.png
cat/wall5.png

sent 1,141,373 bytes received 590 bytes 14,927.62 bytes/sec
total size is 1,139,018 speedup is 1.00

As you can see, it sent the incremental file list both times but just re-performed the backup. These backups were about 5 minutes apart with no changes to the directory or files.

 

[f33dm3bits]

Really strange. Can you do a listing of two files that are the same in your backup directory for the second system in: mnt/GDrive/backup/hudson-2. As from your first post, you are only having this problem with system2 right, not with system 1? I think it may have to do with how rclone handles the syncing of files. How do you sync or mount your google drive with rclone? I'm getting the idea that rclone has a sync option which you use which would cause a new modification time when you sync the Gdrive files. So when you do a rsync again rsynce sees the files as different and then attempts to sync them again. Also can you give show the output of df when your Gdrive is mounted/synced?

 

[userzero]

Sorry for the late response. I've decided to just use my USB drive to back up to. My main Debian computer rsync backs up exactly as it should with incremental back up jobs. Maybe I'll try and tackle this another time with the laptop in question. I do really appreciate the time and interest you've taken in helping me troubleshoot. Thanks much and be safe.

 

[userzero]

Update:

I was using Linux Lite 4.8 when I was having the above problem. Linux Lite 5.0 is out now so I installed it. I repeated the same steps with rclone and rsync and now all is well. No duplicate files. Weird.