Backup script question

RichMiller

New Member
Credits
121
Hello,

I have very limited Linux experience but I have recently turned a raspberry pi into a home & office server (as i'm now working from home for the foreseeable). I feel a bit like the setup I have created is akin to Frankenstein's monster, but it seems to work. I have a series of USB3 external drives connected via a USB hub & a WD Mycloud drive heading into a small linksys switch, along with the pi and PCs.

I am currently trying to figure out a way to backup files from the WD Mycloud onto one of the USB3 drives, I have followed a tutorial on Medium to create a (I am learning as I go here) script as follows; (this is saved into my home/pi folder

#!/bin/bash #typically the first line of the bash scripts

BACKUPTIME=`date +%b-%d-%y` #get the current date

DESTINATION=/mnt/sdd1/PROJECTS ARCHIVE/Test Move To-$BACKUPTIME.tar.gz #create a backup file using the current date in it's name

SOURCEFOLDER=afp://Jupiter.local/Public on Jupiter/Caravan/001 Caravan Projects/Test Move From #the folder that contains the files that we want to backup

tar -cpzf $DESTINATION $SOURCEFOLDER #create the backup



I have used sudo crontab -e in the terminal to create what I hope is a scheduled event, this is the line of text I entered at the bottom of the nano terminal

15 12 * * * /bin/bash /home/pi/backup_script.sh

As you can assume, it's not working! I'm wondering if it has something to do with the pi stumbling with permissions from WD Mycloud, but that's only a very very uneducated hunch, just because the Mycloud is not a simple USB drive, it's attached via an ethernet cable directly into the switch.

I'd be hugely grateful for anyone to sacrifice the time to read this and if possible to lend some advice.
Many thanks

Rich


As a note - my intention is to have the projects folder backed up every night (essentially the same operation as windows copying all files over and choosing 'no' to overwriting).
 
Last edited:


wizardfromoz

Super Moderator
Staff member
Gold Supporter
Credits
7,577
G'day @RichMiller and welcome to linux.org :)

I am moving this Thread to Command Line, where scripting inquiries are better handled.

Good luck

Chris Turner
wizardfromoz
 

khedger

Active Member
Credits
919
Okay,

So when you say it's not working, what exactly do you mean?
Also, what are the permissions on all of the files to be backed up and created - do you have permission to write to them?
You might want to add sudo to your command in the crontab.

keith
 

RichMiller

New Member
Credits
121
Okay,

So when you say it's not working, what exactly do you mean?
Also, what are the permissions on all of the files to be backed up and created - do you have permission to write to them?
You might want to add sudo to your command in the crontab.

keith
Hello! Thanks for replying.

Regarding the not working comment, I had no idea what the issues were as the files in the 'from' folder just weren't copying over. (I've realised from some tutorials that you can see a list of errors if you perform the command in terminal, I was using crontab and couldn't see any errors)

I did some more tinkering last night, and I've ended up scrapping the original script! I now have this;

#!/bin/bash

cp -ar "/mnt/Jupiter/Caravan/001 Caravan Projects/TestMoveFrom/"* "/mnt/sdd1/PROJECTS ARCHIVE/Test/"

And it worked! It's the first time I've ever 'written' anything and seen it function! Honestly I was so chuffed when the files appeared in the folder.

Also, I'd done sudo crontab during the evening! (Think I'm getting the hang of the 0.000001% Linux that I know)

I'd like to ask you a couple of questions about the new script;
1) Will this work the same way as Windows copying files over and selecting 'do not overwrite' anything that hasn't changed?
2) What does the -ar mean? I know I say i've written this, I should know, but I must have grabbed it from a tutorial or forum somewhere, but just can't find it again.

Once again, cheers for helping me out!
Rich


EDIT: Quite a big piece of the puzzle was mounting the WD Mycloud! I wonder if that had anything to do with the original script not working..
 

jglen490

Well-Known Member
Credits
3,116
For 1): Not exactly, but there is a way to do that.
For 2): The -ar means use cp (copy) to create a copy using archive logic that saves all the attributes of the source files (-a) and perform the command recursively (-r) on all subdirectories.

The cp command can be useful, I use the rsync command
Code:
sudo rsync -auv <source> <target>
. The -auv says to archive, using update logic, and do it verbosely. The archive logic preserves all the date and other attributes of the files being copied from the <source> to the <target>. The update logic skips files that already exist at the <target>, unless the date on the <source> file is newer, and the verbose logic sends a lot of output to the terminal screen.

It works great for me!
 

khedger

Active Member
Credits
919
Hmmm.....upon reviewing the 'cp' man page it looks to me that -ar will preserve the directory structure and recursively cp any directories that are 'below' the specified directory.

If either your source directory or destination directory were not in a mounted filesystem, then no 'copy' command will work. If this was the case originally, then I'd test your original command with everything mounted. It should work. 'rsync' is a better solution than 'cp' here. 'rsync' will backup only those files that have changed, which is what you really want to have happen. 'cp' will simply copy over the existing backups everytime you run it. So, again, I'd make sure everything is mounted properly and run the 'rsync' command(s) with 'sudo'. From what you've said so far, 'rsync' should work.

keith
 

JasKinasis

Well-Known Member
Credits
4,104
@RichMiller - I'm going to answer your two questions in reverse order.
Starting with the second question:
What does the -ar mean?
If you want help with any terminal based commands - your first port of call should always be the man pages.

If you run the command man cp you can see the meaning of all of the options for cp and their parameters.

The -a (or --archive) option is the "archive" option.
The man page also tells us that the -a option is the same as running cp -dR --preserve=all.
Where -d (or --no-dereference) means "never follow symbolic links", the -R (or -r, or --recursive) means copy directories recursively. And --preserve=all means that the cp command will attempt to preserve all file attributes.

I say "attempt to preserve the attributes" because:
If you are copying to certain file-systems, then some file attributes might not be preserved.

Also because you are using the -a option, I would say that the -r optiion is unecessary.
I say this because the -a option is equivalent to using -dR --preserve=all. And the -R in that means that it's already recursive, so you can just use -a instead of -ar.

1) Will this work the same way as Windows copying files over and selecting 'do not overwrite' anything that hasn't changed?

No, it will not. But if you use cp's -u, or --update option it will only copy when the source file is newer than the version of the same file at the destination, or when the file doesn't already exist at the destination.

So perhaps cp -au would meet your needs more closely than cp -ar.

EDIT: damn, @jglen490 and @khedger got in there whilst I was still noodling away.

+1 for the rsync suggestion too chaps!
I decided to stick with cp as it was what the OP was using.
 
Last edited:

khedger

Active Member
Credits
919
JasKinasis obviously has more knowledge of the intracacies of the 'cp' command than I do :cool: So it looks like with the -au options the 'cp' command may work for you. All things being equal, if you've gotten 'cp' to work, then 'rsync' should probably work too. Anyway, good luck and let us know if you get things straightened out....

keith
 

RichMiller

New Member
Credits
121
For 1): Not exactly, but there is a way to do that.
For 2): The -ar means use cp (copy) to create a copy using archive logic that saves all the attributes of the source files (-a) and perform the command recursively (-r) on all subdirectories.

The cp command can be useful, I use the rsync command
Code:
sudo rsync -auv <source> <target>
. The -auv says to archive, using update logic, and do it verbosely. The archive logic preserves all the date and other attributes of the files being copied from the <source> to the <target>. The update logic skips files that already exist at the <target>, unless the date on the <source> file is newer, and the verbose logic sends a lot of output to the terminal screen.

It works great for me!
Thank you for your time on this! Does the term 'archive' refer to zipping up a directory? i.e would the rsync command copy everything over, but the files in the target would be zipped files (excuse the total layman's terms here, hope my question is clear)
 

RichMiller

New Member
Credits
121
Hmmm.....upon reviewing the 'cp' man page it looks to me that -ar will preserve the directory structure and recursively cp any directories that are 'below' the specified directory.

If either your source directory or destination directory were not in a mounted filesystem, then no 'copy' command will work. If this was the case originally, then I'd test your original command with everything mounted. It should work. 'rsync' is a better solution than 'cp' here. 'rsync' will backup only those files that have changed, which is what you really want to have happen. 'cp' will simply copy over the existing backups everytime you run it. So, again, I'd make sure everything is mounted properly and run the 'rsync' command(s) with 'sudo'. From what you've said so far, 'rsync' should work.

keith
just googled cp man page, and now my ability to learn this foreign language has grown!
Definitely seems like rsync is a better option, thank you for taking the time to answer. Cheers!
 

RichMiller

New Member
Credits
121
@RichMiller - I'm going to answer your two questions in reverse order.
Starting with the second question:
What does the -ar mean?
If you want help with any terminal based commands - your first port of call should always be the man pages.

If you run the command man cp you can see the meaning of all of the options for cp and their parameters.

The -a (or --archive) option is the "archive" option.
The man page also tells us that the -a option is the same as running cp -dR --preserve=all.
Where -d (or --no-dereference) means "never follow symbolic links", the -R (or -r, or --recursive) means copy directories recursively. And --preserve=all means that the cp command will attempt to preserve all file attributes.

I say "attempt to preserve the attributes" because:
If you are copying to certain file-systems, then some file attributes might not be preserved.

Also because you are using the -a option, I would say that the -r optiion is unecessary.
I say this because the -a option is equivalent to using -dR --preserve=all. And the -R in that means that it's already recursive, so you can just use -a instead of -ar.

1) Will this work the same way as Windows copying files over and selecting 'do not overwrite' anything that hasn't changed?

No, it will not. But if you use cp's -u, or --update option it will only copy when the source file is newer than the version of the same file at the destination, or when the file doesn't already exist at the destination.

So perhaps cp -au would meet your needs more closely than cp -ar.

EDIT: damn, @jglen490 and @khedger got in there whilst I was still noodling away.

+1 for the rsync suggestion too chaps!
I decided to stick with cp as it was what the OP was using.
This post alone must have taken a while to write, so I'm very thankful. Seems like a great community willing to help someone with what feels like a simple problem!

Would a symbolic link be (e.g) a 'shortcut file' in Windows? Say I had a shortcut to a folder on C, then cp wouldn't go down that path if -d was used?

I'm wondering if rsync would be better suited, I used cp because... I knew no better!

Also just learnt how to better format my posts.

Thanks again for the time,
 

khedger

Active Member
Credits
919
No, the 'archive' option (for both commands) has more to do with preserving directory structure and links in the directory being copied.
You'd want to add tar/gzip commands to your script to run after the cp/rsynce.

keith
 

RichMiller

New Member
Credits
121
No, the 'archive' option (for both commands) has more to do with preserving directory structure and links in the directory being copied.
You'd want to add tar/gzip commands to your script to run after the cp/rsynce.

keith
Ah ok, just to clarify i'm not looking to zip anything, just creating a copy of folders & files within on a separate hard drive.

Currently looking into rsync, user jglen490 shared that rsync script and i've just realised he has -auv all together, yet they are different commands for the rsync. an onion skin layer has been peeled back for me!
(having some simple epiphanies over here)

Trying to google but can't find the answer, why would you want Verbose? Is it purely for documentation/error log reasons?
Does anyone know if it would impact the time spent copying GBs (the purpose of this is to backup my projects folder, which has large photoshop files in (usually 1GB+)
 

khedger

Active Member
Credits
919
Verbose will print a list of the files it is processing to the terminal window. You might want to use it to verify that everything you think should be copied is being copied. Or not.....depends on the user.....'-auv' are what are called command options in linuxese. There are a couple of different styles of command arguments. The traditional UNIX style is what you see here, a '-' followed by one or ore options. There was a standard called POSIX instituted in the 80s that included a standardized command format to be used in all POSIX compliant 'flavors' of UNIX (Linux derives a lot of its constituent code and practices from UNIX). Under POSIX the options look more like (these are not exact, just examples of what POSIX options look like) --archive --verbose etc. etc. etc. When reading linux man pages, you'll see that many commands support both 'traditional' and 'POSIX style' options.....

keith
 

RichMiller

New Member
Credits
121
Success!
I scheduled the backup script for 01:00 and woke up this morning to an eye-wateringly beautiful directory backup on the destination drive

At the end of the terminal readout I have a
'rsync warning: some files vanished before they could be transferred (code 24) at main.c(1207) [sender=3.1.3]'

According to https://rsync.samba.org/FAQ.html#10 I should ignore the 'vanished files' warning.
I've also read this happens when rsync is 'wrapped' in a script (which mine is, in a .sh file)

Anyone had experience with this? Should I worry that files have not successfully transferred over? Or just ignore this message as rsync.samba.org states?
(PS - I am using samba as the server for all these files)

Cheers,
Rich
 

RichMiller

New Member
Credits
121
I'm back!

I've spent the last few days wrestling with postfix in order to have the progress of the rsync emailed to me (I've set it to copy over at 1am each morning). Which now works, surprisingly.
However I've just run a test copy of the script and it seems to have stopped at ~1.3GBs of a 1.7GB file.

Just did a google and it seems rsync might have issues handling large files? I found this article, but a lot of it is far too over my head for this point in my life.

It was working earlier in the week, and I can see files have been copied over larger than 1.7GB. But as I've recently set up postfix - I have ended up changing permissions for different things related to it (one example, I was making changes to the hostname file at one point) and I'm now wondering if that has something to do with it?)

Once again, your help and time is appreciated.
Cheers,
Rich


EDIT: as the source directory is a WD Mycloud drive, it appears as an afp link, it seemed like a complicated process but I eventually got it mounted. It felt like there was some to-ing and fro-ing of permission changes in order to get it there.
 
Last edited:

wizardfromoz

Super Moderator
Staff member
Gold Supporter
Credits
7,577

RichMiller

New Member
Credits
121
(This is going to sound like passive aggression, but this is genuinely not) I appreciate the silence for my previous questions! I feel a bit like you've all been here before and knew not to answer, as I found it all out for myself and have learnt a sh*t load in the process! Although I am still a tiny bit worried about my slack attitude to the permissions, but I think it's fine.
Once again, I have a backup of the files at 1am, AND an email from the scary sounding Cron Daemon to let me know what it's copied over.

I adjusted the rsync script to this -arvh and it seems to work in the same was as -u does? last night only 21K was sent over.. i'm assuming that meant it didn't copy over stuff already in the destination folder?

It seems to be functioning!

Cheers,
Rich
 


Staff online

Members online


Top