Shell Programming and Scripting

I have set several cron jobs. I recently added a new cron job that copies a file of last day from another server and is executed each day (for example at 04:00 am) but when I check next day the file hasn't been copied.

I'm working in GNU/Linux CentOS (2.6.32) system.

The files that I need to copy are created in the origin server before 23:00 hours.

I had the cron job like below: (The first line is the job that fails. related script to copy the files is script1.sh)

Code:

	
[root@srvc ~]# crontab -l
0 6 * * * /path/to/scripts/script1.sh
0 5 * * * /path/to/scripts/script3.sh
0 8 * * 1 /path/to/scripts/script4.sh

When I checked next day about 11:00 am I see the file wasn't copied, I edited cron job to be executed at 11:10 am and the file was copied successfully.

Code:

    
[root@srvc ~]# crontab -l
10 11 * * * /path/to/scripts/script1.sh

The script1.sh content is this:

Code:

#!/bin/bash
 
dyear=`date +'%Y' -d "1 day ago"`
dmonth=`date +'%b' -d "1 day ago"`
ddate=`date +%Y-%m-%d -d "1 day ago"`

sshpass -p 'ThePassword' scp -r root@X.X.X.X:/path/to/files/*$ddate* /Destination/path/$dyear/$dmonth/

The files to be copied have in their name the format Logfile.2018-01-17

How to know why cron job fails and how to fix it?

Thanks in advance.`

/path/to/files/*$ddate* is expanded on the calling host i.e. the destination host.
Quote the * characters to expand on the source host.
/path/to/files/\*$ddate\* or
/path/to/files/"*$ddate*" or
"/path/to/files/*$ddate*"
Note that within the " (double quotes) the $ddate is still expanded on the calling host.

Is that behaviour reproducible? What's the difference between the script executing at 6:00h and 11:10h? Anything in the log files? If not, modify the script to log its steps. Is it possible the files does not (yet) exist on the source node at 6:00h?

This was my first thought too. How about doing the following to investigate:

change the script like this:

Code:

#!/bin/bash
set -xv 
dyear=`date +'%Y' -d "1 day ago"`
dmonth=`date +'%b' -d "1 day ago"`
ddate=`date +%Y-%m-%d -d "1 day ago"`

sshpass -p 'ThePassword' scp -r root@X.X.X.X:/path/to/files/*$ddate* /Destination/path/$dyear/$dmonth/

Then modify your crontab like this:

Code:

10 11 * * * /path/to/scripts/script1.sh > /path/to/cronlog.log 2> /path/to/cronlog.err

and have a look at what is logged. My first suspect would be the unquoted globs too, like MadeInGermany already said.

Two things to notice: if you create cron jobs you should ALWAYS redirect their stdout and their stderr - either to a (log-)file or to /dev/null if you are not interested. Otherwise any output the script eventually generates creates a mail to root which you probably want to avoid.

Second, you should really, really do away with sshpass. Even the developers admit that it is ill advised to use it and it is offered just as a last straw effort. When such a process ist started you can see the password in cleartext in the process list - not to mention the script file itself. You might secure the script file against being read by everybody but the output of ps is public information.

You can easily try it yourself: open two terminal windows as a normal user to some host. Issue in one of them:

Code:

sshpass -p 'ThePassword' ssh root@X.X.X.X sleep 1000

Now issue in the other window, while this runs

Code:

ps -fe | grep [s]sh

and you will see the password there.

I hope this helps.

bakunin

Why would it fail at 6:00h but work at 11:10h, then?

Hi to all,

Thanks for your answers.

IMHO the issue is not the expansion of the variable because it worked at 11:10 but not at 06:00 and the script1.sh is the same.

The files to be copied are created before 23:00 hours, so that wouldn't be the reason either.

In order to redirect the log like bakunin saiys, how to know which is the path for cronlog.log and cronlog.err?

This is up to you to chose. For temporary, transient debugging you might select your own home directory, for permanent logging, /var/log lends itself to usage.