RSYNC syntax for pushing file with latest system date


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting RSYNC syntax for pushing file with latest system date
# 1  
Old 08-11-2009
Error RSYNC syntax for pushing file with latest system date

OK, I am a little new to AIX 5.3 and also to scripting. I have a shell script that I wrote and am having difficulty pushing specific files by the system date. Here is my script:

Code:
#!/usr/bin/sh

RSYNC=/usr/local/bin/rsync
SSH=/usr/local/bin/ssh
KEY=<path> somekey.key
RUSER=mike
RHOST=bulldog
RPATH=<path> sync_test (this is the line for the file to be moved to)
LPATH=<path> (this is the line where the file is moving from) 

$RSYNC -arpe "$SSH -i $KEY" $LPATH $RUSER@$RHOST:$RPATH

For security reasons I removed the paths and certain file names

I am thinking there is either a seperate line for the date information or the date is at the end of the line from LPATH.

Any help would be much appreciative!!

-Thanks
# 2  
Old 08-11-2009
Quote:
Originally Posted by tfort73
I have a shell script that I wrote and am having difficulty pushing specific files by the system date. Here is my script:
I'm not sure I understood the objective.
- Do you want to push only files that have been updated?
You're already doing this with --archive..since it assumes --times. So only those files that are newer than the destination will copy. Keep in mind to check the time of your boxes (ntp a huge plus here)

- Do you want only push files that have updated in a timeframe you choose?
If you're wanting to do something that only sends files that have been updated in a timeframe which you set, you might want to consider a staging script to analyze which files you need to send...thn use rsync to send them.

- Maybe I'm way off?

Thanks!
J
# 3  
Old 08-11-2009
re: RSYNC syntax for pushing file with latest system date

On our production server everyday, "1" file will be generated and stored in the starting directory. Now I have to find a way to move the newest file (by date) to be pushed to the DEVL server. Remind you I have to keep all other files in the same directory. So, I figure by "date" or "system date" rysnc will know to move that particular file.

Does this help?

I am looking for the syntax to complete this task

-thanks
# 4  
Old 08-11-2009
Quote:
Originally Posted by tfort73
On our production server everyday, "1" file will be generated and stored in the starting directory. Now I have to find a way to move the newest file (by date) to be pushed to the DEVL server. Remind you I have to keep all other files in the same directory. So, I figure by "date" or "system date" rysnc will know to move that particular file.

Does this help?

I am looking for the syntax to complete this task

-thanks
If I understand correctly I think you're pretty much there. You might want to remove the -e from your rsync which would prevent files from being updated if they did not already exist. You can also drop the -pr from your options..it is redundant. Using -a assumes (-rlptgoD).

So try just:
rsync -a </source/dir/> </desintation/dir/>

Here is an example on how I can keep a dev server in synch with the production. Notice I have 3 files in 3 directories in my "source" for testing. I'm using the same server...but that does not matter here.
Code:
jnetnix@nix:/tmp/Testing$ find ./Srcfiles/ -type f -exec ls -l {} \;
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir1/file1
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir1/file2
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir1/file3
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir2/file1
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir2/file2
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir2/file3
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir3/file1
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir3/file2
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir3/file3
jnetnix@nix:/tmp/Testing$ find ./DevFiles/ -type f -exec ls -l {} \;
jnetnix@nix:/tmp/Testing$

I run rsync, in recursive mode and enable --verbose and --stats for testing.
Code:
jnetnix@nix$ rsync -a \
  --verbose --stats \
  /tmp/Testing/Srcfiles/ localhost:/tmp/Testing/DevFiles/
jnetnix@localhost's password: 
sending incremental file list
./
dir1/
dir1/file1
dir1/file2
dir1/file3
dir2/
dir2/file1
dir2/file2
dir2/file3
dir3/
dir3/file1
dir3/file2
dir3/file3

Number of files: 13
Number of files transferred: 9
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 197
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 576
Total bytes received: 198

sent 576 bytes  received 198 bytes  172.00 bytes/sec
total size is 0  speedup is 0.00

I can now see that my Src and Dev are mirrored.
Code:
jnetnix@nix:/tmp/Testing$ find ./Srcfiles/ -type f -exec ls -l {} \;
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir1/file1
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir1/file2
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir1/file3
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir2/file1
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir2/file2
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir2/file3
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir3/file1
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir3/file2
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./Srcfiles/dir3/file3
jnetnix@nix:/tmp/Testing$ find ./DevFiles/ -type f -exec ls -l {} \;
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./DevFiles/dir1/file1
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./DevFiles/dir1/file2
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./DevFiles/dir1/file3
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./DevFiles/dir2/file1
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./DevFiles/dir2/file2
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./DevFiles/dir2/file3
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./DevFiles/dir3/file1
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./DevFiles/dir3/file2
-rw-r--r-- 1 jnetnix jnetnix 0 2009-08-11 13:20 ./DevFiles/dir3/file3

I can re-run rsync all day long and nothing that is not new will transfer. you could even cron this part once an hour or so.
Code:
jnetnix@nix:/tmp/Testing$ rsync -a \
  --verbose --stats \
  /tmp/Testing/Srcfiles/ localhost:/tmp/Testing/DevFiles/
jnetnix@localhost's password: 
sending incremental file list

Number of files: 13
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 197
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 213
Total bytes received: 15

sent 213 bytes  received 15 bytes  65.14 bytes/sec
total size is 0  speedup is 0.00

If I update a file, or two, they will be the only ones sent to the destination.
Code:
jnetnix@nix:/tmp/Testing$ touch ./Srcfiles/dir1/file4
jnetnix@nix:/tmp/Testing$ touch ./Srcfiles/dir2/file4
jnetnix@nix:/tmp/Testing$ rsync -a \
  --verbose --stats \
  /tmp/Testing/Srcfiles/ localhost:/tmp/Testing/DevFiles/
jnetnix@localhost's password: 
sending incremental file list
dir1/
dir1/file4
dir2/
dir2/file4

Number of files: 15
Number of files transferred: 2
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 219
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 319
Total bytes received: 59

sent 319 bytes  received 59 bytes  108.00 bytes/sec
total size is 0  speedup is 0.00
jnetnix@nix:/tmp/Testing$

Hope it helps...good luck!
-J
# 5  
Old 08-12-2009
Thanks for all that information. I now have a better understanding how RSYNC works. I was able to complete my task.

Thank you!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

To get latest hdfs file system

Hi All, I am having below hdfs file system /user/home/dte=2019_01_30/part_1 /user/home/dte=2019_01_30/part_2 /user/home/dte=2019_01_31/part_1 I need to take the latest month hdfs folder while passing date as parameter. For eg . if i pass as Feb month i.e. 20190201(YYYYMMDD), then... (0 Replies)
Discussion started by: Master_Mind
0 Replies

2. Shell Programming and Scripting

Finding latest file in dir but getting syntax errors

I believe there are couple of syntax issues in my script, couldn't find them :( can someone help me with fixing it to make it work. cd /abcde/ #get the latest filename excluding subdirs filename=`ls -ltr | grep ^- | tail -1 | awk '{print $8}'` #get system date and file timestamp and... (3 Replies)
Discussion started by: simpltyansh
3 Replies

3. UNIX for Dummies Questions & Answers

How to pick the latest file with date as one among the file name.( not exactly present date.)?

i have files like 1)20131112_abc_01.csv and 2)20131113_abc_01.csv and 3)20131113_abc_02.csv when i try to fetch the file in the next day. it shud pick the third file.. plz help me.. and i use `date +"%Y%m%d"` command to fetch..it fetches the current date... (2 Replies)
Discussion started by: applepie
2 Replies

4. Shell Programming and Scripting

Find the latest file based on the date in the filename

Hi, We've a list of files that gets created on a weekly basis and it has got a date and time embedded to it. Below are the examples. I want to find out how to get the latest files get the date and time stamp out of it. Files are PQR123.PLL.M989898.201308012254.gpg... (1 Reply)
Discussion started by: rudoraj
1 Replies

5. Shell Programming and Scripting

Needed shell script to get the latest file based on date

hi i need the shell script to get the file with latest date. for example in my input folder i have files like file_20130212.txt file_20130211.txt now my output folder should have the file with latest date i.e..file_20120212.txt i want to get the latest file .. i.e is should take... (6 Replies)
Discussion started by: hemanthsaikumar
6 Replies

6. UNIX for Dummies Questions & Answers

Selecting the file of latest Date

Hi Folks, I have one query that there is a folder in which daily several logs files are getting created , I reached to that location through putty but what I observer that 10 files of different date are been created with same name , what I need to see is the latest file ...let say the location is ... (5 Replies)
Discussion started by: KAREENA18
5 Replies

7. Shell Programming and Scripting

Grep the Content of a LOG File which has latest Date and Time

Hi All, Need a small help. I have a log file which keeps updating for every Minute with multiple number of lines. I just want to grep few properties which has latest Date and Time to it. How do i do it? I wanted to grep a property by name "Reloading cache with a maximum of" from the... (4 Replies)
Discussion started by: nvindraneel
4 Replies

8. Shell Programming and Scripting

grep latest file based on date.

hi all, not sure if this has been posted b4 but i try to search but not valid. this is my question: when i do a ls -ltr there will be a list generated as follows: -rw-r--r-- 1 root sys 923260 Jan 10 04:38 FilePolling.41025.083TL021.xml -rw-r--r-- 1 root sys 1761337 Jan 10 04:40... (12 Replies)
Discussion started by: lweegp
12 Replies

9. UNIX for Dummies Questions & Answers

get the latest file by reading the date in the filename.

Hi, I grep for a pattern in a list of files. "grep -i -l $pattern *.datx*" it may give me n number of files. say for eg, it gives me 2 files. lock_eicu_20071228_00000000.dat_20071228_05343100 lock_eicu_20080501_00000000.dat_20080501_05343900 out of these 2 files I need to get the... (7 Replies)
Discussion started by: prsshini
7 Replies

10. Shell Programming and Scripting

Loop through files in dir, omit file with latest date

I want to loop through files in a directory but omit the file with the latest date in my list of files. How would I accomplish this? Thanks (2 Replies)
Discussion started by: stringzz
2 Replies
Login or Register to Ask a Question