RSYNC syntax for pushing file with latest system date
OK, I am a little new to AIX 5.3 and also to scripting. I have a shell script that I wrote and am having difficulty pushing specific files by the system date. Here is my script:
Code:
#!/usr/bin/sh
RSYNC=/usr/local/bin/rsync
SSH=/usr/local/bin/ssh
KEY=<path> somekey.key
RUSER=mike
RHOST=bulldog
RPATH=<path> sync_test (this is the line for the file to be moved to)
LPATH=<path> (this is the line where the file is moving from)
$RSYNC -arpe "$SSH -i $KEY" $LPATH $RUSER@$RHOST:$RPATH
For security reasons I removed the paths and certain file names
I am thinking there is either a seperate line for the date information or the date is at the end of the line from LPATH.
I have a shell script that I wrote and am having difficulty pushing specific files by the system date. Here is my script:
I'm not sure I understood the objective.
- Do you want to push only files that have been updated?
You're already doing this with --archive..since it assumes --times. So only those files that are newer than the destination will copy. Keep in mind to check the time of your boxes (ntp a huge plus here)
- Do you want only push files that have updated in a timeframe you choose?
If you're wanting to do something that only sends files that have been updated in a timeframe which you set, you might want to consider a staging script to analyze which files you need to send...thn use rsync to send them.
re: RSYNC syntax for pushing file with latest system date
On our production server everyday, "1" file will be generated and stored in the starting directory. Now I have to find a way to move the newest file (by date) to be pushed to the DEVL server. Remind you I have to keep all other files in the same directory. So, I figure by "date" or "system date" rysnc will know to move that particular file.
On our production server everyday, "1" file will be generated and stored in the starting directory. Now I have to find a way to move the newest file (by date) to be pushed to the DEVL server. Remind you I have to keep all other files in the same directory. So, I figure by "date" or "system date" rysnc will know to move that particular file.
Does this help?
I am looking for the syntax to complete this task
-thanks
If I understand correctly I think you're pretty much there. You might want to remove the -e from your rsync which would prevent files from being updated if they did not already exist. You can also drop the -pr from your options..it is redundant. Using -a assumes (-rlptgoD).
So try just:
rsync -a </source/dir/> </desintation/dir/>
Here is an example on how I can keep a dev server in synch with the production. Notice I have 3 files in 3 directories in my "source" for testing. I'm using the same server...but that does not matter here.
I run rsync, in recursive mode and enable --verbose and --stats for testing.
Code:
jnetnix@nix$ rsync -a \
--verbose --stats \
/tmp/Testing/Srcfiles/ localhost:/tmp/Testing/DevFiles/
jnetnix@localhost's password:
sending incremental file list
./
dir1/
dir1/file1
dir1/file2
dir1/file3
dir2/
dir2/file1
dir2/file2
dir2/file3
dir3/
dir3/file1
dir3/file2
dir3/file3
Number of files: 13
Number of files transferred: 9
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 197
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 576
Total bytes received: 198
sent 576 bytes received 198 bytes 172.00 bytes/sec
total size is 0 speedup is 0.00
I can re-run rsync all day long and nothing that is not new will transfer. you could even cron this part once an hour or so.
Code:
jnetnix@nix:/tmp/Testing$ rsync -a \
--verbose --stats \
/tmp/Testing/Srcfiles/ localhost:/tmp/Testing/DevFiles/
jnetnix@localhost's password:
sending incremental file list
Number of files: 13
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 197
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 213
Total bytes received: 15
sent 213 bytes received 15 bytes 65.14 bytes/sec
total size is 0 speedup is 0.00
If I update a file, or two, they will be the only ones sent to the destination.
Code:
jnetnix@nix:/tmp/Testing$ touch ./Srcfiles/dir1/file4
jnetnix@nix:/tmp/Testing$ touch ./Srcfiles/dir2/file4
jnetnix@nix:/tmp/Testing$ rsync -a \
--verbose --stats \
/tmp/Testing/Srcfiles/ localhost:/tmp/Testing/DevFiles/
jnetnix@localhost's password:
sending incremental file list
dir1/
dir1/file4
dir2/
dir2/file4
Number of files: 15
Number of files transferred: 2
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 219
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 319
Total bytes received: 59
sent 319 bytes received 59 bytes 108.00 bytes/sec
total size is 0 speedup is 0.00
jnetnix@nix:/tmp/Testing$
Hi All,
I am having below hdfs file system
/user/home/dte=2019_01_30/part_1
/user/home/dte=2019_01_30/part_2
/user/home/dte=2019_01_31/part_1
I need to take the latest month hdfs folder while passing date as parameter.
For eg . if i pass as Feb month i.e. 20190201(YYYYMMDD), then... (0 Replies)
I believe there are couple of syntax issues in my script, couldn't find them :(
can someone help me with fixing it to make it work.
cd /abcde/
#get the latest filename excluding subdirs
filename=`ls -ltr | grep ^- | tail -1 | awk '{print $8}'`
#get system date and file timestamp and... (3 Replies)
i have files like
1)20131112_abc_01.csv and
2)20131113_abc_01.csv and
3)20131113_abc_02.csv
when i try to fetch the file in the next day. it shud pick the third file..
plz help me..
and i use `date +"%Y%m%d"` command to fetch..it fetches the current date... (2 Replies)
Hi,
We've a list of files that gets created on a weekly basis and it has got a date and time embedded to it. Below are the examples. I want to find out how to get the latest files get the date and time stamp out of it.
Files are
PQR123.PLL.M989898.201308012254.gpg... (1 Reply)
hi i need the shell script to get the file with latest date.
for example in my input folder i have files like
file_20130212.txt
file_20130211.txt
now my output folder should have the file with latest date i.e..file_20120212.txt
i want to get the latest file .. i.e is should take... (6 Replies)
Hi Folks,
I have one query that there is a folder in which daily several logs files are getting created , I reached to that location through putty but what I observer that 10 files of different date are been created with same name , what I need to see is the latest file ...let say the location is ... (5 Replies)
Hi All,
Need a small help. I have a log file which keeps updating for every Minute with multiple number of lines. I just want to grep few properties which has latest Date and Time to it. How do i do it?
I wanted to grep a property by name "Reloading cache with a maximum of" from the... (4 Replies)
hi all,
not sure if this has been posted b4 but i try to search but not valid.
this is my question:
when i do a ls -ltr there will be a list generated as follows:
-rw-r--r-- 1 root sys 923260 Jan 10 04:38 FilePolling.41025.083TL021.xml
-rw-r--r-- 1 root sys 1761337 Jan 10 04:40... (12 Replies)
Hi,
I grep for a pattern in a list of files.
"grep -i -l $pattern *.datx*"
it may give me n number of files.
say for eg, it gives me 2 files.
lock_eicu_20071228_00000000.dat_20071228_05343100
lock_eicu_20080501_00000000.dat_20080501_05343900
out of these 2 files I need to get the... (7 Replies)
I want to loop through files in a directory but omit the file with the latest date in my list of files. How would I accomplish this?
Thanks (2 Replies)
RSYNC syntax for pushing file with latest system date