Shell Programming and Scripting

Hi Don,
It woking fine

Thanks,
Mohan

---------- Post updated at 04:01 AM ---------- Previous update was at 02:57 AM ----------

Hi Don,
Your script is working fine ,i am getting the required files but i want to copy those files ,i have tried with another pipe to copy this but it throwing error

I can able to copy the files from s3(all files without using filter) to s3 using the below

Code:

aws s3  cp $folder/  $target  --recursive

Moderator's Comments:

Please use CODE tags to display code; not bold tags.

Code:

logdir=/home/hadoop/logs/
logfile=segment_$DATE2_inc.logDATE2=$(date --date='1 day ago' +%Y-%m-%d)
date --date='1 day ago' -u -d  "$(date --date='1 day ago' -u "+%a %b %e 00:00:00 %Z %Y")" +%s000 >unixtime 

x=`eval cat unixtime`

echo "processing of $DATE2 files with Unix timestamp $x " >$logdir/$logfile
 
folder="s3://xx-logs/JaZ/$x"
echo "Source Path of S3 bucket $folder">>$logdir/$logfile
echo $folder
target="s3://xx/test/"
chk="$DATE2 20:30"
#Uisng another pipe to copy only required files 
aws s3 ls $folder/|awk -v start="$chk" '$1" "$2 > start'|aws s3 cp *.gz $target --recursive
#aws s3 cp $test  $target  --recursive
#aws s3  cp $folder/|awk -v start='$DATE2 20:30' '$1 " " $2 > start { print $0 }'/  $target  --recursive


hadoop@ip-172-31-19-240 ~]$ ./xx_process_inc.sh 
s3://xx-logs/JaZ/1451088000000

2015-12-26 20:39:56     334080 file1.gz
2015-12-26 20:44:08     320179 file2.gz
2015-12-26 20:44:13     316953  file3.gz
2015-12-26 21:42:07     305313  file4.gz
2015-12-27 00:42:06     189541  file5.gz

Thanks,
Mohan

I will assume that the code marked in red above is on a line by itself and is not at the end of a line as shown...

I am always glad to hear that code that I have suggested is working. And, disappointed to see that many of the code improvements I suggested have been ignored.

Showing us a command that works is nice. Saying that when you modify it, you get an error without showing us what error(s) you get leaves us with a lot of work to try to duplicate your environment so we can figure out what error you might be getting (or just leaves us with little desire to try to help you). Since most of us volunteering our time to help you here don't have Amazon Web Server accounts, the only way for us to figure out what is going wrong in your environment is for you to explicitly show us exactly what errors you are getting. And, showing us exactly what is being produced by each step in your script so we can figure out what is happening to the data as it passes through your script and seeing where things are going wrong. If you want our help, please help us help you by giving us the data we need to help you analyze your problem(s).

It is immediately obvious that when you pipe a list of long format aws ls utility output to a command that is not a filter (i.e., does not read from standard input), the command at the start of the pipeline will be ignored (unless it contains a syntax error that kills the entire pipeline). Even if that did work; treating a line selected and printed by awk (such as:

Code:

2015-12-26 20:44:13     316953  file3.gz

as a the name of a file is clearly never going to work).

After looking up an aws cp man page, it would appear that you want to create an aws cp command something like:

Code:

aws s3 cp "$folder/" "$target" --recursive --exclude '*' \
    $(aws s3 ls $folder/ |
        awk -v start="$chk" '
            $1" "$2 > start{
                printf(" --include \"%s\"", $NF)
            }
            END{print ""}
        ')

using command substitution (not a pipeline) to add arguments to your aws cp command line. But, without actually seeing the exact format of the output produced by:

Code:

aws s3 ls $folder/

the error messages you got running the last script you showed us, and the exact output you get from:

Code:

aws s3 ls $folder/|
    awk -v start="$chk" '$1" "$2 > start{printf(" --include \"%s\"", $NF)};END{print ""}'

(all in code tags), this is just wild speculation.

Hi Don,
Thanks for your advise ,Btw changed the code as you suggested ,Apologies for not publishing in the post .This below code working fine .I will check your script

Code:

chk="$DATE2 20:30"
 

aws s3 ls $folder/|awk -v start="$chk" '$1" "$2 > start'|awk '{print $NF }'>/home/hadoop/file.txt

for file in `cat /home/hadoop/file.txt`; do
aws s3 cp $folder/$file $target
done

Thanks

You're welcome. (Note that hitting the Thanks button at the lower left corner of posts that you find helpful is also appreciated.)

Again, if you don't need the file /home/hadoop/file.txt elsewhere, your script will run faster if you don't create an unneeded file, write data to that file, and read the data back from that file. And feeding the output of an awk script into another awk script (especially when they're both using the same field separators) is almost always a waste of resources. For example:

Code:

chk="$DATE2 20:30"

aws s3 ls $folder/ |
awk -v start="$chk" '$1" "$2 > start{print $NF}' |
while read -r file; do
    aws s3 cp $folder/$file $target
done

will produce the same output as your script above, but run faster and consume fewer system resources.

Yes Don,it produces the same out put and script ran fast compare to the previous .

Thanks