awk variable search and line count between variable-search pattern

Tags
awk variable pattern, rsync, shell scripts

 
Thread Tools Search this Thread
# 1  
Old 11-22-2016
awk variable search and line count between variable-search pattern

Input:

Code:
|Running the Rsync|Sun Oct 16 22:48:01 BST 2016
|End of the Rsync|Sun Oct 16 22:49:54 BST 2016
|Running the Rsync|Sun Oct 16 22:54:01 BST 2016
|End of the Rsync|Sun Oct 16 22:55:45 BST 2016
|Running the Rsync|Sun Oct 16 23:00:02 BST 2016
|End of the Rsync|Sun Oct 16 23:01:44 BST 2016
|Running the Rsync|Sun Oct 16 23:30:01 BST 2016
|End of the Rsync|Sun Oct 16 23:34:40 BST 2016
|Running the Rsync|Mon Oct 17 00:00:04 BST 2016
|End of the Rsync|Mon Oct 17 00:24:38 BST 2016

Every single run of the Rsync is having a number of files being transferred during the session.

I have successfully extracted time duration of the Rsync duration, but I can't understand how to calculate number of files being transferred during this one particular ( among many ).

Current, output of the script is like :

Code:
|    0  | Sun Nov 20 00:00:01 GMT 2016   | Sun Nov 20 00:02:48 GMT 2016   | 2.783333    |
|    1  | Sun Nov 20 00:30:01 GMT 2016   | Sun Nov 20 00:31:53 GMT 2016   | 1.866667    |
|    2  | Sun Nov 20 01:00:01 GMT 2016   | Sun Nov 20 01:02:24 GMT 2016   | 2.383333    |
|    3  | Sun Nov 20 01:30:01 GMT 2016   | Sun Nov 20 01:32:33 GMT 2016   | 2.533333    |
|    4  | Sun Nov 20 02:00:01 GMT 2016   | Sun Nov 20 02:00:36 GMT 2016   | 0.583333    |
|    5  | Sun Nov 20 02:30:01 GMT 2016   | Sun Nov 20 02:32:15 GMT 2016   | 2.233333    |
|    6  | Sun Nov 20 03:00:01 GMT 2016   | Sun Nov 20 03:00:34 GMT 2016   | 0.550000    |
|    7  | Sun Nov 20 03:30:01 GMT 2016   | Sun Nov 20 03:32:05 GMT 2016   | 2.066667    |
|    8  | Sun Nov 20 04:00:01 GMT 2016   | Sun Nov 20 04:00:34 GMT 2016   | 0.550000    |
|    9  | Sun Nov 20 04:30:01 GMT 2016   | Sun Nov 20 04:31:59 GMT 2016   | 1.966667    |
|   10  | Sun Nov 20 05:00:02 GMT 2016   | Sun Nov 20 05:00:35 GMT 2016   | 0.550000    |


What I'm looking for is the count of files during these Rsync sessions.


If I take a look at the variable search, I found it is impossible for me to make a variable based search.


Code:
awk -F"|" ' BEGIN { SP="Nov 22"; FP="Running the Rsync * "SP; EP="End of the Rsync * "SP; }
( $0 ~ FP ) { ST=$3;}
($0 ~ EP ) { ET=$3; TT= /* calculate time difference between these two instances here"; }
/FP/,/EP/ { x++ }

{ print "Total Time taken durng %s and %s, is %f and Number of files transfered are %d", FP, EP, TT, x }

' /var/slash/input

any help!


Moderator's Comments:
Mod Comment Use code tags, thanks.

Last edited by zaxxon; 11-22-2016 at 05:06 AM..
# 2  
Old 11-22-2016
Scripts aren't very good at extracting information that is not present in the inputs that are provided. If you were to look at the output produced by rsync instead of looking at the messages your script is producing before it runs rsync and after rsync completes, you might be able to accurately count the number of successful and the number of failed attempts to transfer a file. But when that information is not available, there is no way to make it magically appear out of the ether.
# 3  
Old 11-23-2016
@Don,

Between those two starting and ending lines there are thousands of lines stating the file transfer like normal Rsync does. I just skipped them because it might have filled the whole page.

below is the output.

Code:
|Running the Rsync|Sun Oct 16 22:48:01 BST 2016
+ /usr/bin/rsync -P -avz --delete  rootbox:/var/www/images /var/www/images/
receiving incremental file list
images/a55f95e69ed87c1b6fc1/large/
images/a55f95e69ed87c1b6fc1/large/1320.jpg
           0   0%    0.00kB/s    0:00:00
     1991058 100%   12.33MB/s    0:00:00 (xfer#1, to-check=1965/35841)
images/a55f95e69ed87c1b6fc1/large/1321.jpg
           0   0%    0.00kB/s    0:00:00
     1688954 100%    5.37MB/s    0:00:00 (xfer#2, to-check=1964/35841)
images/a55f95e69ed87c1b6fc1/large/1322.jpg
           0   0%    0.00kB/s    0:00:00
     1590928 100%    3.46MB/s    0:00:00 (xfer#3, to-check=1963/35841)
images/a55f95e69ed87c1b6fc1/large/1323.jpg
           0   0%    0.00kB/s    0:00:00
     1701678 100%    2.77MB/s    0:00:00 (xfer#4, to-check=1962/35841)
images/a55f95e69ed87c1b6fc1/large/1324.jpg
           0   0%    0.00kB/s    0:00:00
     1596974 100%    2.11MB/s    0:00:00 (xfer#5, to-check=1961/35841)
images/a55f95e69ed87c1b6fc1/large/1325.jpg
           0   0%    0.00kB/s    0:00:00
|End of the Rsync|Sun Oct 16 22:49:54 BST 2016

# 4  
Old 11-23-2016
With that data, a file copy count could be implemented much easier...
Count the number of appearences of the string "100%", or use the xfer#nn field. What about the sixth file copy action? Was it interrupted? incomplete? Is the log file corrupt?
# 5  
Old 11-23-2016
xfer#xx is only option that I am sure will be accounted for here to check the file transfer count.

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
awk -Search pattern through Variable koti_rama Shell Programming and Scripting 4 06-14-2018 10:30 AM
Grep/awk using a begin search pattern and end search pattern vbabz UNIX for Beginners Questions & Answers 2 05-10-2018 03:20 PM
Search for string in column using variable: awk genome Shell Programming and Scripting 7 02-14-2018 04:56 PM
awk search using variable pchang Shell Programming and Scripting 10 08-05-2015 02:34 PM
How to search, replace and multiply variable within awk? kieranfoley Shell Programming and Scripting 2 07-23-2014 09:04 AM
awk to search for pattern and remove line ncwxpanther Shell Programming and Scripting 1 12-05-2013 05:35 PM
How to search a filename stored in a variable using a pattern? Little Shell Programming and Scripting 8 07-25-2013 01:04 AM
Search the shell variable inside awk jesu Shell Programming and Scripting 3 03-11-2012 12:16 AM
solved -gawk, search for pattern - mark the previous line as a variable? trey85stang Shell Programming and Scripting 0 05-13-2010 03:02 PM
search the pattern in a file and replace with variable already defined previously in csh nehashine Shell Programming and Scripting 2 04-13-2010 06:04 AM
How to assign the Pattern Search string as Input Variable raghunsi Shell Programming and Scripting 1 02-11-2010 06:51 AM
Using a script variable in awk search patterns abhinav192 Shell Programming and Scripting 12 11-20-2009 03:07 AM
Search for awk pattern in unix env variable sdohn Shell Programming and Scripting 6 10-15-2009 01:18 PM
search a string in a line and save it in a variable Ishita Shell Programming and Scripting 5 09-18-2009 08:20 AM
awk using env variable as search argument amit1_x Shell Programming and Scripting 7 02-04-2009 12:09 PM