Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

awk variable search and line count between variable-search pattern

Shell Programming and Scripting


Tags
awk variable pattern, rsync

Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 11-22-2016   -   Original Discussion by busyboy
busyboy's Unix or Linux Image
busyboy busyboy is offline
Registered User
 
Join Date: Jan 2010
Last Activity: 9 May 2018, 9:49 AM EDT
Posts: 208
Thanks: 4
Thanked 9 Times in 9 Posts
awk variable search and line count between variable-search pattern

Input:



Code:
|Running the Rsync|Sun Oct 16 22:48:01 BST 2016
|End of the Rsync|Sun Oct 16 22:49:54 BST 2016
|Running the Rsync|Sun Oct 16 22:54:01 BST 2016
|End of the Rsync|Sun Oct 16 22:55:45 BST 2016
|Running the Rsync|Sun Oct 16 23:00:02 BST 2016
|End of the Rsync|Sun Oct 16 23:01:44 BST 2016
|Running the Rsync|Sun Oct 16 23:30:01 BST 2016
|End of the Rsync|Sun Oct 16 23:34:40 BST 2016
|Running the Rsync|Mon Oct 17 00:00:04 BST 2016
|End of the Rsync|Mon Oct 17 00:24:38 BST 2016

Every single run of the Rsync is having a number of files being transferred during the session.

I have successfully extracted time duration of the Rsync duration, but I can't understand how to calculate number of files being transferred during this one particular ( among many ).

Current, output of the script is like :



Code:
|    0  | Sun Nov 20 00:00:01 GMT 2016   | Sun Nov 20 00:02:48 GMT 2016   | 2.783333    |
|    1  | Sun Nov 20 00:30:01 GMT 2016   | Sun Nov 20 00:31:53 GMT 2016   | 1.866667    |
|    2  | Sun Nov 20 01:00:01 GMT 2016   | Sun Nov 20 01:02:24 GMT 2016   | 2.383333    |
|    3  | Sun Nov 20 01:30:01 GMT 2016   | Sun Nov 20 01:32:33 GMT 2016   | 2.533333    |
|    4  | Sun Nov 20 02:00:01 GMT 2016   | Sun Nov 20 02:00:36 GMT 2016   | 0.583333    |
|    5  | Sun Nov 20 02:30:01 GMT 2016   | Sun Nov 20 02:32:15 GMT 2016   | 2.233333    |
|    6  | Sun Nov 20 03:00:01 GMT 2016   | Sun Nov 20 03:00:34 GMT 2016   | 0.550000    |
|    7  | Sun Nov 20 03:30:01 GMT 2016   | Sun Nov 20 03:32:05 GMT 2016   | 2.066667    |
|    8  | Sun Nov 20 04:00:01 GMT 2016   | Sun Nov 20 04:00:34 GMT 2016   | 0.550000    |
|    9  | Sun Nov 20 04:30:01 GMT 2016   | Sun Nov 20 04:31:59 GMT 2016   | 1.966667    |
|   10  | Sun Nov 20 05:00:02 GMT 2016   | Sun Nov 20 05:00:35 GMT 2016   | 0.550000    |


What I'm looking for is the count of files during these Rsync sessions.


If I take a look at the variable search, I found it is impossible for me to make a variable based search.




Code:
awk -F"|" ' BEGIN { SP="Nov 22"; FP="Running the Rsync * "SP; EP="End of the Rsync * "SP; }
( $0 ~ FP ) { ST=$3;}
($0 ~ EP ) { ET=$3; TT= /* calculate time difference between these two instances here"; }
/FP/,/EP/ { x++ }

{ print "Total Time taken durng %s and %s, is %f and Number of files transfered are %d", FP, EP, TT, x }

' /var/slash/input

any help!


Moderator's Comments:
awk variable search and line count between variable-search pattern Use code tags, thanks.

Last edited by zaxxon; 11-22-2016 at 04:06 AM..
Sponsored Links
    #2  
Old Unix and Linux 11-22-2016   -   Original Discussion by busyboy
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 26 May 2018, 4:08 AM EDT
Location: San Jose, CA, USA
Posts: 11,311
Thanks: 637
Thanked 3,936 Times in 3,367 Posts
Scripts aren't very good at extracting information that is not present in the inputs that are provided. If you were to look at the output produced by rsync instead of looking at the messages your script is producing before it runs rsync and after rsync completes, you might be able to accurately count the number of successful and the number of failed attempts to transfer a file. But when that information is not available, there is no way to make it magically appear out of the ether.
Sponsored Links
    #3  
Old Unix and Linux 11-23-2016   -   Original Discussion by busyboy
busyboy's Unix or Linux Image
busyboy busyboy is offline
Registered User
 
Join Date: Jan 2010
Last Activity: 9 May 2018, 9:49 AM EDT
Posts: 208
Thanks: 4
Thanked 9 Times in 9 Posts
@Don,

Between those two starting and ending lines there are thousands of lines stating the file transfer like normal Rsync does. I just skipped them because it might have filled the whole page.

below is the output.



Code:
|Running the Rsync|Sun Oct 16 22:48:01 BST 2016
+ /usr/bin/rsync -P -avz --delete  rootbox:/var/www/images /var/www/images/
receiving incremental file list
images/a55f95e69ed87c1b6fc1/large/
images/a55f95e69ed87c1b6fc1/large/1320.jpg
           0   0%    0.00kB/s    0:00:00
     1991058 100%   12.33MB/s    0:00:00 (xfer#1, to-check=1965/35841)
images/a55f95e69ed87c1b6fc1/large/1321.jpg
           0   0%    0.00kB/s    0:00:00
     1688954 100%    5.37MB/s    0:00:00 (xfer#2, to-check=1964/35841)
images/a55f95e69ed87c1b6fc1/large/1322.jpg
           0   0%    0.00kB/s    0:00:00
     1590928 100%    3.46MB/s    0:00:00 (xfer#3, to-check=1963/35841)
images/a55f95e69ed87c1b6fc1/large/1323.jpg
           0   0%    0.00kB/s    0:00:00
     1701678 100%    2.77MB/s    0:00:00 (xfer#4, to-check=1962/35841)
images/a55f95e69ed87c1b6fc1/large/1324.jpg
           0   0%    0.00kB/s    0:00:00
     1596974 100%    2.11MB/s    0:00:00 (xfer#5, to-check=1961/35841)
images/a55f95e69ed87c1b6fc1/large/1325.jpg
           0   0%    0.00kB/s    0:00:00
|End of the Rsync|Sun Oct 16 22:49:54 BST 2016

    #4  
Old Unix and Linux 11-23-2016   -   Original Discussion by busyboy
RudiC's Unix or Linux Image
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 26 May 2018, 6:41 AM EDT
Location: Aachen, Germany
Posts: 12,753
Thanks: 415
Thanked 3,921 Times in 3,605 Posts
With that data, a file copy count could be implemented much easier...
Count the number of appearences of the string "100%", or use the xfer#nn field. What about the sixth file copy action? Was it interrupted? incomplete? Is the log file corrupt?
Sponsored Links
    #5  
Old Unix and Linux 11-23-2016   -   Original Discussion by busyboy
busyboy's Unix or Linux Image
busyboy busyboy is offline
Registered User
 
Join Date: Jan 2010
Last Activity: 9 May 2018, 9:49 AM EDT
Posts: 208
Thanks: 4
Thanked 9 Times in 9 Posts
xfer#xx is only option that I am sure will be accounted for here to check the file transfer count.
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How to search a filename stored in a variable using a pattern? Little Shell Programming and Scripting 8 07-25-2013 12:04 AM
solved -gawk, search for pattern - mark the previous line as a variable? trey85stang Shell Programming and Scripting 0 05-13-2010 02:02 PM
search the pattern in a file and replace with variable already defined previously in csh nehashine Shell Programming and Scripting 2 04-13-2010 05:04 AM
How to assign the Pattern Search string as Input Variable raghunsi Shell Programming and Scripting 1 02-11-2010 05:51 AM
Search for awk pattern in unix env variable sdohn Shell Programming and Scripting 6 10-15-2009 12:18 PM



All times are GMT -4. The time now is 07:53 AM.