👤
Home Man
Search
Today's Posts
Register

BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

awk variable search and line count between variable-search pattern

Tags
awk variable pattern, rsync

👤 Login to reply

 
Thread Tools Search this Thread
# 1  
Old 11-22-2016
awk variable search and line count between variable-search pattern

Input:

Code:
|Running the Rsync|Sun Oct 16 22:48:01 BST 2016
|End of the Rsync|Sun Oct 16 22:49:54 BST 2016
|Running the Rsync|Sun Oct 16 22:54:01 BST 2016
|End of the Rsync|Sun Oct 16 22:55:45 BST 2016
|Running the Rsync|Sun Oct 16 23:00:02 BST 2016
|End of the Rsync|Sun Oct 16 23:01:44 BST 2016
|Running the Rsync|Sun Oct 16 23:30:01 BST 2016
|End of the Rsync|Sun Oct 16 23:34:40 BST 2016
|Running the Rsync|Mon Oct 17 00:00:04 BST 2016
|End of the Rsync|Mon Oct 17 00:24:38 BST 2016

Every single run of the Rsync is having a number of files being transferred during the session.

I have successfully extracted time duration of the Rsync duration, but I can't understand how to calculate number of files being transferred during this one particular ( among many ).

Current, output of the script is like :

Code:
|    0  | Sun Nov 20 00:00:01 GMT 2016   | Sun Nov 20 00:02:48 GMT 2016   | 2.783333    |
|    1  | Sun Nov 20 00:30:01 GMT 2016   | Sun Nov 20 00:31:53 GMT 2016   | 1.866667    |
|    2  | Sun Nov 20 01:00:01 GMT 2016   | Sun Nov 20 01:02:24 GMT 2016   | 2.383333    |
|    3  | Sun Nov 20 01:30:01 GMT 2016   | Sun Nov 20 01:32:33 GMT 2016   | 2.533333    |
|    4  | Sun Nov 20 02:00:01 GMT 2016   | Sun Nov 20 02:00:36 GMT 2016   | 0.583333    |
|    5  | Sun Nov 20 02:30:01 GMT 2016   | Sun Nov 20 02:32:15 GMT 2016   | 2.233333    |
|    6  | Sun Nov 20 03:00:01 GMT 2016   | Sun Nov 20 03:00:34 GMT 2016   | 0.550000    |
|    7  | Sun Nov 20 03:30:01 GMT 2016   | Sun Nov 20 03:32:05 GMT 2016   | 2.066667    |
|    8  | Sun Nov 20 04:00:01 GMT 2016   | Sun Nov 20 04:00:34 GMT 2016   | 0.550000    |
|    9  | Sun Nov 20 04:30:01 GMT 2016   | Sun Nov 20 04:31:59 GMT 2016   | 1.966667    |
|   10  | Sun Nov 20 05:00:02 GMT 2016   | Sun Nov 20 05:00:35 GMT 2016   | 0.550000    |


What I'm looking for is the count of files during these Rsync sessions.


If I take a look at the variable search, I found it is impossible for me to make a variable based search.


Code:
awk -F"|" ' BEGIN { SP="Nov 22"; FP="Running the Rsync * "SP; EP="End of the Rsync * "SP; }
( $0 ~ FP ) { ST=$3;}
($0 ~ EP ) { ET=$3; TT= /* calculate time difference between these two instances here"; }
/FP/,/EP/ { x++ }

{ print "Total Time taken durng %s and %s, is %f and Number of files transfered are %d", FP, EP, TT, x }

' /var/slash/input

any help!


Moderator's Comments:
awk variable search and line count between variable-search pattern Use code tags, thanks.

Last edited by zaxxon; 11-22-2016 at 04:06 AM..
# 2  
Old 11-22-2016
Scripts aren't very good at extracting information that is not present in the inputs that are provided. If you were to look at the output produced by rsync instead of looking at the messages your script is producing before it runs rsync and after rsync completes, you might be able to accurately count the number of successful and the number of failed attempts to transfer a file. But when that information is not available, there is no way to make it magically appear out of the ether.
# 3  
Old 11-23-2016
@Don,

Between those two starting and ending lines there are thousands of lines stating the file transfer like normal Rsync does. I just skipped them because it might have filled the whole page.

below is the output.

Code:
|Running the Rsync|Sun Oct 16 22:48:01 BST 2016
+ /usr/bin/rsync -P -avz --delete  rootbox:/var/www/images /var/www/images/
receiving incremental file list
images/a55f95e69ed87c1b6fc1/large/
images/a55f95e69ed87c1b6fc1/large/1320.jpg
           0   0%    0.00kB/s    0:00:00
     1991058 100%   12.33MB/s    0:00:00 (xfer#1, to-check=1965/35841)
images/a55f95e69ed87c1b6fc1/large/1321.jpg
           0   0%    0.00kB/s    0:00:00
     1688954 100%    5.37MB/s    0:00:00 (xfer#2, to-check=1964/35841)
images/a55f95e69ed87c1b6fc1/large/1322.jpg
           0   0%    0.00kB/s    0:00:00
     1590928 100%    3.46MB/s    0:00:00 (xfer#3, to-check=1963/35841)
images/a55f95e69ed87c1b6fc1/large/1323.jpg
           0   0%    0.00kB/s    0:00:00
     1701678 100%    2.77MB/s    0:00:00 (xfer#4, to-check=1962/35841)
images/a55f95e69ed87c1b6fc1/large/1324.jpg
           0   0%    0.00kB/s    0:00:00
     1596974 100%    2.11MB/s    0:00:00 (xfer#5, to-check=1961/35841)
images/a55f95e69ed87c1b6fc1/large/1325.jpg
           0   0%    0.00kB/s    0:00:00
|End of the Rsync|Sun Oct 16 22:49:54 BST 2016

# 4  
Old 11-23-2016
With that data, a file copy count could be implemented much easier...
Count the number of appearences of the string "100%", or use the xfer#nn field. What about the sixth file copy action? Was it interrupted? incomplete? Is the log file corrupt?
# 5  
Old 11-23-2016
xfer#xx is only option that I am sure will be accounted for here to check the file transfer count.
👤 Login to reply

« Previous Thread | Next Thread »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
awk -Search pattern through Variable koti_rama Shell Programming and Scripting 4 06-14-2018 09:30 AM
Grep/awk using a begin search pattern and end search pattern vbabz UNIX for Beginners Questions & Answers 2 05-10-2018 02:20 PM
awk search using variable pchang Shell Programming and Scripting 10 08-05-2015 01:34 PM
awk to search for pattern and remove line ncwxpanther Shell Programming and Scripting 1 12-05-2013 04:35 PM
How to search a filename stored in a variable using a pattern? Little Shell Programming and Scripting 8 07-25-2013 12:04 AM
solved -gawk, search for pattern - mark the previous line as a variable? trey85stang Shell Programming and Scripting 0 05-13-2010 02:02 PM
search the pattern in a file and replace with variable already defined previously in csh nehashine Shell Programming and Scripting 2 04-13-2010 05:04 AM
How to assign the Pattern Search string as Input Variable raghunsi Shell Programming and Scripting 1 02-11-2010 05:51 AM
Search for awk pattern in unix env variable sdohn Shell Programming and Scripting 6 10-15-2009 12:18 PM
search a string in a line and save it in a variable Ishita Shell Programming and Scripting 5 09-18-2009 07:20 AM


All times are GMT -4. The time now is 02:45 PM.

Unix & Linux Forums Content Copyright©1993-2018. All Rights Reserved.
×
UNIX.COM Login
Username:
Password:  
Show Password