awk variable search and line count between variable-search pattern


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk variable search and line count between variable-search pattern
# 1  
Old 11-22-2016
awk variable search and line count between variable-search pattern

Input:

Code:
|Running the Rsync|Sun Oct 16 22:48:01 BST 2016
|End of the Rsync|Sun Oct 16 22:49:54 BST 2016
|Running the Rsync|Sun Oct 16 22:54:01 BST 2016
|End of the Rsync|Sun Oct 16 22:55:45 BST 2016
|Running the Rsync|Sun Oct 16 23:00:02 BST 2016
|End of the Rsync|Sun Oct 16 23:01:44 BST 2016
|Running the Rsync|Sun Oct 16 23:30:01 BST 2016
|End of the Rsync|Sun Oct 16 23:34:40 BST 2016
|Running the Rsync|Mon Oct 17 00:00:04 BST 2016
|End of the Rsync|Mon Oct 17 00:24:38 BST 2016

Every single run of the Rsync is having a number of files being transferred during the session.

I have successfully extracted time duration of the Rsync duration, but I can't understand how to calculate number of files being transferred during this one particular ( among many ).

Current, output of the script is like :

Code:
|    0  | Sun Nov 20 00:00:01 GMT 2016   | Sun Nov 20 00:02:48 GMT 2016   | 2.783333    |
|    1  | Sun Nov 20 00:30:01 GMT 2016   | Sun Nov 20 00:31:53 GMT 2016   | 1.866667    |
|    2  | Sun Nov 20 01:00:01 GMT 2016   | Sun Nov 20 01:02:24 GMT 2016   | 2.383333    |
|    3  | Sun Nov 20 01:30:01 GMT 2016   | Sun Nov 20 01:32:33 GMT 2016   | 2.533333    |
|    4  | Sun Nov 20 02:00:01 GMT 2016   | Sun Nov 20 02:00:36 GMT 2016   | 0.583333    |
|    5  | Sun Nov 20 02:30:01 GMT 2016   | Sun Nov 20 02:32:15 GMT 2016   | 2.233333    |
|    6  | Sun Nov 20 03:00:01 GMT 2016   | Sun Nov 20 03:00:34 GMT 2016   | 0.550000    |
|    7  | Sun Nov 20 03:30:01 GMT 2016   | Sun Nov 20 03:32:05 GMT 2016   | 2.066667    |
|    8  | Sun Nov 20 04:00:01 GMT 2016   | Sun Nov 20 04:00:34 GMT 2016   | 0.550000    |
|    9  | Sun Nov 20 04:30:01 GMT 2016   | Sun Nov 20 04:31:59 GMT 2016   | 1.966667    |
|   10  | Sun Nov 20 05:00:02 GMT 2016   | Sun Nov 20 05:00:35 GMT 2016   | 0.550000    |


What I'm looking for is the count of files during these Rsync sessions.


If I take a look at the variable search, I found it is impossible for me to make a variable based search.


Code:
awk -F"|" ' BEGIN { SP="Nov 22"; FP="Running the Rsync * "SP; EP="End of the Rsync * "SP; }
( $0 ~ FP ) { ST=$3;}
($0 ~ EP ) { ET=$3; TT= /* calculate time difference between these two instances here"; }
/FP/,/EP/ { x++ }

{ print "Total Time taken durng %s and %s, is %f and Number of files transfered are %d", FP, EP, TT, x }

' /var/slash/input

any help!


Moderator's Comments:
Mod Comment Use code tags, thanks.

Last edited by zaxxon; 11-22-2016 at 05:06 AM..
# 2  
Old 11-22-2016
Scripts aren't very good at extracting information that is not present in the inputs that are provided. If you were to look at the output produced by rsync instead of looking at the messages your script is producing before it runs rsync and after rsync completes, you might be able to accurately count the number of successful and the number of failed attempts to transfer a file. But when that information is not available, there is no way to make it magically appear out of the ether.
# 3  
Old 11-23-2016
@Don,

Between those two starting and ending lines there are thousands of lines stating the file transfer like normal Rsync does. I just skipped them because it might have filled the whole page.

below is the output.

Code:
|Running the Rsync|Sun Oct 16 22:48:01 BST 2016
+ /usr/bin/rsync -P -avz --delete  rootbox:/var/www/images /var/www/images/
receiving incremental file list
images/a55f95e69ed87c1b6fc1/large/
images/a55f95e69ed87c1b6fc1/large/1320.jpg
           0   0%    0.00kB/s    0:00:00
     1991058 100%   12.33MB/s    0:00:00 (xfer#1, to-check=1965/35841)
images/a55f95e69ed87c1b6fc1/large/1321.jpg
           0   0%    0.00kB/s    0:00:00
     1688954 100%    5.37MB/s    0:00:00 (xfer#2, to-check=1964/35841)
images/a55f95e69ed87c1b6fc1/large/1322.jpg
           0   0%    0.00kB/s    0:00:00
     1590928 100%    3.46MB/s    0:00:00 (xfer#3, to-check=1963/35841)
images/a55f95e69ed87c1b6fc1/large/1323.jpg
           0   0%    0.00kB/s    0:00:00
     1701678 100%    2.77MB/s    0:00:00 (xfer#4, to-check=1962/35841)
images/a55f95e69ed87c1b6fc1/large/1324.jpg
           0   0%    0.00kB/s    0:00:00
     1596974 100%    2.11MB/s    0:00:00 (xfer#5, to-check=1961/35841)
images/a55f95e69ed87c1b6fc1/large/1325.jpg
           0   0%    0.00kB/s    0:00:00
|End of the Rsync|Sun Oct 16 22:49:54 BST 2016

# 4  
Old 11-23-2016
With that data, a file copy count could be implemented much easier...
Count the number of appearences of the string "100%", or use the xfer#nn field. What about the sixth file copy action? Was it interrupted? incomplete? Is the log file corrupt?
# 5  
Old 11-23-2016
xfer#xx is only option that I am sure will be accounted for here to check the file transfer count.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Need help with how to search a file for a variable string and delete that line

Hi, I have a working script. It does what I am intending it to but a bit confused whether the sed part is supposed to be working or not. Further down is the script with the sed part that should have been working but not and the grep -v part which is the workaround that I am using at the... (10 Replies)
Discussion started by: newbie_01
10 Replies

2. Shell Programming and Scripting

awk -Search pattern through Variable

Hello, We have wrote shell script for multiple file name search pattern. file format: <numner>_<20180809>.txt starting with single number and ending with 8 digits number Command: awk -v string="12_1234" -v serch="^+_+$" "BEGIN{ if (string ~/serch$/) print string }" If sting matches... (4 Replies)
Discussion started by: koti_rama
4 Replies

3. UNIX for Beginners Questions & Answers

Grep/awk using a begin search pattern and end search pattern

I have this fileA TEST FILE ABC this file contains ABC; TEST FILE DGHT this file contains DGHT; TEST FILE 123 this file contains ABC, this file contains DEF, this file contains XYZ, this file contains KLM ; I want to have a fileZ that has only (begin search pattern for will be... (2 Replies)
Discussion started by: vbabz
2 Replies

4. Shell Programming and Scripting

awk search using variable

Hi Forum. I have the following script that I would like to use a variable to search using awk but it's not working as expected: # Define working variables token_search_string=536088 token_search_length=16 This code example works by hardcoding 536088 in the string search: awk -v... (10 Replies)
Discussion started by: pchang
10 Replies

5. Shell Programming and Scripting

How to search a filename stored in a variable using a pattern?

hi, i have a variable which contains some file names delimited by a single space. FNAME="s1.txt s2.lst s3.cvs s4.lst" i have another variable that contains a pattern FILE_PATTERN="*.lst" i want to take the filenames from FNAME variable and assign each file name in to an array say for... (8 Replies)
Discussion started by: Little
8 Replies

6. Shell Programming and Scripting

solved -gawk, search for pattern - mark the previous line as a variable?

Im trying to parse ifconfig with awk and setup a bunch of variables in one shot. But Im having trouble figuring out how to work with data in previous lines. ifconfig output: eth0 Link encap:Ethernet HWaddr 00:50:DA:10:7F:1B inet addr:10.10.10.10 Bcast:10.10.10.127 ... (0 Replies)
Discussion started by: trey85stang
0 Replies

7. Shell Programming and Scripting

search the pattern in a file and replace with variable already defined previously in csh

I want to replace a certain pattern with the variable already defined. e.g. set path_verilog = /home/priya/bin/verilogfile my file contents are : verilog new verilog is defined here verilog_path_comes I am using the below command sed 's/verilog_path_comes/'$path_verilog'/g' <filename>... (2 Replies)
Discussion started by: nehashine
2 Replies

8. Shell Programming and Scripting

How to assign the Pattern Search string as Input Variable

guys, I need to know how to assing pattern matched string as an input command variable. Here it goes' My script is something like this. ./routing.sh <Server> <enable|disable> ## This Script takes an input <Server> variable from this line of the script ## echo $1 | egrep... (1 Reply)
Discussion started by: raghunsi
1 Replies

9. Shell Programming and Scripting

Search for awk pattern in unix env variable

For a Script I need to detemine which field of the unix environment variable SHLIB_PATH has the WALTDB entry. export SHLIB_PATH=/usr/user5/WALTDB/oracle/product/10.2.0/lib32:/usr/TZD/bin.wdad/mug/oracle/lib: echo $SHLIB_PATH | awk -F: '{ print $1 }' Shure gives me the first entry, but... (6 Replies)
Discussion started by: sdohn
6 Replies

10. Shell Programming and Scripting

search a string in a line and save it in a variable

Hi I want to read a file line by line and search for a particular string in each line(say for example string containing @ )and save that string into a variable. Can someone suggest me the way to implement it.I am using K- shell Thanks Ishita (5 Replies)
Discussion started by: Ishita
5 Replies
Login or Register to Ask a Question