Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Time rotation

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Time rotation
# 1  
Old 10-10-2016
Time rotation
Hi,

I'm trying to figure out Rsync Timings from my logs of the script. For this I have enabled a wrapper script to log start time and end time of the script so that I can catch the Rsync lapsed time.

Below are the code lines I'm trying to use for Rsync time .


INPUT is

Code:
|Running the Rsync|Sat Oct  8 11:30:01 BST 2016
|End of the Rsync|Sat Oct  8 11:44:00 BST 2016
|Running the Rsync|Sat Oct  8 11:48:01 BST 2016
|End of the Rsync|Sat Oct  8 12:01:44 BST 2016
|Running the Rsync|Sat Oct  8 12:06:01 BST 2016
|End of the Rsync|Sat Oct  8 12:18:48 BST 2016
|Running the Rsync|Sat Oct  8 12:24:01 BST 2016
|End of the Rsync|Sat Oct  8 12:37:15 BST 2016
|Running the Rsync|Sat Oct  8 12:42:01 BST 2016
|End of the Rsync|Sat Oct  8 12:57:02 BST 2016
|Running the Rsync|Sat Oct  8 13:00:02 BST 2016
|End of the Rsync|Sat Oct  8 13:11:59 BST 2016
|Running the Rsync|Sat Oct  8 13:12:01 BST 2016
|End of the Rsync|Sat Oct  8 13:25:06 BST 2016
|Running the Rsync|Sat Oct  8 13:30:01 BST 2016
|End of the Rsync|Sat Oct  8 13:44:09 BST 2016
|Running the Rsync|Sat Oct  8 13:48:01 BST 2016
|End of the Rsync|Sat Oct  8 14:01:29 BST 2016
|Running the Rsync|Sat Oct  8 14:06:01 BST 2016
|End of the Rsync|Sat Oct  8 14:19:08 BST 2016
|Running the Rsync|Sat Oct  8 14:24:01 BST 2016
|End of the Rsync|Sat Oct  8 14:37:27 BST 2016
|Running the Rsync|Sat Oct  8 14:42:01 BST 2016
|End of the Rsync|Sat Oct  8 14:57:02 BST 2016
|Running the Rsync|Sat Oct  8 15:00:01 BST 2016
|End of the Rsync|Sat Oct  8 15:11:53 BST 2016


I'm using this line to extract timings

Code:
 awk -F"|" '/^\|Running the Rsync/ {XX=$3; cmd = "date +%s -d \"" $3 "\""; cmd | getline X; close(cmd);} 
/^\|End of the Rsync/ {YY=$3; cmd = "date +%s -d \"" $3 "\""; cmd | getline Y; close(cmd);a[y++]=XX"-"YY"-"(Y-X)/60;} END { for ( j in a ) { print a[j] } } ' rsyncout.log




the problem with this code is that even though it runs successfully, the output is not in sequence of its appearance in logs, entries are mixed and I dont get a sequential output.


Here is what I'm getting :


Code:
Sun Oct  9 20:36:01 BST 2016-Sun Oct  9 20:39:40 BST 2016-3.65
Fri Oct  7 12:24:02 BST 2016-Fri Oct  7 12:36:18 BST 2016-12.2667
Thu Oct  6 20:54:01 BST 2016-Thu Oct  6 21:06:01 BST 2016-12
Thu Oct  6 10:54:01 BST 2016-Thu Oct  6 11:04:31 BST 2016-10.5
Mon Oct  3 16:36:01 BST 2016-Mon Oct  3 16:46:36 BST 2016-10.5833
Mon Oct  3 08:18:01 BST 2016-Mon Oct  3 08:25:05 BST 2016-7.06667
Fri Oct  7 12:42:01 BST 2016-Fri Oct  7 12:55:41 BST 2016-13.6667
Thu Oct  6 21:06:01 BST 2016-Thu Oct  6 21:17:50 BST 2016-11.8167
Thu Oct  6 11:06:01 BST 2016-Thu Oct  6 11:17:07 BST 2016-11.1
Mon Oct  3 16:48:01 BST 2016-Mon Oct  3 16:55:58 BST 2016-7.95
Mon Oct  3 08:30:01 BST 2016-Mon Oct  3 08:40:17 BST 2016-10.2667
Fri Oct  7 13:00:01 BST 2016-Fri Oct  7 13:11:34 BST 2016-11.55
Thu Oct  6 21:18:01 BST 2016-Thu Oct  6 21:35:17 BST 2016-17.2667
Thu Oct  6 11:18:02 BST 2016-Thu Oct  6 11:29:48 BST 2016-11.7667
Mon Oct  3 17:00:01 BST 2016-Mon Oct  3 17:08:27 BST 2016-8.43333
Mon Oct  3 08:42:01 BST 2016-Mon Oct  3 08:52:04 BST 2016-10.05
Fri Oct  7 13:12:01 BST 2016-Fri Oct  7 13:24:48 BST 2016-12.7833
Thu Oct  6 21:36:01 BST 2016-Thu Oct  6 21:56:25 BST 2016-20.4
Thu Oct  6 11:30:01 BST 2016-Thu Oct  6 11:42:56 BST 2016-12.9167
Mon Oct  3 17:12:01 BST 2016-Mon Oct  3 17:21:10 BST 2016-9.15
Mon Oct  3 08:54:01 BST 2016-Mon Oct  3 09:02:57 BST 2016-8.93333
Thu Oct  6 22:00:01 BST 2016-Thu Oct  6 22:19:04 BST 2016-19.05
Thu Oct  6 11:48:01 BST 2016-Thu Oct  6 12:00:20 BST 2016-12.3167
Mon Oct  3 17:24:01 BST 2016-Mon Oct  3 17:34:04 BST 2016-10.05
Mon Oct  3 09:06:01 BST 2016-Mon Oct  3 09:15:27 BST 2016-9.43333
Thu Oct  6 12:06:01 BST 2016-Thu Oct  6 12:16:17 BST 2016-10.2667
Mon Oct  3 17:36:01 BST 2016-Mon Oct  3 17:45:19 BST 2016-9.3
Mon Oct  3 09:18:01 BST 2016-Mon Oct  3 09:26:59 BST 2016-8.96667
Mon Oct  3 17:48:01 BST 2016-Mon Oct  3 17:56:47 BST 2016-8.76667
Mon Oct  3 09:30:01 BST 2016-Mon Oct  3 09:40:59 BST 2016-10.9667
Mon Oct  3 18:00:01 BST 2016-Mon Oct  3 18:09:36 BST 2016-9.58333
Mon Oct  3 09:42:01 BST 2016-Mon Oct  3 09:52:36 BST 2016-10.5833
Mon Oct  3 18:12:01 BST 2016-Mon Oct  3 18:20:57 BST 2016-8.93333
Mon Oct  3 09:54:01 BST 2016-Mon Oct  3 10:02:42 BST 2016-8.68333
Mon Oct  3 10:06:01 BST 2016-Mon Oct  3 10:15:18 BST 2016-9.28333



whereas this code works fine but I need average, min,max of the last column at the end this is why I added the result into an array and then used the END block to sum the results ( not implemented so far )


Code:
awk -F"|" '/^\|Running the Rsync/ {XX=$3; cmd = "date +%s -d \"" $3 "\""; cmd | getline X; close(cmd);} /^\|End of the Rsync/ {YY=$3; cmd = "date +%s -d \"" $3 "\""; cmd | getline Y;printf("%s-%s-%d\n",XX,YY,(Y-X)/60 ); close(cmd);} ' rsyncout.log

# 2  
Old 10-10-2016
Try walking the array manually
Code:
for (j = 0; j < y; ++j)
   print a[j]

Juha
# 3  
Old 10-10-2016
As far as i can see there is no single job going over midnight. This makes it rather easy: convert the times to seconds passed since midnight, i.e. the timestamp in the first line of the input:

Code:
11:30:01 ^=  11*3600 +
             30*  60 +
              1*   1 == 41401

From there it is a simple subtraction to get the number of seconds passed during the run.

I hope this helps.

bakunin
# 4  
Old 10-10-2016
Running the date command for every single data line in your log file definitively is an expensive method to determine the seconds for an arithmetic operation, but it works. The next step - esp. if we are talking of large files - would be to implement an algorithm along what bakunin proposed.
Given the log file looks exactly like what your posted in #1, try
Code:
awk -F\| '
function getsec(TM)     {cmd = "date +%s -d \"" TM "\""
                         cmd | getline X
                         close(cmd)
                         return X
                        }
BEGIN   {MIN = 1E100
        }

 (NR%2) {T1 = $3
        }

!(NR%2) {T2   = $3
         DTA  = (getsec(T2) - getsec(T1)) / 60
         SUM += DTA
         CNT++
         MIN  = MIN>DTA?DTA:MIN
         MAX  = MAX<DTA?DTA:MAX 
         printf "%s - %s: %f\n", T2, T1, DTA
        }
END     {printf "Min: %f, Avg: %f, Max: %f\n", MIN, SUM/CNT, MAX
        }
' file
Sat Oct  8 11:44:00 BST 2016 - Sat Oct  8 11:30:01 BST 2016: 13.983333
Sat Oct  8 12:01:44 BST 2016 - Sat Oct  8 11:48:01 BST 2016: 13.716667
Sat Oct  8 12:18:48 BST 2016 - Sat Oct  8 12:06:01 BST 2016: 12.783333
Sat Oct  8 12:37:15 BST 2016 - Sat Oct  8 12:24:01 BST 2016: 13.233333
Sat Oct  8 12:57:02 BST 2016 - Sat Oct  8 12:42:01 BST 2016: 15.016667
Sat Oct  8 13:11:59 BST 2016 - Sat Oct  8 13:00:02 BST 2016: 11.950000
Sat Oct  8 13:25:06 BST 2016 - Sat Oct  8 13:12:01 BST 2016: 13.083333
Sat Oct  8 13:44:09 BST 2016 - Sat Oct  8 13:30:01 BST 2016: 14.133333
Sat Oct  8 14:01:29 BST 2016 - Sat Oct  8 13:48:01 BST 2016: 13.466667
Sat Oct  8 14:19:08 BST 2016 - Sat Oct  8 14:06:01 BST 2016: 13.116667
Sat Oct  8 14:37:27 BST 2016 - Sat Oct  8 14:24:01 BST 2016: 13.433333
Sat Oct  8 14:57:02 BST 2016 - Sat Oct  8 14:42:01 BST 2016: 15.016667
Sat Oct  8 15:11:53 BST 2016 - Sat Oct  8 15:00:01 BST 2016: 11.866667
Min: 11.866667, Avg: 13.446154, Max: 15.016667

This User Gave Thanks to RudiC For This Post:
busyboy
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. OS X (Apple)

Mavericks log rotation

In Mavericks, Apple has apparently moved control of log rotation to ASL. There's a 'ttl' value to determine how long log files will stick around for. I can compress them, change the way they're named, limit them by size, etc. But the one thing I cannot find is how to NOT keep one log file per... (0 Replies)
Discussion started by: jnojr
0 Replies

2. Solaris

Log rotation

Hi All! I seem to have a problem with log rotation, unless I am doing something wrong, I have type the following command for testing purposes to see if the -s option works but he did not: logadm -w /var/adm/messages -C 8 -c -s 512k -t '/var/adm/messages.$n' -z 1 the file is now at this... (7 Replies)
Discussion started by: fretagi
7 Replies

3. UNIX for Dummies Questions & Answers

Log Rotation

Hi Guys, Good morning, I just want to know and collect ideas on this one. Regarding rotation of logs as I've observed it's not consistently functioning. I have a server with 8 Partitions, each partition has a dedicated directory for the logs that is needed and I set it every 5mins (300secs) the... (1 Reply)
Discussion started by: rymnd_12345
1 Replies

4. Red Hat

check ip rotation

Hi all, I configured ip rotation on exim mail server by modifying the /etc/exim.conf file. I want to check whether the ip address rotating or not. Can any body show me how can i find out is it rotating or not. Thanks, (1 Reply)
Discussion started by: mastansaheb
1 Replies

5. Solaris

Log rotation, twice

hi folk, need advise regarding the log rotation, i have the logadm set at 30 2 * * * /usr/sbin/logadm so it supposed to rotate once per day, but now it rotated twice! but someone my log will rotate at 2:30 AM, but then another 2 hours later, it creates a new and rotate a new log again,... (2 Replies)
Discussion started by: dehetoxic
2 Replies

6. Solaris

Log Rotation of Catalina.out

Hi, Recently i received a request to rotate logs of catalina.out (tomcat). The file size was about 807 MB. I used logadm to truncate the log ( -c ) and zip (-z 0) it. Everything worked fine, catalina.out.0.gz was created (22 MB) and the size of original catalina.out became 0kb. After... (2 Replies)
Discussion started by: Mack1982
2 Replies

7. Shell Programming and Scripting

log rotation and autosys

Hi, I current have many apps servers running and need to create a script to rotate logs daily, and then create an autosys job to delete logs that are older than 30 days. I was thrown into this and have no idea what to do, please help me get started, thanks! -----Post Update----- and i will... (6 Replies)
Discussion started by: new2learn09
6 Replies

8. Shell Programming and Scripting

Log rotation script

I have the below script to help with disk space cleanup that finds logs older than a specified number of days (say 10 days). I need it to grab "active" logs as well. Problem is an "active log" will not get archived unless I put in 0 days which I don't want to do, I need to leave the past 10 days,... (2 Replies)
Discussion started by: theninja
2 Replies

9. Shell Programming and Scripting

log rotation

Hello all. Due to some reason I can not use HUP to rotate needed log files. So I use the standard method: cp $file $file.1 cat /dev/null > $file But if Java application in this time writing the output to $file, in the beginning of it appears many "^@^@^@^@^@^@". How to avoid it? Or how... (6 Replies)
Discussion started by: mirusnet
6 Replies

10. HP-UX

Log rotation on HP-UX

Can anyone post a sample log rotate and archive configuration on HP-UX? I really don't know how to do that... :( (3 Replies)
Discussion started by: untamed
3 Replies
Login or Register to Ask a Question