Newbie looking for how to Grep times more than 10 seconds apart


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Newbie looking for how to Grep times more than 10 seconds apart
# 1  
Old 09-18-2016
Newbie looking for how to Grep times more than 10 seconds apart

I am new to grep and Linux and am looking to see if grep can parse out a list of lines that have a difference of more than 10 seconds between the times on each line.

Example

Code:
2016-09-17 19:30:57  INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 19:30:41,
2016-09-17 12:02:26  INFO: [D3B4AEB3] id: 4562079193, time: 2016-09-17 12:02:25,

We need to have the grep script parse out the first line because the difference in times is more than 10 seconds, but not the second line as they are within 10 seconds.

Any assistance would be appreciated.

I do not even know where to start and I spent hours googleing this to only get completely confused.

Thanks in advance.

Mark

Last edited by Scrutinizer; 09-18-2016 at 01:50 AM.. Reason: code tags
# 2  
Old 09-18-2016
Hi Mark,
Welcome to the UNIX & Linux Forums.

For you first question; no, grep can't do this. The grep utility selects lines matching certain fixed strings, basic regular expressions, or extended regular expression. The grep utility is not able to perform arithmetic calculations.

Are the timestamps on a given line always on the same date? The arithmetic needed to compare two HH:MM:SS values is relatively simple when all times are on the same date. If timestamps in a line can cross the midnight barrier, the arithmetic is more complex.

What is the name of your input file?

Please show us what output you want to produce from your sample input file (in CODE tags, please).

What shell are you using?
# 3  
Old 09-18-2016
Like Don Cragun said, this cannot be done with grep.

Here is an awk approach you could try, with date change around midnight:

Code:
awk -F, '                                 # set the input field separator (FS) to a comma
  {
    n=split($2,B," ")                     # use split() twice to convert begin time t1 to seconds
    split(B[n], T, ":")
    t1=T[1]*3600 + T[2]*60 + T[3]
    split($1,E," ")                       # use split() twice to convert end time t2 to seconds
    split(E[2], T, ":")
    t2=T[1]*3600 + T[2]*60 + T[3]
  } 
  E[1]>B[n-1] {                           # if there is a date change
    t2+=3600*24                           # add the number of seconds in a day to t2
  }
  (t2-t1)>10                              # if the difference is more than 10 seconds, print the line.
' file


Last edited by Scrutinizer; 09-18-2016 at 10:41 AM.. Reason: Correction in date change check
# 4  
Old 09-18-2016
Hi.

If there is a concern about date differences over days, months, years, then the date-aware package dateutils can be used. The package can be found in many Linux distribution repositories, or at GitHub - hroptatyr/dateutils: nifty command line date and time utilities; fast date calculations and conversion in the shell

We convert the dates into a generic form, then find the absolute value of the difference in seconds, printing the (saved) line if greater than 10:
Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate date/time difference, dconf, ddiff.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C dateutils.dconv dateutils.ddiff

# Function absolute value.
abs() { v1="$1"; [ "$v1" -lt 0 ] && echo "${v1:1}" || echo "$v1" ; }

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

#         1        2     3          4   5           6     7         8          9
#2016-09-17 19:30:57 INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 19:30:41,

pl " Results:"
while read line
do
  read d1 t1 j3 j4 j5 j6 j7 d8 t9 <<< $line
  reference=$( dateutils.dconv "$d1 $t1" )
  other=$( dateutils.dconv "$d8 $t9" )
  db " reference is :$reference:, other is :$other:"
  difference=$( dateutils.ddiff -f "%S%n" $reference $other )
  db " Difference in time is :$difference:"
  positive=$( abs "$difference" )
  db " absolute value of :$difference: is :$positive:"
  [ "$positive" -gt 10 ] && echo "$line"
done < $FILE

exit 0

produciing:
Code:
$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.4 (jessie) 
bash GNU bash 4.3.30
dateutils.dconv dconv 0.3.1
dateutils.ddiff ddiff 0.3.1

-----
 Input data file data1:
2016-09-17 19:30:57  INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 19:30:41,
2016-09-17 12:02:26  INFO: [D3B4AEB3] id: 4562079193, time: 2016-09-17 12:02:25,

-----
 Results:
2016-09-17 19:30:57  INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 19:30:41,

To see inermediate values, interchange the 2 db lines to get (in part):
Code:
 Results:
 db,  reference is :2016-09-17T19:30:57:, other is :2016-09-17T19:30:41:
 db,  Difference in time is :-16:
 db,  absolute value of :-16: is :16:
2016-09-17 19:30:57  INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 19:30:41,
 db,  reference is :2016-09-17T12:02:26:, other is :2016-09-17T12:02:25:
 db,  Difference in time is :-1:
 db,  absolute value of :-1: is :1:

See man pages, results from Google for details ... cheers, drl
# 5  
Old 09-18-2016
@drl: I noticed there is a trailing comma in the the t9 variable. I have not tested with the dateutils packages, but I could imagine that this might have undesirable effects.

Code:
$ read d1 t1 j3 j4 j5 j6 j7 d8 t9 <<< file; echo "$t9"
19:30:41,

This could be mitigated like this:
Code:
$ IFS=$' \t\n,' read d1 t1 j3 j4 j5 j6 j7 d8 t9 <<< file; echo "$t9"
19:30:41

This User Gave Thanks to Scrutinizer For This Post:
# 6  
Old 09-18-2016
Don,

The dates change every day.

They are server log files that log access by IOT devices and the name changes every day.

I do not know what "CODE tags" are nor how to show them. I would want to have the first line be part of the output and not the second line.

I have Ubuntu Server 16.0.4

---------- Post updated at 11:35 AM ---------- Previous update was at 11:18 AM ----------

I want to thank everyone for the suggestions.

I will have to review them and test them to see what works.

Again, I am a newbie to unix and appreciate all the suggestions. I never thought that it would be easy. I am not a programmer, so I have a lot to digest.
# 7  
Old 09-18-2016
Quote:
Originally Posted by Markham
Don,

The dates change every day.

They are server log files that log access by IOT devices and the name changes every day.

I do not know what "CODE tags" are nor how to show them. I would want to have the first line be part of the output and not the second line.

I have Ubuntu Server 16.0.4

---------- Post updated at 11:35 AM ---------- Previous update was at 11:18 AM ----------

I want to thank everyone for the suggestions.

I will have to review them and test them to see what works.

Again, I am a newbie to unix and appreciate all the suggestions. I never thought that it would be easy. I am not a programmer, so I have a lot to digest.
Hi Mark,
I know that dates change every day. What I don't know is whether or not the starting time and ending times in your input data are ever on different dates. As I said before, if the timestamps being compared are always on the same date, your problem is much simpler than if the timestamps can be on different dates. The following tutorial explains how to use CODE and ICODE tags:

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Problem with implementing the times() function in C (struct tms times return zero/negative values)

Hello, i'm trying to implement the times() function and i'm programming in C. I'm using the "struct tms" structure which consists of the fields: The tms_utime structure member is the CPU time charged for the execution of user instructions of the calling process. The tms_stime structure... (1 Reply)
Discussion started by: g_p
1 Replies

2. HP-UX

grep for x for m times ',\{11\}'

All: OS version HP-UX ga016a501 B.11.31 U ia64 from the command prompt -grep for 1 to 11 occurences of "," returns both rows from the command prompt -grep for 11 occurences of "," returns 0 rows - should be 1 row. Any ideas - why? ga016a501 -> grep ',\{1,11\}' test3 | more ... (7 Replies)
Discussion started by: Bill L.
7 Replies

3. Shell Programming and Scripting

perl newbie . &&..programming newbie

Hi, I am new to programming and also to perl..But i know 'perl' can come to my rescue, But I am stuck at many places and need help..any small help is much appreciated... below is the description of what i intend to acheive with my script. I have a files named in this format... (13 Replies)
Discussion started by: xytiz
13 Replies

4. Shell Programming and Scripting

perl newbie . &&..programming newbie (question 2)

Hello everyone, I am having to do a lot of perl scripting these days and I am learning a lot. I have this problem I want to move files from a folder and all its sub folders to one parent folder, they are all .gz files.. there is folder1\folder2\*.gz and there are about 50 folders... (1 Reply)
Discussion started by: xytiz
1 Replies

5. UNIX for Dummies Questions & Answers

UNIX newbie NEWBIE question!

Hello everyone, Just started UNIX today! In our school we use solaris. I just want to know how do I setup Solaris 10 not the GUI one, the one where you have to type the commands like ECHO, ls, pwd, etc... I have windows xp and I also have vmware. I hope I am not missing anything! :p (4 Replies)
Discussion started by: Hanamachi
4 Replies

6. UNIX for Dummies Questions & Answers

Newbie Help with Grep or Awk .. Easy one ...

I have this output: uniquemember=uid=315kthatch,ou=people,ou=client315,dc=paisleyhosting,dc=com and i want the output to be just this: 315kthatch I need it to be generic tho, because I have hundreds of lines of output, and the preceding numbers are not always 315. So I would need... (3 Replies)
Discussion started by: kthatch
3 Replies

7. AIX

how would you know your server was rebooted 3 times or 5 times

Is there such location or command to know how many times did you reboot your server in that particular day?in AIX. (3 Replies)
Discussion started by: kenshinhimura
3 Replies

8. UNIX for Dummies Questions & Answers

Simple newbie grep question

How come grep testfile1 won't find anything in testfile1 (even though the characters sd are there in great quantity), but grep '' testfile1 will find plenty? Do the single quotes prevent the shell from interpreting the testfile1 is interpreted as: grep *test whether or not characters sd exist*... (5 Replies)
Discussion started by: doubleminus
5 Replies

9. Shell Programming and Scripting

GREP Searching for a newbie...

Hi, I really need some help with GREP searching... I need to find all occurances of a file reference and remove two characters from the end of the reference. For example, here are a few lines showing the text: <image file="STRAIGHT_004CR.jpg" ALT="STRAIGHT_004CR.jpg" /> <image... (8 Replies)
Discussion started by: steveglevin
8 Replies

10. AIX

grep to give how many times each lines were found

Lets say I have a file containing string patterns to be looked for inside a file. I would normaly do : grep -if MyFilePattern FiletoSearchInto if I use the -c it gives how many total lines it found out of my whole pattern file, but what if i want grep to report how many times it found each... (4 Replies)
Discussion started by: Browser_ice
4 Replies
Login or Register to Ask a Question