Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Newbie looking for how to Grep times more than 10 seconds apart

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Newbie looking for how to Grep times more than 10 seconds apart
# 22  
Old 09-18-2016
Quote:
Originally Posted by Don Cragun
In your earlier samples, the timestamps were in the 2nd field on a line and on the last field on a line. With your new data, the starting timestamp is not even close to being in the last field on the input line. If the timestamps are in the 2nd and 9th fields (with fields separated by sequences of one or more space or tab characters), the following should work:
Code:
awk '
{    split($2, t, ":")
    e = t[1] * 3600 + t[2] * 60 + t[3]
    split($9, t, ":")
    s = t[1] * 3600 + t[2] * 60 + t[3]
}
e - s > 10
' Example2

producing the output:
Code:
2016-09-17 10:30:57  INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 07:30:55, lat: 51.00000, lon: -112.00000, speed: 0.0, course: 0.0

with your new sample data.

And, please go back and look at that tutorial on how to post threads in the UNIX & Linux Forums and how to use CODE tags. There is absolutely nothing in there about using UNIX commands; only about how to format posts submitted to this forum in a manner that will allow people reading your posts to see leading spaces and tabs in the sample input, output, and code you show us and to see the difference between single spaces and sequences of spaces and tabs in the sample input, output, and code that you show us.

Don

Don, I will try to understand it better. It is starting to make sense to me about the spaces and formatting. Again us neophytes think everything is the same.

Mark
# 23  
Old 09-18-2016
Hi.

@Scrutinizer: dateutils seem to be able to scan entire lines and find stuff that corresponds to various date/time formats. However, if some characters confuse the programs, then your code will be very useful, thanks.

@Markham: Use the simplest solution, so if your data is all on the same day, use one of those. However, using programs that understand date-formatted data may be useful for someone else looking at this that has longer durations. Using the additional data with debugging turned on:
Code:
$ ./s1 data2

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.4 (jessie) 
bash GNU bash 4.3.30
dateutils.dconv dconv 0.3.1
dateutils.ddiff ddiff 0.3.1

-----
 Input data file data2:
2016-09-17 10:30:36  INFO: [D3B4AEB3] id: 4562079193, time: 2016-09-17 10:30:35, lat: 51.00000, lon: -112.00000, speed: 64.3, course: 130.0
2016-09-17 10:30:57  INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 07:30:55, lat: 51.00000, lon: -112.00000, speed: 0.0, course: 0.0
2016-09-17 10:30:57  INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 10:30:55, lat: 51.00000, lon: -112.00000, speed: 0.0, course: 0.0
2016-09-17 10:31:16  INFO: [D3B4AEB3] id: 4562079193, time: 2016-09-17 10:31:15, lat: 51.00000, lon: -112.00000, speed: 64.3, course: 130.0

-----
 Results:
 db,  reference is :2016-09-17T10:30:36:, other is :2016-09-17T10:30:35:
 db,  Difference in time is :-1:
 db,  absolute value of :-1: is :1:
 db,  reference is :2016-09-17T10:30:57:, other is :2016-09-17T07:30:55:
 db,  Difference in time is :-10802:
 db,  absolute value of :-10802: is :10802:
2016-09-17 10:30:57  INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 07:30:55, lat: 51.00000, lon: -112.00000, speed: 0.0, course: 0.0
 db,  reference is :2016-09-17T10:30:57:, other is :2016-09-17T10:30:55:
 db,  Difference in time is :-2:
 db,  absolute value of :-2: is :2:
 db,  reference is :2016-09-17T10:31:16:, other is :2016-09-17T10:31:15:
 db,  Difference in time is :-1:
 db,  absolute value of :-1: is :1:

Best wishes ... cheers, drl
# 24  
Old 09-30-2016
Slight modification needed
We just ran into an instance where there was an overlap from the previous day.

How hard is it to make the existing code (#21 by Don) to work with both present date and previous date?

Thanks in advance.

Mark
# 25  
Old 09-30-2016
As long as the dates are either identical or the date in field #1 is exactly one day after the date in field #8, the change is pretty simple:
Code:
awk '
{	split($2, t, ":")
	e = ($1 != $8) * 86400 + t[1] * 3600 + t[2] * 60 + t[3]
	split($9, t, ":")
	s = t[1] * 3600 + t[2] * 60 + t[3]
}
e - s > 10
' Example3

If Example3 contains:
Code:
2016-09-17 10:30:36  INFO: [D3B4AEB3] id: 4562079193, time: 2016-09-17 10:30:35, lat: 51.00000, lon: -112.00000, speed: 64.3, course: 130.0
2016-09-17 10:30:57  INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 07:30:55, lat: 51.00000, lon: -112.00000, speed: 0.0, course: 0.0
2016-09-17 10:30:57  INFO: [D3B4AEB3] id: 4562079217, time: 2016-09-17 10:30:55, lat: 51.00000, lon: -112.00000, speed: 0.0, course: 0.0
2016-09-17 10:31:16  INFO: [D3B4AEB3] id: 4562079194, time: 2016-09-17 10:31:15, lat: 51.00000, lon: -112.00000, speed: 64.3, course: 130.0
2016-09-18 00:00:06  INFO: [D3B4AEB3] id: 4562079195, time: 2016-09-17 23:59:59, lat: 51.00000, lon: -112.00000, speed: 64.3, course: 130.0
2016-09-18 00:00:06  INFO: [D3B4AEB3] id: 4562079196, time: 2016-09-17 23:59:55, lat: 51.00000, lon: -112.00000, speed: 64.3, course: 130.0

the output produced is:
Code:
2016-09-17 10:30:57  INFO: [D3B4AEB3] id: 4562079216, time: 2016-09-17 07:30:55, lat: 51.00000, lon: -112.00000, speed: 0.0, course: 0.0
2016-09-18 00:00:06  INFO: [D3B4AEB3] id: 4562079196, time: 2016-09-17 23:59:55, lat: 51.00000, lon: -112.00000, speed: 64.3, course: 130.0

Last edited by Don Cragun; 09-30-2016 at 10:59 PM.. Reason: Remove command line from sample output.
# 26  
Old 09-30-2016
Don,

The dates in Field #1 are NEVER more than one day after the dates in Field #8

Thanks. I tried it and it was able to find the ones that I needed.

Mark
# 27  
Old 09-30-2016
Hi.

No changes necessary with dateutils approach. Here are the results from running the script in post #4 for differing days in differing months in differing years:
Code:
$ ./s1 data3

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.6 (jessie) 
bash GNU bash 4.3.30
dateutils.dconv dconv 0.3.1
dateutils.ddiff ddiff 0.3.1

-----
 Input data file data3:
2016-01-01 00:00:01  INFO: [D3B4AEB3] id: 4562079193, time: 2015-12-31 23:59:58, lat: 51.00000, lon: -112.00000, speed: 64.3, course: 130.0
2016-01-01 00:00:01  INFO: [D3B4AEB3] id: 4562079193, time: 2015-12-31 23:59:40, lat: 51.00000, lon: -112.00000, speed: 64.3, course: 130.0

-----
 Results:
 db,  reference is :2016-01-01T00:00:01:, other is :2015-12-31T23:59:58:
 db,  Difference in time is :-3:
 db,  absolute value of :-3: is :3:
 db,  reference is :2016-01-01T00:00:01:, other is :2015-12-31T23:59:40:
 db,  Difference in time is :-21:
 db,  absolute value of :-21: is :21:
2016-01-01 00:00:01  INFO: [D3B4AEB3] id: 4562079193, time: 2015-12-31 23:59:40, lat: 51.00000, lon: -112.00000, speed: 64.3, course: 130.0

Best wishes ... cheers, drl
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Problem with implementing the times() function in C (struct tms times return zero/negative values)

Hello, i'm trying to implement the times() function and i'm programming in C. I'm using the "struct tms" structure which consists of the fields: The tms_utime structure member is the CPU time charged for the execution of user instructions of the calling process. The tms_stime structure... (1 Reply)
Discussion started by: g_p
1 Replies

2. HP-UX

grep for x for m times ',\{11\}'

All: OS version HP-UX ga016a501 B.11.31 U ia64 from the command prompt -grep for 1 to 11 occurences of "," returns both rows from the command prompt -grep for 11 occurences of "," returns 0 rows - should be 1 row. Any ideas - why? ga016a501 -> grep ',\{1,11\}' test3 | more ... (7 Replies)
Discussion started by: Bill L.
7 Replies

3. Shell Programming and Scripting

perl newbie . &&..programming newbie

Hi, I am new to programming and also to perl..But i know 'perl' can come to my rescue, But I am stuck at many places and need help..any small help is much appreciated... below is the description of what i intend to acheive with my script. I have a files named in this format... (13 Replies)
Discussion started by: xytiz
13 Replies

4. Shell Programming and Scripting

perl newbie . &&..programming newbie (question 2)

Hello everyone, I am having to do a lot of perl scripting these days and I am learning a lot. I have this problem I want to move files from a folder and all its sub folders to one parent folder, they are all .gz files.. there is folder1\folder2\*.gz and there are about 50 folders... (1 Reply)
Discussion started by: xytiz
1 Replies

5. UNIX for Dummies Questions & Answers

UNIX newbie NEWBIE question!

Hello everyone, Just started UNIX today! In our school we use solaris. I just want to know how do I setup Solaris 10 not the GUI one, the one where you have to type the commands like ECHO, ls, pwd, etc... I have windows xp and I also have vmware. I hope I am not missing anything! :p (4 Replies)
Discussion started by: Hanamachi
4 Replies

6. UNIX for Dummies Questions & Answers

Newbie Help with Grep or Awk .. Easy one ...

I have this output: uniquemember=uid=315kthatch,ou=people,ou=client315,dc=paisleyhosting,dc=com and i want the output to be just this: 315kthatch I need it to be generic tho, because I have hundreds of lines of output, and the preceding numbers are not always 315. So I would need... (3 Replies)
Discussion started by: kthatch
3 Replies

7. AIX

how would you know your server was rebooted 3 times or 5 times

Is there such location or command to know how many times did you reboot your server in that particular day?in AIX. (3 Replies)
Discussion started by: kenshinhimura
3 Replies

8. UNIX for Dummies Questions & Answers

Simple newbie grep question

How come grep testfile1 won't find anything in testfile1 (even though the characters sd are there in great quantity), but grep '' testfile1 will find plenty? Do the single quotes prevent the shell from interpreting the testfile1 is interpreted as: grep *test whether or not characters sd exist*... (5 Replies)
Discussion started by: doubleminus
5 Replies

9. Shell Programming and Scripting

GREP Searching for a newbie...

Hi, I really need some help with GREP searching... I need to find all occurances of a file reference and remove two characters from the end of the reference. For example, here are a few lines showing the text: <image file="STRAIGHT_004CR.jpg" ALT="STRAIGHT_004CR.jpg" /> <image... (8 Replies)
Discussion started by: steveglevin
8 Replies

10. AIX

grep to give how many times each lines were found

Lets say I have a file containing string patterns to be looked for inside a file. I would normaly do : grep -if MyFilePattern FiletoSearchInto if I use the -c it gives how many total lines it found out of my whole pattern file, but what if i want grep to report how many times it found each... (4 Replies)
Discussion started by: Browser_ice
4 Replies
Login or Register to Ask a Question