generating reports based on time field of network data


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting generating reports based on time field of network data
# 1  
Old 08-20-2008
generating reports based on time field of network data

hi i have data extracted in the following format ranging around 300000 to 800000 records in a text file , the format is of network data .
No. Time Source Destination Protocol
1 1998-06-05 17:20:23.569905 HP_61:aa:c9 HP_61:aa:c9 LLC
2 1998-06-05 17:20:24.569709 HP_61:aa:c9 HP_61:aa:c9 LLC
3 1998-06-05 17:20:25.547310 Cisco_04:41:bc Cisco_04:41:bc LOOP
4 1998-06-05 17:20:26.068756 HP_61:aa:c9 HP_61:aa:c9 TCP
5 1998-06-05 17:20:27.068809 HP_61:aa:c9 HP_61:aa:c9 UDP
6 1998-06-05 17:20:28.069107 HP_61:aa:c9 HP_61:aa:c9 FTP
7 1998-06-05 17:20:29.068810 HP_61:aa:c9 HP_61:aa:c9 FTP
8 1998-06-05 17:20:30.069417 HP_61:aa:c9 HP_61:aa:c9 TELNET
9 1998-06-05 17:20:31.071067 HP_61:aa:c9 HP_61:aa:c9 SMTP
10 1998-06-05 17:20:32.068868 HP_61:aa:c9 HP_61:aa:c9 SMTP
11 1998-06-05 17:20:33.069603 HP_61:aa:c9 HP_61:aa:c9 TELNET
12 1998-06-05 17:20:34.070245 HP_61:aa:c9 HP_61:aa:c9 LLC
13 1998-06-05 17:20:35.069411 HP_61:aa:c9 HP_61:aa:c9 LLC
14 1998-06-05 17:20:35.547414 Cisco_04:41:bc Cisco_04:41:bc LOOP
15 1998-06-05 17:20:36.572394 HP_61:aa:c9 HP_61:aa:c9 LLC
16 1998-06-05 17:20:37.570132 HP_61:aa:c9 HP_61:aa:c9 LLC

so on ......................

i want the output in following format -- for a day --- in a fixed interval of time
i should be prompt for start date and end date , then start time and end time and duration which is HH ( hours) or MM (minutes ) or SS ( seconds) --- this is offline analysis i am making ,
what is below displayed is what i have extracted from a awk script seperately for a 10 mins duration on a certain date.... but the problem i faced is inserting columns correspondingly -- like some time in a certain duration of time there might not be any count of a certain protocol packet then that entry will not be there in that duration then i will be having trouble in using either join or paste----

so which is better ( what kind of scripting ) can any one help me to do this

also since here the missing ones are left blank -- i want to replace that with 0
since i willl be using that for further analysis ( to add or to generate graph)

ofcourse if we have fixed number of protocols with the first field then join will be easy -- but extracting that count from the initial file is what i am finding a bit tricky

could any one please help me


17:20-7:30 17:30-17:40 17:40-17:50 17:50-18:00 18:00-18:10





ARP 6 20 22 42 20 DNS
204 212 92 50 FTP
593 303 183
FTP-DATA
487 1691 131
HTTP
354 596 528 297 ICMP 2 2 2 2 2 LLC 542 456 441 489 528 LOOP 58 60 60 60 60 NTP 18 20 18 18 20 SMTP
760 1600 409 303 TCP
3333 4939 3640 2622 TELNET

55 77 36 SSL




IP




ICAP





626 6289 9939 5671 3938
# 2  
Old 08-20-2008
there is some problem while i copied the output from spreadsheet -- this looks like below

1st row is 10 seconds duration
protocol 17:20-7:30 17:30-17:40 17:40-17:50 17:50-18:00 18:00-18:10
ARP 10 20 30 15 20
UDP 7 3 50 60 44
TELNET 1000 333 333 333 333
ICMP
SMTP
DNS
NetBIOS

etc .....
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Cron job scheduled is running once, but reports are generating twice

Team, Hope you all are doing fine I have one admin server which is being used dedicately to run cron jobs on hourly basis, fetching the details from Database which is in a different server.These cronjob are run on every hourly/5 minutes basis depending as per end user requirement.The script... (12 Replies)
Discussion started by: whizkidash
12 Replies

2. UNIX for Beginners Questions & Answers

Pivoting data based on a header field

Hi Team, Could you please help me with the below scenario. I have a file which is in the below format. Zipcode,001,001f,002,002f,003,003f,004,004f,005,005f,006,006f,007,007f 0050, ,0, ,0, ,0, ,1,*,7, ,7, ,7 0060, ,0, ,0, ,7, ,0,*,7, ,0, ,0 Would need the output as below. First field... (1 Reply)
Discussion started by: saj
1 Replies

3. UNIX for Dummies Questions & Answers

Generating all possible combinations of values in field 1 (awk)

Input: A|1 B|2 C|3 D|4 Output: A+B|3 A+C|4 A+D|5 B+C|5 B+D|6 C+D|7 A+B+C|6 A+B+D|7 A+C+D|8 B+C+D|9 A+B+C+D|10 I only managed to get the output for pairs of $1 values (i.e. combination of length 2): (4 Replies)
Discussion started by: beca123456
4 Replies

4. Shell Programming and Scripting

Matching and Merging csv data fields based on a common field

Dear List, I have a file of csv data which has a different line per compliance check per host. I do not want any omissions from this csv data file which looks like this: date,hostname,status,color,check 02-03-2012,COMP1,FAIL,Yellow,auth_pass_change... (3 Replies)
Discussion started by: landossa
3 Replies

5. Shell Programming and Scripting

extract data in a csv file based on a certain field.

I have a csv file that I need to extract some data from depending on another field after reading info from another text file. The text file would say have 592560 in it. The csv file may have some data like so Field 1 Field2 Field3 Field4 Field5 Field6 20009756 1 ... (9 Replies)
Discussion started by: GroveTuckey
9 Replies

6. Linux

Generating apache log reports

Hello all, I'm trying to find some tool on generating reports based on apache access_log files (of Common format). I found some of them (awstats, lire/logreport, weblog expert, apache logs viewer, etc..) but they generate some global and general report about the log file. Also some perl... (0 Replies)
Discussion started by: enux
0 Replies

7. Red Hat

sarg issue while generating reports for squid

I installed sarg from sarg rpm and i am facing issue while generating sarg reports and getting this time different error below sarg -l /var/log/squid/access.log SARG: Records in file: 242332, reading: 0.00% SARG: Records in file: 242332, reading: 2.06% SARG: Records in file: 242332, reading:... (0 Replies)
Discussion started by: mail4vij
0 Replies

8. Shell Programming and Scripting

Generating formatted reports from log files

Given that I have a log file of the format: DATE ID LOG_LEVEL | EVENT 2009-07-23T14:05:11Z T-4030097550 D | MessX 2009-07-23T14:10:44Z T-4030097550 D | MessY 2009-07-23T14:34:08Z T-7298651656 D | MessX 2009-07-23T14:41:00Z T-7298651656 D | MessY 2009-07-23T15:05:10Z T-4030097550 D | MessZ... (5 Replies)
Discussion started by: daccad
5 Replies

9. Shell Programming and Scripting

Moving files which are generating time to time

Hi all, I always getting great response from this forum, that why i am putting again.... I am working in a company which is giving ATM support.In one of my production server a lot of files are getting generated every day. I want to move these files to another name. The file name which is... (4 Replies)
Discussion started by: Renjesh
4 Replies
Login or Register to Ask a Question