Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Need script to pull multiple field from log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need script to pull multiple field from log file
# 1  
Old 01-21-2010
Need script to pull multiple fields from log file
I am hoping to get some help with a script to pull certain fields from a log file.

Code:
User update (xx6xxx P) rpt (yy6yyy B) 2010/01/20 21:36:01.298
Remote client forward start streamid 85af  2010/01/20 21:36:01.307
   rpt2 (ZZ6ZZZ G) rpt1 (YY6YYY B) urcall (CQCQCQ  ) mycall (W1AW)
   user message (hamradio  message ) mycall (W1AW) streamid 85af
   hdr f 400000 r2 XX6XXX G r1 YY6YYY B ur CQCQCQ   my KB6OYA P/91AD fc 5462 OK
Stream end   streamid 85af cnt 272 missed 0  2010/01/20 21:36:06.701

Above is a snip from a log file. I am looking to pull the info from in between the "()" after the field "mycall". I am also looking to pull the field after "user message" and the time stamp at the end of the line starting with "Stream end". This information changes frequently depending on use of the system.

Should look like this:

W1AW hamradio message 2010/01/20 21:36:06.

Also, the data after the mycall should not print out twice.


Any help would be appreciated.

Thanks

Ted
Last edited by TedSD; 01-21-2010 at 04:18 AM.. Reason: use code tags please, ty
# 2  
Old 01-21-2010
The format of the log file is not clear.
Is there one message set per logfile or are there multiple messages per logfile?
Is there anything else in the logfile in a different format?
Is the message set one long message wrapped or multiple distinct lines?
# 3  
Old 01-22-2010
This log file is generated as users transmit on a ham radio data repeater.
Below is another sample from the log file. I separated the sections that denote a given set of information that I need to pull data from. The log file is a constant stream of information. The data sets below are somewhat unique in that they dont occur unless users are accessing the system.

When a user starts a transmission the log generates a line starting with "Remote client start" or "Remoter gateway tx start" then a stream id and a time stamp. When the user stops transmitting the log generates a line starting with "Stream end" or "Remote gateway tx end" , streamid then time stamp.
Im looking to grep or awk the information after "mycall" on the line starting with "user message", in between the "()", the "user message" between "()" and the time stamp at the end of "Stream end" line.

None of the data should occur twice in the final output. That way I have a user list with no duplicates when the script is run.

I wish I could articulate what I looking for a little better.


Code:
User update (KI6KQU S) rpt (KI6KQU S) 2010/01/21 19:38:34.765
User update (W6SAT   ) rpt (KI6KQU B) 2010/01/21 19:39:05.595

Remote client forward start streamid 2511  2010/01/21 19:39:05.604
   rpt2 (KI6KQU G) rpt1 (KI6KQU B) urcall (CQCQCQ  ) mycall (W6SAT   /TED )
   user message (ID-800              ) mycall (W6SAT   ) streamid 2511
Stream end   streamid 2511 cnt 18 missed 0  2010/01/21 19:39:05.917

Remote client forward start streamid 8235  2010/01/21 19:39:12.860
   rpt2 (KI6KQU G) rpt1 (KI6KQU B) urcall (CQCQCQ  ) mycall (W6SAT   /TED )
   user message (ID-800              ) mycall (W6SAT   ) streamid 8235
Stream end   streamid 8235 cnt 19 missed 0  2010/01/21 19:39:13.202

Remote gateway tx start (REF012 A) streamid 0019  hdr BAD 2010/01/21 19:39:20.152
   rpt2 (KI6MGN G) rpt1 (REF012 A) urcall (CQCQCQ  ) mycall (W6SAT   /TED )
   user message (ID-800              ) mycall (W6SAT   ) streamid 0019
Remote gateway tx end   (REF012 A) streamid 0019 cnt 22 missed 0 2010/01/21 19:39:20.541

Remote gateway tx start (REF012 A) streamid 7d0b  hdr BAD 2010/01/21 19:39:25.024
   rpt2 (KI6MGN G) rpt1 (REF012 A) urcall (CQCQCQ  ) mycall (W6SAT   /TED )
   user message (ID-800              ) mycall (W6SAT   ) streamid 7d0b
Remote gateway tx end   (REF012 A) streamid 7d0b cnt 14 missed 0 2010/01/21 19:39:25.289

Linked gateway list: 2010/01/21 19:39:33.380
 remote gateway 209.112.244.26:20001  call REF012 A status Linked to B
Last heard list: 2010/01/21 19:39:33

Thanks for the help

Ted
# 4  
Old 01-22-2010
First write a quick code for your reference:

Code:
$ awk 'BEGIN{RS="";FS="\n"} /mycall/ {split($3,a,"[()]"); print a[4],a[2] }' urfile
W6SAT    ID-800
W6SAT    ID-800
W6SAT    ID-800
W6SAT    ID-800

What's the rule to export date? Second one in every section?
# 5  
Old 01-22-2010
Quote:
Originally Posted by rdcwayx
First write a quick code for your reference:

What's the rule to export date? Second one in every section?
Yes it the one that signals the end of the transmission. Second one.

Thanks for the help
Last edited by TedSD; 01-22-2010 at 03:49 AM..
# 6  
Old 01-23-2010
Code:
$ awk 'BEGIN{RS="";FS="\n"} /mycall/ {split($3,a,"[()]"); split ($4,b,"[ .]"); c=length(b);print a[4],a[2],b[c-2],b[c-1] }' urfile
W6SAT    ID-800               2010/01/21 19:39:05
W6SAT    ID-800               2010/01/21 19:39:13
W6SAT    ID-800               2010/01/21 19:39:20
W6SAT    ID-800               2010/01/21 19:39:25

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to pull multiple XML tags from the same XML file in Shell.?

I'm searching for the names of a TV show in the XML file I've attached at the end of this post. What I'm trying to do now is pull out/list the data from each of the <SeriesName> tags throughout the document. Currently, I'm only able to get data the first instance of that XML field using the... (9 Replies)
Discussion started by: hungryd
9 Replies

2. Shell Programming and Scripting

Command/script to match a field and print the next field of each line in a file.

Hello, I have a text file in the below format: Source Destination State Lag Status CQA02W2K12pl:D:\CAQA ... (10 Replies)
Discussion started by: pocodot
10 Replies

3. Linux

How do I format a Date field of a .CSV file with multiple commas in a string field?

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 "12","B000QRIGJ4","4432","string with quotes, and with a comma, and colon: in... (3 Replies)
Discussion started by: dhruuv369
3 Replies

4. UNIX for Dummies Questions & Answers

Pull out multiple lines with grep patternfile

Hi, I'm trying to get lines from a file using identifiers in the first two columns. I have used: cat MasterFile.txt | grep -f Pattern.txt and the lines I want display on screen. If I try to put them in a file the file is created but stays empty: cat MasterFile.txt | grep -f Pattern.txt... (14 Replies)
Discussion started by: FGPonce
14 Replies

5. Shell Programming and Scripting

Access log field - using awk to pull date/time

hey guys. the following line is a line taken from apache's access_log 10.10.10.10 - jdoe "GET /images/down.gif HTTP/1.1" 304 I'm concerned about the field that has the date and time in it. if assuming the delimiter in the file is a space, then the fourth field will always have the date... (5 Replies)
Discussion started by: SkySmart
5 Replies

6. Shell Programming and Scripting

Script to pull hashes out of large text file

I am attempting to write a script that will pull out NTLM hashes from a text file that contains about 500,000 lines of data. Not all accounts contain hashes and I only need the ones that do contain hashes. Here is a sample of what the data looks like: There are thousands of other lines in... (6 Replies)
Discussion started by: chango77747
6 Replies

7. Shell Programming and Scripting

Please do help: Perl Script to pull out rows from a CSV file

I have CSV file that contains data in the format as shown below: ABC, 67, 56, 67, 78, 89, 76, 55 PDR, 85, 83, 83, 72, 82, 89, 83 MPG, 86, 53, 54, 65, 23, 54, 75 .. .. .. .. I want to create a script that will pull out the rows from the above sheet and paste it into another CSV file.... (12 Replies)
Discussion started by: pankajusc
12 Replies

8. Shell Programming and Scripting

Archive log Pull script from one server to another server

Hi all, Iam looking out for a shell script which pulls archive from Server A to Server B. And at the same time, we dont want the archives which are already pulled in Server B to be removed from Server A. Please help me on this, I have been trying on this for a quiet few time. ... (3 Replies)
Discussion started by: vivi.raghav
3 Replies

9. Shell Programming and Scripting

How do you use pull data from multiple lines to do a for statement?

Guys I am having a problem with being able to find missing monitors in a configuration check script I am trying to create for accountability purposes for managing a large number of systems. What I am trying to do is run a script that will look at the raw config data in a file and pull all the pool... (7 Replies)
Discussion started by: scottzx7rr
7 Replies

10. UNIX for Dummies Questions & Answers

Pull a file from a remote server through a shell script

Hi, I am writing a shell script to pull a file from a remote server (Let say its a windows based remote server). One of my criteria is to pull a file only if it is not empty. We have done a similar script to push a file from our end to a remote server and before pushing it we check for the... (2 Replies)
Discussion started by: sashankkrk
2 Replies
Login or Register to Ask a Question