Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk/sed to extract column bases on partial match

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk/sed to extract column bases on partial match
# 1  
Old 04-07-2011
awk/sed to extract column based on partial match
Hi
I have a log file which has outputs like the one below

Code:
[2011-04-06T17:57:15.661-07:00]  [octetstring] [NOTIFICATION] [OVD-20044] [com.octetstring.accesslog] [tid: 15]  [ecid: 0000IwhTc8C8pmP_IdH7if1Dauw_000H5q,0] [arg: 24196] [arg: 1] [arg: 0]  [arg: 9] [arg: 3712] [arg: 0] [arg: 486183328] [arg: 2147483648] conn=24,196  op=1 RESULT err=0 tag=0 nentries=9 etime=3,712 dbtime=0  mem=486,183,328/2,147,483,648




Now most of the time I am only interested in the time ( the first column) and a column that begins with etime i.e etime=someNumber. This column seems to shift around so I cant do a simple awk based on the column number. Is there a way to search for the word etime and output the etime=Xyz?

Hope I communicated what I was trying to do.I basically need to extract the field based on a partial match.

Thanks
Last edited by pkabali; 04-07-2011 at 02:57 AM..
# 2  
Old 04-07-2011
For the time:
Code:
sed 's/^\[\([^]]*\)].*/\1/' file

For the etime:
Code:
sed 's/.*etime=\([^ ]*\) .*/\1/' file

Both separate by a space:
Code:
sed 's/^\[\([^]]*\)].*etime=\([^ ]*\) .*/\1 \2/' file

# 3  
Old 04-07-2011
Hi Franklin52
Thank for posting but the solution doesnt seem to be working. It is not only printing entire lines it is also printing lines without etimes.

I would require an output like
Code:
[2011-04-06T17:57:15.661-07:00] etime=3,712

or better yet
Code:
2011-04-06 17:57:15 etime=3,712

for lines with etimes, and nothing for lines without etimes

currently i am using
Code:
grep -etime filename | awk '{print $1 " " $32}'

But as I explained earlier the etime field moves around and hence I miss out on certain output.

Thnaks
# 4  
Old 04-07-2011
The solution works fine with the line of your first post, probably the other lines of the file are formatted differently.
# 5  
Old 04-07-2011
Hi Franklin
Could you please explain what you are doing with the statements? I am relatively new to sed and awk and would appreciate the help. This way I can try and figure out if the code you wrote needs to be modified in a certain way for it to work on my system or file.
Would it help if I attached a few more lines of logs?
Thanks
# 6  
Old 04-07-2011
you can use awk just fine. since the time you want seems to contain 2 ":", we will use that as the regex pattern to capture.
Code:
$ awk '{for(i=1;i<=NF;i++) if($i~/.*:.*:.*|etime/) {print $i} }' file
[2011-04-06T17:57:15.661-07:00]
etime=3,712

Or if you have Ruby(1.9+)
Code:
$ ruby -ane '$F.each{|x| puts x if x.count(":")>1 or x[/etime/]}' file
[2011-04-06T17:57:15.661-07:00]
etime=3,712

# 7  
Old 04-07-2011
Code:
awk '{ if ( match($0,/etime=?[0-9]*,?[0-9]+/)) print substr($1,2,10)" "substr($1,13,8), substr($0,RSTART,RLENGTH) }' filename

Assumes that $1 will always be string in that specific format.
Code:
[2011-04-06T17:57:15.661-07:00]

Regards.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to update file based on partial match in field1 and exact match in field2

I am trying to create a cronjob that will run on startup that will look at a list.txt file to see if there is a later version of a database using database.txt as the source. The matching lines are written to output. $1 in database.txt will be in list.txt as a partial match. $2 of database.txt... (2 Replies)
Discussion started by: cmccabe
2 Replies

2. Shell Programming and Scripting

awk unique count of partial match with semi-colon

Trying to get the unique count of the below input, but if the text in beginning of $5 is a partial match to another line in the file then it is not unique. awk awk '!seen++ {n++} END {print n}' input 7 input chr1 159174749 159174770 chr1:159174749-159174770 ACKR1 chr1 ... (2 Replies)
Discussion started by: cmccabe
2 Replies

3. Shell Programming and Scripting

awk partial string match and add specific fields

Trying to combine strings that are a partial match to another in $1 (usually below it). If a match is found than the $2 value is added to the $2 value of the match and the $3 value is added to the $3 value of the match. I am not sure how to do this and need some expert help. Thank you :). file ... (2 Replies)
Discussion started by: cmccabe
2 Replies

4. UNIX for Dummies Questions & Answers

How to substitute for the partial match?

Hi I have a question and hope I can get answer here. Thank you in advance. I have two files: file1: aa X bb Y cc Z file2: cc A bb B dd C aa D bb E If the 1st column match in both file1 and file2, the 2nd column in file2 will be replaced by the 2nd column in file1. If there is no... (2 Replies)
Discussion started by: yuejian
2 Replies

5. Shell Programming and Scripting

awk Print New Column For Every Two Lines and Match On Multiple Column Values to print another column

Hi, My input files is like this axis1 0 1 10 axis2 0 1 5 axis1 1 2 -4 axis2 2 3 -3 axis1 3 4 5 axis2 3 4 -1 axis1 4 5 -6 axis2 4 5 1 Now, these are my following tasks 1. Print a first column for every two rows that has the same value followed by a string. 2. Match on the... (3 Replies)
Discussion started by: jacobs.smith
3 Replies

6. Shell Programming and Scripting

for each different entry in column 1 extract maximum values from column 2 in unix/awk

Hello, I have 2 columns (1st column has multiple entries but the corresponding values in the column 2 may be the same or different.) however I want to extract unique values for each entry in column 1 by assigning the max value from column 2 SDF4 -0.211654 SDF4 0.978068 ... (1 Reply)
Discussion started by: Diya123
1 Replies

7. Shell Programming and Scripting

AWK - Print partial line/partial field

Hello, this is probably a simple request but I've been toying with it for a while. I have a large list of devices and commands that were run with a script, now I have lines such as: a-router-hostname-C#show ver I want to print everything up to (and excluding) the # and everything after it... (3 Replies)
Discussion started by: ippy98
3 Replies

8. Shell Programming and Scripting

Awk or Sed, fubd match in column, then edit column.

FILE A: 9780743551526,(Abridged) 9780743551779,(Unabridged) 9780743582469,(Abridged) 9780743582483,(Unabridged) 9780743563468,(Abridged) 9780743563475,(Unabridged) FILE B: c3saCandyland 9780743518321 "CANDYLAND" "MCBAIN, ED" 2001 c3sbCandyland 9780743518321 ... (7 Replies)
Discussion started by: glev2005
7 Replies

9. Shell Programming and Scripting

Partial average of a column with awk

Hello, Let's assume I have 100 files FILE_${m} (0<m<101). Each of them contains 100 lines and 10 columns. I'd like to get in a file called "result" the average value of column 3, ONLY between lines 11 and 17, in order to plot that average as a function of the parameter m. So far I can compute... (6 Replies)
Discussion started by: DMini
6 Replies

10. Shell Programming and Scripting

awk partial match and filter records

Hi, I am having file which contains around 15 columns, i need to fetch column 3,12,14 based on the condition that column 3 starts with 40464 this is the sample data how to achieve that (3 Replies)
Discussion started by: aemunathan
3 Replies
Login or Register to Ask a Question