sed / awk / grep to extract information from log


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sed / awk / grep to extract information from log
# 1  
Old 10-14-2017
sed / awk / grep to extract information from log

Hi all, I have a query that runs that outputs data in the following format -

Code:
01/09/12 11:43:40,ADMIN,4,77,Application Group Load: Name(TESTED) LoadId(5137-1-0-1XX-15343-15343) File(/dir/dir/File.T03.CI2.RYR.2012009.11433350806.ARD) InputSize(5344) OutputSize(1359) Rows(2) Time(1.9960) Appl(C99I2),T,1437884,

There are several thousand entries like this going back years in my logs. I need to extract the LoadId Field, which might physically be in a different location because ADMIN could change, the TESTED field could change, etc. But, LoadId will always be-

Code:
LoadId(nnnn-n-n-1FAA-NNNNN-NNNNN)

Last two parts might vary.

What is the most efficient way to run through this with the korn shell. I am no expert in any of this. Usually, I'd dump this data into excel and do text to columns, but I need an automated way.

Help is appreciated
# 2  
Old 10-14-2017
Code:
$ grep -o "LoadId([^)]*)" file
LoadId(5137-1-0-1XX-15343-15343)

Or with sed
Code:
$ sed -n "/LoadId/{s/.*\(LoadId([^)]*)\).*/\1/;p;}" file
LoadId(5137-1-0-1XX-15343-15343)

This User Gave Thanks to Scott For This Post:
# 3  
Old 10-15-2017
A short sed (but maybe slower if there are many lines without LoadId)
Code:
sed -n 's/.*\(LoadId([^)]*)\).*/\1/p' file

The /p modifier prints if a substitution was done.
# 4  
Old 10-15-2017
Quote:
Originally Posted by Scott
Code:
$ grep -o "LoadId([^)]*)" file
LoadId(5137-1-0-1XX-15343-15343)

Or with sed
Code:
$ sed -n "/LoadId/{s/.*\(LoadId([^)]*)\).*/\1/;p;}" file
LoadId(5137-1-0-1XX-15343-15343)

Hello Scott,

The grep did not work for some reason, though I am going to investigate for future uses.

the sed worked perfectly.
# 5  
Old 10-15-2017
Hi.

-o is a GNU Grep option, so it should work on Linux, or if you're using Solaris, you could try /usr/sfw/bin/ggrep (or /usr/xpg4/bin/grep (it's a while since I've used Solaris, so I don't remember which of those locations its in!)), but the sed will work regardless.

Cheers.
# 6  
Old 10-16-2017
Quote:
Originally Posted by Scott
Code:
$ grep -o "LoadId([^)]*)" file
LoadId(5137-1-0-1XX-15343-15343)

Or with sed
Code:
$ sed -n "/LoadId/{s/.*\(LoadId([^)]*)\).*/\1/;p;}" file
LoadId(5137-1-0-1XX-15343-15343)

Wondering how I can take this a bit further.

I am looking to get only the information in the ( ) after load ID. So I would like to get 5137-1-0-1XX-15343-15343 by itself.
# 7  
Old 10-16-2017
Just move the part you don't want outside of the remembered expression.

e.g.
Code:
$ sed -n "/LoadId/{s/.*LoadId(\([^)]*\).*/\1/;p;}" file
5137-1-0-1XX-15343-15343

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk script to extract transcript information from gff3 file

I need help to extract transcript information from gff3 file. Here is the input Chr01 JGI gene 82773 86941 . - . ID=Potri.001G000900;Name=Potri.001G000900 Chr01 JGI mRNA 82793 86530 . - . ID=PAC:27047814;Name=Potri.001G000900.1;pacid=27047814;longest=1;Parent=Potri.001G000900... (6 Replies)
Discussion started by: Maduranga
6 Replies

2. Shell Programming and Scripting

Include information in certain columns using grep and awk

HI all, I have data in a file that looks like this: 1 HOW _ NNP NNP _ 3 nn _ _ 2 DRUGS _ NNP NNP _ 3 nn _ _ 3 ACT _ NNP NNP _ 0 null _ _ 4 : _ ... (3 Replies)
Discussion started by: owwow14
3 Replies

3. Shell Programming and Scripting

Extract various information from a log file

Hye ShamRock If you can help me with this difficult task for me then it will save my day Logs : ================================================================================================================== ... (4 Replies)
Discussion started by: SilvesterJ
4 Replies

4. Shell Programming and Scripting

extract information from a log file (last days)

I'm still new to bash script , I have a log file and I want to extract the items within the last 5 days . and also within the last 10 hours the log file is like this : it has 14000 items started from march 2002 to january 2003 awk '{print $4}' < *.log |uniq -c|sort -g|tail -10 but... (14 Replies)
Discussion started by: matarsak
14 Replies

5. Shell Programming and Scripting

Router ping log extract data from it Awk/Sed/grep

Hi, I am new to this world.. Using expect i loging to router and checking ping response to my links. I need to genarate report using this output and that report contains only three file link name, packet loss, latency. my output of script is like below: -bash-3.00$ monmw/mwbkp... (2 Replies)
Discussion started by: jkmistry
2 Replies

6. Shell Programming and Scripting

Use grep sed or awk to extract string from log file and put into CSV

I'd like to copy strings from a log file and put them into a CSV. The strings could be on different line numbers, depending on size of log. Example Log File: File = foo.bat Date = 11/11/11 User = Foo Bar Size = 1024 ... CSV should look like: "foo.bat","11/11/11","Foo Bar","1024" (7 Replies)
Discussion started by: chipperuga
7 Replies

7. Shell Programming and Scripting

Extract information from Log file formatted

Good evening! Trying to make a shell script to parse log file and show only required information. log file has 44 fields and alot of lines, each columns separated by ":". log file is like: first_1:3:4:5:6:1:3:4:5:something:notinterested second_2:3:4:3:4:2 first_1:3:4:6:6:7:8 I am interested... (3 Replies)
Discussion started by: dummie55
3 Replies

8. UNIX for Advanced & Expert Users

bash/grep/awk/sed: How to extract every appearance of text between two specific strings

I have a text wich looks like this: clid=2 cid=6 client_database_id=35 client_nickname=Peter client_type=0|clid=3 cid=22 client_database_id=57 client_nickname=Paul client_type=0|clid=5 cid=22 client_database_id=7 client_nickname=Mary client_type=0|clid=6 cid=22 client_database_id=6... (3 Replies)
Discussion started by: Pioneer1976
3 Replies

9. UNIX for Advanced & Expert Users

WEB Server Log File Analysis using awk/sed/grep

I'm trying to find a way to show large page sizes (page size in K) from multiple web server log files. Essentially I want to show only rows from a file where a specific column is larger than some value. Has anyone ever done this type of log analysis? If so, a snippet of code would be very... (2 Replies)
Discussion started by: mike_cataldo@ad
2 Replies

10. Shell Programming and Scripting

AWK to extract information

Hi all, I am working on a shell script to extract information from a file that has output from Oracle sqlplus. The problem is that the output of a single line is spread across multiple lines and i do not know as how to extract the particular filed at ones,which spans multiple lines.... (2 Replies)
Discussion started by: harris2107
2 Replies
Login or Register to Ask a Question