How to extract part of string from all log files.?

Login or Register to Ask a Question and Join Our Community

How to extract part of string from all log files.?

Tags

Login to Discuss or Reply to this Discussion in Our Community

Top Forums Shell Programming and Scripting How to extract part of string from all log files.?

07-19-2016

Banned

74, 1

Join Date: May 2015

Last Activity: 25 March 2019, 8:12 AM EDT

Posts: 74

Thanks Given: 3

Thanked 1 Time in 1 Post

How to extract part of string from all log files.?

Hi All,

Let us say we have 5 log files, extract the data from all log files and save the output in a file.

Code:

home/log/first.log
home/log/second.log
home/log/third.log
home/log/four.log
home/log/five.log

I want to extract the following text from the log files and save the output in a file.

Code:

numberoffiles=5 numberofRows=3459
Time taken: 10.65 seconds

The log file look like this

Code:

/ABC/RTE/AD_900_VOP_123/OPP
/ABC/RTE/TRE/AD_900_VOP_145/BBB
/ABC/RTE/AN_900_VFP_124/FBF
/ABC/RTE/HD_900_FOP_153/WEW
/ABD/RDV/AD_900_VOP_123/OPP
/ABC/RTE/WD_900_VOP_123/GRR/TRD
/ABC/RTE/RTD/AR_900_VOP_443/SDD

Thanks in Advance.

Moderator's Comments:

Mod Comment

Use code tags, thanks.

---------- Post updated at 03:43 PM ---------- Previous update was at 03:10 PM ----------

Sorry for violating code tags.

Please help me.

Thanks

Last edited by rbatte1; 07-20-2016 at 05:26 AM.. Reason: Code tags for output

ROCK_PLSQL

View Public Profile for ROCK_PLSQL

Find all posts by ROCK_PLSQL

07-19-2016

Registered User

11,728, 1,345

Join Date: Feb 2004

Last Activity: 8 May 2020, 9:07 AM EDT

Location: NM

Posts: 11,728

Thanks Given: 903

Thanked 1,345 Times in 1,201 Posts

Try this:

Code:

>output.txt
for filename in /home/log/*.log
do
   echo "$filename"
  tr -s '[,\]\[]' ' ' < filename | 
  awk '/numberoffiles/ {print $4, $5}
         /Time taken/ {print} '
  echo " "
done > output.txt

- tr replaces brackets and commas with a space
- awk prints 4th and 5th columns when numberoffiles appears in the line
--- -- prints line when"Time taken" appears.

jim mcnamara

View Public Profile for jim mcnamara

Find all posts by jim mcnamara

07-19-2016

Banned

74, 1

Join Date: May 2015

Last Activity: 25 March 2019, 8:12 AM EDT

Posts: 74

Thanks Given: 3

Thanked 1 Time in 1 Post

Hi,

Thanks for your response.

I have tried this only file names are creating and data is not coming.

output.txt

Code:

/home/log/first.log

Please help me.

Thanks

Moderator's Comments:

Mod Comment

Please wrap all code, files, input & output/errors in CODE tags.
It makes it far easier to read and preserves spaces for indenting or fixed width data.

Last edited by rbatte1; 07-20-2016 at 05:27 AM.. Reason: Added CODE tags

ROCK_PLSQL

View Public Profile for ROCK_PLSQL

Find all posts by ROCK_PLSQL

07-19-2016

Registered User

11,728, 1,345

Join Date: Feb 2004

Last Activity: 8 May 2020, 9:07 AM EDT

Location: NM

Posts: 11,728

Thanks Given: 903

Thanked 1,345 Times in 1,201 Posts

You have to help us here.
What is the name and kind of UNIX/Linux you have?
Please try these commands and post the output.

Code:

uname -a
cat /etc/release

What shell are you using? i.e., bash, ksh ...

I can guess what is wrong but that will take forever.

jim mcnamara

View Public Profile for jim mcnamara

Find all posts by jim mcnamara

07-19-2016

Banned

74, 1

Join Date: May 2015

Last Activity: 25 March 2019, 8:12 AM EDT

Posts: 74

Thanks Given: 3

Thanked 1 Time in 1 Post

Hi,

I found the issue.

$ is missing in front of filename.

I have added data is coming in the file.

But one extra bracket is coming.

Code:

[numberoffiles=8 numberofRows=112
Time taken: 19.43 seconds

one more thing instead of position can we do based on the string only.

Let us say in the log some more fields added as below it will be problem.

Code:

for filename in /home/log/*.log
do
   echo "$filename"
  tr -s '[,\]\[]' ' ' < $filename | 
  awk '/numberoffiles/ {print $4, $5}
         /Time taken/ {print} '
  echo " "
done > output.txt

Please help me.

Thanks

Last edited by ROCK_PLSQL; 07-21-2016 at 04:23 AM.. Reason: Corrected spelling and highlighted the $

ROCK_PLSQL

View Public Profile for ROCK_PLSQL

Find all posts by ROCK_PLSQL

07-19-2016

Registered User

11,728, 1,345

Join Date: Feb 2004

Last Activity: 8 May 2020, 9:07 AM EDT

Location: NM

Posts: 11,728

Thanks Given: 903

Thanked 1,345 Times in 1,201 Posts

Sorry about the error - I was away from where I could test the script.

I will show you a template of how to deal with this stuff in the context I already have.
You have to enumerate all of the fields on the line with numberoffiles
I changed the code to be verbose so you can modify it for the next change that we do not know about yet.

Change the awk script:

Code:

tr -s '[,\]\[]' ' ' < $filename | 
     awk '
     {  if (index($0, "stats" ) > 0)
        {
         { for(i=1;i<=NF; i++)                               
             {               
                if( $(i) ~ /numberoffiles/ || $(i) ~ /numberofRows/ ) 
                {
                 printf("%s ", $(i) )
                }
             }
             printf("\n")
         }
       }
       if (index($0, "Time")> 0 )  
       {
         print $0
       } 
      
     } '

What this does is to step through each field and see what the field compares to.
When a match is found it print just the field. Notice it uses printf() which lets us
use output format specifiers just like in the C language.

This User Gave Thanks to jim mcnamara For This Post:

jim mcnamara

View Public Profile for jim mcnamara

Find all posts by jim mcnamara

07-19-2016

Registered User

15,129, 5,008

Join Date: Jul 2012

Last Activity: 4 May 2020, 4:31 PM EDT

Location: Aachen, Germany

Posts: 15,129

Thanks Given: 735

Thanked 5,008 Times in 4,483 Posts

Try also

Code:

awk '
BEGIN                   {SRCH1 = "numberoffiles[^ ]*"
                         SRCH2 = "numberofRows[^ ]*"
                        }
                        {gsub (/[][,]/, "")
                        }
FNR == 1                {FND = 0
                        }
match ($0, SRCH1)       {RST = RSTART
                         RLN = RLENGTH  
                         match ($0, SRCH2)
                         print FILENAME
                         print substr ($0, RST, RLN), substr ($0, RSTART, RLENGTH)
                         FND = 1
                        }
/Time/ && FND
' /home/log/*.log

This User Gave Thanks to RudiC For This Post:

RudiC

View Public Profile for RudiC

Find all posts by RudiC

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Need help to extract part of the string

Hi, I have a string with name as 20140412-s1-Potopolive_promos_20140412. So I want to extract only Potopolive string. Could you please help me the command. O/p : Potopolive Thx in advance

2. Shell Programming and Scripting

Unzip all the files with subdirectories present and append a part of string from the main .zip files

Hi frnds, My requirement is I have a zip file with name say eg: test_ABC_UH_ccde2a_awdeaea_20150422.zip within that there are subdirectories on each directory we again have .zip files and in that we have files like mama20150422.gz and so on. Iam in need of a bash script so that it unzips...

3. Shell Programming and Scripting

Extract number part from the string in ksh 88

I have to extract number part (Date and timestamp part ) from the following 3 strings AB_XYZA_20130930183017.log AB_DY_XYZA_20130930183017.log AB_GZU_20130930183017.log Output should be 20130930183017 Please help me to get the string like above Thanks

4. Shell Programming and Scripting

Extract part of a string

I have a file with 100s of lines of text. I want to perform an extraction of this line: Info bpzs(pid=2652) using 1000 bits I have not been able to extract it. Should I try expr match or is there another method? This line has data both in front of and in back of it. I do not have grep -o...

5. Shell Programming and Scripting

extract part of string using sed

Hi, I have the following string and I need to extract the date from it: TextHere,2012/07/11,1 I tried using sed: sed -n 's#^.*\({4}/{2}/{2}\).*$#\1#p' But it doesn't return anything. The following line doesn't even return '2012': sed -n 's/^.*\({4}\).*$/\1/p' I used to use grep -o...

6. Shell Programming and Scripting

Get a part of a String from a log file

Hey there, I'm searched for one day your forum for a similar solution. Didn't find one, sorry :( Now here is what I'm searching for. I have the following multiple lines from a log (Edit: log name is SystemOut.log - this is not my day. I Apologize): 000042e2 1_BLA_Yab I logging.LogInfo...

7. Shell Programming and Scripting

extract last part of string.

Hi, I have a string like this BUNDLE=/home/bob/flx/user.bun how to extract only the the last part ie, only user.bun

8. Shell Programming and Scripting

get the part of log between 2 string

Hi Let's say I have a log file with this format: Marker1 End of Marker1 .... Marker1 End of Marker1 .... Marker1 End of Marker1 how to get the last part between End of Marker1 and Marker1?

9. Shell Programming and Scripting

How to extract certain part of log file?

Hi there, I'm having some problem with UNIX scripting (ksh), perhaps somebody can help me out? For example: ------------ Sample content of my log file (text file): -------------------------------------- File1: .... info_1 ... info_2 ... info_3 ... File2: .... info_1 ... info_2 ......

10. Shell Programming and Scripting

Extract Part of string from 3rd field $3 using AWK

I'm executing "wc -lc" command in a c shell script to get record count and byte counts and writing them to a file. I get the result with the full pathname of the file. But I do not want the path name to be printed in the output file. I heard that using Awk we can get this but I don't have any...

Login or Register to Ask a Question