Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Feeding information in columns of LOG file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Feeding information in columns of LOG file
# 1  
Old 11-20-2011
Feeding information in columns of LOG file
Dear all,

I want to write a shell script to easy my job. Here is the description of task:
I have several files (d1.log, d2.log, d3.log etc)
with the common text as
Code:
=======using photon counter ======
tot pho is           = 29596
nomatch pho is       = 1350
phoeta pho is        = 1220
nospike_counter is   = 1220
et cut passed pho    = 674
ecal cut passed pho  = 566
hcal cut passed pho  = 538
trk cut passed pho   = 242
sieie cut passed pho = 377
hoe cut passed pho   = 580
pixsed cut passed pho= 888
phoID passed pho     = 41

Only values of variables will change in the different log files.

I want to grep "counter" and print the next 8 lines, which I am doing using:
Code:
 awk '/counter/ {p = 9}
     p > 0   {print $0; p--}' d1.log > LOG

I can do it for all files.

But Finally, I this information feed in a LOg file soemthing like this.
Code:
=======using photon counter ======
tot pho is           = 29596           34788
nomatch pho is       = 1350          46758
phoeta pho is        = 1220             2387
nospike_counter is   = 1220          876
et cut passed pho    = 674            879 
ecal cut passed pho  = 566           34
hcal cut passed pho  = 538            45
trk cut passed pho   = 242            45
sieie cut passed pho = 377            56
hoe cut passed pho   = 580            98
pixsed cut passed pho= 888           87
phoID passed pho     = 41              6

and similarly all d*.log file information in subsequent column of LOG file. In the end, I need to add this numbers on row basics and printed in the last column of the LOG file

If I could make a shell script, would be great.

Thanks in advance
pooja..
# 2  
Old 11-20-2011
Hi,

1. You want more than 8 lines to be printed I guess, 12 lines?
--Yes it is correct.

2. You want to print the lines after "=== using photon counter===" right, not just any counter?
--true.

3. The last column, is it the sum of all the related values found in the log files?
--No, presently is is the NUMBERS from different d*.log file.
Please check the desired output form given below.

4. You want the accumulated info in one single file?
Yes, in single LOG file.

Need some more clarity on the desired output
desired output
Code:
=======using photon counter ======
tot pho is           =  7            4              11
nomatch pho is       = 1         2              3
phoeta pho is        = 1             2            3
nospike_counter is   = 0          2             2
et cut passed pho    = 6            4           10
ecal cut passed pho  = 2           1            3
hcal cut passed pho  = 8            1           9
trk cut passed pho   = 2            1            3
sieie cut passed pho = 3            4            7
hoe cut passed pho   = 5            4            9
pixsed cut passed pho= 8           4            12
phoID passed pho     = 1            1            2

Where,
red - d1.log
blue - d2.log
similarly there will be columns for other log file numbers
bold are the addition of variables like "tot pho is" .


I hope , it is clear now? Please let me know if it is still confusing.

Thanks
pooja..
# 3  
Old 11-20-2011
Try this...
Code:
awk '/using photon counter/{p=12;next}
p>0{
        split($0,arr,"=")
        h[$1"HDR"]=arr[1]
        a[$1]=a[$1]" "$NF
        sum[$1]+=$NF
        p--
}
END{
        for(i in a)
                print h[i"HDR"]"="a[i]" "sum[i]
}' *.log > LOG

--ahamed
This User Gave Thanks to ahamed101 For This Post:
nrjrasaxena
# 4  
Old 11-20-2011
Yes, Smilie

THANKs a lot ahamed..
pooja..

---------- Post updated at 07:02 AM ---------- Previous update was at 06:54 AM ----------

Hi again,
A simple question please.
Will it be possible to provide the information of "no of lines" & "LOG file name" information from terminal ?
& also If i could mention that the files it need to read should be *.log not any other file having "counter" text.

Pooja..
# 5  
Old 11-20-2011
Code:
awk '
/using photon counter/{p=lines;next}
...
' lines=12 *.log > any_log_name

*.log will make sure only files with log extension are considered for processing.

--ahamed
# 6  
Old 11-20-2011
Hi,
There is some confusion. IT is working all fine, but somehow its changing the order of the text.
for example, here is the text in d*log file
Code:
=======using photon counter ======
tot pho is           = 419
nomatch pho is       = 20
phoeta pho is        = 19
nospike_counter is   = 19
et cut passed pho    = 10
ecal cut passed pho  = 10
hcal cut passed pho  = 9
trk cut passed pho   = 5
sieie cut passed pho = 4
hoe cut passed pho   = 11
pixsed cut passed pho= 14
phoID passed pho     = 1

wheres, I am getting following in the LOG file:
Code:
sieie cut passed pho = 4 4 8
phoID passed pho     = 1 1 2
ecal cut passed pho  = 10 10 20
pixsed cut passed pho= 14 14 28
hoe cut passed pho   = 11 11 22
hcal cut passed pho  = 9 9 18
tot pho is           = 419 419 838
phoeta pho is        = 19 19 38
et cut passed pho    = 10 10 20
nospike_counter is   = 19 19 38
trk cut passed pho   = 5 5 10
nomatch pho is       = 20 20 40

The script I am using is following:
Code:
#!/bin/bash                                                                                                                                                            
awk '/using photon counter/{p=12;next}                                                                                                                                 
p>0{                                                                                                                                                                   
        split($0,arr,"=")                                                                                                                                              
        h[$1"HDR"]=arr[1]                                                                                                                                              
        a[$1]=a[$1]" "$NF                                                                                                                                              
        sum[$1]+=$NF                                                                                                                                                   
        p--                                                                                                                                                            
}                                                                                                                                                                      
END{                                                                                                                                                                   
        for(i in a)                                                                                                                                                    
                print h[i"HDR"]"="a[i]" "sum[i]                                                                                                                        
}' *.log > LOG

The order is important for comparison..Smilie

pooja..
# 7  
Old 11-20-2011
Try this...
Code:
awk '/using photon counter/{p=lines;next}
p>0{
        split($0,arr,"=")
        if(!h[$1"HDR"])h[$1"HDR"]=arr[1]
        a[$1]=a[$1]" "$NF
        if(j<lines) b[++j]=$1
        sum[$1]+=$NF
        p--
}END{
        for(i=1;i<=j;i++)
                print h[b[i]"HDR"]"="a[b[i]]" "sum[b[i]]
}' lines=12 *.log

--ahamed
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Getting information out of a log file

Hello guys, I am trying to filter some information out of a log file (example shortcut) ===== fspCIV0 /vol/vol0 -sec=sys,rw=fspsanp42.net,root=fspsanp42.net,nosuid ===== fcvCIS01 /vol/ARDW -sec=sys,rw /vol/ARDW -sec=sys,rw /vol/ARDW -sec=sys,rw,nosuid /vol/ARDW -sec=sys,rw... (2 Replies)
Discussion started by: linux_scrub
2 Replies

2. Shell Programming and Scripting

Include information in certain columns using grep and awk

HI all, I have data in a file that looks like this: 1 HOW _ NNP NNP _ 3 nn _ _ 2 DRUGS _ NNP NNP _ 3 nn _ _ 3 ACT _ NNP NNP _ 0 null _ _ 4 : _ ... (3 Replies)
Discussion started by: owwow14
3 Replies

3. Shell Programming and Scripting

Remove lines with unique information in indicated columns

Hi, I have the 3-column, tab-separated following data: dot is-big 2 dot is-round 3 dot is-gray 4 cat is-big 3 hot in-summer 5 I want to remove all of those lines in which the values of Columns 1 and 2 are identical. In this way, the results would be as follows: dot is-big 2 cat... (4 Replies)
Discussion started by: owwow14
4 Replies

4. Shell Programming and Scripting

Parsing text file and feeding it into an executable

Hello, everyone. I am working wtihin AIX 5.3, and I need to do the following: read the each line of file BASE.txt do XK {line contents} if XK's output begins with "BASE", then append line contents to file "output.txt" continue until end of file Here is what I tried(unsuccessfuly): ... (4 Replies)
Discussion started by: Mordaris
4 Replies

5. Shell Programming and Scripting

Grep'ing information from a log file on SUN OS 5

Hi Guys, I'm trying to write an script that will be launched by a user. The script will look at a log file and check for alerts with the date (supplied by user) and a machine's hostname (also supplied by the user). I'm trying to get the output formatted just like the log file. The logfile looks... (5 Replies)
Discussion started by: illgetit
5 Replies

6. Shell Programming and Scripting

Extract various information from a log file

Hye ShamRock If you can help me with this difficult task for me then it will save my day Logs : ================================================================================================================== ... (4 Replies)
Discussion started by: SilvesterJ
4 Replies

7. Shell Programming and Scripting

extract information from a log file (last days)

I'm still new to bash script , I have a log file and I want to extract the items within the last 5 days . and also within the last 10 hours the log file is like this : it has 14000 items started from march 2002 to january 2003 awk '{print $4}' < *.log |uniq -c|sort -g|tail -10 but... (14 Replies)
Discussion started by: matarsak
14 Replies

8. Shell Programming and Scripting

Extract information from Log file formatted

Good evening! Trying to make a shell script to parse log file and show only required information. log file has 44 fields and alot of lines, each columns separated by ":". log file is like: first_1:3:4:5:6:1:3:4:5:something:notinterested second_2:3:4:3:4:2 first_1:3:4:6:6:7:8 I am interested... (3 Replies)
Discussion started by: dummie55
3 Replies

9. UNIX for Dummies Questions & Answers

Routing a verbose information to a log file

Hi All, In the script mentioned below uses a sftp command to login to the remote host and pull the files from the remote server. we want to log this inf to a log file . But it is not happening, the verbose information is just displayed on the screen but not to the log file when I run this... (1 Reply)
Discussion started by: raghuveer2008
1 Replies

10. Shell Programming and Scripting

Perl: adding columns in CSV file with information in each

Hi Wise UNIX Crew, I want to add 3 different columns to the file in which: 1. The first new column pulls in today's date and time 2. Second column one has a '0' 3. Third column has the word 'ANY' going down the column If my file content is as follows: "7","a","abc",123"... (1 Reply)
Discussion started by: dolo21taf
1 Replies
Login or Register to Ask a Question