Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Reading string and adding the values in the final logfile

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Reading string and adding the values in the final logfile
# 1  
Old 11-13-2012
Reading string and adding the values in the final logfile
Dear all,
I have small script which seems to be working but seems to have some bug.
It suppose to read commonTxt and then print the noOfLines in outputFile.
It is working for most of the txt but unable to add some of the variables values.
Can somebody please spend looking at the thread and reply.

Thanks in advance,

Script is following
Code:
#!/bin/bash                                                                                                                                           
date
FileName=DataFileName
LINES4LOG=38
TEXT=Procss

MakeFinalLog() {
    echo '=====Making Final Log ====='
    awk '{if($0~text){p=lines;next}}                                                                                                                 
                                                                                                                                                      
p>0{                                                                                                                                                  
       split($0,arr,"=")                                                                                                                              
       if(!h[arr[1]"HDR"])h[arr[1]"HDR"]=arr[1]                                                                                                       
       a[arr[1]]=a[arr[1]]" "$NF                                                                                                                      
       if(j<lines) b[++j]=arr[1]                                                                                                                      
       sum[arr[1]]+=$NF                                                                                                                               
       p--                                                                                                                                            
}                                                                                                                                                     
END{                                                                                                                                                  
 for(i=1;i<=j;i++)                                                                                                                                    
     print h[b[i]"HDR"]"="a[b[i]]" "sum[b[i]]                                                                                                         
}' lines=$LINES4LOG text="$TEXT" *.list > $FileName"_log.txt"
echo '++++++ Done ++++++  '$FileName"_log.txt"
}

if [ "$1" = "output" ]; then
    SetEnv
    MakeFinalLog
    exit 0
fi


And here is the OutPutFile (DataFileName_log.txt), and as you can see the text in red is not being added up, I am not sure why.
Code:
DataFileName_log.txtSample Count  = 0 0 0
nPU weighted   = 441063 441530 882593
Pass vtx trk   = 442356 442358 884714
GenLevel       = 0 0 0
Pass   HLT     = 442356 442358 884714
   eff = inf 0.00493042 0.812471 inf 0.00483545 0.804582 1.62682
          = === === === === 0
 Pass Spike cut       = 649724 648483 1298207
 Pass Eta cut         = 623956 622832 1246788
 Pass Pt cut          = 185356 184933 370289
Pass EleID     = 60081 59976 120057
Passed DiLep   = 2181 2139 4320
   eff = inf 0.00493042 0.812471 inf 0.00483545 0.804582 1.62682
Pass Z mass    = 1865 1813 3678
 Fail GenEle match = 117 115 232
Passed Zs      = 1772 1721 3493
   eff = inf 0.00493042 0.812471 inf 0.00483545 0.804582 1.62682
Passed ZsW     = 8.10845e+06 8.03766e+06 16146110
Passed ZsW2    = 0 0 0
Sum LepID      = 7.99145e+07 7.76145e+07 157529000
Sum Pileup     = 7.95332e+07 7.88391e+07 158372300
          = === === === === 0
Spike                                      : 2089= 2089 2089
pt                                          : 1281= 1281 1281
pt, eta                                     : 1169= 1169 1169
dR                                          : 1080= 1080 1080
EleVeto                                     : 1029= 1029 1029
EleVeto, HoE                                : 752= 752 752
EleVeto, HoE, sie                           : 197= 197 197
EleVeto, HoE, sie, ChargdHad                : 10= 10 10
EleVeto, HoE, sie, ChargdHad, NeuHad        : 9= 9 9 18
EleVeto, HoE, sie, ChargdHad, NeuHad, phopf : 8= 8 8 16
----------------------------------------------------------= ---------------------------------------------------------- ---------------------------------------------------------- 0
PhoID pass events = 8 8 16
Passed events  = 7 8 15
  eff = 0.00395034 0.00464846 0.0085988
Passed PU evt  = 7.40527 6.24361 13.6489
Passed Yields  = 0.75497 0.636537 1.39151

Here are the two list files that I m trying to read
File1.list
Code:
Procss: TTBb_0
Sample Count  = 0
nPU weighted   = 441063
Pass vtx trk   = 442356
GenLevel       = 0
Pass   HLT     = 442356
   eff = inf
          =====Electron Cut Efficiency ===
 Pass Spike cut       = 649724
 Pass Eta cut         = 623956
 Pass Pt cut          = 185356
Pass EleID     = 60081
Passed DiLep   = 2181
   eff = 0.00493042
Pass Z mass    = 1865
 Fail GenEle match = 117
Passed Zs      = 1772
   eff = 0.812471
Passed ZsW     = 8.10845e+06
Passed ZsW2    = 0
Sum LepID      = 7.99145e+07
Sum Pileup     = 7.95332e+07
          =====Pho Cut Efficiency ===
Spike                                      : 2089
pt                                          : 1281
pt, eta                                     : 1169
dR                                          : 1080
EleVeto                                     : 1029
EleVeto, HoE                                : 752
EleVeto, HoE, sie                           : 197
EleVeto, HoE, sie, ChargdHad                : 10
EleVeto, HoE, sie, ChargdHad, NeuHad        : 9
EleVeto, HoE, sie, ChargdHad, NeuHad, phopf : 8
----------------------------------------------------------
PhoID pass events = 8
Passed events  = 7
  eff = 0.00395034
Passed PU evt  = 7.40527
Passed Yields  = 0.75497


File2.list
Code:
Procss: TTBb_0
Sample Count  = 0
nPU weighted   = 441530
Pass vtx trk   = 442358
GenLevel       = 0
Pass   HLT     = 442358
   eff = inf
          =====Electron Cut Efficiency ===
 Pass Spike cut       = 648483
 Pass Eta cut         = 622832
 Pass Pt cut          = 184933
Pass EleID     = 59976
Passed DiLep   = 2139
   eff = 0.00483545
Pass Z mass    = 1813
 Fail GenEle match = 115
Passed Zs      = 1721
   eff = 0.804582
Passed ZsW     = 8.03766e+06
Passed ZsW2    = 0
Sum LepID      = 7.76145e+07
Sum Pileup     = 7.88391e+07
          =====Pho Cut Efficiency ===
Spike                                      : 2077
pt                                          : 1304
pt, eta                                     : 1198
dR                                          : 1095
EleVeto                                     : 1038
EleVeto, HoE                                : 729
EleVeto, HoE, sie                           : 199
EleVeto, HoE, sie, ChargdHad                : 12
EleVeto, HoE, sie, ChargdHad, NeuHad        : 9
EleVeto, HoE, sie, ChargdHad, NeuHad, phopf : 8
----------------------------------------------------------
PhoID pass events = 8
Passed events  = 8
  eff = 0.00464846
Passed PU evt  = 6.24361
Passed Yields  = 0.636537

# 2  
Old 11-13-2012
Hope i got your requirement right..Smilie

Assuming some points here..

Code:
awk 'NR==FNR && $NF ~ /[0-9]/{X[NR]=$NF;next}{if(X[FNR]){$NF=$NF" "X[FNR]" "$NF+X[FNR]}}1' file1 file2


pamu
# 3  
Old 11-13-2012
Quote:
Originally Posted by pamu
Hope i got your requirement right..Smilie

Assuming some points here..

Code:
awk 'NR==FNR && $NF ~ /[0-9]/{X[NR]=$NF;next}{if(X[FNR]){$NF=$NF" "X[FNR]" "$NF+X[FNR]}}1' file1 file2


pamu
Hi Pamu,
Thanks for looking into it. There is still problem,
1.) it does work only for 2 files, whereas I need to run on some 1000 files.
2.) The content I showed for file1.list and file2.list was part of the file. The whole file has lot of trash. If there could be choice of taking some commonTxt and then further noOfLines options, would be great.


I heard, it is some festival in India. Hope you enjoying.

Greetings,
emily,
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Logfile monitoring with logfile replacement

Bonjour, I've wrote a script to monitor a logfile in realtime. It is working almost perfeclty except for two things. The script use the following technique : tail -fn0 $logfile | \ while read line ; do ... some stuff done First one, I'd like a way to end the monitoring script if a... (3 Replies)
Discussion started by: Warluck
3 Replies

2. Shell Programming and Scripting

Adding filename and line number from multiple files to final file

Hi all, I have 20 files (file001.txt upto file020.txt) and I want to read them from 3rd line upto end of file (line 1002). But in the final file they should appear to start from line 1. I need following kind of output in a single file: Filename Line number 2ndcolumn 4thcolumn I... (14 Replies)
Discussion started by: bioinfo
14 Replies

3. Shell Programming and Scripting

Adding of two column values

Hi cat /tmp/xx.txt 1 4 1 5 1 6 2 1 2 1 2 1 i want to add the values of 2nd column resepect to 1st column values..for 1 in 1st column i need sum of all the values in 2nd column ..pls tell me hw to do it?? (8 Replies)
Discussion started by: Aditya.Gurgaon
8 Replies

4. Shell Programming and Scripting

Logfile Reading

I am reading through a log file that has multiple entries: xx-xxxx-xxx-130111090001 <XML> ... ... </XML> ... ... ... xx-xxxx-xxx-130111100001 <XML> ... ... </XML> There are 2 parts: 1) Take the XML statement and parse out. I have that with a sed script sed -n '/<XML>/,/XML>/p'... (6 Replies)
Discussion started by: SivaAfi
6 Replies

5. Shell Programming and Scripting

Reading and removing lines from logfile

Hello everbody. I'm trying to write a ksh script to find errors inside a logfile. But the main problem I found is that the "Logfile.txt" will got new lines very often. The software generates about 1000 lines per hour. (yes, ~24000 lines/day) "Logfile.txt" (with just very few lines) #... (0 Replies)
Discussion started by: killuayoukai
0 Replies

6. Shell Programming and Scripting

looking for string in logfile

I have a shell script that used to look for a particular sting in the last line of a log file. Howeve this string now has moved to the 3rd or 4th line from bottom. Can anyone point me to how i easiest chage this one to look within the last frew lines (5 or so) for the particular string rather... (6 Replies)
Discussion started by: jjlinux
6 Replies

7. Shell Programming and Scripting

Adding text in final line

Dear Friends, I have a flat file where last line of it has word D$mhtt I want to add a space and back slash after it. Also wanna add -S "J" in the last line. Following example will make it clear. I have this in the last line of file D$mhtt I want D$mhtt \ -S "J" Please... (5 Replies)
Discussion started by: anushree.a
5 Replies

8. Shell Programming and Scripting

Adding values concatenating values

I have the following script in a shell # The start of a filename file=$(ls -tr $EMT*.dat | tail -1) # Select the latest file echo $file file_seq=$( < /u02/sct/banner/bandev2/xxxxxx/misc/EFTSQL.dat) echo $file_seq file2 = '$file_seq + 1' echo $file2 It is reading a file EFTSQL.dat... (3 Replies)
Discussion started by: rechever
3 Replies
Login or Register to Ask a Question