Good evening! Trying to make a shell script to parse log file and show only required information.
log file has 44 fields and alot of lines, each columns separated by ":".
log file is like:
I am interested only in 5 fields which are : $3,$6,$14,$37,$38
So what i need is to parse that log file and display on screen something like this:
Code:
echo "Group Jobs WallClock CPU TotalMem Memory"
first_1 2 4 6 7 8
second_2 1 3 4 8 3
The problem is that log file has group names repeated and what i have to do is just to read every line , pick unique group name and sum number of times its repeats on other lines so i get number of Jobs. Any help would be appreciated.
---------- Post updated at 06:33 PM ---------- Previous update was at 04:55 PM ----------
I think the point you thought not easy is, how to count the line number with same first field. This can definitely be programmed using awk. Notice awk is not just an executable program, but you can make complicated code with condition and variables. just like: if $1 == 'first_1' then u=$u+1... Please read awk manual.
I think the point you thought not easy is, how to count the line number with same first field. This can definitely be programmed using awk. Notice awk is not just an executable program, but you can make complicated code with condition and variables. just like: if $1 == 'first_1' then u=$u+1... Please read awk manual.
Ok but what if i have like 100 different names, like first_1,second_1,something_, etc. I really cant use conditions and variables.
---------- Post updated at 04:58 AM ---------- Previous update was at 04:37 AM ----------
Quote:
Originally Posted by rdcwayx
Code:
awk -F":" ' BEGIN{print "Group\tJobs\tWallClock\tCPU\tTotalMem\tMemory"}
{Jobs[$1]+=$3;WC[$1]+=$6;CPU[$1]+=$14;TM[$1]+=$37;Mem[$1]+=$38}
END{for ( i in Jobs) printf "%s\t%d\t%d\t%d\t%d\t%d\n",i, Jobs[i],WC[i],CPU[i],TM[i],Mem[i]} ' $1
Thank you, ill try it! also i was trying to do something like :
Code:
awk -F":" '
NF==44 {
# $3 - group
# $6 - jobs
# $14 - wallclock
# $37 - CPU
# $38 - TotalMem
#1024*$38/$37 - Memory
++gsum[$3]
++gcnt
jsum[$3]+=$14
jcnt+=$14
csum[$3]+=$37
ccnt+=$37
tmsum[$3]+=$38
tmcnt+=$38
}
END {
printf("Gruop\tJobs\tWallclock\tCPU\tTotalMem\tMemory\n")
for(i in jsum) {
printf("%-10s\t%.2f%\t%.2f%.2f\t.2f\t.2f\n", i, gsum[i]*100/gcnt, jsum[i]*100/jcnt,csum[i]*100/ccnt[i]*100,tmsum[i]*100/tmcnt,1024*tmsum[i]/csum[i])
}
}' $1
---------- Post updated at 07:41 AM ---------- Previous update was at 04:58 AM ----------
Code:
#$3 - Group
#$6 - Jobs
#$14 - Wallclock
#$37 - CPU
#$38 - TotalMem
#1024*$38/$37 - MedMem
echo ""
date +"DATA: %m-%d-%y"
echo ""
awk -F":" ' BEGIN{
print "Group\tN. Jobss\tWallClock(s)\tCPU(s)\tTotalMem(GB)\tMedMem(MB)"}
{Jobs[$1]+=$3;WC[$1]+=$6;CPU[$1]+=$14/1000;TM[$1]+=$37;Mem[$1]+=1024*$38/$37}
END{for ( i in Jobs) printf "%s\t%d\t%d\t%d\t%d\t%.2f\n",i, Jobs[i],WC[i],CPU[i],TM[i]/1024,Mem[i]} ' $1
This works but still not properly
---------- Post updated at 02:30 PM ---------- Previous update was at 07:41 AM ----------
Code:
#$3 - Grupo
#$6 - N.Tarefas
#$14 - Wallclock
#$37 - CPU
#$38 - MemTotal
#1024*$38/$37 - Mem #estamos dividir pq temos k apresentar em GB
echo ""
date +"DATA: %m-%d-%y"
echo ""
#awk -F":" permite ir ao ficheiro neste caso $1 ( ou seja ficheiro com nome que vem depois de chamar o script
# ./ge-stat.sh <accounting-20110226> e ler cada campo separado por ":"
#Jobs[$3]++ array com os nomes dos grupos, cada vez quando nome repete-se ele mete no arrray
#WC[$3]+=$6 - WallClock de cada grupo quando repete-se soma-se os valores do mesmo grupo. Esses valores
#vai buscar ao campo $6 do ficheiro accounting
#$CPU,$TM,$Mem - igual ao wallclock, vai somar os valores com mesmo group name.
awk -F":" ' BEGIN{
print "Grupo\tN. Tarefas\tWallClock(s)\tCPU(s)\tMemTotal(GB)\tMem(MB)"}
{Jobs[$3]++;WC[$3]+=$6;CPU[$3]+=$14;TM[$3]+=$37;Mem[$3]+=1024*$38/$37}
END{for ( i in Jobs) printf "%s\t%d\t%d\t%d\t%d\t%.2f\n",i, Jobs[i],WC[i],CPU[i],TM[i]/1024,Mem[i]} ' $1
now works properly, sorry for foreign language, just ignore it )
Also if accounting does have some intro lines which are not interested should use {getline;getline;}
Thx to all for help!
Hi all, I have a query that runs that outputs data in the following format -
01/09/12 11:43:40,ADMIN,4,77,Application Group Load: Name(TESTED) LoadId(5137-1-0-1XX-15343-15343) File(/dir/dir/File.T03.CI2.RYR.2012009.11433350806.ARD) InputSize(5344) OutputSize(1359) Rows(2) Time(1.9960)... (8 Replies)
In a particular directory, there can be 1000 files like below.
filename is job901.ksh
#!/bin/ksh
cront -x << EOJ
submit file=$PRODPATH/scripts/genReport.sh maxdelay=30
&node=xnode01
tname=job901
&pfile1=/prod/mldata/data/test1.dat
... (17 Replies)
Gents,
If is possible please help.
I have a big file (example attached) which contends exactly same value in column, but from column 2 to 6 these values are diff. I will like to compile for all records all columns like the example attached in .csv format (output.rar ).. The last column in the... (11 Replies)
Hello!
I need help :) I have a file like this:
AA BC FG
RF TT GH
DD FF HH
(a few number of rows and three columns) and I want to put the letters of each column in a variable step by step in order to give them as input in another script. So I would like to obtain:
for the 1° loop:... (11 Replies)
Hi, i have a file like this:
<Iteration>
<Iteration_iter-num>3</Iteration_iter-num>
<Iteration_query-ID>lcl|3_0</Iteration_query-ID>
<Iteration_query-def>G383C4U01EQA0A length=197</Iteration_query-def>
<Iteration_query-len>197</Iteration_query-len>
... (9 Replies)
Hye ShamRock
If you can help me with this difficult task for me then it will save my day
Logs :
==================================================================================================================
... (4 Replies)
I'm still new to bash script , I have a log file and I want to extract the items within the last 5 days . and also within the last 10 hours
the log file is like this : it has 14000 items started from march 2002 to january 2003
awk '{print $4}' < *.log |uniq -c|sort -g|tail -10
but... (14 Replies)
Hi to all,
I got this content/pattern from file http.log.20110808.gz
mail1 httpd: Account Notice: close igchung@abc.com 2011/8/7 7:37:36 0:00:03 0 0 1
mail1 httpd: Account Information: login sastria9@abc.com proxy sid=gFp4DLm5HnU
mail1 httpd: Account Notice: close sastria9@abc.com... (16 Replies)
Hi,
Following is sample portion of the file;
<JDBCConnectionPool DriverName="oracle.jdbc.OracleDriver"
MaxCapacity="10" Name="MyApp_DevPool"
PasswordEncrypted="{3DES}7tXFH69Xg1c="
Properties="user=MYAPP_ADMIN" ShrinkingEnabled="false"
... (12 Replies)
Hello All,
I need some assistance to extract a piece of information from a huge file.
The file is like this one :
database information
ccccccccccccccccc
ccccccccccccccccc
ccccccccccccccccc
ccccccccccccccccc
os information
cccccccccccccccccc
cccccccccccccccccc... (2 Replies)