append data to each line

10-19-2009

Registered User

59, 0

Join Date: Oct 2008

Last Activity: 4 May 2011, 5:21 PM EDT

Posts: 59

Thanks Given: 0

Thanked 0 Times in 0 Posts

append data to each line

Hi guys,

I need to investigate a memory leak on a solaris server, so what I have done is pmap'd each process on the system with a script which tar'd the directory every hour in cron. Now I need to write a script to process the pmap data.

So what I have is about 100 directories

Code:

[root@localhost ~]# ll pmaptest/pmapdata2009101
pmapdata20091012122344/ pmapdata20091013080000/ pmapdata20091013230000/ pmapdata20091014140000/ pmapdata20091015050000/ pmapdata20091015200001/
pmapdata20091012180000/ pmapdata20091013090000/ pmapdata20091014000000/ pmapdata20091014150000/ pmapdata20091015060000/ pmapdata20091015210000/
pmapdata20091012190000/ pmapdata20091013100000/ pmapdata20091014010000/ pmapdata20091014160000/ pmapdata20091015070000/ pmapdata20091015220000/
.......

inside each directory is the output from the pmap on that process

Code:

[root@localhost pmapdata]# ll
total 1172
-rw-r--r-- 1 root root    21 2009-10-12 17:00 pmap0.txt
-rw-r--r-- 1 root root 21394 2009-10-12 17:00 pmap11746.txt
-rw-r--r-- 1 root root  3148 2009-10-12 17:00 pmap1294.txt
-rw-r--r-- 1 root root  8578 2009-10-12 17:00 pmap13176.txt
-rw-r--r-- 1 root root  9316 2009-10-12 17:00 pmap14005.txt
-rw-r--r-- 1 root root  2862 2009-10-12 17:00 pmap14006.txt
-rw-r--r-- 1 root root  2134 2009-10-12 17:00 pmap141.txt
-rw-r--r-- 1 root root 13857 2009-10-12 17:00 pmap14467.txt
-rw-r--r-- 1 root root 13959 2009-10-12 17:00 pmap14468.txt
-rw-r--r-- 1 root root 20210 2009-10-12 17:00 pmap14469.txt
-rw-r--r-- 1 root root 13003 2009-10-12 17:00 pmap14470.txt
-rw-r--r-- 1 root root  7652 2009-10-12 17:00 pmap16406.txt
-rw-r--r-- 1 root root  6032 2009-10-12 17:00 pmap16436.txt
-rw-r--r-- 1 root root  6032 2009-10-12 17:00 pmap16438.txt

the format of the filename is pmap<pid>.txt

inside each file looks like this

Code:

[root@localhost pmapdata]# cat pmap985.txt
 4423 1388 grep $pid
985:    /usr/local/sbin/xinetd -f /usr/local/etc/xinetd.conf
 Address  Kbytes     RSS    Anon  Locked Mode   Mapped File
08045000      12      12       4       - rw---    [ stack ]
08050000     212     140       -       - r-x--  xinetd
08094000       4       4       -       - rwx--  xinetd
08095000      44      40      20       - rwx--    [ heap ]
FED20000      64       4       -       - rwx--    [ anon ]
FED40000      64       8       -       - rwx--    [ anon ]
FED60000    1080    1080       -       - r-x--  libc.so.1
FEE7E000      32      32      20       - rw---  libc.so.1
FEE86000       8       4       -       - rw---  libc.so.1
FEE90000     516     516       -       - r-x--  libnsl.so.1
FEF21000      20      20       4       - rw---  libnsl.so.1
FEF26000      32       8       -       - rw---  libnsl.so.1
FEF30000      44      44       -       - r-x--  libsocket.so.1
FEF40000      24      12       4       - rwx--    [ anon ]
FEF4B000       4       4       -       - rw---  libsocket.so.1
FEF50000     268      84       -       - r-x--  libm.so.2
FEFA2000      16       8       -       - rwx--  libm.so.2
FEFC0000       4       4       -       - rwx--    [ anon ]
FEFC3000     160     160       -       - r-x--  ld.so.1
FEFF0000       4       4       4       - rwx--    [ anon ]
FEFF8000       4       4       -       - rwxs-    [ anon ]
FEFFB000       8       8       4       - rwx--  ld.so.1
FEFFD000       4       4       4       - rwx--  ld.so.1
-------- ------- ------- ------- -------
total Kb    2628    2204      64       -

What I am interested in is the anon for each pmap is the total at the bottom for the Anon column. I want to output this data in csv format for use inside an excel spreadsheet where I can graph the data.

So the output format need to be

Code:

pid,anonmemtotal,anonmemtotal,anonmemtotal,anonmemtotal
pid,anonmemtotal,anonmemtotal,anonmemtotal,anonmemtotal
pid,anonmemtotal,anonmemtotal,anonmemtotal,anonmemtotal

etc

I have started writing a script to do that which is below, but I cant figure out how I will get the pid and mem total appended to the same line each time?

Code:

#!/bin/bash
ls -l /root/pmaptest/ | awk '{print $8}' > /root/pmapdirs.txt
for dir in `cat /root/pmapdirs.txt`
do
cat /root/pmaptest/pmapdata$dir/var/tmp/pmapdata/psefo.txt | awk '{print $1}' > /root/pmaptest/pmapdata$dir/var/tmp/pmapdata/pids.txt
for pid in `cat /root/pmaptest/pmapdata$dir/var/tmp/pmapdata/pids.txt`
do
tail -1 /root/pmaptest/pmapdata$dir/var/tmp/pmapdata/pmap$pid.txt  | awk {'print $5'}
done
done

Last edited by pludi; 10-19-2009 at 06:07 AM.. Reason: code tags, please...

borderblaster

View Public Profile for borderblaster

Find all posts by borderblaster

10-20-2009

Registered User

2,759, 420

Join Date: Jun 2006

Last Activity: 13 September 2015, 8:58 PM EDT

Posts: 2,759

Thanks Given: 44

Thanked 420 Times in 408 Posts

Code:

for i in `find /root/pmaptest -type f -name pmap*.txt`
do
  PID=`echo "$i" |awk -F/ '{print $NF}' |sed -e 's/^pmap//' -e 's/\.txt//' `
  anonmemtotal=`tail -1 $i |awk '{print $5}' `
  echo $PID "," $anonmemtotal
done

you can create another script (or add one more function) to merge anonmemtotal into same PID.

rdcwayx

View Public Profile for rdcwayx

Find all posts by rdcwayx

10-20-2009

Registered User

156, 1

Join Date: Jul 2009

Last Activity: 7 March 2010, 1:01 AM EST

Posts: 156

Thanks Given: 0

Thanked 1 Time in 1 Post

If I understand what you are looking to do, this awk script will take a list of pmap files of the form you presented and print a csv line with the pid and anon values found in all the pmap files. The anon values are in the order of the pmap files given on the command line.

(

Code:

FNR == 1) {

        getline   # skip to the pid
        this_pid = strtonum($1)
        if( this_pid in pid_list)
                pid_list[this_pid] = pid_list[this_pid] ","     # add a comma
        else
                pid_list[this_pid] = "%d,"              # set the format string
        next
}
/^total/ {              # look for total line

        pid_list[this_pid] = pid_list[this_pid] $5      # add anon total
}

END {
        for( i in pid_list )
                printf pid_list[i] "\n" , i
}

To use it, put the above code in a file (pid.awk in my example).
Then use a command of this form

Code:

awk -f pid.awk <list_of_pmap_file>

The <list_of_pmap_file> should be a wildcard spec or back tick command to produce a list of files in the order you want them processed.

jp2542a

View Public Profile for jp2542a

Find all posts by jp2542a

Shell Programming and Scripting

append data to each line

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove new line starting with a numeric value and append it to the previous line

Discussion started by: nms

2. Shell Programming and Scripting

To append new data at the end of each line based on substring of last column

Discussion started by: null7

3. Shell Programming and Scripting

Append Next line with current Line bassed on condition

Discussion started by: rramkrishnas

4. UNIX for Dummies Questions & Answers

How to remove fields space and append next line to previous line.?

Discussion started by: cumeh1624

5. Shell Programming and Scripting

find a certain line and append text to the end of the line

Discussion started by: peachclift

6. Shell Programming and Scripting

Append next line to previous line when one pattern not found

Discussion started by: kannansr621

7. Shell Programming and Scripting

Match data based on two fields, and append to a line

Discussion started by: jamessmith01

8. Programming

Append data to smallint data in informix4gl?

Discussion started by: dvah

9. Shell Programming and Scripting

how to append line of of data to file

Discussion started by: meggae

10. Shell Programming and Scripting

Joining lines in reverse. append line 1 to line 2.

Discussion started by: dwalley