Visit Our UNIX and Linux User Community


append data to each line


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting append data to each line
# 1  
Old 10-19-2009
append data to each line

Hi guys,

I need to investigate a memory leak on a solaris server, so what I have done is pmap'd each process on the system with a script which tar'd the directory every hour in cron. Now I need to write a script to process the pmap data.

So what I have is about 100 directories
Code:
[root@localhost ~]# ll pmaptest/pmapdata2009101
pmapdata20091012122344/ pmapdata20091013080000/ pmapdata20091013230000/ pmapdata20091014140000/ pmapdata20091015050000/ pmapdata20091015200001/
pmapdata20091012180000/ pmapdata20091013090000/ pmapdata20091014000000/ pmapdata20091014150000/ pmapdata20091015060000/ pmapdata20091015210000/
pmapdata20091012190000/ pmapdata20091013100000/ pmapdata20091014010000/ pmapdata20091014160000/ pmapdata20091015070000/ pmapdata20091015220000/
.......

inside each directory is the output from the pmap on that process
Code:
[root@localhost pmapdata]# ll
total 1172
-rw-r--r-- 1 root root    21 2009-10-12 17:00 pmap0.txt
-rw-r--r-- 1 root root 21394 2009-10-12 17:00 pmap11746.txt
-rw-r--r-- 1 root root  3148 2009-10-12 17:00 pmap1294.txt
-rw-r--r-- 1 root root  8578 2009-10-12 17:00 pmap13176.txt
-rw-r--r-- 1 root root  9316 2009-10-12 17:00 pmap14005.txt
-rw-r--r-- 1 root root  2862 2009-10-12 17:00 pmap14006.txt
-rw-r--r-- 1 root root  2134 2009-10-12 17:00 pmap141.txt
-rw-r--r-- 1 root root 13857 2009-10-12 17:00 pmap14467.txt
-rw-r--r-- 1 root root 13959 2009-10-12 17:00 pmap14468.txt
-rw-r--r-- 1 root root 20210 2009-10-12 17:00 pmap14469.txt
-rw-r--r-- 1 root root 13003 2009-10-12 17:00 pmap14470.txt
-rw-r--r-- 1 root root  7652 2009-10-12 17:00 pmap16406.txt
-rw-r--r-- 1 root root  6032 2009-10-12 17:00 pmap16436.txt
-rw-r--r-- 1 root root  6032 2009-10-12 17:00 pmap16438.txt

the format of the filename is pmap<pid>.txt

inside each file looks like this
Code:
[root@localhost pmapdata]# cat pmap985.txt
 4423 1388 grep $pid
985:    /usr/local/sbin/xinetd -f /usr/local/etc/xinetd.conf
 Address  Kbytes     RSS    Anon  Locked Mode   Mapped File
08045000      12      12       4       - rw---    [ stack ]
08050000     212     140       -       - r-x--  xinetd
08094000       4       4       -       - rwx--  xinetd
08095000      44      40      20       - rwx--    [ heap ]
FED20000      64       4       -       - rwx--    [ anon ]
FED40000      64       8       -       - rwx--    [ anon ]
FED60000    1080    1080       -       - r-x--  libc.so.1
FEE7E000      32      32      20       - rw---  libc.so.1
FEE86000       8       4       -       - rw---  libc.so.1
FEE90000     516     516       -       - r-x--  libnsl.so.1
FEF21000      20      20       4       - rw---  libnsl.so.1
FEF26000      32       8       -       - rw---  libnsl.so.1
FEF30000      44      44       -       - r-x--  libsocket.so.1
FEF40000      24      12       4       - rwx--    [ anon ]
FEF4B000       4       4       -       - rw---  libsocket.so.1
FEF50000     268      84       -       - r-x--  libm.so.2
FEFA2000      16       8       -       - rwx--  libm.so.2
FEFC0000       4       4       -       - rwx--    [ anon ]
FEFC3000     160     160       -       - r-x--  ld.so.1
FEFF0000       4       4       4       - rwx--    [ anon ]
FEFF8000       4       4       -       - rwxs-    [ anon ]
FEFFB000       8       8       4       - rwx--  ld.so.1
FEFFD000       4       4       4       - rwx--  ld.so.1
-------- ------- ------- ------- -------
total Kb    2628    2204      64       -

What I am interested in is the anon for each pmap is the total at the bottom for the Anon column. I want to output this data in csv format for use inside an excel spreadsheet where I can graph the data.

So the output format need to be
Code:
pid,anonmemtotal,anonmemtotal,anonmemtotal,anonmemtotal
pid,anonmemtotal,anonmemtotal,anonmemtotal,anonmemtotal
pid,anonmemtotal,anonmemtotal,anonmemtotal,anonmemtotal

etc

I have started writing a script to do that which is below, but I cant figure out how I will get the pid and mem total appended to the same line each time?
Code:
#!/bin/bash
ls -l /root/pmaptest/ | awk '{print $8}' > /root/pmapdirs.txt
for dir in `cat /root/pmapdirs.txt`
do
cat /root/pmaptest/pmapdata$dir/var/tmp/pmapdata/psefo.txt | awk '{print $1}' > /root/pmaptest/pmapdata$dir/var/tmp/pmapdata/pids.txt
for pid in `cat /root/pmaptest/pmapdata$dir/var/tmp/pmapdata/pids.txt`
do
tail -1 /root/pmaptest/pmapdata$dir/var/tmp/pmapdata/pmap$pid.txt  | awk {'print $5'}
done
done


Last edited by pludi; 10-19-2009 at 06:07 AM.. Reason: code tags, please...
# 2  
Old 10-20-2009
Code:
for i in `find /root/pmaptest -type f -name pmap*.txt`
do
  PID=`echo "$i" |awk -F/ '{print $NF}' |sed -e 's/^pmap//' -e 's/\.txt//' `
  anonmemtotal=`tail -1 $i |awk '{print $5}' `
  echo $PID "," $anonmemtotal
done

you can create another script (or add one more function) to merge anonmemtotal into same PID.
# 3  
Old 10-20-2009
If I understand what you are looking to do, this awk script will take a list of pmap files of the form you presented and print a csv line with the pid and anon values found in all the pmap files. The anon values are in the order of the pmap files given on the command line.


(
Code:
FNR == 1) {

        getline   # skip to the pid
        this_pid = strtonum($1)
        if( this_pid in pid_list)
                pid_list[this_pid] = pid_list[this_pid] ","     # add a comma
        else
                pid_list[this_pid] = "%d,"              # set the format string
        next
}
/^total/ {              # look for total line

        pid_list[this_pid] = pid_list[this_pid] $5      # add anon total
}

END {
        for( i in pid_list )
                printf pid_list[i] "\n" , i
}

To use it, put the above code in a file (pid.awk in my example).
Then use a command of this form

Code:
awk -f pid.awk <list_of_pmap_file>

The <list_of_pmap_file> should be a wildcard spec or back tick command to produce a list of files in the order you want them processed.

Previous Thread | Next Thread
Test Your Knowledge in Computers #418
Difficulty: Easy
'JavaScript' is a trademark of Oracle Corporation in the United States.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove new line starting with a numeric value and append it to the previous line

Hi, i have a file with multiple entries. After some tests with sed i managed to get the file output as follows: lsn=X-LINK-IN0,apc=661:0,state=avail,avail/links=1/1, 00,2110597,2094790,0,81,529,75649011,56435363, lsn=TM1ITP1-AM1ITP1-LS,apc=500:0,state=avail,avail/links=1/1,... (5 Replies)
Discussion started by: nms
5 Replies

2. Shell Programming and Scripting

To append new data at the end of each line based on substring of last column

Hi guys, I need to append new data at the end of each line of the files. This new data is based on substring (3rd fields) of last column. Input file xxx.csv: U1234|1-5X|orange|1-5X|Act|1-5X|0.1 /sac/orange 12345 0 U5678|1-7X|grape|1-7X|Act|1-7X|0.1 /sac/grape 5678 0... (5 Replies)
Discussion started by: null7
5 Replies

3. Shell Programming and Scripting

Append Next line with current Line bassed on condition

Hi, I have an XML file and I am tring to extract some data form it, after lot of data cleaning process, I ended up with an issue, and need your urgent support. my current input data in below format: <Node>xxxxxx <Node>yyyyy</Node> <Node>zzzzzz <Node>12345</node> I need... (9 Replies)
Discussion started by: rramkrishnas
9 Replies

4. UNIX for Dummies Questions & Answers

How to remove fields space and append next line to previous line.?

awk 'BEGIN{FS = "«"} NR == 1 {p = $0; next} NF > 1 {print p; p = $0} NF <= 1 {p = (p " " $0)} END {print p}' input.txt > output.txt This is what the input data file looks like with broken lines Code: 29863 «890000000 «543209911 «CHNGOHG «000000001 «055 ... (4 Replies)
Discussion started by: cumeh1624
4 Replies

5. Shell Programming and Scripting

find a certain line and append text to the end of the line

After I create printer queues in AIX, I have to append a filter file location within that printers custom file. within lets say test_queue.txt I need to find the row that starts with :699 and then I need to append on the end the string /usr/local/bin/k_portrait.sh. Now I've gotten the sed... (2 Replies)
Discussion started by: peachclift
2 Replies

6. Shell Programming and Scripting

Append next line to previous line when one pattern not found

Hi, I need help for below scenario.I have a flat file which is having records seperated by delimiters which will represent each record for oracle table.My Control file will consider each line as one record for that table. Some of the lines are aligned in two/three lines so that records are... (4 Replies)
Discussion started by: kannansr621
4 Replies

7. Shell Programming and Scripting

Match data based on two fields, and append to a line

I need to write a program to do something like a 'vlookup' in excel. I want to match data from file2 based on two fields (where both match) in file1, and for matching lines, add the data from two of the fields from file2 to file1. If anyone knows something in perl or awk that can do this, I'd be... (20 Replies)
Discussion started by: jamessmith01
20 Replies

8. Programming

Append data to smallint data in informix4gl?

Hi, I have an smallint variable, say "a", i would like to prefix it with "0" in certain conditions. Is it possible to achieve that with this datatype? For instance, a=9 --> a=09 Many thanks (1 Reply)
Discussion started by: dvah
1 Replies

9. Shell Programming and Scripting

how to append line of of data to file

hai..i am new to unix..and i've currently learn shell script.. i have this small problem where i would like to save every data from log file into user directory if the data is equal to the name of the user.. i manage to do that with below script.. i would like to ask if there is any solutions so... (1 Reply)
Discussion started by: meggae
1 Replies

10. Shell Programming and Scripting

Joining lines in reverse. append line 1 to line 2.

Hi I have used many times the various methods to append two lines together in a file. This time I want to append the 1st line to the second and repeat for the complete file.... an example This is the file owns the big brown dog joe owns the small black dog jim What I want is ... (7 Replies)
Discussion started by: dwalley
7 Replies

Featured Tech Videos