Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

format evaperf output

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting format evaperf output
# 1  
Old 02-13-2011
format evaperf output
I want to split the blocks in the input.log file to one file per block type:

all blocks starting with
Code:
Time,Device,Path ID,Target ID,LUN,Product ID,Product Rev.,Ctlr,Serial,Hardware Ver.,Name,Node

should go to file1.txt

all blocks starting with
Code:
Time,Total Host Req/s,Total Host MB/s,Node

should go to file2.txt

and all blocks starting with
Code:
Time,CPU %,Data %,Ctlr,Serial,Node

should go to file3.txt

the input file is very big (around 1gb). I have tried to do it with below script, but it runs too slow. I expect that awk can do it much faster, but don't know how to make it.
Code:
in=n
rm file*.log
cat input.log | while read line
do
  case $line in
    "Time,Device,Path ID,Target ID,LUN,Product ID,Product Rev.,Ctlr,Serial,Hardware Ver.,Name,Node")
      logfile=file1.txt
      in=y;;
    "Time,Total Host Req/s,Total Host MB/s,Node")
      logfile=file2.txt
      in=y;;
    "Time,CPU %,Data %,Ctlr,Serial,Node")
      logfile=file3.txt
      in=y;;
    *) if [ "${in}" = "y" ]
       then
         if [ "${line}" = "" ]
         then
           in=n
         else
           echo $line >>${logfile}
         fi
       fi
    ;;
  esac
done

input.log:
Code:
EVAPerf Version from RPC Interface: 9.3.0
Time,Device,Path ID,Target ID,LUN,Product ID,Product Rev.,Ctlr,Serial,Hardware Ver.,Name,Node
12/Feb/2011 00:05:06,\\.\Scsi2:,0,4,0,HSV210,6220,L80B(B),PB5A7D1AATL80B,210,5000-1FE1-500B-0FB0,5000-1FE1-500B-0FB0
12/Feb/2011 00:05:06,\\.\Scsi2:,0,5,0,HSV210,6220,L809(A),PB5A7D1AATL809,210,5000-1FE1-500B-0FB0,5000-1FE1-500B-0FB0
 
Time,Total Host Req/s,Total Host MB/s,Node
12/Feb/2011 00:05:06,6777,291.79,5000-1FE1-500B-0FB0
 
Time,CPU %,Data %,Ctlr,Serial,Node
12/Feb/2011 00:05:06,35,34,L80B,PB5A7D1AATL80B,5000-1FE1-500B-0FB0
12/Feb/2011 00:05:06,18,18,L809,PB5A7D1AATL809,5000-1FE1-500B-0FB0
 
EVAPerf Version from RPC Interface: 9.3.0
Time,Device,Path ID,Target ID,LUN,Product ID,Product Rev.,Ctlr,Serial,Hardware Ver.,Name,Node
12/Feb/2011 00:06:05,\\.\Scsi2:,0,4,0,HSV210,6220,L80B(B),PB5A7D1AATL80B,210,5000-1FE1-500B-0FB0,5000-1FE1-500B-0FB0
12/Feb/2011 00:06:05,\\.\Scsi2:,0,5,0,HSV210,6220,L809(A),PB5A7D1AATL809,210,5000-1FE1-500B-0FB0,5000-1FE1-500B-0FB0
 
Time,Total Host Req/s,Total Host MB/s,Node
12/Feb/2011 00:06:06,7003,297.49,5000-1FE1-500B-0FB0
 
Time,CPU %,Data %,Ctlr,Serial,Node
12/Feb/2011 00:06:06,34,33,L80B,PB5A7D1AATL80B,5000-1FE1-500B-0FB0
12/Feb/2011 00:06:06,21,20,L809,PB5A7D1AATL809,5000-1FE1-500B-0FB0
 
.
.
.

Last edited by Scott; 02-13-2011 at 05:24 PM.. Reason: Please use code tags and indent code
# 2  
Old 02-13-2011
What Operating System and version do you have?
What Shell are you using?
What is the runtime of the script?
What is the runtime you require?
# 3  
Old 02-13-2011
What Operating System and version do you have?
=> HP-UX 11.31, but it could also be executed on SuSe linux Version11.
What Shell are you using?
=> bourne shell
What is the runtime of the script?
=> estimated 10 hours
What is the runtime you require?
=> 1 hour
# 4  
Old 02-13-2011
seems there is a space in the empty line.

Code:
awk 'BEGIN{RS=" \n";FS="\n"}
/^Time,Device,/ {print > "file1.txt"}
/^Time,Total Host/ {print > "file2.txt"}
/^Time,CPU %,/ {print > "file3.txt"} ' input.log

if there is no space in empty line,
Code:
awk 'BEGIN{RS="";FS="\n"}
/^Time,Device,/ {print > "file1.txt"}
/^Time,Total Host/ {print > "file2.txt"}
/^Time,CPU %,/ {print > "file3.txt"} ' input.log

# 5  
Old 02-14-2011
Thanks, I have tested it on the small input.log file in this thread and it does work.

But when I do run it on a real file, it fails with below error
Code:
$ awk 'BEGIN{RS="";FS="\n"}
> /^Time,Device,/ {print > "file1.txt"}
> /^Time,Total Host/ {print > "file2.txt"}
> /^Time,CPU %,/ {print > "file3.txt"} ' Sat_8K_OC.csv
awk: Input line EVAPerf Version from cannot be longer than 3,000 bytes.
The source line number is 1.

The file is 178 mb big and I have 6 files in total.

I forgot to mention that the block start (^Time line) should be dropped if possible.
Last edited by Franklin52; 02-14-2011 at 05:05 AM.. Reason: Please use code tags, thank you
# 6  
Old 02-14-2011
I think that "sed" will be much faster than the Shell version - even with three passes of the file.
I have assumed that each data block ends with a line containing only a single space character:

Code:
echo "Pass 1"
sed -n "/^Time\,Device\,/,\ \$ p" input.log > file1.txt
echo "Pass 2"
sed -n "/^Time\,Total Host Req\/s\,/,\ \$ p" input.log > file2.txt
echo "Pass 3"
sed -n "/^Time\,CPU %\,/,\ \$ p" input.log > file3.txt


If you have enough machine resources, the three passes could of course be executed concurrenly.
Last edited by methyl; 02-14-2011 at 08:34 AM.. Reason: filename wrong
# 7  
Old 02-14-2011
Hi

It is very fast, 15 sek for one 178 mb log file , but it does only list the first line for each block.

/fsmik
Last edited by fsmik; 02-14-2011 at 12:10 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Format output

Hello Team, I have the following details in a txt file (please note the spaces and tabs) C1 C2 C3 ------------------ --------------- ------------- abc, xyz 2 8 pqr 2 ... (9 Replies)
Discussion started by: H squared
9 Replies

2. Shell Programming and Scripting

Format with output

I have written some scripts that resulted in the table below (Column1 is ITEM, Column2 is Group, Column3 is Category and Column4 is Quantity) but I want the output in another format: Input: K123 X CATA 3 K123 Y CATA 4 K123 Z CATA 2 K123 X CATB 5 K123 Y CATB 2 K123 Z CATB 2 B65 M CATB... (7 Replies)
Discussion started by: aydj
7 Replies

3. Shell Programming and Scripting

Script to generate Excel file or to SQL output data to Excel format/tabular format

Hi , i am generating some data by firing sql query with connecting to the database by my solaris box. The below one should be the header line of my excel ,here its coming in separate row. TO_CHAR(C. CURR_EMP_NO ---------- --------------- LST_NM... (6 Replies)
Discussion started by: dani1234
6 Replies

4. UNIX for Dummies Questions & Answers

Format Output

Hello Learned People, Good evening. I have absolutely no idea as how to do this & I admit that I do not know anything in unix. I must learn this one of these days. Im a help desk incharge today & someone gave me this file to be formatted & I have a file like this. vi NewFile.txt 232... (8 Replies)
Discussion started by: RTAN
8 Replies

5. Shell Programming and Scripting

Dynamic output file generation using a input text file with predefined output format

Hi, I have two files , one file with data file with attributes that need to be sent to another file to generate a predefined format. Example: File.txt AP|{SSHA}VEEg42CNCghUnGhCVg== APVG3|{SSHA}XK|"password" AP3|{SSHA}XK|"This is test" .... etc --------- test.sh has... (1 Reply)
Discussion started by: hudson03051nh
1 Replies

6. UNIX for Advanced & Expert Users

format df -k output

i am running df -k command on aix machine. i got the output like this. i need to store into file and send that file into microsoft excel.i need to allign properly. Filesystem 512 blocks Free % Used Iused %Iused Mounted on /dev/hd4 262144 126488 52% ... (9 Replies)
Discussion started by: wintercoat
9 Replies

7. Shell Programming and Scripting

How to format output

Dear all, I have written a program which access database and displays the values returned by the query . There are 10 columns to be displayed in one row. But am ending with 5 lines displayed on 1st line and the next 5 in the 2nd line. Ex : 21608 10-20-2007 148 Al's Appliance... (7 Replies)
Discussion started by: uday542
7 Replies

8. Shell Programming and Scripting

capturing output from top and format output

Hi all, I'd like to capture the output from the 'top' command to monitor my CPU and Mem utilisation.Currently my command isecho date `top -b -n1 | grep -e Cpu -e Mem` I get the output in 3 separate lines.Tue Feb 24 15:00:03 Cpu(s): 3.4% us, 8.5% sy .. .. Mem: 1011480k total, 226928k used, ....... (4 Replies)
Discussion started by: new2ss
4 Replies

9. Shell Programming and Scripting

format output

I have a text file that I want a script to parse and grab only the relavent stuff. No idea where to start.. This is the text file.212 0.00000 ? -> (multicast) ETHER Type=2000 (Unknown), size = 344 bytes 0: 0100 0ccc cccc 000b 5f95 0fbe 014a aaaa ........_....J..... (7 Replies)
Discussion started by: Tornado
7 Replies

10. UNIX for Dummies Questions & Answers

ls output format

below is the output is ls -l -rw-r--r-- 1 tonyt staff 3212 Apr 17 1999 file1 -rw-r--r-- 1 tonyt staff 4541 Mar 3 21:08 file2 why the date format is not the same? (6 Replies)
Discussion started by: tonyt
6 Replies
Login or Register to Ask a Question