awk performance issue


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk performance issue
# 1  
Old 07-20-2012
awk performance issue

Hi,
I have the code below as

Code:
cat <filename> | tr '~' '\n' | sed '/^$/ d' | sed "s/*/|/g" > <filename>
awk -F\| -vt=`date +%m%d%y%H%M%S%s` '$1=="ST",$1=="SE"{if($1=="ST"){close(f);f="214_edifile_"t"" ++i} ; $1=$1; print>f}' OFS=\| <filename>

This script replaces some characters and extracts the records starting from ST to SE and writes that in a seperate file. It works fine for smaller file, but when trying input file of bigger size the script is giving only partial output.

Can you tell me what could be the reason, is there any size limit of input file in UNIX OS please suggest to resolve.
# 2  
Old 07-20-2012
The first line is using the same file as input and output. This lead to undefined behavior. Just use a different file as output and process that later one with awk.
This User Gave Thanks to jlliagre For This Post:
# 3  
Old 07-20-2012
You could probably take out the $1=$1 since the input and output field separators are the same.
This User Gave Thanks to Ygor For This Post:
# 4  
Old 07-20-2012
Both the approaches worked fine.

Thanks to you both !
# 5  
Old 07-23-2012
I have one more clarification, with the above commands I am getting the desired result. But at the end of last line in each file generated I get a extra space. When viewing the file in vi editor I dont see the space, but can see it in textpad when ftp'ng to windows machine.

I am not sure how this space is there, and is there a way to remove that.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

awk script performance issue

Hello All, I have the below excerpt of code in my shell script and it taking long time to complete, though it prints the output quickly. Is there a way to make it come out once it finds the first instance as the file size of 4.7 GB it could be going through all lines of the data file to find for... (3 Replies)
Discussion started by: Ariean
3 Replies

2. AIX

Performance issue

Hi, We have 2 lpars on p6 blade. One of the lpar is having 3 core cpu with 5gb memory running sybase as database. An EOD process takes 25 min. to complete. Now we have an lpar on P7 server with entitled cpu capacity of 2 with 16 Gb memory and sybase as database. The EOD process which takes... (17 Replies)
Discussion started by: vjm
17 Replies

3. UNIX for Dummies Questions & Answers

Performance issue

hi I am having a performance issue with the following requirement i have to create a permutation and combination on a set of three files such that each record in each file is picked and the output is redirected in a specific format but it is taking around 70 odd hours to prepare a combination... (7 Replies)
Discussion started by: mad_man12
7 Replies

4. Shell Programming and Scripting

Performance issue or something else?

Hi All, I have the following script which I use in Nagios to check the health of the applications, the problem with it is that the curl part ($TOTAL) does not return anything after running for 2-3 hrs, even though from command line the script runs fine but not from Nagios. There are 17... (1 Reply)
Discussion started by: jacki
1 Replies

5. Solaris

Performance issue

Hi Gurus, I am beginner in solaris and want to know what are the things we need to check for performance monitoring on our solairs OS. for DISK,CPU and MEMORY. Also how we do ipforwarding in slaris Many thanks for your help Pradeep P (4 Replies)
Discussion started by: ppandey21
4 Replies

6. Shell Programming and Scripting

performance issue using gzcat, awk and sort

hi all, I was able to do a script to gather a few files and sort them. here it is: #!/usr/bin/ksh ls *mainFile* |cut -c20-21 | sort > temp set -A line_array i=0 file_name='temp' while read file_line do line_array=${file_line} let i=${i}+1 (5 Replies)
Discussion started by: naoseionome
5 Replies

7. Shell Programming and Scripting

Performance issue with awk script.

Hi, The below awk script is taking about 1 hour to fetch just 11 records(columns). There are about 48000 records. The script file name is take_first_uniq.sh #!/bin/ksh if then while read line do first=`echo $line | awk -F"|" '{print $1$2$3}'` while read line2 do... (4 Replies)
Discussion started by: RRVARMA
4 Replies

8. Shell Programming and Scripting

performance issue

I want to read a file. is it good to use File I/O or shell script?? which one is the best option? (1 Reply)
Discussion started by: vishwaraj
1 Replies

9. UNIX for Advanced & Expert Users

performance issue

Hi, on a linux server I have the following : vmstat 2 10 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 4 0 675236 39836 206060 1617660 3 3 3 6 8 7 1 1 ... (1 Reply)
Discussion started by: big123456
1 Replies

10. UNIX for Advanced & Expert Users

Performance issue

Hello all, I just stuck up in an uncertain situation related to network performance... I am trying to access one of my remote client unix machine from a distant location.. The client machine is Ultra-5_10 , with SunOS 5.5.1 The ndd result ( hme1 )shows that the machine is hooked to a... (5 Replies)
Discussion started by: shibz
5 Replies
Login or Register to Ask a Question