Awk Performance Issues


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Awk Performance Issues
# 1  
Old 09-02-2009
Question Awk Performance Issues

Hi All,

I'm facing an issue in my awk script.
The script is processing a large text file having the details of a number of persons, each person's details being written from 100 to 250 tags as given below:
100 START|
101klklk|
...
245 opr|
246 55|
250 END|
100 START|
...
245 pp|
246 65|
250 END|
In the code it writes each individual person's info into a temporary file and then process it for customization

We increased the tags from 250 to 450 leaving almost 200 tags blank.
like from 245 tag to 250 where moved to 445 to 450.(tags from 245-444 will not be coming in the file for now)
But after doing this rearrangment awk script has performance issues of almost double the time. So did changing these tags/identifiers degraded performance?
# 2  
Old 09-03-2009
If I'm reading this right, you roughly doubled the length of the file you're processing, and now it takes roughly twice as long...

Sounds about right to me...

If you could post your script, or at least the core sections I'm sure someone could assist in some performance tuning...
# 3  
Old 09-04-2009
Hi ,

Thanks for the update.
We found out the issue, its not due to the tag rearrangment.
But due to the removal of the temporary files in the code like given below:
{if(system("test -r fact_consbill")==0){system("rm fact_cc")}}
{if(system("test -s fact_consbill")==0){system("rm fact_bon")}}

But if we interchange this code then the code is taking the same as the old one, so is there any impact on the test variable regarding the names?

Thanks
# 4  
Old 09-04-2009
instead of spawning a next shell with "test -r", use this function and check for the return value:
Code:
# Function to test for the existance of a file
#
function fileExists(file,   dummy,ret)
{
  ret=0;
  if ((getline dummy < file) > 0) {
  # file exists and can be read
      ret = 1;
      close(file);
  }
   return ret;
}

# 5  
Old 09-07-2009
Hi,
Changing the removal using getline certainly helped in performance.
Thanks a lottt Smilie

Thanks
P
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. What is on Your Mind?

Baiduspider and Forum Performance Issues

For years we blocked Baiduspider due to the fact their bots do not obey the robots.txt directive and can really hurt site performance when they unleash 100 bots on the site each pulling pages many times per second. Last year, I unblocked Baiduspider's IP addresses, and now the problem is back. ... (1 Reply)
Discussion started by: Neo
1 Replies

2. UNIX for Dummies Questions & Answers

Hard Disk Performance issues Suse 11 SP1

hi guys right now I have 6 Virtual Machines (VMs) running on Vmware ESXi 5.1 and attached to Storage SAN. All these run Suse Linux 11 SP1 x64. All of a sudden 1 of these VMs is running very slow making high CPU usage and I see al wait % kinda high 40-50%. Apparently since I don't own this... (5 Replies)
Discussion started by: karlochacon
5 Replies

3. AIX

AIX 6.1 Memory Performance issues

Good Day Everyone, Just wonder anyone has encounter AIX 6.1 Memory Performance issues ? What I have in my current scenario is we have 3 datastage servers (Segregate server and EE jobs - for those who know Datastage achitect) and 2 db servers(running HA to load balance 4 nodes partitions for... (3 Replies)
Discussion started by: ckwan
3 Replies

4. Solaris

Getcwd performance issues

Hello everyone, recently we have been experiencing performance issues with chmod. We managed to narrow it down to getcwd. The following folder exists: /Folder1/subfol1/subfol2/subfol3 cd /Folder1/subfol1/subfol2/subfol3 truss -D pwd 2>&1 | grep getcwd 0.0001... (4 Replies)
Discussion started by: KotekBury
4 Replies

5. Solaris

zfs send receive performance issues

I 'm trying to clone a zfs file system pool/u01 to a new file system called newpool/u01 using following commands zfs list zfs snapshot pool/u01@new zfs send pool/u01@new | zfs -F receive newpool/u01 Its a 100G file system snapshot and copied to same server on different pool and... (9 Replies)
Discussion started by: fugitive
9 Replies

6. AIX

Performance issues for LPAR with GPFS 3.4

Hi, We have GPFS 3.4 Installed on two AIX 6.1 Nodes. We have 3 GPFS Mount points: /abc01 4TB (Comprises of 14 x 300GB disks from XIV SAN) /abc02 4TB (Comprises of 14 x 300GB disks from XIV SAN) /abc03 1TB ((Comprises of Multiple 300GB disks from XIV SAN) Now these 40... (1 Reply)
Discussion started by: aixromeo
1 Replies

7. Solaris

Solaris 11 Express NAT performance issues

Hi all, I decided to replace my linux router/firewall with Solaris 11 express. This is a pppoe connection directly to my server...no router boxes. I got everything setup, but the performance is terrible on the NAT....really slow. A web page that loads on the server instantly will take... (3 Replies)
Discussion started by: vectox
3 Replies

8. Programming

performance issues of calling a function in if condition

Hi, I have written a program in C and have to test the return value of the functions. So the normal way of doin this wud b int rc rc=myfunction(input); if(rc=TRUE){ } else{ } But instead of doing this I have called the function in the if() condition. Does this have any... (2 Replies)
Discussion started by: sidmania
2 Replies

9. Solaris

raidctl performance issues

using the internal 2 drives mirror was created using raidctl on 100's of our servers . sometime when one drive fails we dont face any issue & we replace the drive with out any problem . but sometimes when one drive fails , system becomes unresponsive and doesnot allow us to login , the only way to... (1 Reply)
Discussion started by: skamal4u
1 Replies

10. Shell Programming and Scripting

shell script performance issues --Urgent

I need help in awk please help immediatly. This below function is taking lot of time Please help me to fine tune it so that it runs faster. The file count is around 3million records # Process Body processbody() { #set -x while read line do ... (18 Replies)
Discussion started by: icefish
18 Replies
Login or Register to Ask a Question