Improve script - slow process with big files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Improve script - slow process with big files
# 1  
Old 01-26-2017
Improve script - slow process with big files

Gents,

Please can u help me to improve this script to be more faster, it works perfectly but for big files take a lot time to end the job..

I see the problem is in the step (while) and in this part the script takes a lot time..

Please if you can find a best way to do will be great.

Input file = 16.txt
output files = 16.ss01 16.xx01
rawsps = script

Attached. ( information ).

Thanks for your help.
# 2  
Old 01-26-2017
Hi jiam,

thanks for not inserting the files into this forum because it is too much.

Futhermore it's annoying for me needing to download anything, install an unpacking program prior to reading the files. I politely ask you if you please use a pasting service. Maybe this one:

New paste • Fedora Project Pastebin

What's regarding the script:

Even I'm not acquainted in the use of csh, these are my
recommendations:
  • You have a lot external awk-calls in your loop. That's one reason making your program very slow. Like this one:
Code:
        set lineinfo = `cat info_records.list|head -$i|tail -1`
        set tap = `echo $lineinfo|awk '{print $1;}'`
        set rec = `echo $lineinfo|awk '{print $2;}'`
        set lin = `echo $lineinfo|awk '{print $3;}'`
        set pnt = `echo $lineinfo|awk '{print $4;}'`
        set spx = `echo $lineinfo|awk '{print $5;}'`
        set spy = `echo $lineinfo|awk '{print $6;}'`
        set spz = `echo $lineinfo|awk '{print $7;}'`
        set tim = `echo $lineinfo|awk '{print $8;}'`
        set lfr = `echo $lineinfo|awk '{print $9;}'`
        set lto = `echo $lineinfo|awk '{print $10;}'`

If you create one awk-program it will be a lot faster.

Regards,
stomp

Last edited by rbatte1; 01-26-2017 at 06:01 AM..
This User Gave Thanks to stomp For This Post:
# 3  
Old 01-26-2017
Hello Stomp.

Thanks for your answer,

Can you help me to generate the awk program please
# 4  
Old 01-26-2017
To expand on what stomp already said, your script seems to read the input file n+1 times, once to detect the count of reports ( = n) and their respective location in it (one sed and one awk invocation), then using a shell loop to "extract" and analyse the respective single reports, invoking awk 13 times, and sed 4 times per loop (i.e. 130 awks and 40 seds for the sample file with 10 reports).

No surprise this will be somewhat lengthy on large files with many reports...
This User Gave Thanks to RudiC For This Post:
# 5  
Old 01-26-2017
Hi RudiC,

Sometimes I have files with more of 20000 lines in this case it takes long time to end the job.
Kindly, could you please help me to improve it.
Appreciate your help.
# 6  
Old 01-26-2017
The csh shell has many known issues and I would strongly recommending that you don't use it.

I've not read the whole thing, but assuming that you are working through the file, line by line and this design is actually reading the whole file each and every time round your loop, calling 10 processes to split up your line, how about this construct instead:-
Code:
while read tap rec lin pnt spx spy spz tim lfr lto unused
do
   whatever_you_need_here
done < info_records.list

Okay, so this is sh / ksh / bash, but it is far neater and has far lower overheads. I've not got a true csh available and the manual page I have uses all sorts of bash phrasing so it is only imitating some csh scripting so I can't really test anything I write in csh.

You might be better with this csh mangle instead of calling awk all over the place:-
Code:
foreach line ( "`cat input_file`" )
   set parsed = ($line)
   set tap = $parsed[1]
   set rec = $parsed[2]
   set lin = $parsed[3]
   :
   :
end

This reads the file once, and for each line it splits up the record to the variables you want without calling external commands (except the cat that I can't find a way to remove)

Overall though, it is worth the effort to convert to sh based scripts. I hope your code is not riddled with goto statements like I have had to decipher before. That poor programming can leave serious headaches in re-designing.


I hope that this helps,
Robin
These 2 Users Gave Thanks to rbatte1 For This Post:
# 7  
Old 01-26-2017
You might want to try this one. Due to the input file structure, it must be read twice - once to identify the respective reports, another time to extract the data and produce the output files.
As you can see below, the X output exactly matches your sample output. The S file doesn't as I don't understand your date/time function and thus can't replicate it. Nor more shell loops, no sed, and just two awk invocations, I'd guess it should save serious amounts of time. Please report back.

Code:
awk -F: '
BEGIN   {FMT = "%d %d %d %d %11.1f %11.1f %11.1f %s %010d %010d\n"
         for (n = split ("Tape_Nb:File_Nb:Line_Name:Point_Number:Cog_Easting:Cog_Northing:Cog_Elevation:Tb_GPS_Time", IX); n>0; n--) SRCH[IX[n]]
        }

$1 == "Observer_Report "        {if (flag)      printf FMT,     OUT[IX[1]], OUT[IX[2]], OUT[IX[3]], OUT[IX[4]],
                                                                OUT[IX[5]], OUT[IX[6]], OUT[IX[7]], OUT[IX[8]], from, to
                                 delete OUT
                                 from = NR
                                 flag = 1
                                }

                {gsub (/[       ]/, _)
                 to = NR
                }

$1 in SRCH      {OUT[$1] = $2
                 if ($1 ~ /Tb_GPS_Time/)        OUT[$1] = substr($2,2,16)
                }

END     {printf FMT,    OUT[IX[1]], OUT[IX[2]], OUT[IX[3]], OUT[IX[4]],
                        OUT[IX[5]], OUT[IX[6]], OUT[IX[7]], OUT[IX[8]], from, to
        }
' /tmp/16.txt |
awk -F[:-\(] '
BEGIN           {HD1 = "H26 5678901234567890123456789012345678901234567890123456789012345678901234567890"
                 HD2 = "H26      1         2         3         4         5         6         7          "
                }
NR == 1         {print HD1 RS HD2 > XFILE
                 print HD1 RS HD2 > SFILE
                 }

FNR == NR       {OR[NR] = $0
                 MX = NR
                 next
                }
FNR > NXTREP ||
FNR == 1        {n = split (OR[++OCNT], T, " ")
                 NXTREP = T[n] + 0
                 printf "S%10.2f%10.2f%3d1                     %9.1f%10.1f%6.1f%09d\n", T[3], T[4], 1, T[5], T[6], T[7], T[8] > SFILE
                }

                {sub (/^[       ]*/, _)
                 sub (/ *: */, ":")
                }


$1 ~ /^Live_Seis/       {DATA = 1
                         sub (/Live_Seis[^:]*:/, _)
                        }
/[^0-9:() -]/           {DATA = 0
                        }
DATA                    {printf "X%6d%8d11%10.2f%10.2f%1d%5d%5d1%10.2f%10.2f%10.2f1\n", T[1], T[2], T[3], T[4], 1, $4, $5, $1, $2, $3 > XFILE 
                        }
' XFILE="xfile" SFILE="sfile" - /tmp/16.txt

diff xfile /tmp/16.xx01   # no output from diff -> no difference!


Last edited by RudiC; 01-26-2017 at 04:30 PM.. Reason: typo
These 2 Users Gave Thanks to RudiC For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script search, improve performance with large files

Hello, For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies

2. Solaris

Rsync quite slow (using very little cpu): how to improve its speed?

I have "inherited" a OmniOS (illumos based) server. I noticed rsync is significantly slower in respect to my reference, FreeBSD 12-CURRENT, running on exactly same hardware. Using same hardware, same command with same source and target disks, OmniOS r151026 gives: test@omniosce:~# time... (11 Replies)
Discussion started by: priyadarshan
11 Replies

3. HP-UX

Script execution is very slow when trying to find all files and their owners on HP-UX box

Hi, I have a HP-UX server were I need to list all the files in the entire file system, their directory path, last modified date, owner and group. I do not need to search the file contents. I created the script given below and I am excluding directories and files of type tmp, temp and log. The... (4 Replies)
Discussion started by: Adyan Faruqi
4 Replies

4. UNIX for Dummies Questions & Answers

How do I slow down a process?

Hello, I've been searching for something that slows down a process for some time now. Slow down as in make time pass by slower. I have rarely turned to asking a forum in the past but at this point I've given up. For example: if I made a program that would print "Hello" in 5 seconds, I would use... (18 Replies)
Discussion started by: Nathan1
18 Replies

5. Shell Programming and Scripting

Very big text file - Too slow!

Hello everyone, suppose there is a very big text file (>800 mb) that each line contains an article from wikipedia. Each article begins with a tag (<..>) containing its url. Currently there are 10^6 articles in the file. I want to take random N articles, eliminate all non-alpharithmetic... (14 Replies)
Discussion started by: fedonMan
14 Replies

6. UNIX for Advanced & Expert Users

sed working slow on big files

HI Experts , I'm using the following code to remove spaces appearing at the end of the file. sed "s/*$//g" <filename> > <new_filename> mv <new_filename> <filename> this is working fine for volumes upto 20-25 GB. for the bigger files it is taking more time that it is required... (5 Replies)
Discussion started by: sumoka
5 Replies

7. Shell Programming and Scripting

egrep is very slow : How to improve performance

We have an egrep search in a while loop. egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new ${PICKUP_DIR}/new_update is 210 MB file In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other... (7 Replies)
Discussion started by: hidnana
7 Replies

8. AIX

How to send big files over slow network?

Hi, I am trying to send oracle archives over WAN and it is taking hell a lot of time. To reduce the time, I tried to gzip the files and send over to the other side. That seems to reduce the time. Does anybody have experienced this kind of problem and any possible ways to reduce the time. ... (1 Reply)
Discussion started by: giribt
1 Replies

9. Shell Programming and Scripting

bash script working for small size files but not for big size files.

Hi, I have one file stat. Stat file contents are as follows: for example. H50768020040913,00260100,507680,13,0000000643,0000000643,00000,0000 H50769520040808,00260100,507695,13,0000000000,0000000000,00000,0000 H50770620040611,00260100,507706,13,0000000000,0000000000,00000,0000 Now i... (1 Reply)
Discussion started by: davidpreml
1 Replies

10. UNIX for Advanced & Expert Users

looking for solution to improve process replicate files to remote loc.

looking for solution to replicate 1.5GB files to a remote location... Currently, this process looks like the following: move 1.5GB files into a staging area. compress files. rsync files to remote server. remove compressed files. I have performed some timings, and compress seems more... (5 Replies)
Discussion started by: mr_manny
5 Replies
Login or Register to Ask a Question