speeding up bash script with "while read line" Post: 302526461

Sponsored Content

Top Forums Shell Programming and Scripting speeding up bash script with "while read line" Post 302526461 by Corona688 on Tuesday 31st of May 2011 02:53:13 PM

05-31-2011

Registered User

Quote:

Originally Posted by origamisven

Hello everybody,
I'm still slowly treading my way into bash scripting (without any prior programming experience) and hence my code is mostly what some might call "creative" if they meant well Smilie

I have created a script that serves its purpose but it does so very slowly, since it needs to work its way through ~1million lines of text file input and does so with a "while read line" loop, which slows down the process terribly.

It's not the read that's slow, it's "echo stuff | awk". You do that twice, so for a file with a million lines, you're running two million separate instances of awk! awk is a full-fledged language in its own right which you're loading, running, and quitting each time you use it. It's capable of processing more than one line, which is the efficient way to use it -- instead of 99% time spent loading/quitting, most time will be spent actually processing. You might as well do it all in awk.

You also have lots of useless use of backticks. Why do var=`echo stuff` when you can just do var=stuff ?

Incidentally, the shell can split variables by itself. You could do while read V1 V2 V3 V4 V5 V6 V7 to get rid of that first awk.

Code:

awk '
# The BEGIN block gets run once, before any lines are read
BEGIN { x=0 MOLprev=0 }
# This block gets run once per line
{
  if ((MOLprev == 9999) && (MOLprev != $5))
  {
    x++;
    SCALE=x*10000;
  }

  if (MOLprev > $5) 
            printf("%4s%7s%5s%4s%6s%12s%8s%8s\n",$1,$2,$3,$4,$5+SCALE,$6,$7,$8)
  else 
            printf("%4s%7s%5s%4s%6s%12s%8s%8s\n",$1,$2,$3,$4,$5,$6,$7,$8); 

  MOLprev=MOLcur;
}' < inputfile > outputfile

This User Gave Thanks to Corona688 For This Post:

Corona688

View Public Profile for Corona688

Visit Corona688's homepage!

Find all posts by Corona688

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha

2. Shell Programming and Scripting

script to read a line with spaces bet " " and write to a file

Hi, I need a command in UNIX KSH below is the description... MAPPING DESCRIPTION ="Test Mapping for the calid inputs" ISVALID ="YES" NAME ="m_test_xml" OBJECTVERSION ="1" VERSIONNUMBER ="1" unix ksh command to read the DESCRIPTION and write to a file Test Mapping for the calid inputs...

3. Shell Programming and Scripting

read -p "prompt text" foo say "read: bad option(s)" in Bourne-Shell

Hallo, i need a Prompting read in my script: read -p "Enter your command: " command But i always get this Error: -p: is not an identifier When I run these in c-shell i get this error /usr/bin/read: read: bad option(s) How can I use a Prompt in the read command?

4. Shell Programming and Scripting

read -n1 -r -p "Press..." key / produces error in bash shell script

Hello! Sorry, for my not so perfect english! I want to stop bash shell script execution until any key is pressed. This line in a bash shell script read -n1 -r -p "Press any key to continue..." key produces this error When I run this from the command line usera@lynx:~$ read...

5. Shell Programming and Scripting

Simplify Bash Script Using "sed" Or "awk"

Input file: 2 aux003.net3.com error12 6 awn0117.net1.com error13 84 aux008 error14 29 aux001.ha.ux.isd.com error12 209 aux002.vm.ux.isd.com error34 21 alx0027.vm.net2.com error12 227 dux001.net5.com error123 22 us008.dot.net2.com error121 13 us009.net2.com error129Expected Output: 2...

6. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone...

7. UNIX for Dummies Questions & Answers

"Help with bash script" - "License Server and Patch Updates"

Hi All, I'm completely new to bash scripting and still learning my way through albeit vey slowly. I need to know where to insert my server names', my ip address numbers through out the script alas to no avail. I'm also searching on how to save .sh (bash shell) script properly....

8. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing �To� e-mail address and column 3 contains �cc� e-mail address to include with same email. Sample input file, email.txt Below is an sample code where...

9. Shell Programming and Scripting

Failure: if grep "$Var" "$line" inside while read line loop

Hi everybody, I am new at Unix/Bourne shell scripting and with my youngest experiences, I will not become very old with it :o My code: #!/bin/sh set -e set -u export IFS= optl="Optl" LOCSTORCLI="/opt/lsi/storcli/storcli" ($LOCSTORCLI /c0 /vall show | grep RAID | cut -d " "...

10. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing...

LEARN ABOUT DEBIAN

igawk

IGAWK(1)							 Utility Commands							  IGAWK(1)

NAME

       igawk - gawk with include files

SYNOPSIS

       igawk [ all gawk options ] -f program-file [ -- ] file ...
       igawk [ all gawk options ] [ -- ] program-text file ...

DESCRIPTION

       Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).

       AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like

	      @include getopt.awk

       in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.

OPTIONS

       See gawk(1) for a full description of the AWK language and the options that gawk supports.

EXAMPLES

       cat << EOF > test.awk
       @include getopt.awk

       BEGIN {
	    while (getopt(ARGC, ARGV, "am:q") != -1)
		 ...
       }
       EOF

       igawk -f test.awk

SEE ALSO

       gawk(1)

       Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.

AUTHOR

       Arnold Robbins (arnold@skeeve.com).

Free Software Foundation					    Nov 3 1999								  IGAWK(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Discussion started by: Lokesha

2. Shell Programming and Scripting

script to read a line with spaces bet " " and write to a file

Discussion started by: perlamohan

3. Shell Programming and Scripting

read -p "prompt text" foo say "read: bad option(s)" in Bourne-Shell

Discussion started by: wiseguy

4. Shell Programming and Scripting

read -n1 -r -p "Press..." key / produces error in bash shell script

Discussion started by: linuxinho