Sponsored Content
Top Forums Shell Programming and Scripting speeding up bash script with "while read line" Post 302526461 by Corona688 on Tuesday 31st of May 2011 02:53:13 PM
Old 05-31-2011
Quote:
Originally Posted by origamisven
Hello everybody,
I'm still slowly treading my way into bash scripting (without any prior programming experience) and hence my code is mostly what some might call "creative" if they meant well Smilie

I have created a script that serves its purpose but it does so very slowly, since it needs to work its way through ~1million lines of text file input and does so with a "while read line" loop, which slows down the process terribly.
It's not the read that's slow, it's "echo stuff | awk". You do that twice, so for a file with a million lines, you're running two million separate instances of awk! awk is a full-fledged language in its own right which you're loading, running, and quitting each time you use it. It's capable of processing more than one line, which is the efficient way to use it -- instead of 99% time spent loading/quitting, most time will be spent actually processing. You might as well do it all in awk.

You also have lots of useless use of backticks. Why do var=`echo stuff` when you can just do var=stuff ?

Incidentally, the shell can split variables by itself. You could do while read V1 V2 V3 V4 V5 V6 V7 to get rid of that first awk.

Code:
awk '
# The BEGIN block gets run once, before any lines are read
BEGIN { x=0 MOLprev=0 }
# This block gets run once per line
{
  if ((MOLprev == 9999) && (MOLprev != $5))
  {
    x++;
    SCALE=x*10000;
  }

  if (MOLprev > $5) 
            printf("%4s%7s%5s%4s%6s%12s%8s%8s\n",$1,$2,$3,$4,$5+SCALE,$6,$7,$8)
  else 
            printf("%4s%7s%5s%4s%6s%12s%8s%8s\n",$1,$2,$3,$4,$5,$6,$7,$8); 

  MOLprev=MOLcur;
}' < inputfile > outputfile

This User Gave Thanks to Corona688 For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies

2. Shell Programming and Scripting

script to read a line with spaces bet " " and write to a file

Hi, I need a command in UNIX KSH below is the description... MAPPING DESCRIPTION ="Test Mapping for the calid inputs" ISVALID ="YES" NAME ="m_test_xml" OBJECTVERSION ="1" VERSIONNUMBER ="1" unix ksh command to read the DESCRIPTION and write to a file Test Mapping for the calid inputs... (3 Replies)
Discussion started by: perlamohan
3 Replies

3. Shell Programming and Scripting

read -p "prompt text" foo say "read: bad option(s)" in Bourne-Shell

Hallo, i need a Prompting read in my script: read -p "Enter your command: " command But i always get this Error: -p: is not an identifier When I run these in c-shell i get this error /usr/bin/read: read: bad option(s) How can I use a Prompt in the read command? (9 Replies)
Discussion started by: wiseguy
9 Replies

4. Shell Programming and Scripting

read -n1 -r -p "Press..." key / produces error in bash shell script

Hello! Sorry, for my not so perfect english! I want to stop bash shell script execution until any key is pressed. This line in a bash shell script read -n1 -r -p "Press any key to continue..." key produces this error When I run this from the command line usera@lynx:~$ read... (4 Replies)
Discussion started by: linuxinho
4 Replies

5. Shell Programming and Scripting

Simplify Bash Script Using "sed" Or "awk"

Input file: 2 aux003.net3.com error12 6 awn0117.net1.com error13 84 aux008 error14 29 aux001.ha.ux.isd.com error12 209 aux002.vm.ux.isd.com error34 21 alx0027.vm.net2.com error12 227 dux001.net5.com error123 22 us008.dot.net2.com error121 13 us009.net2.com error129Expected Output: 2... (4 Replies)
Discussion started by: sQew
4 Replies

6. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

7. UNIX for Dummies Questions & Answers

"Help with bash script" - "License Server and Patch Updates"

Hi All, I'm completely new to bash scripting and still learning my way through albeit vey slowly. I need to know where to insert my server names', my ip address numbers through out the script alas to no avail. I'm also searching on how to save .sh (bash shell) script properly.... (25 Replies)
Discussion started by: profileuser
25 Replies

8. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

9. Shell Programming and Scripting

Failure: if grep "$Var" "$line" inside while read line loop

Hi everybody, I am new at Unix/Bourne shell scripting and with my youngest experiences, I will not become very old with it :o My code: #!/bin/sh set -e set -u export IFS= optl="Optl" LOCSTORCLI="/opt/lsi/storcli/storcli" ($LOCSTORCLI /c0 /vall show | grep RAID | cut -d " "... (5 Replies)
Discussion started by: Subsonic66
5 Replies

10. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies
RBASH(1)						      General Commands Manual							  RBASH(1)

NAME
rbash - restricted bash, see bash(1) RESTRICTED SHELL
If bash is started with the name rbash, or the -r option is supplied at invocation, the shell becomes restricted. A restricted shell is used to set up an environment more controlled than the standard shell. It behaves identically to bash with the exception that the follow- ing are disallowed or not performed: o changing directories with cd o setting or unsetting the values of SHELL, PATH, ENV, or BASH_ENV o specifying command names containing / o specifying a file name containing a / as an argument to the . builtin command o specifying a filename containing a slash as an argument to the -p option to the hash builtin command o importing function definitions from the shell environment at startup o parsing the value of SHELLOPTS from the shell environment at startup o redirecting output using the >, >|, <>, >&, &>, and >> redirection operators o using the exec builtin command to replace the shell with another command o adding or deleting builtin commands with the -f and -d options to the enable builtin command o using the enable builtin command to enable disabled shell builtins o specifying the -p option to the command builtin command o turning off restricted mode with set +r or set +o restricted. These restrictions are enforced after any startup files are read. When a command that is found to be a shell script is executed, rbash turns off any restrictions in the shell spawned to execute the script. SEE ALSO
bash(1) GNU Bash-4.0 2004 Apr 20 RBASH(1)
All times are GMT -4. The time now is 08:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy