Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

On how to select the right tool for a given task

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting On how to select the right tool for a given task
# 15  
Old 10-20-2008
I agree with perderabo that any use of an external program is costly in terms of system calls. So in fact we are not talking about sed (or awk) but about the usage of any program external to the shell.

I doubt that any of us can write a meaningful shellscript without any calls to external programs. I even doubt that it is possible to write such a script without any calls to external programs inside loops.

And as an example for an operation on a variable which is probably easier (and faster) to do in an external program than with shell means: perform a rot13 encryption on the contents of a string variable. In external programs this is one call (here with tr, with sed or awk its equally simple):

print - $var | tr 'abcdefghijklmnopqrstuvwxyz' 'nopqrstuvwxyzabcdefghijklm'

The same in shell variable expansion??

bakunin
This User Gave Thanks to bakunin For This Post:
editha1000
# 16  
Old 10-20-2008
Quote:
Originally Posted by bakunin
I doubt that any of us can write a meaningful shellscript without any calls to external programs. I even doubt that it is possible to write such a script without any calls to external programs inside loops.

I frequently write scripts that do not call any external commands.
Quote:

And as an example for an operation on a variable which is probably easier (and faster) to do in an external program than with shell means: perform a rot13 encryption on the contents of a string variable. In external programs this is one call (here with tr, with sed or awk its equally simple):

print - $var | tr 'abcdefghijklmnopqrstuvwxyz' 'nopqrstuvwxyzabcdefghijklm'

The same in shell variable expansion??

Of course not; you are not doing something that can be done with parameter expansion.

But it can easily be done entirely within the shell, and for short strings (up to ~30 characters), I do that. For longer strings, it is faster to use tr.
# 17  
Old 10-20-2008
Quote:
Originally Posted by ghostdog74
i fully understand cfajohnson's point of view that "sed (or any external command) should not be used to chop a single string".
I am arguing that that's not true because
1) for some situations, using sed's regexp is easier, especially to a person versed in the art of regexp. in this situation, development time is more important than execution time.

For someone who knows parameter expansion, both development time and execution time will be saved by using it.
Quote:
2) processor time/resources are so cheap nowadays that its insignificant to even mention.

Real time is as significant as it ever was. Using external commands unnecessarily can turn a snappy script into a slug.
# 18  
Old 10-20-2008
Quote:
Originally Posted by bakunin
I doubt that any of us can write a meaningful shellscript without any calls to external programs. I even doubt that it is possible to write such a script without any calls to external programs inside loops.
So, datecalc is not meaningful? I'm hurt... Smilie

When I get some time I'll benchmark rot13 in pure ksh vs tr. My feeling is that ksh will be a win for any size string. I wonder if you realize how much work is done to execute a fork() and an exec(). And remember that ksh compiles the code of a loop and then executes the compiled code. It does not repeatedly rescan the source code.
# 19  
Old 10-20-2008
Quote:
Originally Posted by Perderabo
So, datecalc is not meaningful? I'm hurt... :(
Sorry! ;-)) Of course it is meaningful. I was thinking about the scripts i was writing and couldn't find a single one without any external program. You are correct in pointing out datecalc to me, because things like this (which are quite "abstracted" from concrete problems) i have put into external ksh functions and use these so frequently that i don't think about how they are constructed. I shouldn't have looked into my scripts but my functions. Anyway, i stand corrected and offer my apologies.

Quote:
When I get some time I'll benchmark rot13 in pure ksh vs tr.
I have done that and this was the real surprise for me. Here are the scripts i used (yes, i used ksh93. In AIX the default shell is ksh88 still, but i didn't want to reinvent the substring function of ksh93):

Code:
#!/bin/ksh93

exec 3>&1
exec 1>/dev/null
typeset -i iCnt=1

while [ $iCnt -le 1000 ] ; do
     var="abcde"
     print - $var | tr 'abcdefghijklmnopqrstuvwxyz' 'mnopqrstuvwxyzabcdefghijklm
'
     (( iCnt += 1 ))
done

exec 1>&3
exit 0

Code:
#!/bin/ksh93

exec 3>&1
exec 1>/dev/null
typeset -i iCnt=1

while [ $iCnt -le 1000 ] ; do
     var="abcde"
     typeset -i j=1
     while [ $j -le ${#var} ] ; do
          case ${var:$j:1} in
               a) print -f "%s" "m" ;;
               b) print -f "%s" "n" ;;
               c) print -f "%s" "o" ;;
               d) print -f "%s" "p" ;;
               e) print -f "%s" "q" ;;
               f) print -f "%s" "r" ;;
               g) print -f "%s" "s" ;;
               h) print -f "%s" "t" ;;
               i) print -f "%s" "u" ;;
               j) print -f "%s" "v" ;;
               k) print -f "%s" "w" ;;
               l) print -f "%s" "x" ;;
               m) print -f "%s" "y" ;;
               n) print -f "%s" "z" ;;
               o) print -f "%s" "a" ;;
               p) print -f "%s" "b" ;;
               q) print -f "%s" "c" ;;
               r) print -f "%s" "d" ;;
               s) print -f "%s" "e" ;;
               t) print -f "%s" "f" ;;
               u) print -f "%s" "g" ;;
               v) print -f "%s" "h" ;;
               w) print -f "%s" "i" ;;
               x) print -f "%s" "j" ;;
               y) print -f "%s" "k" ;;
               z) print -f "%s" "l" ;;
          esac
          (( j += 1 ))
     done
     print -f "\n"
     (( iCnt += 1 ))
done

exec 1>&3
exit 0

The the "external" version is "xtime1.sh", the "shell-only" version xtime2.sh:

Code:
bakunin@aix5200-10-03-0744/~# time ./xtime1.sh

real    0m4.70s
user    0m0.33s
sys     0m4.07s

bakunin@aix5200-10-03-0744/~# time ./xtime2.sh

real    0m0.12s
user    0m0.12s
sys     0m0.00s

I reasoned that the tr command has to be executed completely while the long case..esac construct is not completely used because i used characters from the beginning of the alphabet. So changed "$var" above to the complete alphabet and ran again:

Code:
bakunin@aix5200-10-03-0744/~# time ./xtime1.sh

real    0m4.82s
user    0m0.29s
sys     0m4.20s
bakunin@aix5200-10-03-0744/~# time ./xtime2.sh

real    0m0.77s
user    0m0.69s
sys     0m0.06s

You see that the length of the string indeed has some impact but not nearly as much as i would have expected. The tr-version was not impacted by the length of the string, because the most expensive part is the call of tr itself and the work it does is negligeable by comparison.

So also in this case i stand corrected. I heavily understimated the "cost" of system calls and would have supposed that - because the implementation is way longer - there would be a tradeoff. Obviously this is not nearly the case.

Anyway, it was interesting to find out all this and i am glad i still can learn something about shell programming every day. ;-))

bakunin
This User Gave Thanks to bakunin For This Post:
editha1000
# 20  
Old 10-20-2008
Quote:
Originally Posted by cfajohnson

For someone who knows parameter expansion, both development time and execution time will be saved by using it.
it works both ways, therefore its just different strokes.
Quote:
Real time is as significant as it ever was. Using external commands unnecessarily can turn a snappy script into a slug.
[/INDENT]
i don't know about you, but if real time is a concern, i don't write shell scripts.
# 21  
Old 10-20-2008
Quote:
Originally Posted by bakunin
Anyway, it was interesting to find out all this and i am glad i still can learn something about shell programming every day. ;-))

bakunin
like i said, if time is a concern, i won't write shell scripts. Here's an awk version (which i hope its correct), if you consider awk an "external tool" as well.

Code:
BEGIN{
  i=1
  while ( i <= 1000 ){
    var="abcde"
    j=1
    while ( j <= length(var) ){
        if ( substr(var,j,1) == "a" ){
            printf "%s", "m"
        }else if (substr(var,j,1) == "b"){
            printf "%s", "n" 
        }else if(substr(var,j,1) == "c") {
            printf "%s", "o" 
        }else if(substr(var,j,1) == "d") {
            printf "%s", "p" 
        }else if(substr(var,j,1) == "e") {
            printf "%s", "q" 
        }else if(substr(var,j,1) == "f") {
            printf "%s", "r" 
        }else if(substr(var,j,1) == "g") {
            printf "%s", "s" 
        }else if(substr(var,j,1) == "h") {
            printf "%s", "t" 
        }else if(substr(var,j,1) == "i") {
            printf "%s", "u" 
        }else if(substr(var,j,1) == "j") {
            printf "%s", "v" 
        }else if(substr(var,j,1) == "k") {
            printf "%s", "w" 
        }else if(substr(var,j,1) == "l") {
            printf "%s", "x" 
        }else if(substr(var,j,1) == "m") {
            printf "%s", "y" 
        }else if(substr(var,j,1) == "n") {
            printf "%s", "z" 
        }else if(substr(var,j,1) == "o") {
            printf "%s", "a" 
        }else if(substr(var,j,1) == "p") {
            printf "%s", "b" 
        }else if(substr(var,j,1) == "q") {
            printf "%s", "c" 
        }else if(substr(var,j,1) == "r") {
            printf "%s", "d" 
        }else if(substr(var,j,1) == "s") {
            printf "%s", "e" 
        }else if( substr(var,j,1) == "t") {
            printf "%s", "f" 
        }else if(substr(var,j,1) == "u") {
            printf "%s", "g" 
        }else if(substr(var,j,1) == "v") {
            printf "%s", "h" 
        }else if(substr(var,j,1) == "w") {
            printf "%s", "i" 
        }else if(substr(var,j,1) == "x") {
            printf "%s", "j" 
        }else if(substr(var,j,1) == "y") {
            printf "%s", "k" 
        }else if(substr(var,j,1) == "z") {
            printf "%s", "l" 
        }
        j++
    }
    print
    i++
  }

}

output:
Code:
# time awk -f testawk > /dev/null

real    0m0.022s
user    0m0.020s
sys     0m0.000s

I don't have ksh, so i used bash
Code:
iCnt=1
while [ $iCnt -le 1000 ] ; do
     var="abcde"
     j=0
     while [ $j -le ${#var} ] ; do
          case ${var:$j:1} in
               a) printf "%s" "m" ;;
               b) printf "%s" "n" ;;
               c) printf "%s" "o" ;;
               d) printf "%s" "p" ;;
               e) printf "%s" "q" ;;
               f) printf "%s" "r" ;;
               g) printf "%s" "s" ;;
               h) printf "%s" "t" ;;
               i) printf "%s" "u" ;;
               j) printf "%s" "v" ;;
               k) printf "%s" "w" ;;
               l) printf "%s" "x" ;;
               m) printf "%s" "y" ;;
               n) printf "%s" "z" ;;
               o) printf "%s" "a" ;;
               p) printf "%s" "b" ;;
               q) printf "%s" "c" ;;
               r) printf "%s" "d" ;;
               s) printf "%s" "e" ;;
               t) printf "%s" "f" ;;
               u) printf "%s" "g" ;;
               v) printf "%s" "h" ;;
               w) printf "%s" "i" ;;
               x) printf "%s" "j" ;;
               y) printf "%s" "k" ;;
               z) printf "%s" "l" ;;
          esac
          (( j += 1 ))
     done
     echo
     (( iCnt += 1 ))
done

I think it resembles the ksh version so here's the test
Code:
# time ./test.sh >  testshell.txt

real    0m1.041s
user    0m0.956s
sys     0m0.080s

# time awk -f testawk > testawk.txt

real    0m0.022s
user    0m0.020s
sys     0m0.000s

# diff testawk.txt testshell.txt
#

I may be wrong Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Homework & Coursework Questions

[HELP] Easy task

I have a simple task for my school work. I'm new with unix, so i need help. I need to write a scenario. Task is. From created txt file read first 3 words and create a 3 catalogs with those 3 words. 2 of those new catalogs should be transferred to other directory. If someone could help me just... (1 Reply)
Discussion started by: justynykas
1 Replies

2. Shell Programming and Scripting

Task

Hi experts, I have a problem with the below shell task: I need to modify the file creatin a paired row , per each row which matches filter (e.g. number of nonempty columns = 5) Output should look like this: second row is original one from the input, first row(red) is pairing row, it's... (29 Replies)
Discussion started by: hernand
29 Replies

3. Shell Programming and Scripting

task

Hi all, I'm newbie and stuck here. Thanks for any help. Input(txt file) a b X c d Y e f Z g h W Requested output: a b X Y c d Y X e f Z W g h W Z Please use code tags when posting data and code samples! (10 Replies)
Discussion started by: hernand
10 Replies

4. Shell Programming and Scripting

last task for my script

hi, infile- create table salary ( occupation_code char(40), earnings decimal(10,2), occ_yearend integer ); outfile- salary:create table salary salary:( occupation_code char(40), salary: earnings decimal(10,2), salary: occ_yearend integer salary:); Thanks. (4 Replies)
Discussion started by: dvah
4 Replies

5. Boot Loaders

Reboot and Select Proper Boot device or insert Boot media in select Boot device and press a key

Hello, I have kubuntu on my laptop and now I decided to switch to Windows 7. I made the bios settings properly (first choice is boot from cd\vd) but I see the error " reboot and select proper Boot device or insert Boot media in select Boot device and press a key " I have tried CD and... (0 Replies)
Discussion started by: rpf
0 Replies

6. Programming

Help with task daemon

believe it or not but this is my first c program (i've worked with java, C#, php though) I am trying to make a daemon that checks if mplayer is running(it's for a projection room) and if it is not then to run mplayer with a file.. So far it's not working and I don't know why Help and comments... (5 Replies)
Discussion started by: james2432
5 Replies

7. Shell Programming and Scripting

Parse an XML task list to create each task.xml file

I have an task definition listing xml file that contains a list of tasks such as <TASKLIST <TASK definition="Completion date" id="Taskname1" Some other <CODE name="Code12" <Parameter pname="Dog" input="5.6" units="feet" etc /Parameter> <Parameter... (3 Replies)
Discussion started by: MissI
3 Replies

8. Shell Programming and Scripting

comment and Uncomment single task out of multiple task

I have a file contains TASK gsnmpproxy { CommandLine = $SMCHOME/bin/gsnmpProxy.exe } TASK gsnmpdbgui { CommandLine = $SMCHOME/bin/gsnmpdbgui.exe I would like to comment and than uncomment specific task eg TASK gsnmpproxy Pls suggest how to do in shell script (9 Replies)
Discussion started by: madhusmita
9 Replies

9. UNIX for Dummies Questions & Answers

process vs task

Hi, I am new to this forum and unix too. I have just started learning unix. As I was going through the first chapter, I read that unix is multitasking, multiprogramming, multiprocessing and multiuser OS. My question is: Is there any difference between a TASK and a PROCESS. How are PROCESS... (2 Replies)
Discussion started by: hana
2 Replies
Login or Register to Ask a Question